refactor: update binary field types and conversions in RPC and protofiles (#2619)

Signed-off-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>

.agents/skills/code-change-verification/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: code-change-verification
+description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
+---
+
+# Code Change Verification
+
+Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
+
+## Quick Start
+
+1. Read the scope: commit, PR, patch, or file list.
+2. Map each changed area by risk and user impact.
+3. Inspect each risky change in context.
+4. Report findings first, ordered by severity.
+5. Close with residual risks and verification recommendations.
+
+## Core Workflow
+
+### 1) Scope and assumptions
+- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
+- If context is missing, state assumptions before deeper analysis.
+- Focus only on requested scope; avoid reviewing unrelated files.
+
+### 2) Risk map
+- Prioritize in this order:
+  - Data correctness and user-visible behavior
+  - API/contract compatibility
+  - Security and authz/authn boundaries
+  - Concurrency and lifecycle correctness
+  - Performance and resource usage
+- Give higher priority to stateful paths, migration logic, defaults, and error handling.
+
+### 3) Evidence-based inspection
+- Read each modified hunk with neighboring context.
+- Trace call paths and call-site expectations.
+- Check for:
+  - invariant breaks and missing guards
+  - unchecked assumptions and null/empty/error-path handling
+  - stale tests, fixtures, and configs
+  - hidden coupling to shared helpers/constants/features
+- If a point is uncertain, mark it as an open question instead of guessing.
+
+### 4) Findings-first output
+- Order findings by severity:
+  - P0: critical failure, security breach, or data loss risk
+  - P1: high-impact regression
+  - P2: medium risk correctness gap
+  - P3: low risk/quality debt
+- For each finding include:
+  - Severity
+  - `path:line` reference
+  - concise issue statement
+  - impact and likely failure mode
+  - specific fix or mitigation
+  - validation step to confirm
+- If no issues exist, explicitly state `No findings` and why.
+
+### 5) Close
+- Report assumptions and unknowns.
+- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
+
+## Output Template
+
+1. Findings
+2. No findings (if applicable)
+3. Assumptions / Unknowns
+4. Recommended verification steps
+
+## Finding Template
+
+- `[P1] Missing timeout for downstream call`
+  - Location: `path/to/file.rs:123`
+  - Issue: ...
+  - Impact: ...
+  - Fix suggestion: ...
+  - Validation: ...
+

.agents/skills/code-change-verification/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Code Change Verification"
+  short_description: "Prioritize risks and verify code changes before merge."
+  default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

.agents/skills/pr-creation-checker/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: pr-creation-checker
+description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
+---
+
+# PR Creation Checker
+
+Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
+
+## Read sources of truth first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Use `Makefile` and `.config/make/` for local quality commands.
+- Use `.github/workflows/ci.yml` for CI expectations.
+- Do not restate long command matrices or template sections from memory when the files exist.
+
+## Workflow
+
+1. Collect PR context
+- Confirm base branch, current branch, change goal, and scope.
+- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
+- Confirm whether the branch includes only intended changes.
+
+2. Inspect change scope
+- Review the diff and summarize what changed.
+- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
+- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
+- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
+
+3. Verify readiness requirements
+- Require `make pre-commit` before marking the PR ready.
+- If `make` is unavailable, use the equivalent commands from `.config/make/`.
+- Add scope-specific verification commands when the changed area needs more than the baseline.
+- If required checks fail, stop and return `BLOCKED`.
+
+4. Draft PR metadata
+- Write the PR title in English using Conventional Commits and keep it within 72 characters.
+- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
+- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
+- Keep the PR body in English.
+- Use the exact section headings from `.github/pull_request_template.md`.
+- Fill non-applicable sections with `N/A`.
+- Include verification commands in the PR description.
+- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
+- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
+
+5. Prepare reviewer context
+- Summarize why the change exists.
+- Summarize what was verified.
+- Call out risks, rollout notes, config impact, and rollback notes when applicable.
+- Mention assumptions or missing context instead of guessing.
+
+6. Prepare CLI-safe output
+- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
+- Return a ready-to-save PR body plus a short title.
+- If not ready, return blockers first and list the minimum steps needed to unblock.
+
+## Output format
+
+### Status
+- `READY` or `BLOCKED`
+
+### Title
+- `<type>(<scope>): <summary>`
+
+### PR Body
+- Reproduce the repository template headings exactly.
+- Fill every section.
+- Omit local absolute paths unless explicitly required.
+
+### Verification
+- List each command run.
+- State pass/fail.
+
+### Risks
+- List breaking changes, config changes, migration impact, or `N/A`.
+
+## Blocker rules
+
+- Return `BLOCKED` if `make pre-commit` has not passed.
+- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
+- Return `BLOCKED` if required template sections are missing.
+- Return `BLOCKED` if the title/body is not in English.
+- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
+- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
+
+## Reference
+
+- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "PR Creation Checker"
+  short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
+  default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -0,0 +1,15 @@
+# PR Readiness Checklist
+
+- Confirm the branch is based on current `main`.
+- Confirm the diff matches the stated scope.
+- Confirm no secrets, logs, temp files, or unrelated refactors are included.
+- Confirm `make pre-commit` passed, or document why it could not run.
+- Confirm extra verification commands are listed for risky changes.
+- Confirm the PR title uses Conventional Commits and stays within 72 characters.
+- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+- Confirm the PR body is in English.
+- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
+- Confirm non-applicable sections are filled with `N/A`.
+- Confirm the PR body does not include local absolute paths unless explicitly required.
+- Confirm multiline GitHub CLI commands use `--body-file`.
+- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception.

.agents/skills/test-coverage-improver/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: test-coverage-improver
+description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
+---
+
+# Test Coverage Improver
+
+Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
+
+## Usage assumptions
+- Focus scope is either changed lines/files, a module, or the whole repository.
+- Coverage artifact must be generated or provided in a supported format.
+- If required context is missing, call out assumptions explicitly before proposing work.
+
+## Workflow
+
+1. Define scope and baseline
+   - Confirm target language, framework, and branch.
+   - Confirm whether the scope is changed files only or full-repo.
+
+2. Produce coverage snapshot
+   - Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
+   - JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
+   - Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
+   - Collect total, per-file, and changed-line coverage.
+
+3. Rank highest-risk gaps
+   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
+   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
+   - Keep shortlist to 5–8 gaps.
+   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
+
+4. Propose high-impact tests
+   - For each shortlisted gap, output:
+     - Intent and expected behavior.
+     - Normal, edge, and failure scenarios.
+     - Assertions and side effects to verify.
+     - Setup needs (fixtures, mocks, integration dependencies).
+     - Estimated effort (`S/M/L`).
+
+5. Close with validation plan
+   - State which gaps remain after proposals.
+   - Provide concrete verification command and acceptance threshold.
+   - List assumptions or blockers (environment, fixtures, flaky dependencies).
+
+## Output template
+
+### Coverage Snapshot
+- total / branch coverage
+- changed-file coverage
+- top missing regions by size
+
+### Top Gaps (ranked)
+- `path:line-range` | risk score | why critical
+
+### Test Proposals
+- `path:line-range`
+  - Test name
+  - scenarios
+  - assertions
+  - effort
+
+### Validation Plan
+- command
+- pass criteria
+- remaining risk

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Test Coverage Improver"
+  short_description: "Find top uncovered risk areas and propose high-impact tests."
+  default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -0,0 +1,25 @@
+# Coverage Gap Prioritization Guide
+
+Use this rubric for each uncovered area.
+
+Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
+
+- Criticality:
+  - 5: authz/authn, data-loss, payment/consistency path
+  - 4: state mutation, cache invalidation, scheduling
+  - 3: error handling + fallbacks in user-visible flows
+  - 2: parsing/format conversion paths
+  - 1: logging-only or low-impact utilities
+
+- CoverageDebt:
+  - 0: 0–5 uncovered lines
+  - 1: 6–20 uncovered lines
+  - 2: 21–40 uncovered lines
+  - 3: 41+ uncovered lines
+
+- Volatility:
+  - 1: stable legacy code with few recent edits
+  - 2: changed in last 2 releases
+  - 3: touched in last 30 days or currently in active PR
+
+Sort by score descending, then by business impact.

.cargo/config.toml

@@ -0,0 +1,26 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# RustFS Cargo configuration
+
+# Enable tokio_unstable cfg for dial9-tokio-telemetry support
+# This allows dial9 to hook into Tokio's internal runtime events
+[build]
+# Enable Tokio unstable features required by dial9-tokio-telemetry for runtime tracing.
+# See: https://docs.rs/tokio/latest/tokio/#unstable-features
+rustflags = ["--cfg", "tokio_unstable"]
+
+# Enable frame pointers for CPU profiling (Linux only, optional but recommended)
+# Uncomment the following line for better CPU profiling data
+# rustflags = ["--cfg", "tokio_unstable", "-C", "force-frame-pointers=yes"]

.config/make/build-docker-buildx-dev.mak

@@ -0,0 +1,64 @@
+## —— Development/Source builds using direct buildx commands ---------------------------------------
+
+.PHONY: docker-dev
+docker-dev: ## Build dev multi-arch image (cannot load locally)
+	@echo "🏗️ Building multi-architecture development Docker images with buildx..."
+	@echo "💡 This builds from source code and is intended for local development and testing"
+	@echo "⚠️  Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
+	$(DOCKER_CLI) buildx build \
+		--platform linux/amd64,linux/arm64 \
+		--file $(DOCKERFILE_SOURCE) \
+		--tag rustfs:source-latest \
+		--tag rustfs:dev-latest \
+		.
+
+.PHONY: docker-dev-local
+docker-dev-local: ## Build dev single-arch image (local load)
+	@echo "🏗️ Building single-architecture development Docker image for local use..."
+	@echo "💡 This builds from source code for the current platform and loads locally"
+	$(DOCKER_CLI) buildx build \
+		--file $(DOCKERFILE_SOURCE) \
+		--tag rustfs:source-latest \
+		--tag rustfs:dev-latest \
+		--load \
+		.
+
+.PHONY: docker-dev-push
+docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
+	@if [ -z "$(REGISTRY)" ]; then \
+		echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
+		exit 1; \
+	fi
+	@echo "🚀 Building and pushing multi-architecture development Docker images..."
+	@echo "💡 Pushing to registry: $(REGISTRY)"
+	$(DOCKER_CLI) buildx build \
+		--platform linux/amd64,linux/arm64 \
+		--file $(DOCKERFILE_SOURCE) \
+		--tag $(REGISTRY)/rustfs:source-latest \
+		--tag $(REGISTRY)/rustfs:dev-latest \
+		--push \
+		.
+
+.PHONY: dev-env-start
+dev-env-start: ## Start development container environment
+	@echo "🚀 Starting development environment..."
+	$(DOCKER_CLI) buildx build \
+		--file $(DOCKERFILE_SOURCE) \
+		--tag rustfs:dev \
+		--load \
+		.
+	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
+	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
+	$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
+		-p 9010:9010 -p 9000:9000 \
+		-v $(shell pwd):/workspace \
+		-it rustfs:dev
+
+.PHONY: dev-env-stop
+dev-env-stop: ## Stop development container environment
+	@echo "🛑 Stopping development environment..."
+	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
+	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
+
+.PHONY: dev-env-restart
+dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment

.config/make/build-docker-buildx-production.mak

@@ -0,0 +1,41 @@
+## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
+
+.PHONY: docker-buildx
+docker-buildx: ## Build production multi-arch image (no push)
+	@echo "🏗️ Building multi-architecture production Docker images with buildx..."
+	./docker-buildx.sh
+
+.PHONY: docker-buildx-push
+docker-buildx-push: ## Build and push production multi-arch image
+	@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
+	./docker-buildx.sh --push
+
+.PHONY: docker-buildx-version
+docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
+	@if [ -z "$(VERSION)" ]; then \
+		echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
+		exit 1; \
+	fi
+	@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
+	./docker-buildx.sh --release $(VERSION)
+
+.PHONY: docker-buildx-push-version
+docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
+	@if [ -z "$(VERSION)" ]; then \
+		echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
+		exit 1; \
+	fi
+	@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
+	./docker-buildx.sh --release $(VERSION) --push
+
+.PHONY: docker-buildx-production-local
+docker-buildx-production-local: ## Build production single-arch image locally
+	@echo "🏗️ Building single-architecture production Docker image locally..."
+	@echo "💡 Alternative to docker-buildx.sh for local testing"
+	$(DOCKER_CLI) buildx build \
+		--file $(DOCKERFILE_PRODUCTION) \
+		--tag rustfs:production-latest \
+		--tag rustfs:latest \
+		--load \
+		--build-arg RELEASE=latest \
+		.

.config/make/build-docker-production.mak

@@ -0,0 +1,16 @@
+## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
+
+.PHONY: docker-build-production
+docker-build-production: ## Build single-arch production image
+	@echo "🏗️ Building single-architecture production Docker image..."
+	@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
+	$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
+
+.PHONY: docker-build-source
+docker-build-source: ## Build single-arch source image
+	@echo "🏗️ Building single-architecture source Docker image..."
+	@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
+	DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
+		--build-arg BUILDKIT_INLINE_CACHE=1 \
+		-f $(DOCKERFILE_SOURCE) -t rustfs:source .
+

.config/make/build-docker.mak

@@ -0,0 +1,22 @@
+## —— Docker-based build (alternative approach) ----------------------------------------------------
+
+# Usage: make BUILD_OS=ubuntu22.04 build-docker
+# Output: target/ubuntu22.04/release/rustfs
+
+.PHONY: build-docker
+build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
+build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
+build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
+build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
+	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
+	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
+	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
+
+.PHONY: docker-inspect-multiarch
+docker-inspect-multiarch: ## Check image architecture support
+	@if [ -z "$(IMAGE)" ]; then \
+		echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
+		exit 1; \
+	fi
+	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
+	docker buildx imagetools inspect $(IMAGE)

.config/make/build.mak

@@ -0,0 +1,55 @@
+## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
+
+.PHONY: build
+build: ## Build RustFS binary (includes console by default)
+	@echo "🔨 Building RustFS using build-rustfs.sh script..."
+	./build-rustfs.sh
+
+.PHONY: build-dev
+build-dev: ## Build RustFS in Development mode
+	@echo "🔨 Building RustFS in development mode..."
+	./build-rustfs.sh --dev
+
+.PHONY: build-musl
+build-musl: ## Build x86_64 musl version
+	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
+	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	./build-rustfs.sh --platform x86_64-unknown-linux-musl
+
+.PHONY: build-gnu
+build-gnu: ## Build x86_64 GNU version
+	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
+	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
+
+.PHONY: build-musl-arm64
+build-musl-arm64: ## Build aarch64 musl version
+	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
+	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	./build-rustfs.sh --platform aarch64-unknown-linux-musl
+
+.PHONY: build-gnu-arm64
+build-gnu-arm64: ## Build aarch64 GNU version
+	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
+	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	./build-rustfs.sh --platform aarch64-unknown-linux-gnu
+
+
+.PHONY: build-cross-all
+build-cross-all: core-deps ## Build binaries for all architectures
+	@echo "🔧 Building all target architectures..."
+	@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
+	@echo "🔨 Generating protobuf code..."
+	cargo run --bin gproto || true
+
+	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
+	./build-rustfs.sh --platform x86_64-unknown-linux-musl
+
+	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
+	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
+
+	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
+	./build-rustfs.sh --platform aarch64-unknown-linux-musl
+
+	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
+	./build-rustfs.sh --platform aarch64-unknown-linux-gnu

.config/make/check.mak

@@ -0,0 +1,24 @@
+## —— Check and Inform Dependencies ----------------------------------------------------------------
+
+# Fatal check
+# Checks all required dependencies and exits with error if not found
+# (e.g., cargo, rustfmt)
+check-%:
+	@command -v $* >/dev/null 2>&1 || { \
+		echo >&2 "❌ '$*' is not installed."; \
+		exit 1; \
+	}
+
+# Warning-only check
+# Checks for optional dependencies and issues a warning if not found
+# (e.g., cargo-nextest for enhanced testing)
+warn-%:
+	@command -v $* >/dev/null 2>&1 || { \
+		echo >&2 "⚠️ '$*' is not installed."; \
+	}
+
+# For checking dependencies use check-<dep-name> or warn-<dep-name>
+.PHONY: core-deps fmt-deps test-deps
+core-deps: check-cargo ## Check core dependencies
+fmt-deps: check-rustfmt ## Check lint and formatting dependencies
+test-deps: warn-cargo-nextest ## Check tests dependencies

.config/make/deploy.mak

@@ -0,0 +1,6 @@
+## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
+
+.PHONY: deploy-dev
+deploy-dev: build-musl ## Deploy to dev server
+	@echo "🚀 Deploying to dev server: $${IP}"
+	./scripts/dev_deploy.sh $${IP}

.config/make/help.mak

@@ -0,0 +1,38 @@
+## —— Help, Help Build and Help Docker -------------------------------------------------------------
+
+
+.PHONY: help
+help: ## Shows This Help Menu
+	echo -e "$$HEADER"
+	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} ; {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}' | sed -e 's/\[36m##/\n[32m##/'
+
+.PHONY: help-build
+help-build: ## Shows RustFS build help
+	@echo ""
+	@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
+	@echo ""
+	@echo "🔧 Direct usage of build-rustfs.sh script:"
+	@echo ""
+	@echo "	./build-rustfs.sh --help                                # View script help"
+	@echo "	./build-rustfs.sh --no-console                          # Build without console resources"
+	@echo "	./build-rustfs.sh --force-console-update                # Force update console resources"
+	@echo "	./build-rustfs.sh --dev                                 # Development mode build"
+	@echo "	./build-rustfs.sh --sign                                # Sign binary files"
+	@echo "	./build-rustfs.sh --platform x86_64-unknown-linux-gnu   # Specify target platform"
+	@echo "	./build-rustfs.sh --skip-verification                   # Skip binary verification"
+	@echo ""
+
+.PHONY: help-docker
+help-docker: ## Shows docker environment and suggestion help
+	@echo ""
+	@echo "📋 Environment Variables:"
+	@echo "	REGISTRY                 Image registry address (required for push)"
+	@echo "	DOCKERHUB_USERNAME       Docker Hub username"
+	@echo "	DOCKERHUB_TOKEN          Docker Hub access token"
+	@echo "	GITHUB_TOKEN             GitHub access token"
+	@echo ""
+	@echo "💡 Suggestions:"
+	@echo "	Production use:          Use docker-buildx* commands (based on precompiled binaries)"
+	@echo "	Local development:       Use docker-dev* commands (build from source)"
+	@echo "	Development environment: Use dev-env-* commands to manage dev containers"
+	@echo ""

.config/make/lint-fmt.mak

@@ -0,0 +1,22 @@
+## —— Code quality and Formatting ------------------------------------------------------------------
+
+.PHONY: fmt
+fmt: core-deps fmt-deps ## Format code
+	@echo "🔧 Formatting code..."
+	cargo fmt --all
+
+.PHONY: fmt-check
+fmt-check: core-deps fmt-deps ## Check code formatting
+	@echo "📝 Checking code formatting..."
+	cargo fmt --all --check
+
+.PHONY: clippy-check
+clippy-check: core-deps ## Run clippy checks
+	@echo "🔍 Running clippy checks..."
+	cargo clippy --fix --allow-dirty
+	cargo clippy --all-targets --all-features -- -D warnings
+
+.PHONY: compilation-check
+compilation-check: core-deps ## Run compilation check
+	@echo "🔨 Running compilation check..."
+	cargo check --all-targets

.config/make/pre-commit.mak

@@ -0,0 +1,11 @@
+## —— Pre Commit Checks ----------------------------------------------------------------------------
+
+.PHONY: setup-hooks
+setup-hooks: ## Set up git hooks
+	@echo "🔧 Setting up git hooks..."
+	chmod +x .git/hooks/pre-commit
+	@echo "✅ Git hooks setup complete!"
+
+.PHONY: pre-commit
+pre-commit: fmt clippy-check compilation-check test ## Run pre-commit checks
+	@echo "✅ All pre-commit checks passed!"

.config/make/tests.mak

@@ -0,0 +1,22 @@
+## —— Tests and e2e test ---------------------------------------------------------------------------
+
+TEST_THREADS ?= 1
+
+.PHONY: test
+test: core-deps test-deps ## Run all tests
+	@echo "🧪 Running tests..."
+	@if command -v cargo-nextest >/dev/null 2>&1; then \
+		cargo nextest run --all --exclude e2e_test; \
+	else \
+		echo "ℹ️ cargo-nextest not found; falling back to 'cargo test'"; \
+		cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
+	fi
+	cargo test --all --doc
+
+.PHONY: e2e-server
+e2e-server: ## Run e2e-server tests
+	sh $(shell pwd)/scripts/run.sh
+
+.PHONY: probe-e2e
+probe-e2e: ## Probe e2e tests
+	sh $(shell pwd)/scripts/probe.sh

.docker/README.md

@@ -1,261 +1,131 @@
-# RustFS Docker Images
+# RustFS Docker Infrastructure
 
-This directory contains Docker configuration files and supporting infrastructure for building and running RustFS container images.
+This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
 
-## 📁 Directory Structure
+## 📂 Directory Structure
 
-```
+| Directory | Description | Status |
-rustfs/
+| :--- | :--- | :--- |
-├── Dockerfile           # Production image (Alpine + pre-built binaries)
+| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
-├── Dockerfile.source    # Development image (Debian + source build)
+| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
-├── docker-buildx.sh     # Multi-architecture build script
+| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
-├── Makefile             # Build automation with simplified commands
+| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
-└── .docker/             # Supporting infrastructure
-    ├── observability/   # Monitoring and observability configs
-    ├── compose/         # Docker Compose configurations
-    ├── mqtt/            # MQTT broker configs
-    └── openobserve-otel/ # OpenObserve + OpenTelemetry configs
-```
 
-## 🎯 Image Variants
+---
 
-### Core Images
+## 📄 Root Directory Files
 
-| Image | Base OS | Build Method | Size | Use Case |
+The following files in the project root are essential for Docker operations:
-|-------|---------|--------------|------|----------|
-| `production` (default) | Alpine 3.18 | GitHub Releases | Smallest | Production deployment |
-| `source` | Debian Bookworm | Source build | Medium | Custom builds with cross-compilation |
-| `dev` | Debian Bookworm | Development tools | Large | Interactive development |
 
-## 🚀 Usage Examples
+### Build Scripts & Dockerfiles
 
-### Quick Start (Production)
+| File | Description | Usage |
+| :--- | :--- | :--- |
+| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
+| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
+| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
+| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
 
+### Docker Compose Configurations
+
+| File | Description | Usage |
+| :--- | :--- | :--- |
+| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
+| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
+
+---
+
+## 🌟 Observability Stack (Recommended)
+
+Located in: [`.docker/observability/`](observability/README.md)
+
+We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
+
+### Components
+- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
+- **Traces**: Tempo (Storage) + Jaeger (UI)
+- **Logs**: Loki
+- **Ingestion**: OpenTelemetry Collector
+
+### Key Features
+- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
+- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
+- **High Performance**: Optimized configurations for batching, compression, and memory management.
+
+### Quick Start
 ```bash
-# Default production image (Alpine + GitHub Releases)
+cd .docker/observability
-docker run -p 9000:9000 rustfs/rustfs:latest
+docker compose up -d
-
-# Specific version
-docker run -p 9000:9000 rustfs/rustfs:1.2.3
 ```
 
-### Complete Tag Strategy Examples
+---
 
+## 🧪 Specialized Environments
+
+Located in: [`.docker/compose/`](compose/README.md)
+
+These configurations are tailored for specific testing scenarios that require complex topologies.
+
+### Distributed Cluster (4-Nodes)
+Simulates a real-world distributed environment with 4 RustFS nodes running locally.
 ```bash
-# Stable Releases
+docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
-docker run rustfs/rustfs:1.2.3           # Main version (production)
-docker run rustfs/rustfs:1.2.3-production # Explicit production variant
-docker run rustfs/rustfs:1.2.3-source   # Source build variant
-docker run rustfs/rustfs:latest          # Latest stable
-
-# Prerelease Versions
-docker run rustfs/rustfs:1.3.0-alpha.2  # Specific alpha version
-docker run rustfs/rustfs:alpha           # Latest alpha
-docker run rustfs/rustfs:beta            # Latest beta
-docker run rustfs/rustfs:rc              # Latest release candidate
-
-# Development Versions
-docker run rustfs/rustfs:dev             # Latest main branch development
-docker run rustfs/rustfs:dev-13e4a0b     # Specific commit
-docker run rustfs/rustfs:dev-latest      # Latest development
-docker run rustfs/rustfs:main-latest     # Main branch latest
 ```
 
-### Development Environment
+### Integrated Observability Test
-
+A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
 ```bash
-# Quick setup using Makefile (recommended)
+docker compose -f .docker/compose/docker-compose.observability.yaml up -d
-make docker-dev-local     # Build development image locally
-make dev-env-start        # Start development container
-
-# Manual Docker commands
-docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs/rustfs:latest-dev
-
-# Build from source locally
-docker build -f Dockerfile.source -t rustfs:custom .
-
-# Development with hot reload
-docker-compose up rustfs-dev
 ```
 
-## 🏗️ Build Arguments and Scripts
+---
 
-### Using Makefile Commands (Recommended)
+## 📡 MQTT Integration
 
-The easiest way to build images using simplified commands:
+Located in: [`.docker/mqtt/`](mqtt/README.md)
 
+Provides an EMQX broker for testing RustFS MQTT features.
+
+### Quick Start
 ```bash
-# Development images (build from source)
+cd .docker/mqtt
-make docker-dev-local             # Build for local use (single arch)
+docker compose up -d
-make docker-dev                   # Build multi-arch (for CI/CD)
-make docker-dev-push REGISTRY=xxx # Build and push to registry
-
-# Production images (using pre-built binaries)
-make docker-buildx                # Build multi-arch production images
-make docker-buildx-push           # Build and push production images
-make docker-buildx-version VERSION=v1.0.0  # Build specific version
-
-# Development environment
-make dev-env-start                # Start development container
-make dev-env-stop                 # Stop development container
-make dev-env-restart              # Restart development container
-
-# Help
-make help-docker                  # Show all Docker-related commands
 ```
+- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
+- **MQTT Port**: `1883`
 
-### Using docker-buildx.sh (Advanced)
+---
 
-For direct script usage and advanced scenarios:
+## 👁️ Alternative: OpenObserve
 
+Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
+
+For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
+
+### Quick Start
 ```bash
-# Build latest version for all architectures
+cd .docker/openobserve-otel
-./docker-buildx.sh
+docker compose up -d
-
-# Build and push to registry
-./docker-buildx.sh --push
-
-# Build specific version
-./docker-buildx.sh --release v1.2.3
-
-# Build and push specific version
-./docker-buildx.sh --release v1.2.3 --push
 ```
 
-### Manual Docker Builds
+---
 
-All images support dynamic version selection:
+## 🔧 Common Operations
 
+### Cleaning Up
+To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
 ```bash
-# Build production image with latest release
+docker compose down -v
-docker build --build-arg RELEASE="latest" -t rustfs:latest .
-
-# Build from source with specific target
-docker build -f Dockerfile.source \
-  --build-arg TARGETPLATFORM="linux/amd64" \
-  -t rustfs:source .
-
-# Development build
-docker build -f Dockerfile.source -t rustfs:dev .
 ```
 
-## 🔧 Binary Download Sources
+### Viewing Logs
-
+To follow logs for a specific service:
-### Unified GitHub Releases
-
-The production image downloads from GitHub Releases for reliability and transparency:
-
-- ✅ **production** → GitHub Releases API with automatic latest detection
-- ✅ **Checksum verification** → SHA256SUMS validation when available
-- ✅ **Multi-architecture** → Supports amd64 and arm64
-
-### Source Build
-
-The source variant compiles from source code with advanced features:
-
-- 🔧 **Cross-compilation** → Supports multiple target platforms via `TARGETPLATFORM`
-- ⚡ **Build caching** → sccache for faster compilation
-- 🎯 **Optimized builds** → Release optimizations with LTO and symbol stripping
-
-## 📋 Architecture Support
-
-All variants support multi-architecture builds:
-
-- **linux/amd64** (x86_64)
-- **linux/arm64** (aarch64)
-
-Architecture is automatically detected during build using Docker's `TARGETARCH` build argument.
-
-## 🔐 Security Features
-
-- **Checksum Verification**: Production image verifies SHA256SUMS when available
-- **Non-root User**: All images run as user `rustfs` (UID 1000)
-- **Minimal Runtime**: Production image only includes necessary dependencies
-- **Secure Defaults**: No hardcoded credentials or keys
-
-## 🛠️ Development Workflow
-
-### Quick Start with Makefile (Recommended)
-
 ```bash
-# 1. Start development environment
+docker compose logs -f [service_name]
-make dev-env-start
-
-# 2. Your development container is now running with:
-#    - Port 9000 exposed for RustFS
-#    - Port 9010 exposed for admin console
-#    - Current directory mounted as /workspace
-
-# 3. Stop when done
-make dev-env-stop
 ```
 
-### Manual Development Setup
+### Checking Status
-
+To see the status of all running containers:
 ```bash
-# Build development image from source
+docker compose ps
-make docker-dev-local
-
-# Or use traditional Docker commands
-docker build -f Dockerfile.source -t rustfs:dev .
-
-# Run with development tools
-docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs:dev bash
-
-# Or use docker-compose for complex setups
-docker-compose up rustfs-dev
 ```
-
-### Common Development Tasks
-
-```bash
-# Build and test locally
-make build                        # Build binary natively
-make docker-dev-local            # Build development Docker image
-make test                        # Run tests
-make fmt                         # Format code
-make clippy                      # Run linter
-
-# Get help
-make help                        # General help
-make help-docker                 # Docker-specific help
-make help-build                  # Build-specific help
-```
-
-## 🚀 CI/CD Integration
-
-The project uses GitHub Actions for automated multi-architecture Docker builds:
-
-### Automated Builds
-
-- **Tags**: Automatic builds triggered on version tags (e.g., `v1.2.3`)
-- **Main Branch**: Development builds with `dev-latest` and `main-latest` tags
-- **Pull Requests**: Test builds without registry push
-
-### Build Variants
-
-Each build creates three image variants:
-
-- `rustfs/rustfs:v1.2.3` (production - Alpine-based)
-- `rustfs/rustfs:v1.2.3-source` (source build - Debian-based)
-- `rustfs/rustfs:v1.2.3-dev` (development - Debian-based with tools)
-
-### Manual Builds
-
-Trigger custom builds via GitHub Actions:
-
-```bash
-# Use workflow_dispatch to build specific versions
-# Available options: latest, main-latest, dev-latest, v1.2.3, dev-abc123
-```
-
-## 📦 Supporting Infrastructure
-
-The `.docker/` directory contains supporting configuration files:
-
-- **observability/** - Prometheus, Grafana, OpenTelemetry configs
-- **compose/** - Multi-service Docker Compose setups
-- **mqtt/** - MQTT broker configurations
-- **openobserve-otel/** - Log aggregation and tracing setup
-
-See individual README files in each subdirectory for specific usage instructions.

.docker/compose/README.md

@@ -1,80 +1,44 @@
-# Docker Compose Configurations
+# Specialized Docker Compose Configurations
 
-This directory contains specialized Docker Compose configurations for different use cases.
+This directory contains specialized Docker Compose configurations for specific testing scenarios.
+
+## ⚠️ Important Note
+
+**For Observability:**
+We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
+
+The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
 
 ## 📁 Configuration Files
 
-This directory contains specialized Docker Compose configurations and their associated Dockerfiles, keeping related files organized together.
+### Cluster Testing
 
-### Main Configuration (Root Directory)
+- **`docker-compose.cluster.yaml`**
+  - **Purpose**: Simulates a 4-node RustFS distributed cluster.
+  - **Use Case**: Testing distributed storage logic, consensus, and failover.
+  - **Nodes**: 4 RustFS instances.
+  - **Storage**: Uses local HTTP endpoints.
 
-- **`../../docker-compose.yml`** - **Default Production Setup**
+### Legacy / Minimal Observability
-  - Complete production-ready configuration
-  - Includes RustFS server + full observability stack
-  - Supports multiple profiles: `dev`, `observability`, `cache`, `proxy`
-  - Recommended for most users
 
-### Specialized Configurations
+- **`docker-compose.observability.yaml`**
-
+  - **Purpose**: A minimal observability setup.
-- **`docker-compose.cluster.yaml`** - **Distributed Testing**
+  - **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
-  - 4-node cluster setup for testing distributed storage
-  - Uses local compiled binaries
-  - Simulates multi-node environment
-  - Ideal for development and cluster testing
-
-- **`docker-compose.observability.yaml`** - **Observability Focus**
-  - Specialized setup for testing observability features
-  - Includes OpenTelemetry, Jaeger, Prometheus, Loki, Grafana
-  - Uses `../../Dockerfile.source` for builds
-  - Perfect for observability development
 
 ## 🚀 Usage Examples
 
-### Production Setup
-
-```bash
-# Start main service
-docker-compose up -d
-
-# Start with development profile
-docker-compose --profile dev up -d
-
-# Start with full observability
-docker-compose --profile observability up -d
-```
-
 ### Cluster Testing
 
-```bash
+To start a 4-node cluster for distributed testing:
-# Build and start 4-node cluster (run from project root)
-cd .docker/compose
-docker-compose -f docker-compose.cluster.yaml up -d
-
-# Or run directly from project root
-docker-compose -f .docker/compose/docker-compose.cluster.yaml up -d
-```
-
-### Observability Testing
 
 ```bash
-# Start observability-focused environment (run from project root)
+# From project root
-cd .docker/compose
+docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
-docker-compose -f docker-compose.observability.yaml up -d
-
-# Or run directly from project root
-docker-compose -f .docker/compose/docker-compose.observability.yaml up -d
 ```
 
-## 🔧 Configuration Overview
+### (Deprecated) Minimal Observability
 
-| Configuration | Nodes | Storage | Observability | Use Case |
+```bash
-|---------------|-------|---------|---------------|----------|
+# From project root
-| **Main** | 1 | Volume mounts | Full stack | Production |
+docker compose -f .docker/compose/docker-compose.observability.yaml up -d
-| **Cluster** | 4 | HTTP endpoints | Basic | Testing |
+```
-| **Observability** | 4 | Local data | Advanced | Development |
-
-## 📝 Notes
-
-- Always ensure you have built the required binaries before starting cluster tests
-- The main configuration is sufficient for most use cases
-- Specialized configurations are for specific testing scenarios

.docker/compose/docker-compose.observability.yaml

@@ -13,62 +13,126 @@
 # limitations under the License.
 
 services:
+  # --- Observability Stack ---
+
+  tempo-init:
+    image: busybox:latest
+    command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
+    volumes:
+      - tempo-data:/var/tempo
+    user: root
+    networks:
+      - rustfs-network
+    restart: "no"
+
+  tempo:
+    image: grafana/tempo:latest
+    user: "10001"
+    command: [ "-config.file=/etc/tempo.yaml" ]
+    volumes:
+      - ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200" # tempo
+      - "4317"      # otlp grpc
+      - "4318"      # otlp http
+    restart: unless-stopped
+    networks:
+      - rustfs-network
+
   otel-collector:
-    image: otel/opentelemetry-collector-contrib:0.129.1
+    image: otel/opentelemetry-collector-contrib:latest
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
+      - ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
     ports:
-      - 1888:1888
+      - "1888:1888"   # pprof
-      - 8888:8888
+      - "8888:8888"   # Prometheus metrics for Collector
-      - 8889:8889
+      - "8889:8889"   # Prometheus metrics for application indicators
-      - 13133:13133
+      - "13133:13133" # health check
-      - 4317:4317
+      - "4317:4317"   # OTLP gRPC
-      - 4318:4318
+      - "4318:4318"   # OTLP HTTP
-      - 55679:55679
+      - "55679:55679" # zpages
     networks:
       - rustfs-network
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+
   jaeger:
-    image: jaegertracing/jaeger:2.8.0
+    image: jaegertracing/jaeger:latest
     environment:
       - TZ=Asia/Shanghai
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
+      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - jaeger-data:/badger
     ports:
-      - "16686:16686"
+      - "16686:16686" # Web UI
-      - "14317:4317"
+      - "14269:14269" # Admin/Metrics
-      - "14318:4318"
     networks:
       - rustfs-network
+
   prometheus:
-    image: prom/prometheus:v3.4.2
+    image: prom/prometheus:latest
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml
+      - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-data:/prometheus
     ports:
       - "9090:9090"
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yml'
+      - '--web.enable-otlp-receiver'
+      - '--web.enable-remote-write-receiver'
+      - '--enable-feature=promql-experimental-functions'
+      - '--storage.tsdb.path=/prometheus'
+      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
+      - '--web.console.templates=/usr/share/prometheus/consoles'
     networks:
       - rustfs-network
+
   loki:
-    image: grafana/loki:3.5.1
+    image: grafana/loki:latest
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/loki-config.yaml:/etc/loki/local-config.yaml
+      - ../../.docker/observability/loki.yaml:/etc/loki/local-config.yaml:ro
+      - loki-data:/loki
     ports:
       - "3100:3100"
     command: -config.file=/etc/loki/local-config.yaml
     networks:
       - rustfs-network
+
   grafana:
-    image: grafana/grafana:12.0.2
+    image: grafana/grafana:latest
     ports:
       - "3000:3000" # Web UI
     environment:
       - GF_SECURITY_ADMIN_PASSWORD=admin
+      - GF_SECURITY_ADMIN_USER=admin
       - TZ=Asia/Shanghai
+      - GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
+      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
     networks:
       - rustfs-network
+    volumes:
+      - ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
+      - ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+
+  # --- RustFS Cluster ---
 
   node1:
     build:
@@ -79,13 +143,15 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
       - RUSTFS_OBS_LOGGER_LEVEL=debug
     platform: linux/amd64
     ports:
-      - "9001:9000" # Map port 9001 of the host to port 9000 of the container
+      - "9001:9000"
     networks:
       - rustfs-network
+    depends_on:
+      - otel-collector
 
   node2:
     build:
@@ -96,13 +162,15 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
       - RUSTFS_OBS_LOGGER_LEVEL=debug
     platform: linux/amd64
     ports:
-      - "9002:9000" # Map port 9002 of the host to port 9000 of the container
+      - "9002:9000"
     networks:
       - rustfs-network
+    depends_on:
+      - otel-collector
 
   node3:
     build:
@@ -113,13 +181,15 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
       - RUSTFS_OBS_LOGGER_LEVEL=debug
     platform: linux/amd64
     ports:
-      - "9003:9000" # Map port 9003 of the host to port 9000 of the container
+      - "9003:9000"
     networks:
       - rustfs-network
+    depends_on:
+      - otel-collector
 
   node4:
     build:
@@ -130,13 +200,21 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
       - RUSTFS_OBS_LOGGER_LEVEL=debug
     platform: linux/amd64
     ports:
-      - "9004:9000" # Map port 9004 of the host to port 9000 of the container
+      - "9004:9000"
     networks:
       - rustfs-network
+    depends_on:
+      - otel-collector
+
+volumes:
+  prometheus-data:
+  tempo-data:
+  loki-data:
+  jaeger-data:
 
 networks:
   rustfs-network:

.docker/mqtt/README.md

@@ -0,0 +1,30 @@
+# MQTT Broker (EMQX)
+
+This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
+
+## 🚀 Quick Start
+
+To start the EMQX broker:
+
+```bash
+docker compose up -d
+```
+
+## 📊 Access
+
+- **Dashboard**: [http://localhost:18083](http://localhost:18083)
+- **Default Credentials**: `admin` / `public`
+- **MQTT Port**: `1883`
+- **WebSocket Port**: `8083`
+
+## 🛠️ Configuration
+
+The `docker-compose.yml` file sets up a single-node EMQX instance.
+
+- **Persistence**: Data is not persisted by default (for testing).
+- **Network**: Uses the default bridge network.
+
+## 📝 Notes
+
+- This setup is intended for development and testing purposes.
+- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).

.docker/nginx/nginx.conf

@@ -0,0 +1,82 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+worker_processes auto;
+pid /var/run/nginx.pid;
+
+events {
+    worker_connections 1024;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+
+    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
+                    '$status $body_bytes_sent "$http_referer" '
+                    '"$http_user_agent" "$http_x_forwarded_for"';
+
+    access_log /var/log/nginx/access.log main;
+    error_log /var/log/nginx/error.log warn;
+
+    sendfile on;
+    keepalive_timeout 65;
+
+    # RustFS Server Block
+    server {
+        listen 80;
+        server_name localhost;
+
+        # Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
+        # return 301 https://$host$request_uri;
+
+        location / {
+            proxy_pass http://rustfs:9000;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+
+            # S3 specific headers
+            proxy_set_header X-Amz-Date $http_x_amz_date;
+            proxy_set_header Authorization $http_authorization;
+
+            # Disable buffering for large uploads
+            proxy_request_buffering off;
+            client_max_body_size 0;
+        }
+
+        location /rustfs/console {
+            proxy_pass http://rustfs:9001;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+        }
+    }
+
+    # SSL Configuration (Example)
+    # server {
+    #     listen 443 ssl;
+    #     server_name localhost;
+    #
+    #     ssl_certificate /etc/nginx/ssl/server.crt;
+    #     ssl_certificate_key /etc/nginx/ssl/server.key;
+    #
+    #     location / {
+    #         proxy_pass http://rustfs:9000;
+    #         ...
+    #     }
+    # }
+}

.docker/observability/.gitignore

@@ -0,0 +1,5 @@
+jaeger-data/*
+loki-data/*
+prometheus-data/*
+tempo-data/*
+grafana-data/*

.docker/observability/README.md

@@ -1,109 +1,85 @@
-# Observability
+# RustFS Observability Stack
 
-This directory contains the observability stack for the application. The stack is composed of the following components:
+This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
 
-- Prometheus v3.2.1
+## Components
-- Grafana 11.6.0
-- Loki 3.4.2
-- Jaeger 2.4.0
-- Otel Collector 0.120.0 # 0.121.0 remove loki
 
-## Prometheus
+The stack is composed of the following best-in-class open-source components:
 
-Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an
+- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
-intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to
+- **Grafana** (v11.1.0): The leading platform for observability visualization.
-either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be
+- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
-used to visualize the collected data.
+- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
+- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
+- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
 
-## Grafana
+## Architecture
 
-Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts,
+1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
-graphs, and alerts for the web when connected to supported data sources.
+2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
+    - **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
+    - **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
+    - **Logs** -> **Loki**
+3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
 
-## Loki
+## Features
 
-Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is
+- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
-designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of
+- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
-labels for each log stream.
+  - Jump from a Metric spike to relevant Traces.
+  - Jump from a Trace to relevant Logs.
+- **High Performance**: Optimized configurations for batching, compression, and memory management.
+- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
 
-## Jaeger
+## Quick Start
 
-Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and
+### Prerequisites
-troubleshooting microservices-based distributed systems, including:
 
-- Distributed context propagation
+- Docker
-- Distributed transaction monitoring
+- Docker Compose
-- Root cause analysis
-- Service dependency analysis
-- Performance / latency optimization
 
-## Otel Collector
+### Deploy
 
-The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry
+Run the following command to start the entire stack:
-data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source
-observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.
-
-## How to use
-
-To deploy the observability stack, run the following command:
-
-- docker latest version
 
 ```bash
-docker compose -f docker-compose.yml -f docker-compose.override.yml up -d
+docker compose up -d
 ```
 
-- docker compose v2.0.0 or before
+### Access Dashboards
+
+| Service        | URL                                              | Credentials       | Description                    |
+| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
+| **Grafana**    | [http://localhost:3000](http://localhost:3000)   | `admin` / `admin` | Main visualization hub.        |
+| **Prometheus** | [http://localhost:9090](http://localhost:9090)   | -                 | Metric queries and status.     |
+| **Jaeger UI**  | [http://localhost:16686](http://localhost:16686) | -                 | Secondary trace visualization. |
+| **Tempo**      | [http://localhost:3200](http://localhost:3200)   | -                 | Tempo status/metrics.          |
+
+## Configuration
+
+### Data Persistence
+
+Data is stored in the following Docker volumes:
+
+- `prometheus-data`: Prometheus metrics
+- `tempo-data`: Tempo traces (WAL and Blocks)
+- `loki-data`: Loki logs (Chunks and Rules)
+- `jaeger-data`: Jaeger traces (Badger DB)
+
+To clear all data:
 
 ```bash
-docke-compose -f docker-compose.yml -f docker-compose.override.yml up -d
+docker compose down -v
 ```
 
-To access the Grafana dashboard, navigate to `http://localhost:3000` in your browser. The default username and password
+### Customization
-are `admin` and `admin`, respectively.
-
-To access the Jaeger dashboard, navigate to `http://localhost:16686` in your browser.
-
-To access the Prometheus dashboard, navigate to `http://localhost:9090` in your browser.
-
-## How to stop
-
-To stop the observability stack, run the following command:
-
-```bash
-docker compose -f docker-compose.yml -f docker-compose.override.yml down
-```
-
-## How to remove data
-
-To remove the data generated by the observability stack, run the following command:
-
-```bash
-docker compose -f docker-compose.yml -f docker-compose.override.yml down -v
-```
-
-## How to configure
-
-To configure the observability stack, modify the `docker-compose.override.yml` file. The file contains the following
-
-```yaml
-services:
-  prometheus:
-    environment:
-      - PROMETHEUS_CONFIG_FILE=/etc/prometheus/prometheus.yml
-    volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml
-
-  grafana:
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-    volumes:
-      - ./grafana/provisioning:/etc/grafana/provisioning
-```
-
-The `prometheus` service mounts the `prometheus.yml` file to `/etc/prometheus/prometheus.yml`. The `grafana` service
-mounts the `grafana/provisioning` directory to `/etc/grafana/provisioning`. You can modify these files to configure the
-observability stack.
 
+- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
+- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
+- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
 
+## Troubleshooting
 
+- **Service Health**: Check the health of services using `docker compose ps`.
+- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
+- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.

.docker/observability/README_ZH.md

@@ -1,27 +1,85 @@
-## 部署可观测性系统
+# RustFS 可观测性技术栈
 
-OpenTelemetry Collector 提供了一个厂商中立的遥测数据处理方案，用于接收、处理和导出遥测数据。它消除了为支持多种开源可观测性数据格式（如
+本目录包含 RustFS 的全面可观测性技术栈，旨在提供对应用程序性能、日志和追踪的深入洞察。
-Jaeger、Prometheus 等）而需要运行和维护多个代理/收集器的必要性。
 
-### 快速部署
+## 组件
 
-1. 进入 `.docker/observability` 目录
+该技术栈由以下一流的开源组件组成：
-2. 执行以下命令启动服务：
+
+- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
+- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
+- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
+- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
+- **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
+- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
+
+## 架构
+
+1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
+2.  **处理与导出**: Collector 处理数据（批处理、内存限制）并将其导出到相应的后端：
+    -   **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
+    -   **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
+    -   **日志** -> **Loki**
+3.  **可视化**: **Grafana** 连接到所有后端（Prometheus, Tempo, Loki, Jaeger），提供统一的仪表盘体验。
+
+## 特性
+
+-   **完全持久化**: 所有数据（指标、日志、追踪）都持久化到 Docker 卷，确保重启后无数据丢失。
+-   **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
+    -   从指标峰值跳转到相关追踪。
+    -   从追踪跳转到相关日志。
+-   **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
+-   **标准化协议**: 完全基于 OpenTelemetry 标准构建。
+
+## 快速开始
+
+### 前置条件
+
+-   Docker
+-   Docker Compose
+
+### 部署
+
+运行以下命令启动整个技术栈：
 
 ```bash
-docker compose -f docker-compose.yml  up -d
+docker compose up -d
 ```
 
-### 访问监控面板
+### 访问仪表盘
 
-服务启动后，可通过以下地址访问各个监控面板：
+| 服务 | URL | 凭据 | 描述 |
+| :--- | :--- | :--- | :--- |
+| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
+| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
+| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
+| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
 
-- Grafana: `http://localhost:3000` (默认账号/密码：`admin`/`admin`)
+## 配置
-- Jaeger: `http://localhost:16686`
-- Prometheus: `http://localhost:9090`
 
-## 配置可观测性
+### 数据持久化
 
-```shell
+数据存储在以下 Docker 卷中：
-export RUSTFS_OBS_ENDPOINT="http://localhost:4317" # OpenTelemetry Collector 地址
+
+-   `prometheus-data`: Prometheus 指标
+-   `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
+-   `loki-data`: Loki 日志 (Chunks 和 Rules)
+-   `jaeger-data`: Jaeger 追踪 (Badger DB)
+
+要清除所有数据：
+
+```bash
+docker compose down -v
 ```
+
+### 自定义
+
+-   **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
+-   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
+-   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
+
+## 故障排除
+
+-   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
+-   **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
+-   **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态，检查 `http://localhost:1888/debug/pprof/` 进行性能分析。

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -0,0 +1,270 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  rustfs:
+    security_opt:
+      - "no-new-privileges:true"
+    image: rustfs/rustfs:latest
+    container_name: rustfs-server
+    ports:
+      - "9000:9000" # S3 API port
+      - "9001:9001" # Console port
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+    volumes:
+      - rustfs-data:/data/rustfs
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+    depends_on:
+      otel-collector:
+        condition: service_started
+
+  rustfs-init:
+    image: alpine
+    container_name: rustfs-init
+    volumes:
+      - rustfs-data:/data
+    networks:
+      - otel-network
+    command: >
+      sh -c "
+        chown -R 10001:10001 /data &&
+        echo 'Volume Permissions fixed' &&
+        exit 0
+      "
+    restart: no
+
+  # --- Tracing ---
+
+  tempo:
+    image: grafana/tempo:latest
+    container_name: tempo
+    command: [ "-config.file=/etc/tempo.yaml" ]
+    volumes:
+      - ./tempo.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200" # tempo
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - redpanda
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  redpanda:
+    image: redpandadata/redpanda:latest # for tempo ingest
+    container_name: redpanda
+    ports:
+      - "9092:9092"
+    networks:
+      - otel-network
+    restart: unless-stopped
+    command: >
+      redpanda start --overprovisioned
+      --mode=dev-container
+      --kafka-addr=PLAINTEXT://0.0.0.0:9092
+      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
+
+  jaeger:
+    image: jaegertracing/jaeger:latest
+    container_name: jaeger
+    environment:
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
+      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - ./jaeger.yaml:/etc/jaeger/config.yml
+      - jaeger-data:/badger
+    ports:
+      - "16686:16686" # Web UI
+      - "14269:14269" # Admin/Metrics
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  # --- Metrics ---
+
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: prometheus
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-data:/prometheus
+    ports:
+      - "9090:9090"
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--web.enable-otlp-receiver" # Enable OTLP
+      - "--web.enable-remote-write-receiver" # Enable remote write
+      - "--enable-feature=promql-experimental-functions" # Enable info()
+      - "--storage.tsdb.retention.time=30d"
+    restart: unless-stopped
+    networks:
+      - otel-network
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Logging ---
+
+  loki:
+    image: grafana/loki:latest
+    container_name: loki
+    volumes:
+      - ./loki.yaml:/etc/loki/loki.yaml:ro
+      - loki-data:/loki
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/loki.yaml
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  # --- Collection ---
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    ports:
+      - "1888:1888" # pprof
+      - "8888:8888" # Prometheus metrics for Collector
+      - "8889:8889" # Prometheus metrics for application indicators
+      - "13133:13133" # health check
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
+      - "55679:55679" # zpages
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Profiles ---
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    container_name: pyroscope
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - otel-network
+    restart: unless-stopped
+
+  # --- Visualization ---
+
+  grafana:
+    image: grafana/grafana:latest
+    container_name: grafana
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+      - GF_SECURITY_ADMIN_USER=admin
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/etc/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+    healthcheck:
+      test:
+        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  rustfs-data:
+  tempo-data:
+  jaeger-data:
+  prometheus-data:
+  loki-data:
+  grafana-data:
+
+networks:
+  otel-network:
+    driver: bridge
+    name: "network_otel"
+    ipam:
+      config:
+        - subnet: 172.28.0.0/16
+    driver_opts:
+      com.docker.network.enable_ipv6: "true"

.docker/observability/docker-compose-tempo-ha-override.yml

@@ -0,0 +1,62 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Docker Compose override file for High Availability Tempo setup
+#
+# Usage:
+# docker-compose -f docker-compose-example-for-rustfs.yml \
+#                -f docker-compose-tempo-ha-override.yml up
+
+services:
+  # Override Tempo to use high-availability configuration
+  tempo:
+    volumes:
+      - ./tempo-ha.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200"   # Tempo HTTP
+      - "4317:4317"   # OTLP gRPC
+      - "4318:4318"   # OTLP HTTP
+      - "7946:7946"   # Memberlist
+      - "14250:14250" # Jaeger gRPC
+      - "14268:14268" # Jaeger Thrift HTTP
+      - "9411:9411"   # Zipkin
+    environment:
+      - TEMPO_MEMBERLIST_BIND_PORT=7946
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    depends_on:
+      - redpanda
+
+volumes:
+  tempo-data:
+    driver: local
+    driver_opts:
+      type: tmpfs
+      device: tmpfs
+      o: "size=4g"  # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
+
+# Network configuration remains the same
+# networks:
+#   otel-network:
+#     driver: bridge
+#     name: "network_otel"
+#     ipam:
+#       config:
+#         - subnet: 172.28.0.0/16
+

.docker/observability/docker-compose.yml

@@ -14,97 +14,219 @@
 
 services:
 
-  tempo-init:
+  # --- Tracing ---
-    image: busybox:latest
-    command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
-    volumes:
-      - ./tempo-data:/var/tempo
-    user: root
-    networks:
-      - otel-network
-    restart: "no"
-
   tempo:
-    image: grafana/tempo:latest
+    image: grafana/tempo:2.10.3
-    user: "10001" # The container must be started with root to execute chown in the script
+    container_name: tempo
-    command: [ "-config.file=/etc/tempo.yaml" ] # This is passed as a parameter to the entry point script
+    depends_on:
+      - redpanda
+    command: [ "-config.file=/etc/tempo.yaml" ]
     volumes:
       - ./tempo.yaml:/etc/tempo.yaml:ro
-      - ./tempo-data:/var/tempo
+      - tempo-data:/var/tempo
     ports:
       - "3200:3200" # tempo
-      - "24317:4317" # otlp grpc
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+      - "7946" # memberlist
+    networks:
+      - otel-network
     restart: unless-stopped
-    networks:
+    healthcheck:
-      - otel-network
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
-
+      interval: 10s
-  otel-collector:
+      timeout: 5s
-    image: otel/opentelemetry-collector-contrib:latest
+      retries: 3
-    environment:
+      start_period: 15s
-      - TZ=Asia/Shanghai
+  redpanda:
-    volumes:
+    image: redpandadata/redpanda:latest
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
     ports:
-      - "1888:1888"
+      - "9092:9092" # Kafka API for clients
-      - "8888:8888"
+    command: >
-      - "8889:8889"
+      redpanda start --overprovisioned
-      - "13133:13133"
+      --mode=dev-container
-      - "4317:4317"
+      --kafka-addr=PLAINTEXT://0.0.0.0:9092
-      - "4318:4318"
+      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
-      - "55679:55679"
+
-    networks:
+  redpanda-console:
-      - otel-network
+    image: docker.redpanda.com/redpandadata/console:latest
+    environment:
+      - CONFIG_FILEPATH=/etc/redpanda/redpanda-console-config.yaml
+    volumes:
+      - ./redpanda-console.yaml:/etc/redpanda/redpanda-console-config.yaml
+    ports:
+      - "8080:8080"
+    depends_on:
+      - redpanda
+
+  vulture:
+    image: grafana/tempo-vulture:latest
+    restart: always
+    command:
+      [
+        "-prometheus-listen-address=:8080",
+        "-tempo-query-url=http://tempo:3200",
+        "-tempo-push-url=http://tempo:4317",
+      ]
+    depends_on:
+      - tempo
+
   jaeger:
     image: jaegertracing/jaeger:latest
+    container_name: jaeger
     environment:
-      - TZ=Asia/Shanghai
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
+      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - ./jaeger.yaml:/etc/jaeger/config.yml
+      - jaeger-data:/badger
     ports:
-      - "16686:16686"
+      - "16686:16686" # Web UI
-      - "14317:4317"
+      - "14269:14269" # Admin/Metrics
-      - "14318:4318"
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
     networks:
       - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  # --- Metrics ---
+
   prometheus:
     image: prom/prometheus:latest
-    environment:
+    container_name: prometheus
-      - TZ=Asia/Shanghai
     volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-data:/prometheus
     ports:
       - "9090:9090"
     command:
-      - '--config.file=/etc/prometheus/prometheus.yml'
+      - "--config.file=/etc/prometheus/prometheus.yml"
-      - '--web.enable-otlp-receiver' # Enable OTLP
+      - "--web.enable-otlp-receiver" # Enable OTLP
-      - '--enable-feature=promql-experimental-functions' # Enable info()
+      - "--web.enable-remote-write-receiver" # Enable remote write
+      - "--enable-feature=promql-experimental-functions" # Enable info()
+      - "--storage.tsdb.retention.time=30d"
+    restart: unless-stopped
     networks:
       - otel-network
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Logging ---
+
   loki:
     image: grafana/loki:latest
-    environment:
+    container_name: loki
-      - TZ=Asia/Shanghai
     volumes:
-      - ./loki-config.yaml:/etc/loki/local-config.yaml
+      - ./loki.yaml:/etc/loki/loki.yaml:ro
+      - loki-data:/loki
     ports:
       - "3100:3100"
-    command: -config.file=/etc/loki/local-config.yaml
+    command: -config.file=/etc/loki/loki.yaml
     networks:
       - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  # --- Collection ---
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    ports:
+      - "1888:1888" # pprof
+      - "8888:8888" # Prometheus metrics for Collector
+      - "8889:8889" # Prometheus metrics for application indicators
+      - "13133:13133" # health check
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
+      - "55679:55679" # zpages
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Profiles ---
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    container_name: pyroscope
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - otel-network
+    restart: unless-stopped
+
+  # --- Visualization ---
+
   grafana:
     image: grafana/grafana:latest
+    container_name: grafana
     ports:
-      - "3000:3000"  # Web UI
+      - "3000:3000"
-    volumes:
-      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
     environment:
       - GF_SECURITY_ADMIN_PASSWORD=admin
-      - TZ=Asia/Shanghai
+      - GF_SECURITY_ADMIN_USER=admin
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/etc/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
     networks:
       - otel-network
+    restart: unless-stopped
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+    healthcheck:
+      test:
+        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
 
+volumes:
+  tempo-data:
+  jaeger-data:
+  prometheus-data:
+  loki-data:
+  grafana-data:
 
 networks:
   otel-network:
     driver: bridge
-    name: "network_otel_config"
+    name: "network_otel"
+    ipam:
+      config:
+        - subnet: 172.28.0.0/16
     driver_opts:
       com.docker.network.enable_ipv6: "true"

.docker/observability/grafana-datasources.yaml

@@ -1,108 +0,0 @@
-apiVersion: 1
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    uid: prometheus
-    access: proxy
-    orgId: 1
-    url: http://prometheus:9090
-    basicAuth: false
-    isDefault: false
-    version: 1
-    editable: false
-    jsonData:
-      httpMethod: GET
-  - name: Tempo
-    type: tempo
-    access: proxy
-    orgId: 1
-    url: http://tempo:3200
-    basicAuth: false
-    isDefault: true
-    version: 1
-    editable: false
-    apiVersion: 1
-    uid: tempo
-    jsonData:
-      httpMethod: GET
-      serviceMap:
-        datasourceUid: prometheus
-      streamingEnabled:
-        search: true
-      tracesToLogsV2:
-        # Field with an internal link pointing to a logs data source in Grafana.
-        # datasourceUid value must match the uid value of the logs data source.
-        datasourceUid: 'loki'
-        spanStartTimeShift: '-1h'
-        spanEndTimeShift: '1h'
-        tags: [ 'job', 'instance', 'pod', 'namespace' ]
-        filterByTraceID: false
-        filterBySpanID: false
-        customQuery: true
-        query: 'method="$${__span.tags.method}"'
-        tracesToMetrics:
-          datasourceUid: 'prom'
-          spanStartTimeShift: '-1h'
-          spanEndTimeShift: '1h'
-          tags: [ { key: 'service.name', value: 'service' }, { key: 'job' } ]
-          queries:
-            - name: 'Sample query'
-              query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
-        tracesToProfiles:
-          datasourceUid: 'grafana-pyroscope-datasource'
-          tags: [ 'job', 'instance', 'pod', 'namespace' ]
-          profileTypeId: 'process_cpu:cpu:nanoseconds:cpu:nanoseconds'
-          customQuery: true
-          query: 'method="$${__span.tags.method}"'
-        serviceMap:
-          datasourceUid: 'prometheus'
-        nodeGraph:
-          enabled: true
-        search:
-          hide: false
-        traceQuery:
-          timeShiftEnabled: true
-          spanStartTimeShift: '-1h'
-          spanEndTimeShift: '1h'
-        spanBar:
-          type: 'Tag'
-          tag: 'http.path'
-        streamingEnabled:
-          search: true
-  - name: Jaeger
-    type: jaeger
-    uid: Jaeger
-    url: http://jaeger:16686
-    basicAuth: false
-    access: proxy
-    readOnly: false
-    isDefault: false
-    jsonData:
-      tracesToLogsV2:
-        # Field with an internal link pointing to a logs data source in Grafana.
-        # datasourceUid value must match the uid value of the logs data source.
-        datasourceUid: 'loki'
-        spanStartTimeShift: '1h'
-        spanEndTimeShift: '-1h'
-        tags: [ 'job', 'instance', 'pod', 'namespace' ]
-        filterByTraceID: false
-        filterBySpanID: false
-        customQuery: true
-        query: 'method="$${__span.tags.method}"'
-      tracesToMetrics:
-        datasourceUid: 'prom'
-        spanStartTimeShift: '1h'
-        spanEndTimeShift: '-1h'
-        tags: [ { key: 'service.name', value: 'service' }, { key: 'job' } ]
-        queries:
-          - name: 'Sample query'
-            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
-      nodeGraph:
-        enabled: true
-      traceQuery:
-        timeShiftEnabled: true
-        spanStartTimeShift: '1h'
-        spanEndTimeShift: '-1h'
-      spanBar:
-        type: 'None'

.docker/observability/grafana/provisioning/dashboards/dashboard.yml

@@ -0,0 +1,25 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: 1
+
+providers:
+  - name: "default"
+    orgId: 1
+    folder: ""
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    options:
+      path: /etc/grafana/dashboards

.docker/observability/grafana/provisioning/datasources.yaml

@@ -0,0 +1,97 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    uid: prometheus
+    access: proxy
+    orgId: 1
+    url: http://prometheus:9090
+    isDefault: true
+    version: 1
+    editable: false
+    jsonData:
+      httpMethod: GET
+      exemplarTraceIdDestinations:
+        - name: trace_id
+          datasourceUid: tempo
+
+  - name: Tempo
+    type: tempo
+    uid: tempo
+    access: proxy
+    orgId: 1
+    url: http://tempo:3200
+    isDefault: false
+    version: 1
+    editable: false
+    jsonData:
+      httpMethod: GET
+      serviceMap:
+        datasourceUid: prometheus
+      tracesToLogs:
+        datasourceUid: loki
+        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
+        mappedTags: [ { key: 'service.name', value: 'app' } ]
+        spanStartTimeShift: '1s'
+        spanEndTimeShift: '-1s'
+        filterByTraceID: true
+        filterBySpanID: false
+      tracesToMetrics:
+        datasourceUid: prometheus
+        tags: [ { key: 'service.name' }, { key: 'job' } ]
+        queries:
+          - name: 'Service-Level Latency'
+            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
+          - name: 'Service-Level Calls'
+            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
+          - name: 'Service-Level Errors'
+            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
+      nodeGraph:
+        enabled: true
+
+  - name: Loki
+    type: loki
+    uid: loki
+    orgId: 1
+    url: http://loki:3100
+    isDefault: false
+    version: 1
+    editable: false
+    jsonData:
+      derivedFields:
+        - datasourceUid: tempo
+          matcherRegex: 'trace_id=(\w+)'
+          name: 'TraceID'
+          url: '$${__value.raw}'
+
+  - name: Jaeger
+    type: jaeger
+    uid: jaeger
+    url: http://jaeger:16686
+    access: proxy
+    isDefault: false
+    editable: false
+    jsonData:
+      tracesToLogs:
+        datasourceUid: loki
+        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
+        mappedTags: [ { key: 'service.name', value: 'app' } ]
+        spanStartTimeShift: '1s'
+        spanEndTimeShift: '-1s'
+        filterByTraceID: true
+        filterBySpanID: false

.docker/observability/grafana/provisioning/datasources/datasources.yaml

@@ -0,0 +1,98 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    uid: prometheus
+    url: http://prometheus:9090
+    access: proxy
+    isDefault: true
+    editable: false
+    jsonData:
+      httpMethod: GET
+      exemplarTraceIdDestinations:
+        - name: trace_id
+          datasourceUid: tempo
+
+  - name: Tempo
+    type: tempo
+    uid: tempo
+    access: proxy
+    url: http://tempo:3200
+    isDefault: false
+    editable: false
+    jsonData:
+      httpMethod: GET
+      serviceMap:
+        datasourceUid: prometheus
+      tracesToLogs:
+        datasourceUid: loki
+        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
+        mappedTags: [ { key: 'service.name', value: 'app' } ]
+        spanStartTimeShift: '-1h'
+        spanEndTimeShift: '1h'
+        filterByTraceID: true
+        filterBySpanID: false
+      tracesToMetrics:
+        datasourceUid: prometheus
+        tags: [ { key: 'service.name' }, { key: 'job' } ]
+        queries:
+          - name: 'Service-Level Latency'
+            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
+          - name: 'Service-Level Calls'
+            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
+          - name: 'Service-Level Errors'
+            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
+      nodeGraph:
+        enabled: true
+
+  - name: Loki
+    type: loki
+    uid: loki
+    url: http://loki:3100
+    basicAuth: false
+    isDefault: false
+    editable: false
+    jsonData:
+      derivedFields:
+        - datasourceUid: tempo
+          matcherRegex: 'trace_id=(\w+)'
+          name: 'TraceID'
+          url: '$${__value.raw}'
+
+  - name: Jaeger
+    type: jaeger
+    uid: jaeger
+    url: http://jaeger:16686
+    access: proxy
+    isDefault: false
+    editable: false
+    jsonData:
+      tracesToLogs:
+        datasourceUid: loki
+        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
+        mappedTags: [ { key: 'service.name', value: 'app' } ]
+        spanStartTimeShift: '1s'
+        spanEndTimeShift: '-1s'
+        filterByTraceID: true
+        filterBySpanID: false
+
+  - name: Pyroscope
+    type: grafana-pyroscope-datasource
+    url: http://pyroscope:4040
+    jsonData:
+      minStep: '15s'

.docker/observability/jaeger-config.yaml

@@ -1,112 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-service:
-  extensions: [ jaeger_storage, jaeger_query, remote_sampling, healthcheckv2 ]
-  pipelines:
-    traces:
-      receivers: [ otlp, jaeger, zipkin ]
-      processors: [ batch, adaptive_sampling ]
-      exporters: [ jaeger_storage_exporter ]
-  telemetry:
-    resource:
-      service.name: jaeger
-    metrics:
-      level: detailed
-      readers:
-        - pull:
-            exporter:
-              prometheus:
-                host: 0.0.0.0
-                port: 8888
-    logs:
-      level: debug
-    # TODO Initialize telemetry tracer once OTEL released new feature.
-    # https://github.com/open-telemetry/opentelemetry-collector/issues/10663
-
-extensions:
-  healthcheckv2:
-    use_v2: true
-    http:
-
-  # pprof:
-  #   endpoint: 0.0.0.0:1777
-  # zpages:
-  #   endpoint: 0.0.0.0:55679
-
-  jaeger_query:
-    storage:
-      traces: some_store
-      traces_archive: another_store
-    ui:
-      config_file: ./cmd/jaeger/config-ui.json
-      log_access: true
-    # The maximum duration that is considered for clock skew adjustments.
-    # Defaults to 0 seconds, which means it's disabled.
-    max_clock_skew_adjust: 0s
-    grpc:
-      endpoint: 0.0.0.0:16685
-    http:
-      endpoint: 0.0.0.0:16686
-
-  jaeger_storage:
-    backends:
-      some_store:
-        memory:
-          max_traces: 1000000
-      another_store:
-        memory:
-          max_traces: 1000000
-    metric_backends:
-      some_metrics_storage:
-        prometheus:
-          endpoint: http://prometheus:9090
-          normalize_calls: true
-          normalize_duration: true
-
-  remote_sampling:
-    # You can either use file or adaptive sampling strategy in remote_sampling
-    # file:
-    #   path: ./cmd/jaeger/sampling-strategies.json
-    adaptive:
-      sampling_store: some_store
-      initial_sampling_probability: 0.1
-    http:
-    grpc:
-
-receivers:
-  otlp:
-    protocols:
-      grpc:
-      http:
-
-  jaeger:
-    protocols:
-      grpc:
-      thrift_binary:
-      thrift_compact:
-      thrift_http:
-
-  zipkin:
-
-processors:
-  batch:
-  # Adaptive Sampling Processor is required to support adaptive sampling.
-  # It expects remote_sampling extension with `adaptive:` config to be enabled.
-  adaptive_sampling:
-
-exporters:
-  jaeger_storage_exporter:
-    trace_storage: some_store
-

.docker/observability/jaeger.yaml

@@ -0,0 +1,74 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+service:
+  extensions: [jaeger_storage, jaeger_query]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [jaeger_storage_exporter, spanmetrics]
+    metrics/spanmetrics:
+      receivers: [spanmetrics]
+      exporters: [prometheus]
+  telemetry:
+    resource:
+      service.name: jaeger
+    metrics:
+      level: detailed
+      readers:
+        - pull:
+            exporter:
+              prometheus:
+                host: 0.0.0.0
+                port: 8888
+    logs:
+      level: DEBUG
+
+extensions:
+  jaeger_query:
+    storage:
+      traces: some_storage
+      metrics: some_metrics_storage
+  jaeger_storage:
+    backends:
+      some_storage:
+        memory:
+          max_traces: 100000
+    metric_backends:
+      some_metrics_storage:
+        prometheus:
+          endpoint: http://prometheus:9090
+          normalize_calls: true
+          normalize_duration: true
+
+connectors:
+  spanmetrics:
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: "0.0.0.0:4317"
+      http:
+        endpoint: "0.0.0.0:4318"
+
+processors:
+  batch:
+
+exporters:
+  jaeger_storage_exporter:
+    trace_storage: some_storage
+  prometheus:
+    endpoint: "0.0.0.0:8889"

.docker/observability/loki.yaml

@@ -11,22 +11,21 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
 auth_enabled: false
 
 server:
   http_listen_port: 3100
-  grpc_listen_port: 9096
+  grpc_listen_port: 9095
-  log_level: debug
+  log_level: info
   grpc_server_max_concurrent_streams: 1000
 
 common:
   instance_addr: 127.0.0.1
-  path_prefix: /tmp/loki
+  path_prefix: /loki
   storage:
     filesystem:
-      chunks_directory: /tmp/loki/chunks
+      chunks_directory: /loki/chunks
-      rules_directory: /tmp/loki/rules
+      rules_directory: /loki/rules
   replication_factor: 1
   ring:
     kvstore:
@@ -39,9 +38,6 @@ query_range:
         enabled: true
         max_size_mb: 100
 
-limits_config:
-  metric_aggregation_enabled: true
-
 schema_config:
   configs:
     - from: 2020-10-24
@@ -52,27 +48,16 @@ schema_config:
         prefix: index_
         period: 24h
 
+limits_config:
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+  allow_structured_metadata: true
+  max_line_size: 256KB
+
 pattern_ingester:
   enabled: true
   metric_aggregation:
     loki_address: localhost:3100
 
-ruler:
-  alertmanager_url: http://localhost:9093
-
 frontend:
   encoding: protobuf
-
-
-# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
-# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
-#
-# Statistics help us better understand how Loki is used, and they show us performance
-# levels for most users. This helps us prioritize features and documentation.
-# For more information on what's sent, look at
-# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
-# Refer to the buildReport method to see what goes into a report.
-#
-# If you would like to disable reporting, uncomment the following lines:
-#analytics:
-#  reporting_enabled: false

.docker/observability/otel-collector-config.yaml

@@ -15,66 +15,102 @@
 receivers:
   otlp:
     protocols:
-      grpc: # OTLP gRPC 接收器
+      grpc:
         endpoint: 0.0.0.0:4317
-      http: # OTLP HTTP 接收器
+      http:
         endpoint: 0.0.0.0:4318
 
 processors:
-  batch: # 批处理处理器，提升吞吐量
+  batch:
-    timeout: 5s
+    timeout: 1s
-    send_batch_size: 1000
+    send_batch_size: 1024
   memory_limiter:
     check_interval: 1s
-    limit_mib: 512
+    limit_mib: 1024
+    spike_limit_mib: 256
+  transform/logs:
+    log_statements:
+      - context: log
+        statements:
+          - set(attributes["message"], body.string)
+          - set(attributes["log.body"], body.string)
 
 exporters:
-  otlp/traces: # OTLP 导出器，用于跟踪数据
+  otlp/tempo:
-    endpoint: "jaeger:4317"  # Jaeger 的 OTLP gRPC 端点
+    endpoint: "tempo:4317"
-    tls:
-      insecure: true  # 开发环境禁用 TLS，生产环境需配置证书
-  otlp/tempo: # OTLP 导出器，用于跟踪数据
-    endpoint: "tempo:4317"  # tempo 的 OTLP gRPC 端点
-    tls:
-      insecure: true  # 开发环境禁用 TLS，生产环境需配置证书
-  prometheus: # Prometheus 导出器，用于指标数据
-    endpoint: "0.0.0.0:8889"  # Prometheus 刮取端点
-    namespace: "rustfs"  # 指标前缀
-    send_timestamps: true  # 发送时间戳
-    # enable_open_metrics: true
-  otlphttp/loki: # Loki 导出器，用于日志数据
-    endpoint: "http://loki:3100/otlp/v1/logs"
     tls:
       insecure: true
+    compression: gzip
+    retry_on_failure:
+      enabled: true
+      initial_interval: 1s
+      max_interval: 30s
+      max_elapsed_time: 300s
+    sending_queue:
+      enabled: true
+      num_consumers: 10
+      queue_size: 5000
+
+  otlp/jaeger:
+    endpoint: "jaeger:4317"
+    tls:
+      insecure: true
+    compression: gzip
+    retry_on_failure:
+      enabled: true
+      initial_interval: 1s
+      max_interval: 30s
+      max_elapsed_time: 300s
+    sending_queue:
+      enabled: true
+      num_consumers: 10
+      queue_size: 5000
+
+  prometheus:
+    endpoint: "0.0.0.0:8889"
+    send_timestamps: true
+    metric_expiration: 5m
+    resource_to_telemetry_conversion:
+      enabled: true
+
+  otlphttp/loki:
+    endpoint: "http://loki:3100/otlp"
+    tls:
+      insecure: true
+    compression: gzip
+
 extensions:
   health_check:
+    endpoint: 0.0.0.0:13133
   pprof:
+    endpoint: 0.0.0.0:1888
   zpages:
+    endpoint: 0.0.0.0:55679
+
 service:
-  extensions: [ health_check, pprof, zpages ]  # 启用扩展
+  extensions: [ health_check, pprof, zpages ]
   pipelines:
     traces:
       receivers: [ otlp ]
-      processors: [ memory_limiter,batch ]
+      processors: [ memory_limiter, batch ]
-      exporters: [ otlp/traces,otlp/tempo ]
+      exporters: [ otlp/tempo, otlp/jaeger ]
     metrics:
       receivers: [ otlp ]
       processors: [ batch ]
       exporters: [ prometheus ]
     logs:
       receivers: [ otlp ]
-      processors: [ batch ]
+      processors: [ batch, transform/logs ]
       exporters: [ otlphttp/loki ]
   telemetry:
     logs:
-      level: "info"  # Collector 日志级别
+      level: "info"
+      encoding: "json"
     metrics:
-      level: "detailed" # 可以是 basic, normal, detailed
+      level: "normal"
       readers:
-        - periodic:
+        - pull:
             exporter:
-              otlp:
+              prometheus:
-                protocol: http/protobuf
+                host: '0.0.0.0'
-                endpoint: http://otel-collector:4318
+                port: 8888
-
-

.docker/observability/prometheus.yml

@@ -14,20 +14,48 @@
 
 global:
   scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
+  evaluation_interval: 15s
+  external_labels:
+    cluster: 'rustfs-dev'  # Label to identify the cluster
+    replica: '1'  # Replica identifier
 
 scrape_configs:
   - job_name: 'otel-collector'
     static_configs:
       - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
-  - job_name: 'otel-metrics'
+    scrape_interval: 10s
+
+  - job_name: 'rustfs-app-metrics'
     static_configs:
       - targets: [ 'otel-collector:8889' ]  # Application indicators
+    scrape_interval: 15s
+    metric_relabel_configs:
+      - source_labels: [ __name__ ]
+        regex: 'go_.*'
+        action: drop  # Drop Go runtime metrics if not needed
+
   - job_name: 'tempo'
     static_configs:
       - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
 
+  - job_name: 'jaeger'
+    static_configs:
+      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
+
+  - job_name: 'loki'
+    static_configs:
+      - targets: [ 'loki:3100' ]
+
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: [ 'localhost:9090' ]
+
+  - job_name: 'vulture'
+    static_configs:
+      - targets:
+          - 'vulture:8080'
+
 otlp:
-  # Recommended attributes to be promoted to labels.
   promote_resource_attributes:
     - service.instance.id
     - service.name
@@ -46,10 +74,8 @@ otlp:
     - k8s.pod.name
     - k8s.replicaset.name
     - k8s.statefulset.name
-  # Ingest OTLP data keeping all characters in metric/label names.
   translation_strategy: NoUTF8EscapingWithSuffixes
 
 storage:
-  # OTLP is a push-based protocol, Out of order samples is a common scenario.
   tsdb:
-    out_of_order_time_window: 30m
+    out_of_order_time_window: 30m

.docker/observability/redpanda-console.yaml

@@ -0,0 +1,3 @@
+kafka:
+  brokers:
+    - redpanda:9092

.docker/observability/tempo-data/.gitignore

@@ -1 +0,0 @@
-*

.docker/observability/tempo-ha.yaml

@@ -0,0 +1,286 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
+# Features:
+# - Distributed architecture with multiple components
+# - Kafka-based ingestion for fault tolerance
+# - Replication factor of 3 for data resilience
+# - Query frontend for load balancing
+# - Metrics generation from traces
+# - WAL for durability
+
+partition_ring_live_store: true
+stream_over_http_enabled: true
+
+server:
+  http_listen_port: 3200
+  http_server_read_timeout: 30s
+  http_server_write_timeout: 30s
+  grpc_server_max_recv_msg_size: 4194304  # 4MB
+  grpc_server_max_send_msg_size: 4194304
+  log_level: info
+  log_format: json
+
+# Memberlist configuration for distributed mode
+memberlist:
+  node_name: tempo
+  bind_port: 7946
+  join_members:
+    - tempo:7946
+  retransmit_factor: 4
+  node_timeout: 15s
+  retransmit_interval: 300ms
+  dead_node_reclaim_time: 30s
+
+# Distributor configuration - receives traces and routes to ingesters
+distributor:
+  ingester_write_path_enabled: true
+  kafka_write_path_enabled: true
+  rate_limit_bytes: 10MB
+  rate_limit_enabled: true
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:4317"
+          max_concurrent_streams: 0
+          max_receive_message_size: 4194304
+        http:
+          endpoint: "0.0.0.0:4318"
+          cors:
+            allowed_origins:
+              - "*"
+            max_age: 86400
+    jaeger:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:14250"
+        thrift_http:
+          endpoint: "0.0.0.0:14268"
+    zipkin:
+      endpoint: "0.0.0.0:9411"
+  ring:
+    kvstore:
+      store: memberlist
+    heartbeat_timeout: 5s
+    replication_factor: 3
+    heartbeat_interval: 5s
+
+# Ingester configuration - stores traces and querying
+ingester:
+  lifecycler:
+    address: tempo
+    ring:
+      kvstore:
+        store: memberlist
+      replication_factor: 3
+      max_cache_freshness_per_sec: 10s
+      heartbeat_interval: 5s
+      heartbeat_timeout: 5s
+    num_tokens: 128
+    tokens_file_path: /var/tempo/tokens.json
+    claim_on_rollout: true
+  trace_idle_period: 20s
+  max_block_bytes: 10_000_000
+  max_block_duration: 10m
+  chunk_size_bytes: 1_000_000
+  chunk_encoding: snappy
+  wal:
+    checkpoint_duration: 5s
+    max_wal_blocks: 4
+  metrics:
+    enabled: true
+    level: block
+    target_info_duration: 15m
+
+# WAL configuration for data durability
+wal:
+  checkpoint_duration: 5s
+  flush_on_shutdown: true
+  path: /var/tempo/wal
+
+# Kafka ingestion configuration - for high throughput scenarios
+ingest:
+  enabled: true
+  kafka:
+    brokers: [ redpanda:9092 ]
+    topic: tempo-ingest
+    encoding: protobuf
+    consumer_group: tempo-ingest-consumer
+    session_timeout: 10s
+    rebalance_timeout: 1m
+    partition: auto
+    verbosity: 2
+
+# Query frontend configuration - distributed querying
+query_frontend:
+  compression: gzip
+  downstream_url: http://localhost:3200
+  log_queries_longer_than: 5s
+  cache_uncompressed_bytes: 100MB
+  max_outstanding_requests_per_tenant: 100
+  max_query_length: 48h
+  max_query_lookback: 30d
+  default_result_cache_ttl: 1m
+  result_cache:
+    cache:
+      enable_fifocache: true
+      default_validity: 1m
+  rf1_after: "1999-01-01T00:00:00Z"
+  mcp_server:
+    enabled: true
+
+# Querier configuration - queries traces
+querier:
+  frontend_worker:
+    frontend_address: localhost:3200
+    grpc_client_config:
+      max_recv_msg_size: 104857600
+  max_concurrent_queries: 20
+  max_metric_bytes_per_trace: 1MB
+
+# Query scheduler configuration - for distributed querying
+query_scheduler:
+  use_scheduler_ring: false
+
+# Metrics generator configuration - generates metrics from traces
+metrics_generator:
+  enabled: true
+  registry:
+    enabled: true
+    external_labels:
+      source: tempo
+      cluster: rustfs-docker-ha
+      environment: production
+  storage:
+    path: /var/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+        resource_to_telemetry_conversion:
+          enabled: true
+  processor:
+    batch:
+      timeout: 10s
+      send_batch_size: 1024
+    memory_limiter:
+      check_interval: 5s
+      limit_mib: 512
+      spike_limit_mib: 128
+  processors:
+    - span-metrics
+    - local-blocks
+    - service-graphs
+  generate_native_histograms: both
+
+# Backend worker configuration
+backend_worker:
+  backend_scheduler_addr: localhost:3200
+  compaction:
+    block_retention: 24h
+    compacted_block_retention: 1h
+  ring:
+    kvstore:
+      store: memberlist
+
+# Backend scheduler configuration
+backend_scheduler:
+  enabled: true
+  provider:
+    compaction:
+      compaction:
+        block_retention: 24h
+        compacted_block_retention: 1h
+        concurrency: 25
+        v2_out_path: /var/tempo/blocks/compaction
+
+# Storage configuration - local backend with proper retention
+storage:
+  trace:
+    backend: local
+    wal:
+      path: /var/tempo/wal
+      checkpoint_duration: 5s
+      flush_on_shutdown: true
+    local:
+      path: /var/tempo/blocks
+      bloom_filter_false_positive: 0.05
+      bloom_shift: 4
+      index:
+        downsample_bytes: 1000000
+        page_size_bytes: 0
+        cache_size_bytes: 0
+    pool:
+      max_workers: 400
+      queue_depth: 10000
+
+# Compactor configuration - manages block compaction
+compactor:
+  compaction:
+    block_retention: 168h  # 7 days
+    compacted_block_retention: 1h
+    concurrency: 25
+    v2_out_path: /var/tempo/blocks/compaction
+    shard_count: 32
+    max_block_bytes: 107374182400  # 100GB
+    max_compaction_objects: 6000000
+    max_time_per_tenant: 5m
+    block_size_bytes: 107374182400
+  ring:
+    kvstore:
+      store: memberlist
+    heartbeat_interval: 5s
+    heartbeat_timeout: 5s
+
+# Limits configuration - rate limiting and quotas
+limits:
+  max_traces_per_user: 10000
+  max_bytes_per_trace: 10485760  # 10MB
+  max_search_bytes_per_trace: 0
+  forgiving_oversize_traces: true
+  rate_limit_bytes: 10MB
+  rate_limit_enabled: true
+  ingestion_burst_size_bytes: 20MB
+  ingestion_rate_limit_bytes: 10MB
+  max_bytes_per_second: 10485760
+  metrics_generator_max_active_series: 10000
+  metrics_generator_max_churned_series: 10000
+  metrics_generator_forta_out_of_order_ttl: 5m
+
+# Override configuration
+overrides:
+  defaults:
+    metrics_generator:
+      processors:
+        - span-metrics
+        - local-blocks
+        - service-graphs
+      generate_native_histograms: both
+      max_active_series: 10000
+      max_churned_series: 10000
+
+# Usage reporting configuration
+usage_report:
+  reporting_enabled: false
+
+# Tracing configuration for debugging
+tracing:
+  enabled: true
+  jaeger:
+    sampler:
+      name: probabilistic
+      param: 0.1
+    reporter_log_spans: false
+

.docker/observability/tempo.yaml

@@ -1,31 +1,59 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+partition_ring_live_store: true
 stream_over_http_enabled: true
+
 server:
   http_listen_port: 3200
   log_level: info
 
-query_frontend:
+memberlist:
-  search:
+  node_name: tempo
-    duration_slo: 5s
+  bind_port: 7946
-    throughput_bytes_slo: 1.073741824e+09
+  join_members:
-    metadata_slo:
+    - tempo:7946
-      duration_slo: 5s
-      throughput_bytes_slo: 1.073741824e+09
-  trace_by_id:
-    duration_slo: 5s
 
 distributor:
   receivers:
     otlp:
       protocols:
         grpc:
-          endpoint: "tempo:4317"
+          endpoint: "0.0.0.0:4317"
+        http:
+          endpoint: "0.0.0.0:4318"
+  #log_received_spans:
+  #    enabled: true
+  #  log_discarded_spans:
+  #    enabled: true
 
-ingester:
+backend_scheduler:
-  max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally
+  provider:
+    compaction:
+      compaction:
+        block_retention: 1h
 
-compactor:
+backend_worker:
+  backend_scheduler_addr: localhost:3200
   compaction:
-    block_retention: 1h # overall Tempo trace retention. set for demo purposes
+    block_retention: 1h
+  ring:
+    kvstore:
+      store: memberlist
+
+querier:
+  query_live_store: true
 
 metrics_generator:
   registry:
@@ -37,19 +65,35 @@ metrics_generator:
     remote_write:
       - url: http://prometheus:9090/api/v1/write
         send_exemplars: true
-  traces_storage:
+
-    path: /var/tempo/generator/traces
+query_frontend:
+  rf1_after: "1999-01-01T00:00:00Z"
+  mcp_server:
+    enabled: true
 
 storage:
   trace:
-    backend: local # backend configuration to use
+    backend: local
     wal:
       path: /var/tempo/wal # where to store the wal locally
     local:
-      path: /var/tempo/blocks
+      path: /var/tempo/blocks # where to store the traces locally
 
 overrides:
   defaults:
     metrics_generator:
-      processors: [ service-graphs, span-metrics, local-blocks ] # enables metrics generator
+      processors: [ "span-metrics", "service-graphs", "local-blocks" ]
-      generate_native_histograms: both
+      generate_native_histograms: both
+
+ingest:
+  enabled: true
+  kafka:
+    address: redpanda:9092
+    topic: tempo-ingest
+
+block_builder:
+  consume_cycle_duration: 30s
+
+usage_report:
+  reporting_enabled: false
+

.docker/openobserve-otel/README.md

@@ -5,71 +5,57 @@
 
 English | [中文](README_ZH.md)
 
-This directory contains the configuration files for setting up an observability stack with OpenObserve and OpenTelemetry
+This directory contains the configuration for an **alternative** observability stack using OpenObserve.
-Collector.
 
-### Overview
+## ⚠️ Note
 
-This setup provides a complete observability solution for your applications:
+For the **recommended** observability stack (Prometheus, Grafana, Tempo, Loki), please see `../observability/`.
 
-- **OpenObserve**: A modern, open-source observability platform for logs, metrics, and traces.
+## 🌟 Overview
-- **OpenTelemetry Collector**: Collects and processes telemetry data before sending it to OpenObserve.
 
-### Setup Instructions
+OpenObserve is a lightweight, all-in-one observability platform that handles logs, metrics, and traces in a single binary. This setup is ideal for:
+- Resource-constrained environments.
+- Quick setup and testing.
+- Users who prefer a unified UI.
 
-1. **Prerequisites**:
+## 🚀 Quick Start
-    - Docker and Docker Compose installed
-    - Sufficient memory resources (minimum 2GB recommended)
 
-2. **Starting the Services**:
+### 1. Start Services
-   ```bash
-   cd .docker/openobserve-otel
-   docker compose -f docker-compose.yml  up -d
-   ```
-
-3. **Accessing the Dashboard**:
-    - OpenObserve UI: http://localhost:5080
-    - Default credentials:
-        - Username: root@rustfs.com
-        - Password: rustfs123
-
-### Configuration
-
-#### OpenObserve Configuration
-
-The OpenObserve service is configured with:
-
-- Root user credentials
-- Data persistence through a volume mount
-- Memory cache enabled
-- Health checks
-- Exposed ports:
-    - 5080: HTTP API and UI
-    - 5081: OTLP gRPC
-
-#### OpenTelemetry Collector Configuration
-
-The collector is configured to:
-
-- Receive telemetry data via OTLP (HTTP and gRPC)
-- Collect logs from files
-- Process data in batches
-- Export data to OpenObserve
-- Manage memory usage
-
-### Integration with Your Application
-
-To send telemetry data from your application, configure your OpenTelemetry SDK to send data to:
-
-- OTLP gRPC: `localhost:4317`
-- OTLP HTTP: `localhost:4318`
-
-For example, in a Rust application using the `rustfs-obs` library:
 
 ```bash
-export RUSTFS_OBS_ENDPOINT=http://localhost:4317
+cd .docker/openobserve-otel
-export RUSTFS_OBS_SERVICE_NAME=yourservice
+docker compose up -d
-export RUSTFS_OBS_SERVICE_VERSION=1.0.0
-export RUSTFS_OBS_ENVIRONMENT=development
 ```
 
+### 2. Access Dashboard
+
+- **URL**: [http://localhost:5080](http://localhost:5080)
+- **Username**: `root@rustfs.com`
+- **Password**: `rustfs123`
+
+## 🛠️ Configuration
+
+### OpenObserve
+
+- **Persistence**: Data is persisted to a Docker volume.
+- **Ports**:
+    - `5080`: HTTP API and UI
+    - `5081`: OTLP gRPC
+
+### OpenTelemetry Collector
+
+- **Receivers**: OTLP (gRPC `4317`, HTTP `4318`)
+- **Exporters**: Sends data to OpenObserve.
+
+## 🔗 Integration
+
+Configure your application to send OTLP data to the collector:
+
+- **Endpoint**: `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC)
+
+Example for RustFS:
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://localhost:4318
+export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
+```

.docker/openobserve-otel/README_ZH.md

@@ -5,71 +5,57 @@
 
 [English](README.md) | 中文
 
-## 中文
+本目录包含使用 OpenObserve 的**替代**可观测性技术栈配置。
 
-本目录包含搭建 OpenObserve 和 OpenTelemetry Collector 可观测性栈的配置文件。
+## ⚠️ 注意
 
-### 概述
+对于**推荐**的可观测性技术栈（Prometheus, Grafana, Tempo, Loki），请参阅 `../observability/`。
 
-此设置为应用程序提供了完整的可观测性解决方案：
+## 🌟 概览
 
-- **OpenObserve**：现代化、开源的可观测性平台，用于日志、指标和追踪。
+OpenObserve 是一个轻量级、一体化的可观测性平台，在一个二进制文件中处理日志、指标和追踪。此设置非常适合：
-- **OpenTelemetry Collector**：收集和处理遥测数据，然后将其发送到 OpenObserve。
+- 资源受限的环境。
+- 快速设置和测试。
+- 喜欢统一 UI 的用户。
 
-### 设置说明
+## 🚀 快速开始
 
-1. **前提条件**：
+### 1. 启动服务
-    - 已安装 Docker 和 Docker Compose
-    - 足够的内存资源（建议至少 2GB）
-
-2. **启动服务**：
-   ```bash
-   cd .docker/openobserve-otel
-   docker compose -f docker-compose.yml  up -d
-   ```
-
-3. **访问仪表板**：
-    - OpenObserve UI：http://localhost:5080
-    - 默认凭据：
-        - 用户名：root@rustfs.com
-        - 密码：rustfs123
-
-### 配置
-
-#### OpenObserve 配置
-
-OpenObserve 服务配置：
-
-- 根用户凭据
-- 通过卷挂载实现数据持久化
-- 启用内存缓存
-- 健康检查
-- 暴露端口：
-    - 5080：HTTP API 和 UI
-    - 5081：OTLP gRPC
-
-#### OpenTelemetry Collector 配置
-
-收集器配置为：
-
-- 通过 OTLP（HTTP 和 gRPC）接收遥测数据
-- 从文件中收集日志
-- 批处理数据
-- 将数据导出到 OpenObserve
-- 管理内存使用
-
-### 与应用程序集成
-
-要从应用程序发送遥测数据，将 OpenTelemetry SDK 配置为发送数据到：
-
-- OTLP gRPC:`localhost:4317`
-- OTLP HTTP:`localhost:4318`
-
-例如，在使用 `rustfs-obs` 库的 Rust 应用程序中：
 
 ```bash
-export RUSTFS_OBS_ENDPOINT=http://localhost:4317
+cd .docker/openobserve-otel
-export RUSTFS_OBS_SERVICE_NAME=yourservice
+docker compose up -d
-export RUSTFS_OBS_SERVICE_VERSION=1.0.0
+```
-export RUSTFS_OBS_ENVIRONMENT=development
+
-```
+### 2. 访问仪表盘
+
+- **URL**: [http://localhost:5080](http://localhost:5080)
+- **用户名**: `root@rustfs.com`
+- **密码**: `rustfs123`
+
+## 🛠️ 配置
+
+### OpenObserve
+
+- **持久化**: 数据持久化到 Docker 卷。
+- **端口**:
+    - `5080`: HTTP API 和 UI
+    - `5081`: OTLP gRPC
+
+### OpenTelemetry Collector
+
+- **接收器**: OTLP (gRPC `4317`, HTTP `4318`)
+- **导出器**: 将数据发送到 OpenObserve。
+
+## 🔗 集成
+
+配置您的应用程序将 OTLP 数据发送到收集器：
+
+- **端点**: `http://localhost:4318` (HTTP) 或 `localhost:4317` (gRPC)
+
+RustFS 示例：
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://localhost:4318
+export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
+```

.envrc

@@ -0,0 +1 @@
+use flake

.github/AGENTS.md

@@ -0,0 +1,36 @@
+# GitHub Workflow Instructions
+
+Applies to `.github/` and repository pull-request operations.
+
+## Pull Requests
+
+- PR titles and descriptions must be in English.
+- Use `.github/pull_request_template.md` for every PR body.
+- Keep all template section headings.
+- Use `N/A` for non-applicable sections.
+- Include verification commands in the PR details.
+- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
+- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
+- Recommended pattern:
+  - `cat > /tmp/pr_body.md <<'EOF'`
+  - `...markdown...`
+  - `EOF`
+  - `gh pr create ... --body-file /tmp/pr_body.md`
+
+## CI Alignment
+
+When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`.
+
+Current `test-and-lint` gate includes:
+
+- `cargo nextest run --all --exclude e2e_test`
+- `cargo test --all --doc`
+- `cargo test -p rustfs get_object_chunk_fast_path`
+- `cargo test -p rustfs materialize_chunk_stream_before_commit`
+- `touch rustfs/build.rs`
+- `cargo build -p rustfs --bins --jobs 2`
+- `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes`
+- `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path`
+- `cargo fmt --all --check`
+- `cargo clippy --all-targets --all-features -- -D warnings`
+- `./scripts/check_layer_dependencies.sh`

.github/actions/setup/action.yml

@@ -52,24 +52,19 @@ runs:
         sudo apt-get install -y \
           musl-tools \
           build-essential \
-          lld \
-          libdbus-1-dev \
-          libwayland-dev \
-          libwebkit2gtk-4.1-dev \
-          libxdo-dev \
           pkg-config \
           libssl-dev
 
     - name: Install protoc
       uses: arduino/setup-protoc@v3
       with:
-        version: "31.1"
+        version: "33.1"
         repo-token: ${{ inputs.github-token }}
 
     - name: Install flatc
-      uses: Nugine/setup-flatc@v1
+      uses: Nugine/setup-flatc@v1.2.4
       with:
-        version: "25.2.10"
+        version: "25.9.23"
 
     - name: Install Rust toolchain
       uses: dtolnay/rust-toolchain@stable

.github/cla.yml

@@ -0,0 +1,17 @@
+enabled: true
+
+document:
+  version: v1
+  url: https://github.com/rustfs/cla/blob/main/cla/v1.md
+
+signing:
+  mode: comment
+  comment_pattern: I have read and agree to the CLA.
+
+registry:
+  type: json-repo
+  repository: rustfs/cla
+  path_prefix: signatures
+
+status:
+  check_name: CLA Check

.github/copilot-instructions.md

@@ -0,0 +1 @@
+../AGENTS.md

.github/dependabot.yml

@@ -22,8 +22,31 @@ updates:
   - package-ecosystem: "cargo" # See documentation for possible values
     directory: "/" # Location of package manifests
     schedule:
-      interval: "monthly"
+      interval: "weekly"
+      day: "monday"
+      timezone: "Asia/Shanghai"
+      time: "08:00"
+    assignees:
+      - "heihutu"
+    reviewers:
+      - "houseme"
+      - "overtrue"
+      - "majinghe"
+    ignore:
+      - dependency-name: "object_store"
+        versions: [ "0.13.x" ]
+      - dependency-name: "libunftp"
+        versions: [ "0.23.x" ]
+      - dependency-name: "ratelimit"
+        versions: [ "1.x" ]
     groups:
+      s3s:
+        update-types:
+          - "minor"
+          - "patch"
+        patterns:
+          - "s3s"
+          - "s3s-*"
       dependencies:
         patterns:
-          - "*"
+          - "*"

.github/pull_request_template.md

@@ -34,4 +34,4 @@ Pull Request Template for RustFS
 
 ---
 
-Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)) and sign the CLA if this is your first contribution.
+Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.

.github/s3tests/README.md

@@ -0,0 +1,103 @@
+# S3 Compatibility Tests Configuration
+
+This directory contains the configuration for running [Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
+
+## Configuration File
+
+The `s3tests.conf` file is based on the official `s3tests.conf.SAMPLE` from the ceph/s3-tests repository. It uses environment variable substitution via `envsubst` to configure the endpoint and credentials.
+
+### Key Configuration Points
+
+- **Host**: Set via `${S3_HOST}` environment variable (e.g., `rustfs-single` for single-node, `lb` for multi-node)
+- **Port**: 9000 (standard RustFS port)
+- **Credentials**: Uses `${S3_ACCESS_KEY}` and `${S3_SECRET_KEY}` from workflow environment
+- **TLS**: Disabled (`is_secure = False`)
+
+## Test Execution Strategy
+
+### Network Connectivity Fix
+
+Tests run inside a Docker container on the `rustfs-net` network, which allows them to resolve and connect to the RustFS container hostnames. This fixes the "Temporary failure in name resolution" error that occurred when tests ran on the GitHub runner host.
+
+### Performance Optimizations
+
+1. **Parallel Execution**: Uses `pytest-xdist` with `-n 4` to run tests in parallel across 4 workers
+2. **Load Distribution**: Uses `--dist=loadgroup` to distribute test groups across workers
+3. **Fail-Fast**: Uses `--maxfail=50` to stop after 50 failures, saving time on catastrophic failures
+
+### Feature Filtering
+
+Tests are filtered using pytest markers (`-m`) to skip features not yet supported by RustFS:
+
+- `lifecycle` - Bucket lifecycle policies
+- `versioning` - Object versioning
+- `s3website` - Static website hosting
+- `bucket_logging` - Bucket logging
+- `encryption` / `sse_s3` - Server-side encryption
+- `cloud_transition` / `cloud_restore` - Cloud storage transitions
+- `lifecycle_expiration` / `lifecycle_transition` - Lifecycle operations
+
+This filtering:
+1. Reduces test execution time significantly (from 1+ hour to ~10-15 minutes)
+2. Focuses on features RustFS currently supports
+3. Avoids hundreds of expected failures
+
+## Running Tests Locally
+
+### Single-Node Test
+
+```bash
+# Set credentials
+export S3_ACCESS_KEY=rustfsadmin
+export S3_SECRET_KEY=rustfsadmin
+
+# Start RustFS container
+docker run -d --name rustfs-single \
+  --network rustfs-net \
+  -e RUSTFS_ADDRESS=0.0.0.0:9000 \
+  -e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
+  -e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
+  -e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
+  rustfs-ci
+
+# Generate config
+export S3_HOST=rustfs-single
+envsubst < .github/s3tests/s3tests.conf > /tmp/s3tests.conf
+
+# Run tests
+docker run --rm \
+  --network rustfs-net \
+  -v /tmp/s3tests.conf:/etc/s3tests.conf:ro \
+  python:3.12-slim \
+  bash -c '
+    apt-get update -qq && apt-get install -y -qq git
+    git clone --depth 1 https://github.com/ceph/s3-tests.git /s3-tests
+    cd /s3-tests
+    pip install -q -r requirements.txt pytest-xdist
+    S3TEST_CONF=/etc/s3tests.conf pytest -v -n 4 \
+      s3tests/functional/test_s3.py \
+      -m "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption and not sse_s3"
+  '
+```
+
+## Test Results Interpretation
+
+- **PASSED**: Test succeeded, feature works correctly
+- **FAILED**: Test failed, indicates a potential bug or incompatibility
+- **ERROR**: Test setup failed (e.g., network issues, missing dependencies)
+- **SKIPPED**: Test skipped due to marker filtering
+
+## Adding New Feature Support
+
+When adding support for a new S3 feature to RustFS:
+
+1. Remove the corresponding marker from the filter in `.github/workflows/e2e-s3tests.yml`
+2. Run the tests to verify compatibility
+3. Fix any failing tests
+4. Update this README to reflect the newly supported feature
+
+## References
+
+- [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
+- [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/)
+- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/)

.github/s3tests/s3tests.conf

@@ -0,0 +1,193 @@
+# RustFS s3-tests configuration
+# Based on: https://github.com/ceph/s3-tests/blob/master/s3tests.conf.SAMPLE
+#
+# Usage:
+#   Single-node: S3_HOST=rustfs-single envsubst < s3tests.conf > /tmp/s3tests.conf
+#   Multi-node:  S3_HOST=lb envsubst < s3tests.conf > /tmp/s3tests.conf
+
+[DEFAULT]
+## this section is just used for host, port and bucket_prefix
+
+# host set for RustFS - will be substituted via envsubst
+host = ${S3_HOST}
+
+# port for RustFS
+port = 9000
+
+## say "False" to disable TLS
+is_secure = False
+
+## say "False" to disable SSL Verify
+ssl_verify = False
+
+[fixtures]
+## all the buckets created will start with this prefix;
+## {random} will be filled with random characters to pad
+## the prefix to 30 characters long, and avoid collisions
+bucket prefix = rustfs-{random}-
+
+# all the iam account resources (users, roles, etc) created
+# will start with this name prefix
+iam name prefix = s3-tests-
+
+# all the iam account resources (users, roles, etc) created
+# will start with this path prefix
+iam path prefix = /s3-tests/
+
+[s3 main]
+# main display_name
+display_name = RustFS Tester
+
+# main user_id
+user_id = rustfsadmin
+
+# main email
+email = tester@rustfs.local
+
+# zonegroup api_name for bucket location
+api_name = default
+
+## main AWS access key
+access_key = ${S3_ACCESS_KEY}
+
+## main AWS secret key
+secret_key = ${S3_SECRET_KEY}
+
+## replace with key id obtained when secret is created, or delete if KMS not tested
+#kms_keyid = 01234567-89ab-cdef-0123-456789abcdef
+
+## Storage classes
+#storage_classes = "LUKEWARM, FROZEN"
+
+## Lifecycle debug interval (default: 10)
+#lc_debug_interval = 20
+## Restore debug interval (default: 100)
+#rgw_restore_debug_interval = 60
+#rgw_restore_processor_period = 60
+
+[s3 alt]
+# alt display_name
+display_name = RustFS Alt Tester
+
+## alt email
+email = alt@rustfs.local
+
+# alt user_id
+user_id = rustfsalt
+
+# alt AWS access key (must be different from s3 main for many tests)
+access_key = ${S3_ALT_ACCESS_KEY}
+
+# alt AWS secret key
+secret_key = ${S3_ALT_SECRET_KEY}
+
+#[s3 cloud]
+## to run the testcases with "cloud_transition" for transition
+## and "cloud_restore" for restore attribute.
+## Note: the waiting time may have to tweaked depending on
+## the I/O latency to the cloud endpoint.
+
+## host set for cloud endpoint
+# host = localhost
+
+## port set for cloud endpoint
+# port = 8001
+
+## say "False" to disable TLS
+# is_secure = False
+
+## cloud endpoint credentials
+# access_key = 0555b35654ad1656d804
+# secret_key = h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
+
+## storage class configured as cloud tier on local rgw server
+# cloud_storage_class = CLOUDTIER
+
+## Below are optional -
+
+## Above configured cloud storage class config options
+# retain_head_object = false
+# allow_read_through = false    # change it to enable read_through
+# read_through_restore_days = 2
+# target_storage_class = Target_SC
+# target_path = cloud-bucket
+
+## another regular storage class to test multiple transition rules,
+# storage_class = S1
+
+[s3 tenant]
+# tenant display_name
+display_name = RustFS Tenant Tester
+
+# tenant user_id
+# Note: Using same user_id as main to avoid teardown failures.
+# RustFS does not currently support multi-tenancy, so the tenant client
+# effectively operates as the main user. This ensures nuke_prefixed_buckets()
+# in s3-tests teardown can successfully clean up resources.
+user_id = rustfsadmin
+
+# tenant AWS access key
+access_key = ${S3_ACCESS_KEY}
+
+# tenant AWS secret key
+secret_key = ${S3_SECRET_KEY}
+
+# tenant email
+email = tenant@rustfs.local
+
+# tenant name
+# Note: Empty tenant name to avoid multi-tenant path issues during teardown.
+# When s3-tests calls get_tenant_client(), it uses this tenant value in requests.
+# An empty value makes the tenant client behave like the main client, preventing
+# "bucket not found" errors when teardown tries to clean up test buckets.
+tenant =
+
+#following section needs to be added for all sts-tests
+[iam]
+#used for iam operations in sts-tests
+#email
+email = s3@rustfs.local
+
+#user_id
+user_id = rustfsiam
+
+#access_key
+access_key = ${S3_ACCESS_KEY}
+
+#secret_key
+secret_key = ${S3_SECRET_KEY}
+
+#display_name
+display_name = RustFS IAM User
+
+# iam account root user for iam_account tests
+[iam root]
+access_key = ${S3_ACCESS_KEY}
+secret_key = ${S3_SECRET_KEY}
+user_id = RGW11111111111111111
+email = account1@rustfs.local
+
+# iam account root user in a different account than [iam root]
+[iam alt root]
+access_key = ${S3_ACCESS_KEY}
+secret_key = ${S3_SECRET_KEY}
+user_id = RGW22222222222222222
+email = account2@rustfs.local
+
+#following section needs to be added when you want to run Assume Role With Webidentity test
+[webidentity]
+#used for assume role with web identity test in sts-tests
+#all parameters will be obtained from ceph/qa/tasks/keycloak.py
+#token=<access_token>
+
+#aud=<obtained after introspecting token>
+
+#sub=<obtained after introspecting token>
+
+#azp=<obtained after introspecting token>
+
+#user_token=<access token for a user, with attribute Department=[Engineering, Marketing>]
+
+#thumbprint=<obtained from x509 certificate>
+
+#KC_REALM=<name of the realm>

.github/workflows/audit.yml

@@ -40,11 +40,11 @@ env:
 jobs:
   security-audit:
     name: Security Audit
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     timeout-minutes: 15
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Install cargo-audit
         uses: taiki-e/install-action@v2
@@ -57,7 +57,7 @@ jobs:
 
       - name: Upload audit results
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v6
         with:
           name: security-audit-results-${{ github.run_number }}
           path: audit-results.json
@@ -65,14 +65,14 @@ jobs:
 
   dependency-review:
     name: Dependency Review
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     if: github.event_name == 'pull_request'
     permissions:
       contents: read
       pull-requests: write
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Dependency Review
         uses: actions/dependency-review-action@v4

.github/workflows/build.yml

@@ -23,6 +23,7 @@
 #
 # Manual Parameters:
 # - build_docker: Build and push Docker images (default: true)
+# - platforms: Comma-separated platform IDs or 'all' (default: all)
 
 name: Build and Release
 
@@ -44,22 +45,6 @@ on:
       - "**/*.svg"
       - ".gitignore"
       - ".dockerignore"
-  pull_request:
-    branches: [ main ]
-    paths-ignore:
-      - "**.md"
-      - "**.txt"
-      - ".github/**"
-      - "docs/**"
-      - "deploy/**"
-      - "scripts/dev_*.sh"
-      - "LICENSE*"
-      - "README*"
-      - "**/*.png"
-      - "**/*.jpg"
-      - "**/*.svg"
-      - ".gitignore"
-      - ".dockerignore"
   schedule:
     - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
   workflow_dispatch:
@@ -69,6 +54,11 @@ on:
         required: false
         default: true
         type: boolean
+      platforms:
+        description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
+        required: false
+        default: "all"
+        type: string
 
 permissions:
   contents: read
@@ -83,7 +73,7 @@ jobs:
   # Build strategy check - determine build type based on trigger
   build-check:
     name: Build Strategy Check
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       build_type: ${{ steps.check.outputs.build_type }}
@@ -92,7 +82,7 @@ jobs:
       is_prerelease: ${{ steps.check.outputs.is_prerelease }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
 
@@ -154,56 +144,78 @@ jobs:
           echo "  - Is prerelease: $is_prerelease"
 
   # Build RustFS binaries
+  prepare-platform-matrix:
+    name: Prepare Platform Matrix
+    runs-on: ubicloud-standard-2
+    outputs:
+      matrix: ${{ steps.select.outputs.matrix }}
+      selected: ${{ steps.select.outputs.selected }}
+    steps:
+      - name: Select target platforms
+        id: select
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
+          selected="$(echo "${selected}" | tr -d '[:space:]')"
+          if [[ -z "${selected}" ]]; then
+            selected="all"
+          fi
+
+          all='{"include":[
+            {"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
+          ]}'
+
+          if [[ "${selected}" == "all" ]]; then
+            matrix="$(jq -c . <<<"${all}")"
+          else
+            unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | ($all.include | map(.target_id)) as $known
+              | [$req[] | select(( $known | index(.) ) == null)]
+            ')"
+            if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
+              echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
+              echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
+              exit 1
+            fi
+
+            matrix="$(jq -c --arg selected "${selected}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | .include |= map(select(.target_id as $id | ($req | index($id))))
+            ' <<<"${all}")"
+          fi
+
+          echo "selected=${selected}" >> "$GITHUB_OUTPUT"
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
+          echo "Selected platforms: ${selected}"
+
   build-rustfs:
     name: Build RustFS
-    needs: [ build-check ]
+    needs: [ build-check, prepare-platform-matrix ]
-    if: needs.build-check.outputs.should_build == 'true'
+    if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
     timeout-minutes: 60
     env:
-      RUSTFLAGS: ${{ matrix.cross == 'false' && '-C target-cpu=native' || '' }}
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      # Always enable Tokio unstable features (required by dial9-tokio-telemetry).
+      # The RUSTFLAGS env var takes precedence over .cargo/config.toml [build] rustflags,
+      # so we must include --cfg tokio_unstable here explicitly; otherwise an empty
+      # RUSTFLAGS value would shadow the config-file flag and silently break tracing.
+      RUSTFLAGS: "--cfg tokio_unstable ${{ matrix.rustflags }}"
     strategy:
       fail-fast: false
-      matrix:
+      matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
-        include:
-          # Linux builds
-          - os: ubuntu-latest
-            target: x86_64-unknown-linux-musl
-            cross: false
-            platform: linux
-          - os: ubuntu-latest
-            target: aarch64-unknown-linux-musl
-            cross: true
-            platform: linux
-          - os: ubuntu-latest
-            target: x86_64-unknown-linux-gnu
-            cross: false
-            platform: linux
-          - os: ubuntu-latest
-            target: aarch64-unknown-linux-gnu
-            cross: true
-            platform: linux
-          # macOS builds
-          - os: macos-latest
-            target: aarch64-apple-darwin
-            cross: false
-            platform: macos
-          - os: macos-latest
-            target: x86_64-apple-darwin
-            cross: false
-            platform: macos
-          # Windows builds (temporarily disabled)
-          - os: windows-latest
-            target: x86_64-pc-windows-msvc
-            cross: false
-            platform: windows
-          #- os: windows-latest
-          #  target: aarch64-pc-windows-msvc
-          #  cross: true
-          #  platform: windows
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
 
@@ -309,19 +321,26 @@ jobs:
               ;;
           esac
 
+          # Normalize version used for package filenames
+          PACKAGE_VERSION="${VERSION}"
+          if [[ "$PACKAGE_VERSION" == v* ]]; then
+            PACKAGE_VERSION="${PACKAGE_VERSION#v}"
+          fi
+
           # Generate package name based on build type
           if [[ -n "$VARIANT" ]]; then
             ARCH_WITH_VARIANT="${ARCH}-${VARIANT}"
           else
             ARCH_WITH_VARIANT="${ARCH}"
           fi
+          PACKAGE_BASENAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}"
 
           if [[ "$BUILD_TYPE" == "development" ]]; then
             # Development build: rustfs-${platform}-${arch}-${variant}-dev-${short_sha}.zip
             PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}-dev-${SHORT_SHA}"
           else
             # Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
-            PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}-v${VERSION}"
+            PACKAGE_NAME="${PACKAGE_BASENAME}-v${PACKAGE_VERSION}"
           fi
 
           # Create zip packages for all platforms
@@ -391,7 +410,7 @@ jobs:
           if [[ "$BUILD_TYPE" == "release" ]] || [[ "$BUILD_TYPE" == "prerelease" ]]; then
             # Create latest version filename
             # Convert from rustfs-linux-x86_64-musl-v1.0.0 to rustfs-linux-x86_64-musl-latest
-            LATEST_FILE="${PACKAGE_NAME%-v*}-latest.zip"
+            LATEST_FILE="${PACKAGE_BASENAME}-latest.zip"
 
             echo "🔄 Creating latest version: ${PACKAGE_NAME}.zip -> $LATEST_FILE"
             cp "${PACKAGE_NAME}.zip" "$LATEST_FILE"
@@ -404,7 +423,7 @@ jobs:
             # Development builds (only main branch triggers development builds)
             # Create main-latest version filename
             # Convert from rustfs-linux-x86_64-dev-abc123 to rustfs-linux-x86_64-main-latest
-            MAIN_LATEST_FILE="${PACKAGE_NAME%-dev-*}-main-latest.zip"
+            MAIN_LATEST_FILE="${PACKAGE_BASENAME}-main-latest.zip"
 
             echo "🔄 Creating main-latest version: ${PACKAGE_NAME}.zip -> $MAIN_LATEST_FILE"
             cp "${PACKAGE_NAME}.zip" "$MAIN_LATEST_FILE"
@@ -442,7 +461,7 @@ jobs:
           echo "📊 Version: ${VERSION}"
 
       - name: Upload to GitHub artifacts
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v6
         with:
           name: ${{ steps.package.outputs.package_name }}
           path: "rustfs-*.zip"
@@ -454,7 +473,7 @@ jobs:
           OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
           OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
           OSS_REGION: cn-beijing
-          OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
+          OSS_ENDPOINT: https://oss-accelerate.aliyuncs.com
         shell: bash
         run: |
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
@@ -532,7 +551,7 @@ jobs:
     name: Build Summary
     needs: [ build-check, build-rustfs ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     steps:
       - name: Build completion summary
         shell: bash
@@ -584,7 +603,7 @@ jobs:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     permissions:
       contents: write
     outputs:
@@ -592,7 +611,7 @@ jobs:
       release_url: ${{ steps.create.outputs.release_url }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
 
@@ -670,16 +689,16 @@ jobs:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     permissions:
       contents: write
       actions: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Download all build artifacts
-        uses: actions/download-artifact@v5
+        uses: actions/download-artifact@v7
         with:
           path: ./artifacts
           pattern: rustfs-*
@@ -751,7 +770,7 @@ jobs:
     name: Update Latest Version
     needs: [ build-check, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/')
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     steps:
       - name: Update latest.json
         env:
@@ -801,12 +820,12 @@ jobs:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     permissions:
       contents: write
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Update release notes and publish
         env:

.github/workflows/ci.yml

@@ -4,7 +4,7 @@
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
-#     http://www.apache.org/licenses/LICENSE-2.0
+#      http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
@@ -19,7 +19,6 @@ on:
     branches: [ main ]
     paths-ignore:
       - "**.md"
-      - "**.txt"
       - "docs/**"
       - "deploy/**"
       - "scripts/dev_*.sh"
@@ -39,7 +38,6 @@ on:
     branches: [ main ]
     paths-ignore:
       - "**.md"
-      - "**.txt"
       - "docs/**"
       - "deploy/**"
       - "scripts/dev_*.sh"
@@ -55,6 +53,8 @@ on:
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
       - ".github/workflows/performance.yml"
+  merge_group:
+    types: [checks_requested]
   schedule:
     - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
   workflow_dispatch:
@@ -62,17 +62,23 @@ on:
 permissions:
   contents: read
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 env:
   CARGO_TERM_COLOR: always
   RUST_BACKTRACE: 1
+  CARGO_BUILD_JOBS: 2
 
 jobs:
+
   skip-check:
     name: Skip Duplicate Actions
     permissions:
       actions: write
       contents: read
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     outputs:
       should_skip: ${{ steps.skip_check.outputs.should_skip }}
     steps:
@@ -83,15 +89,15 @@ jobs:
           concurrent_skipping: "same_content_newer"
           cancel_others: true
           paths_ignore: '["*.md", "docs/**", "deploy/**"]'
-          # Never skip release events and tag pushes
           do_not_skip: '["workflow_dispatch", "schedule", "merge_group", "release", "push"]'
 
-
   typos:
     name: Typos
-    runs-on: ubuntu-latest
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-2
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v6
       - uses: dtolnay/rust-toolchain@stable
       - name: Typos check with custom config file
         uses: crate-ci/typos@master
@@ -100,13 +106,13 @@ jobs:
     name: Test and Lint
     needs: skip-check
     if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-4
     timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
-      - name: Delete huge unnecessary tools folder
-        run: rm -rf /opt/hostedtoolcache
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
@@ -127,35 +133,75 @@ jobs:
       - name: Run clippy lints
         run: cargo clippy --all-targets --all-features -- -D warnings
 
-  e2e-tests:
+      - name: Check layered dependencies
-    name: End-to-End Tests
+        run: ./scripts/check_layer_dependencies.sh
+
+  build-rustfs-debug-binary:
+    name: Build RustFS Debug Binary
     needs: skip-check
     if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-4
     timeout-minutes: 30
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-e2e-${{ hashFiles('**/Cargo.lock') }}
+          cache-shared-key: ci-rustfs-debug-binary-${{ hashFiles('**/Cargo.lock') }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
           github-token: ${{ secrets.GITHUB_TOKEN }}
 
+      - name: Build debug binary
+        run: |
+          touch rustfs/build.rs
+          cargo build -p rustfs --bins --jobs 2
+
+      - name: Upload debug binary
+        uses: actions/upload-artifact@v6
+        with:
+          name: rustfs-debug-binary
+          path: target/debug/rustfs
+          if-no-files-found: error
+          retention-days: 1
+
+  e2e-tests:
+    name: End-to-End Tests
+    needs: [ skip-check, build-rustfs-debug-binary ]
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Clean up previous test run
+        run: |
+          rm -rf /tmp/rustfs
+          rm -f /tmp/rustfs.log
+
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Setup Rust toolchain for s3s-e2e installation
+        uses: dtolnay/rust-toolchain@stable
+
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@v2
         with:
           tool: s3s-e2e
-          git: https://github.com/Nugine/s3s.git
+          git: https://github.com/s3s-project/s3s.git
-          rev: b7714bfaa17ddfa9b23ea01774a1e7bbdbfc2ca3
+          rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
-
-      - name: Build debug binary
-        run: |
-          touch rustfs/build.rs
-          cargo build -p rustfs --bins
 
       - name: Run end-to-end tests
         run: |
@@ -164,8 +210,44 @@ jobs:
 
       - name: Upload test logs
         if: failure()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v6
         with:
           name: e2e-test-logs-${{ github.run_number }}
           path: /tmp/rustfs.log
           retention-days: 3
+
+  s3-implemented-tests:
+    name: S3 Implemented Tests
+    needs: [ skip-check, build-rustfs-debug-binary ]
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Run implemented s3-tests
+        run: |
+          DEPLOY_MODE=binary \
+          RUSTFS_BINARY=./target/debug/rustfs \
+          TEST_MODE=single \
+          MAXFAIL=1 \
+          ./scripts/s3-tests/run.sh
+
+      - name: Upload s3 test artifacts
+        if: always()
+        uses: actions/upload-artifact@v6
+        with:
+          name: s3tests-implemented-${{ github.run_number }}
+          path: artifacts/s3tests-single/**
+          if-no-files-found: ignore
+          retention-days: 3

.github/workflows/cla.yml

@@ -0,0 +1,70 @@
+# Copyright 2026 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: CLA Check
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+  merge_group:
+    types: [checks_requested]
+  issue_comment:
+    types: [created, edited]
+
+permissions:
+  contents: write
+  pull-requests: write
+  issues: write
+  checks: write
+
+jobs:
+  cla:
+    if: ${{ github.event_name != 'issue_comment' || github.event.issue.pull_request }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Report CLA result for merge queue
+        if: github.event_name == 'merge_group'
+        uses: actions/github-script@v8
+        with:
+          script: |
+            await github.rest.checks.create({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              name: 'CLA Check',
+              head_sha: context.sha,
+              status: 'completed',
+              conclusion: 'success',
+              output: {
+                title: 'CLA requirements satisfied for merge queue',
+                summary: 'Queued pull requests must satisfy the required CLA check before they enter the merge queue. This reports the existing CLA result on the merge-group SHA.'
+              }
+            });
+
+      - name: Create token for rustfs/cla
+        if: github.event_name != 'merge_group'
+        id: registry-token
+        uses: actions/create-github-app-token@v3
+        with:
+          app-id: ${{ vars.CLA_BOT_APP_ID }}
+          private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
+          owner: ${{ github.repository_owner }}
+          repositories: cla
+          permission-contents: write
+
+      - name: Run CLA Bot
+        if: github.event_name != 'merge_group'
+        uses: overtrue/cla-bot@v0.0.9
+        with:
+          github-token: ${{ github.token }}
+          registry-token: ${{ steps.registry-token.outputs.token }}

.github/workflows/docker.yml

@@ -66,13 +66,14 @@ env:
   CARGO_TERM_COLOR: always
   REGISTRY_DOCKERHUB: rustfs/rustfs
   REGISTRY_GHCR: ghcr.io/${{ github.repository }}
+  REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
   DOCKER_PLATFORMS: linux/amd64,linux/arm64
 
 jobs:
   # Check if we should build Docker images
   build-check:
     name: Docker Build Check
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       should_push: ${{ steps.check.outputs.should_push }}
@@ -83,7 +84,7 @@ jobs:
       create_latest: ${{ steps.check.outputs.create_latest }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
           # For workflow_run events, checkout the specific commit that triggered the workflow
@@ -162,11 +163,11 @@ jobs:
                 if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
                   build_type="prerelease"
                   is_prerelease=true
-                  # TODO: 临时修改 - 当前允许 alpha 版本也创建 latest 标签
+                  # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                  # 等版本稳定后，需要移除下面这行，恢复原有逻辑（只有稳定版本才创建 latest）
+                  # After the version is stable, you need to remove the following line and restore the original logic (latest is created only for stable versions)
                   if [[ "$version" == *"alpha"* ]]; then
                     create_latest=true
-                    echo "🧪 Building Docker image for prerelease: $version (临时允许创建 latest 标签)"
+                     echo "🧪 Building Docker image for prerelease: $version (temporarily allowing creation of latest tag)"
                   else
                     echo "🧪 Building Docker image for prerelease: $version"
                   fi
@@ -215,11 +216,11 @@ jobs:
               v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                 build_type="prerelease"
                 is_prerelease=true
-                # TODO: 临时修改 - 当前允许 alpha 版本也创建 latest 标签
+                # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                # 等版本稳定后，需要移除下面的 if 块，恢复原有逻辑
+                # After the version is stable, you need to remove the if block below and restore the original logic.
                 if [[ "$input_version" == *"alpha"* ]]; then
                   create_latest=true
-                  echo "🧪 Building with prerelease version: $input_version (临时允许创建 latest 标签)"
+                   echo "🧪 Building with prerelease version: $input_version (temporarily allowing creation of latest tag)"
                 else
                   echo "🧪 Building with prerelease version: $input_version"
                 fi
@@ -264,11 +265,22 @@ jobs:
     name: Build Docker Images
     needs: build-check
     if: needs.build-check.outputs.should_build == 'true'
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     timeout-minutes: 60
+    strategy:
+          fail-fast: false
+          matrix:
+            include:
+              - variant: musl
+                file: Dockerfile
+                suffix: ""
+              - variant: glibc
+                file: Dockerfile.glibc
+                suffix: "-glibc"
+
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Login to Docker Hub
         uses: docker/login-action@v3
@@ -276,12 +288,19 @@ jobs:
           username: ${{ env.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
-      # - name: Login to GitHub Container Registry
+      - name: Login to GitHub Container Registry
-      #   uses: docker/login-action@v3
+        uses: docker/login-action@v3
-      #   with:
+        with:
-      #     registry: ghcr.io
+          registry: ghcr.io
-      #     username: ${{ github.actor }}
+          username: ${{ secrets.GHCR_USERNAME }}
-      #     password: ${{ secrets.GITHUB_TOKEN }}
+          password: ${{ secrets.GHCR_PASSWORD }}
+
+      - name: Login to Quay.io
+        uses: docker/login-action@v3
+        with:
+          registry: quay.io
+          username: ${{ secrets.QUAY_USERNAME }}
+          password: ${{ secrets.QUAY_PASSWORD }}
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -296,6 +315,7 @@ jobs:
           VERSION="${{ needs.build-check.outputs.version }}"
           SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
           CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
+          VARIANT_SUFFIX="${{ matrix.suffix }}"
 
           # Convert version format for Dockerfile compatibility
           case "$VERSION" in
@@ -326,14 +346,15 @@ jobs:
 
           # Generate tags based on build type
           # Only support release and prerelease builds (no development builds)
-          TAGS="${{ env.REGISTRY_DOCKERHUB }}:${VERSION}"
+          TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
+          TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
 
           # Add channel tags for prereleases and latest for stable
           if [[ "$CREATE_LATEST" == "true" ]]; then
-            # TODO: 临时修改 - 当前 alpha 版本也会创建 latest 标签
+            # TODO: Temporary change - the current alpha version will also create the latest tag
-            # 等版本稳定后，这里的逻辑保持不变，但上游的 CREATE_LATEST 设置需要恢复
+            # After the version is stabilized, the logic here remains unchanged, but the upstream CREATE_LATEST setting needs to be restored.
-            # Stable release (以及临时的 alpha 版本)
+            # Stable release (and temporary alpha versions)
-            TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest"
+            TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
           elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
             # Prerelease channel tags (alpha, beta, rc)
             if [[ "$VERSION" == *"alpha"* ]]; then
@@ -345,7 +366,7 @@ jobs:
             fi
 
             if [[ -n "$CHANNEL" ]]; then
-              TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}"
+              TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
             fi
           fi
 
@@ -372,15 +393,15 @@ jobs:
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: Dockerfile
+          file: ${{ matrix.file }}
           platforms: ${{ env.DOCKER_PLATFORMS }}
           push: ${{ needs.build-check.outputs.should_push == 'true' }}
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
           cache-from: |
-            type=gha,scope=docker-binary
+            type=gha,scope=docker-${{ matrix.variant }}
           cache-to: |
-            type=gha,mode=max,scope=docker-binary
+            type=gha,mode=max,scope=docker-${{ matrix.variant }}
           build-args: |
             BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
             VERSION=${{ needs.build-check.outputs.version }}
@@ -404,7 +425,7 @@ jobs:
     name: Docker Build Summary
     needs: [ build-check, build-docker ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     steps:
       - name: Docker build completion summary
         run: |
@@ -429,10 +450,10 @@ jobs:
             "prerelease")
               echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
               echo "⚠️  This is a prerelease image - use with caution"
-              # TODO: 临时修改 - alpha 版本当前会创建 latest 标签
+              # TODO: Temporary change - alpha versions currently create the latest tag
-              # 等版本稳定后，需要恢复下面的提示信息
+              # After the version is stable, you need to restore the following prompt information
               if [[ "$VERSION" == *"alpha"* ]] && [[ "$CREATE_LATEST" == "true" ]]; then
-                echo "🏷️  Latest tag has been created for alpha version (临时措施)"
+                echo "🏷️  Latest tag has been created for alpha version (temporary measures)"
               else
                 echo "🚫 Latest tag NOT created for prerelease"
               fi

.github/workflows/e2e-s3tests.yml

@@ -0,0 +1,443 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: e2e-s3tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      test-mode:
+        description: "Test mode to run"
+        required: true
+        type: choice
+        default: "single"
+        options:
+          - single
+          - multi
+      xdist:
+        description: "Enable pytest-xdist (parallel). '0' to disable."
+        required: false
+        default: "0"
+      maxfail:
+        description: "Stop after N failures (debug friendly)"
+        required: false
+        default: "1"
+      markexpr:
+        description: "pytest -m expression (feature filters)"
+        required: false
+        default: "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"
+
+env:
+  # main user
+  S3_ACCESS_KEY: rustfsadmin
+  S3_SECRET_KEY: rustfsadmin
+  # alt user (must be different from main for many s3-tests)
+  S3_ALT_ACCESS_KEY: rustfsalt
+  S3_ALT_SECRET_KEY: rustfsalt
+
+  S3_REGION: us-east-1
+
+  RUST_LOG: info
+  PLATFORM: linux/amd64
+  BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
+  cancel-in-progress: true
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  s3tests-single:
+    if: github.event.inputs['test-mode'] == 'single'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Cache pip downloads
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-e2e-s3tests-
+
+      - name: Install Python tools
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl tox
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
+      - name: Enable buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build RustFS image (source, cached)
+        run: |
+          DOCKER_BUILDKIT=1 docker buildx build --load \
+            --platform ${PLATFORM} \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
+            -t rustfs-ci \
+            -f Dockerfile.source .
+
+      - name: Create network
+        run: docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
+
+      - name: Remove existing rustfs-single (if any)
+        run: docker rm -f rustfs-single >/dev/null 2>&1 || true
+
+      - name: Start single RustFS
+        run: |
+          docker run -d --name rustfs-single \
+            --network rustfs-net \
+            -p 9000:9000 \
+            -e RUSTFS_ADDRESS=0.0.0.0:9000 \
+            -e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
+            -e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
+            -e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
+            -v /tmp/rustfs-single:/data \
+            rustfs-ci
+
+      - name: Wait for RustFS ready
+        run: |
+          for i in {1..60}; do
+            if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
+              echo "RustFS is ready"
+              exit 0
+            fi
+
+            if [ "$(docker inspect -f '{{.State.Running}}' rustfs-single 2>/dev/null)" != "true" ]; then
+              echo "RustFS container not running" >&2
+              docker logs rustfs-single || true
+              exit 1
+            fi
+
+            sleep 2
+          done
+
+          echo "Health check timed out" >&2
+          docker logs rustfs-single || true
+          exit 1
+
+      - name: Generate s3tests config
+        run: |
+          export S3_HOST=127.0.0.1
+          envsubst < .github/s3tests/s3tests.conf > s3tests.conf
+
+      - name: Provision s3-tests alt user (required by suite)
+        run: |
+          # Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well.
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ACCESS_KEY}" \
+            --secret_key "${S3_SECRET_KEY}" \
+            -X PUT \
+            -H 'Content-Type: application/json' \
+            -d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
+            "http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
+
+          # Explicitly attach built-in policy via policy mapping.
+          # s3-tests relies on alt client being able to ListBuckets during setup cleanup.
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ACCESS_KEY}" \
+            --secret_key "${S3_SECRET_KEY}" \
+            -X PUT \
+            "http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
+
+          # Sanity check: alt user can list buckets (should not be AccessDenied).
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ALT_ACCESS_KEY}" \
+            --secret_key "${S3_ALT_SECRET_KEY}" \
+            -X GET \
+            "http://127.0.0.1:9000/" >/dev/null
+
+      - name: Prepare s3-tests
+        run: |
+          git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
+
+      - name: Run ceph s3-tests (debug friendly)
+        run: |
+          export PATH="$HOME/.local/bin:$PATH"
+          mkdir -p artifacts/s3tests-single
+
+          cd s3-tests
+
+          set -o pipefail
+
+          MAXFAIL="${{ github.event.inputs.maxfail }}"
+          if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
+
+          MARKEXPR="${{ github.event.inputs.markexpr }}"
+          if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
+
+          XDIST="${{ github.event.inputs.xdist }}"
+          if [ -z "$XDIST" ]; then XDIST="0"; fi
+          XDIST_ARGS=""
+          if [ "$XDIST" != "0" ]; then
+            # Add pytest-xdist to requirements.txt so tox installs it inside
+            # its virtualenv. Installing outside tox does NOT work.
+            echo "pytest-xdist" >> requirements.txt
+            XDIST_ARGS="-n $XDIST --dist=loadgroup"
+          fi
+
+          # Run tests from s3tests/functional (boto2+boto3 combined directory).
+          S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
+            tox -- \
+              -vv -ra --showlocals --tb=long \
+              --maxfail="$MAXFAIL" \
+              --junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-single/junit.xml \
+              $XDIST_ARGS \
+              s3tests/functional/test_s3.py \
+              -m "$MARKEXPR" \
+              2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-single/pytest.log
+
+      - name: Collect RustFS logs
+        if: always()
+        run: |
+          mkdir -p artifacts/rustfs-single
+          docker logs rustfs-single > artifacts/rustfs-single/rustfs.log 2>&1 || true
+          docker inspect rustfs-single > artifacts/rustfs-single/inspect.json || true
+
+      - name: Upload artifacts
+        if: always() && env.ACT != 'true'
+        uses: actions/upload-artifact@v6
+        with:
+          name: s3tests-single
+          path: artifacts/**
+
+  s3tests-multi:
+    if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 150
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Cache pip downloads
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-e2e-s3tests-
+
+      - name: Install Python tools
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl tox
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
+      - name: Enable buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build RustFS image (source, cached)
+        run: |
+          DOCKER_BUILDKIT=1 docker buildx build --load \
+            --platform ${PLATFORM} \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
+            -t rustfs-ci \
+            -f Dockerfile.source .
+
+      - name: Prepare cluster compose
+        run: |
+          cat > compose.yml <<'EOF'
+          services:
+            rustfs1:
+              image: rustfs-ci
+              hostname: rustfs1
+              networks: [rustfs-net]
+              environment:
+                RUSTFS_ADDRESS: "0.0.0.0:9000"
+                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
+              volumes:
+                - rustfs1-data:/data
+            rustfs2:
+              image: rustfs-ci
+              hostname: rustfs2
+              networks: [rustfs-net]
+              environment:
+                RUSTFS_ADDRESS: "0.0.0.0:9000"
+                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
+              volumes:
+                - rustfs2-data:/data
+            rustfs3:
+              image: rustfs-ci
+              hostname: rustfs3
+              networks: [rustfs-net]
+              environment:
+                RUSTFS_ADDRESS: "0.0.0.0:9000"
+                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
+              volumes:
+                - rustfs3-data:/data
+            rustfs4:
+              image: rustfs-ci
+              hostname: rustfs4
+              networks: [rustfs-net]
+              environment:
+                RUSTFS_ADDRESS: "0.0.0.0:9000"
+                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
+              volumes:
+                - rustfs4-data:/data
+            lb:
+              image: haproxy:2.9
+              hostname: lb
+              networks: [rustfs-net]
+              ports:
+                - "9000:9000"
+              volumes:
+                - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
+          networks:
+            rustfs-net:
+              name: rustfs-net
+          volumes:
+            rustfs1-data:
+            rustfs2-data:
+            rustfs3-data:
+            rustfs4-data:
+          EOF
+
+          cat > haproxy.cfg <<'EOF'
+          defaults
+            mode http
+            timeout connect 5s
+            timeout client  30s
+            timeout server  30s
+
+          frontend fe_s3
+            bind *:9000
+            default_backend be_s3
+
+          backend be_s3
+            balance roundrobin
+            server s1 rustfs1:9000 check
+            server s2 rustfs2:9000 check
+            server s3 rustfs3:9000 check
+            server s4 rustfs4:9000 check
+          EOF
+
+      - name: Launch cluster
+        run: docker compose -f compose.yml up -d
+
+      - name: Wait for LB ready
+        run: |
+          for i in {1..90}; do
+            if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
+              echo "Load balancer is ready"
+              exit 0
+            fi
+            sleep 2
+          done
+          echo "LB or backend not ready" >&2
+          docker compose -f compose.yml logs --tail=200 || true
+          exit 1
+
+      - name: Generate s3tests config
+        run: |
+          export S3_HOST=127.0.0.1
+          envsubst < .github/s3tests/s3tests.conf > s3tests.conf
+
+      - name: Provision s3-tests alt user (required by suite)
+        run: |
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ACCESS_KEY}" \
+            --secret_key "${S3_SECRET_KEY}" \
+            -X PUT \
+            -H 'Content-Type: application/json' \
+            -d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
+            "http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
+
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ACCESS_KEY}" \
+            --secret_key "${S3_SECRET_KEY}" \
+            -X PUT \
+            "http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
+
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ALT_ACCESS_KEY}" \
+            --secret_key "${S3_ALT_SECRET_KEY}" \
+            -X GET \
+            "http://127.0.0.1:9000/" >/dev/null
+
+      - name: Prepare s3-tests
+        run: |
+          git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
+
+      - name: Run ceph s3-tests (multi, debug friendly)
+        run: |
+          export PATH="$HOME/.local/bin:$PATH"
+          mkdir -p artifacts/s3tests-multi
+
+          cd s3-tests
+
+          set -o pipefail
+
+          MAXFAIL="${{ github.event.inputs.maxfail }}"
+          if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
+
+          MARKEXPR="${{ github.event.inputs.markexpr }}"
+          if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
+
+          XDIST="${{ github.event.inputs.xdist }}"
+          if [ -z "$XDIST" ]; then XDIST="0"; fi
+          XDIST_ARGS=""
+          if [ "$XDIST" != "0" ]; then
+            # Add pytest-xdist to requirements.txt so tox installs it inside
+            # its virtualenv. Installing outside tox does NOT work.
+            echo "pytest-xdist" >> requirements.txt
+            XDIST_ARGS="-n $XDIST --dist=loadgroup"
+          fi
+
+          # Run tests from s3tests/functional (boto2+boto3 combined directory).
+          S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
+            tox -- \
+              -vv -ra --showlocals --tb=long \
+              --maxfail="$MAXFAIL" \
+              --junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-multi/junit.xml \
+              $XDIST_ARGS \
+              s3tests/functional/test_s3.py \
+              -m "$MARKEXPR" \
+              2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-multi/pytest.log
+
+      - name: Collect logs
+        if: always()
+        run: |
+          mkdir -p artifacts/cluster
+          docker compose -f compose.yml logs --no-color > artifacts/cluster/cluster.log 2>&1 || true
+
+      - name: Upload artifacts
+        if: always() && env.ACT != 'true'
+        uses: actions/upload-artifact@v6
+        with:
+          name: s3tests-multi
+          path: artifacts/**

.github/workflows/helm-package.yml

@@ -0,0 +1,94 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Publish helm chart to artifacthub
+
+on:
+  workflow_run:
+    workflows: [ "Build and Release" ]
+    types: [ completed ]
+
+permissions:
+  contents: read
+
+env:
+  new_version: ${{ github.event.workflow_run.head_branch }}
+
+jobs:
+  build-helm-package:
+    runs-on: ubicloud-standard-2
+    # Only run on successful builds triggered by tag pushes (version format: x.y.z or x.y.z-suffix)
+    if: |
+      github.event.workflow_run.conclusion == 'success' && 
+      github.event.workflow_run.event == 'push' &&
+      contains(github.event.workflow_run.head_branch, '.')
+
+    steps:
+      - name: Checkout helm chart repo
+        uses: actions/checkout@v6
+
+      - name: Replace chart app version
+        run: |
+          set -e
+          set -x
+          old_version=$(grep "^appVersion:" helm/rustfs/Chart.yaml | awk '{print $2}')
+          sed -i "s/$old_version/$new_version/g" helm/rustfs/Chart.yaml
+
+      - name: Set up Helm
+        uses: azure/setup-helm@v4.3.0
+
+      - name: Package Helm Chart
+        run: |
+          cp helm/README.md helm/rustfs/
+          package_version=$(echo $new_version | awk -F '-' '{print $2}' | awk -F '.' '{print $NF}') 
+          helm package ./helm/rustfs --destination helm/rustfs/ --version "0.0.$package_version"
+
+      - name: Upload helm package as artifact
+        uses: actions/upload-artifact@v6
+        with:
+          name: helm-package
+          path: helm/rustfs/*.tgz
+          retention-days: 1
+
+  publish-helm-package:
+    runs-on: ubicloud-standard-2
+    needs: [ build-helm-package ]
+
+    steps:
+      - name: Checkout helm package repo
+        uses: actions/checkout@v6
+        with:
+          repository: rustfs/helm
+          token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
+
+      - name: Download helm package
+        uses: actions/download-artifact@v7
+        with:
+          name: helm-package
+          path: ./
+
+      - name: Set up helm
+        uses: azure/setup-helm@v4.3.0
+
+      - name: Generate index
+        run: helm repo index . --url https://charts.rustfs.com
+
+      - name: Push helm package and index file
+        run: |
+          git config --global user.name "${{ secrets.USERNAME }}"
+          git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
+          git status .
+          git add .
+          git commit -m "Update rustfs helm package with $new_version."
+          git push origin main

.github/workflows/issue-translator.yml

@@ -25,7 +25,7 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-4
     steps:
       - uses: usthe/issues-translate-action@v2.7
         with:

.github/workflows/nix-flake-update.yml

@@ -0,0 +1,65 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Update Nix Flake
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 0 * * 0' # Weekly on Sundays
+
+permissions:
+  contents: write
+  pull-requests: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  update-flake:
+    name: Update flake.lock
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Install Nix
+        uses: DeterminateSystems/determinate-nix-action@v3
+
+      - name: Check Nix flake inputs
+        uses: DeterminateSystems/flake-checker-action@v12
+
+      - name: Update flake.lock
+        id: update
+        uses: DeterminateSystems/update-flake-lock@main
+        with:
+          git-author-name: houseme
+          git-author-email: housemecn@gmail.com
+          git-committer-name: houseme
+          git-committer-email: housemecn@gmail.com
+          pr-title: "chore(deps): update flake.lock"
+          pr-labels: |
+            dependencies
+            nix
+            automated
+          commit-msg: "chore(deps): update flake.lock"
+          pr-reviewers: houseme, overtrue, majinghe
+          token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
+
+      - name: Log PR details
+        if: steps.update.outputs.pull-request-number
+        run: |
+          echo "Pull Request created: ${{ steps.update.outputs.pull-request-number }}"

.github/workflows/nix.yml

@@ -0,0 +1,84 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Nix CI
+
+on:
+  push:
+    branches: [ "main" ]
+    paths:
+      - 'flake.nix'
+      - 'flake.lock'
+      - 'Cargo.toml'
+      - 'Cargo.lock'
+      - '.github/workflows/nix.yml'
+  pull_request:
+    branches: [ "main" ]
+    paths:
+      - 'flake.nix'
+      - 'flake.lock'
+      - 'Cargo.toml'
+      - 'Cargo.lock'
+      - '.github/workflows/nix.yml'
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+permissions:
+  contents: read
+
+jobs:
+  nix-validation:
+    name: Nix Build & Check
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Install Nix
+        uses: cachix/install-nix-action@v31
+        with:
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+          extra_nix_config: |
+            experimental-features = nix-command flakes
+
+      - name: Setup Magic Nix Cache
+        uses: DeterminateSystems/magic-nix-cache-action@v13
+
+      - name: Setup Flake Checker
+        uses: DeterminateSystems/flake-checker-action@v12
+
+      - name: Verify Flake
+        run: |
+          echo "Checking flake structure and evaluation..."
+          nix flake show
+          nix flake check --print-build-logs
+
+      - name: Build RustFS
+        run: |
+          echo "Building the default package..."
+          nix build .#default --print-build-logs
+
+      - name: Test Binary
+        run: |
+          echo "Verifying the built binary..."
+          if [ -x "./result/bin/rustfs" ]; then
+              ./result/bin/rustfs --help
+              echo "Binary verification successful."
+          else
+              echo "Error: Binary not found or not executable at ./result/bin/rustfs"
+              exit 1
+          fi

.github/workflows/performance.yml

@@ -40,11 +40,13 @@ env:
 jobs:
   performance-profile:
     name: Performance Profiling
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     timeout-minutes: 30
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
@@ -107,7 +109,7 @@ jobs:
 
       - name: Upload profile data
         if: steps.profiling.outputs.profile_generated == 'true'
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v6
         with:
           name: performance-profile-${{ github.run_number }}
           path: samply-profile.json
@@ -115,11 +117,13 @@ jobs:
 
   benchmark:
     name: Benchmark Tests
-    runs-on: ubuntu-latest
+    runs-on: ubicloud-standard-2
     timeout-minutes: 45
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
@@ -135,7 +139,7 @@ jobs:
           tee benchmark-results.json
 
       - name: Upload benchmark results
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v6
         with:
           name: benchmark-results-${{ github.run_number }}
           path: benchmark-results.json

.github/workflows/stale.yml

@@ -0,0 +1,20 @@
+name: "Mark stale issues"
+on:
+  schedule:
+    - cron: "30 1 * * *"
+
+jobs:
+  stale:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/stale@v9
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          stale-issue-message: 'This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.'
+          stale-issue-label: 'stale'
+          ## Mark if there is no activity for more than 7 days
+          days-before-stale: 7  
+          # If no one responds after 3 days, the tag will be closed.
+          days-before-close: 3   
+          # These tags are exempt and will not close automatically.
+          exempt-issue-labels: 'pinned,security'

.gitignore

@@ -2,9 +2,13 @@
 .DS_Store
 .idea
 .vscode
+.cursor
+.direnv/
 /test
 /logs
 /data
+/docs
+/rustfs-data/
 .devcontainer
 rustfs/static/*
 !rustfs/static/.gitkeep
@@ -12,10 +16,12 @@ vendor
 cli/rustfs-gui/embedded-rustfs/rustfs
 *.log
 deploy/certs/*
+deploy/data/*
 *jsonl
 .env
 .rustfs.sys
-.cargo
+.cargo/
+!.cargo/config.toml
 profile.json
 .docker/openobserve-otel/data
 *.zst
@@ -23,4 +29,27 @@ profile.json
 *.go
 *.pb
 *.svg
-deploy/logs/*.log.*
+deploy/logs/*.log.*
+artifacts/
+# s3-tests local artifacts (root directory only)
+/s3-tests/
+/s3-tests-local/
+/s3tests.conf
+/s3tests.conf.*
+*.events
+*.audit
+*.snappy
+PR_DESCRIPTION.md
+scripts/s3-tests/selected_tests.txt
+docs
+
+# nix stuff
+result*
+*.gz
+rustfs-webdav.code-workspace
+
+.aiexclude
+*.bak
+# Local test/benchmark artifacts
+benchmarks.logs
+tmp/

.pre-commit-config.yaml

@@ -0,0 +1,32 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+  - repo: local
+    hooks:
+      - id: cargo-fmt
+        name: cargo fmt
+        entry: cargo fmt --all --check
+        language: system
+        types: [rust]
+        pass_filenames: false
+
+      - id: cargo-clippy
+        name: cargo clippy
+        entry: cargo clippy --all-targets --all-features -- -D warnings
+        language: system
+        types: [rust]
+        pass_filenames: false
+
+      - id: cargo-check
+        name: cargo check
+        entry: cargo check --all-targets
+        language: system
+        types: [rust]
+        pass_filenames: false
+
+      - id: cargo-test
+        name: cargo test
+        entry: bash -c 'cargo test --workspace --exclude e2e_test && cargo test --all --doc'
+        language: system
+        types: [rust]
+        pass_filenames: false

.rules.md

@@ -1,702 +0,0 @@
-# RustFS Project AI Coding Rules
-
-## 🚨🚨🚨 CRITICAL DEVELOPMENT RULES - ZERO TOLERANCE 🚨🚨🚨
-
-### ⛔️ ABSOLUTE PROHIBITION: NEVER COMMIT DIRECTLY TO MASTER/MAIN BRANCH ⛔️
-
-**🔥 THIS IS THE MOST CRITICAL RULE - VIOLATION WILL RESULT IN IMMEDIATE REVERSAL 🔥**
-
-- **🚫 ZERO DIRECT COMMITS TO MAIN/MASTER BRANCH - ABSOLUTELY FORBIDDEN**
-- **🚫 ANY DIRECT COMMIT TO MAIN BRANCH MUST BE IMMEDIATELY REVERTED**
-- **🚫 NO EXCEPTIONS FOR HOTFIXES, EMERGENCIES, OR URGENT CHANGES**
-- **🚫 NO EXCEPTIONS FOR SMALL CHANGES, TYPOS, OR DOCUMENTATION UPDATES**
-- **🚫 NO EXCEPTIONS FOR ANYONE - MAINTAINERS, CONTRIBUTORS, OR ADMINS**
-
-### 📋 MANDATORY WORKFLOW - STRICTLY ENFORCED
-
-**EVERY SINGLE CHANGE MUST FOLLOW THIS WORKFLOW:**
-
-1. **Check current branch**: `git branch` (MUST NOT be on main/master)
-2. **Switch to main**: `git checkout main`
-3. **Pull latest**: `git pull origin main`
-4. **Create feature branch**: `git checkout -b feat/your-feature-name`
-5. **Make changes ONLY on feature branch**
-6. **Test thoroughly before committing**
-7. **Commit and push to feature branch**: `git push origin feat/your-feature-name`
-8. **Create Pull Request**: Use `gh pr create` (MANDATORY)
-9. **Wait for PR approval**: NO self-merging allowed
-10. **Merge through GitHub interface**: ONLY after approval
-
-### 🔒 ENFORCEMENT MECHANISMS
-
-- **Branch protection rules**: Main branch is protected
-- **Pre-commit hooks**: Will block direct commits to main
-- **CI/CD checks**: All PRs must pass before merging
-- **Code review requirement**: At least one approval needed
-- **Automated reversal**: Direct commits to main will be automatically reverted
-
-## 🎯 Core AI Development Principles
-
-### Five Execution Steps
-
-#### 1. Task Analysis and Planning
-- **Clear Objectives**: Deeply understand task requirements and expected results before starting coding
-- **Plan Development**: List specific files, components, and functions that need modification, explaining the reasons for changes
-- **Risk Assessment**: Evaluate the impact of changes on existing functionality, develop rollback plans
-
-#### 2. Precise Code Location
-- **File Identification**: Determine specific files and line numbers that need modification
-- **Impact Analysis**: Avoid modifying irrelevant files, clearly state the reason for each file modification
-- **Minimization Principle**: Unless explicitly required by the task, do not create new abstraction layers or refactor existing code
-
-#### 3. Minimal Code Changes
-- **Focus on Core**: Only write code directly required by the task
-- **Avoid Redundancy**: Do not add unnecessary logs, comments, tests, or error handling
-- **Isolation**: Ensure new code does not interfere with existing functionality, maintain code independence
-
-#### 4. Strict Code Review
-- **Correctness Check**: Verify the correctness and completeness of code logic
-- **Style Consistency**: Ensure code conforms to established project coding style
-- **Side Effect Assessment**: Evaluate the impact of changes on downstream systems
-
-#### 5. Clear Delivery Documentation
-- **Change Summary**: Detailed explanation of all modifications and reasons
-- **File List**: List all modified files and their specific changes
-- **Risk Statement**: Mark any assumptions or potential risk points
-
-### Core Principles
-- **🎯 Precise Execution**: Strictly follow task requirements, no arbitrary innovation
-- **⚡ Efficient Development**: Avoid over-design, only do necessary work
-- **🛡️ Safe and Reliable**: Always follow development processes, ensure code quality and system stability
-- **🔒 Cautious Modification**: Only modify when clearly knowing what needs to be changed and having confidence
-
-### Additional AI Behavior Rules
-
-1. **Use English for all code comments and documentation** - All comments, variable names, function names, documentation, and user-facing text in code should be in English
-2. **Clean up temporary scripts after use** - Any temporary scripts, test files, or helper files created during AI work should be removed after task completion
-3. **Only make confident modifications** - Do not make speculative changes or "convenient" modifications outside the task scope. If uncertain about a change, ask for clarification rather than guessing
-
-## Project Overview
-
-RustFS is a high-performance distributed object storage system written in Rust, compatible with S3 API. The project adopts a modular architecture, supporting erasure coding storage, multi-tenant management, observability, and other enterprise-level features.
-
-## Core Architecture Principles
-
-### 1. Modular Design
-
-- Project uses Cargo workspace structure, containing multiple independent crates
-- Core modules: `rustfs` (main service), `ecstore` (erasure coding storage), `common` (shared components)
-- Functional modules: `iam` (identity management), `madmin` (management interface), `crypto` (encryption), etc.
-- Tool modules: `cli` (command line tool), `crates/*` (utility libraries)
-
-### 2. Asynchronous Programming Pattern
-
-- Comprehensive use of `tokio` async runtime
-- Prioritize `async/await` syntax
-- Use `async-trait` for async methods in traits
-- Avoid blocking operations, use `spawn_blocking` when necessary
-
-### 3. Error Handling Strategy
-
-- **Use modular, type-safe error handling with `thiserror`**
-- Each module should define its own error type using `thiserror::Error` derive macro
-- Support error chains and context information through `#[from]` and `#[source]` attributes
-- Use `Result<T>` type aliases for consistency within each module
-- Error conversion between modules should use explicit `From` implementations
-- Follow the pattern: `pub type Result<T> = core::result::Result<T, Error>`
-- Use `#[error("description")]` attributes for clear error messages
-- Support error downcasting when needed through `other()` helper methods
-- Implement `Clone` for errors when required by the domain logic
-
-## Code Style Guidelines
-
-### 1. Formatting Configuration
-
-```toml
-max_width = 130
-fn_call_width = 90
-single_line_let_else_max_width = 100
-```
-
-### 2. **🔧 MANDATORY Code Formatting Rules**
-
-**CRITICAL**: All code must be properly formatted before committing. This project enforces strict formatting standards to maintain code consistency and readability.
-
-#### Pre-commit Requirements (MANDATORY)
-
-Before every commit, you **MUST**:
-
-1. **Format your code**:
-   ```bash
-   cargo fmt --all
-   ```
-
-2. **Verify formatting**:
-   ```bash
-   cargo fmt --all --check
-   ```
-
-3. **Pass clippy checks**:
-   ```bash
-   cargo clippy --all-targets --all-features -- -D warnings
-   ```
-
-4. **Ensure compilation**:
-   ```bash
-   cargo check --all-targets
-   ```
-
-#### Quick Commands
-
-Use these convenient Makefile targets for common tasks:
-
-```bash
-# Format all code
-make fmt
-
-# Check if code is properly formatted
-make fmt-check
-
-# Run clippy checks
-make clippy
-
-# Run compilation check
-make check
-
-# Run tests
-make test
-
-# Run all pre-commit checks (format + clippy + check + test)
-make pre-commit
-
-# Setup git hooks (one-time setup)
-make setup-hooks
-```
-
-### 3. Naming Conventions
-
-- Use `snake_case` for functions, variables, modules
-- Use `PascalCase` for types, traits, enums
-- Constants use `SCREAMING_SNAKE_CASE`
-- Global variables prefix `GLOBAL_`, e.g., `GLOBAL_Endpoints`
-- Use meaningful and descriptive names for variables, functions, and methods
-- Avoid meaningless names like `temp`, `data`, `foo`, `bar`, `test123`
-- Choose names that clearly express the purpose and intent
-
-### 4. Type Declaration Guidelines
-
-- **Prefer type inference over explicit type declarations** when the type is obvious from context
-- Let the Rust compiler infer types whenever possible to reduce verbosity and improve maintainability
-- Only specify types explicitly when:
-  - The type cannot be inferred by the compiler
-  - Explicit typing improves code clarity and readability
-  - Required for API boundaries (function signatures, public struct fields)
-  - Needed to resolve ambiguity between multiple possible types
-
-### 5. Documentation Comments
-
-- Public APIs must have documentation comments
-- Use `///` for documentation comments
-- Complex functions add `# Examples` and `# Parameters` descriptions
-- Error cases use `# Errors` descriptions
-- Always use English for all comments and documentation
-- Avoid meaningless comments like "debug 111" or placeholder text
-
-### 6. Import Guidelines
-
-- Standard library imports first
-- Third-party crate imports in the middle
-- Project internal imports last
-- Group `use` statements with blank lines between groups
-
-## Asynchronous Programming Guidelines
-
-### 1. Trait Definition
-
-```rust
-#[async_trait::async_trait]
-pub trait StorageAPI: Send + Sync {
-    async fn get_object(&self, bucket: &str, object: &str) -> Result<ObjectInfo>;
-}
-```
-
-### 2. Error Handling
-
-```rust
-// Use ? operator to propagate errors
-async fn example_function() -> Result<()> {
-    let data = read_file("path").await?;
-    process_data(data).await?;
-    Ok(())
-}
-```
-
-### 3. Concurrency Control
-
-- Use `Arc` and `Mutex`/`RwLock` for shared state management
-- Prioritize async locks from `tokio::sync`
-- Avoid holding locks for long periods
-
-## Logging and Tracing Guidelines
-
-### 1. Tracing Usage
-
-```rust
-#[tracing::instrument(skip(self, data))]
-async fn process_data(&self, data: &[u8]) -> Result<()> {
-    info!("Processing {} bytes", data.len());
-    // Implementation logic
-}
-```
-
-### 2. Log Levels
-
-- `error!`: System errors requiring immediate attention
-- `warn!`: Warning information that may affect functionality
-- `info!`: Important business information
-- `debug!`: Debug information for development use
-- `trace!`: Detailed execution paths
-
-### 3. Structured Logging
-
-```rust
-info!(
-    counter.rustfs_api_requests_total = 1_u64,
-    key_request_method = %request.method(),
-    key_request_uri_path = %request.uri().path(),
-    "API request processed"
-);
-```
-
-## Error Handling Guidelines
-
-### 1. Error Type Definition
-
-```rust
-// Use thiserror for module-specific error types
-#[derive(thiserror::Error, Debug)]
-pub enum MyError {
-    #[error("IO error: {0}")]
-    Io(#[from] std::io::Error),
-
-    #[error("Storage error: {0}")]
-    Storage(#[from] ecstore::error::StorageError),
-
-    #[error("Custom error: {message}")]
-    Custom { message: String },
-
-    #[error("File not found: {path}")]
-    FileNotFound { path: String },
-
-    #[error("Invalid configuration: {0}")]
-    InvalidConfig(String),
-}
-
-// Provide Result type alias for the module
-pub type Result<T> = core::result::Result<T, MyError>;
-```
-
-### 2. Error Helper Methods
-
-```rust
-impl MyError {
-    /// Create error from any compatible error type
-    pub fn other<E>(error: E) -> Self
-    where
-        E: Into<Box<dyn std::error::Error + Send + Sync>>,
-    {
-        MyError::Io(std::io::Error::other(error))
-    }
-}
-```
-
-### 3. Error Context and Propagation
-
-```rust
-// Use ? operator for clean error propagation
-async fn example_function() -> Result<()> {
-    let data = read_file("path").await?;
-    process_data(data).await?;
-    Ok(())
-}
-
-// Add context to errors
-fn process_with_context(path: &str) -> Result<()> {
-    std::fs::read(path)
-        .map_err(|e| MyError::Custom {
-            message: format!("Failed to read {}: {}", path, e)
-        })?;
-    Ok(())
-}
-```
-
-## Performance Optimization Guidelines
-
-### 1. Memory Management
-
-- Use `Bytes` instead of `Vec<u8>` for zero-copy operations
-- Avoid unnecessary cloning, use reference passing
-- Use `Arc` for sharing large objects
-
-### 2. Concurrency Optimization
-
-```rust
-// Use join_all for concurrent operations
-let futures = disks.iter().map(|disk| disk.operation());
-let results = join_all(futures).await;
-```
-
-### 3. Caching Strategy
-
-- Use `LazyLock` for global caching
-- Implement LRU cache to avoid memory leaks
-
-## Testing Guidelines
-
-### 1. Unit Tests
-
-```rust
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use test_case::test_case;
-
-    #[tokio::test]
-    async fn test_async_function() {
-        let result = async_function().await;
-        assert!(result.is_ok());
-    }
-
-    #[test_case("input1", "expected1")]
-    #[test_case("input2", "expected2")]
-    fn test_with_cases(input: &str, expected: &str) {
-        assert_eq!(function(input), expected);
-    }
-}
-```
-
-### 2. Integration Tests
-
-- Use `e2e_test` module for end-to-end testing
-- Simulate real storage environments
-
-### 3. Test Quality Standards
-
-- Write meaningful test cases that verify actual functionality
-- Avoid placeholder or debug content like "debug 111", "test test", etc.
-- Use descriptive test names that clearly indicate what is being tested
-- Each test should have a clear purpose and verify specific behavior
-- Test data should be realistic and representative of actual use cases
-
-## Cross-Platform Compatibility Guidelines
-
-### 1. CPU Architecture Compatibility
-
-- **Always consider multi-platform and different CPU architecture compatibility** when writing code
-- Support major architectures: x86_64, aarch64 (ARM64), and other target platforms
-- Use conditional compilation for architecture-specific code:
-
-```rust
-#[cfg(target_arch = "x86_64")]
-fn optimized_x86_64_function() { /* x86_64 specific implementation */ }
-
-#[cfg(target_arch = "aarch64")]
-fn optimized_aarch64_function() { /* ARM64 specific implementation */ }
-
-#[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
-fn generic_function() { /* Generic fallback implementation */ }
-```
-
-### 2. Platform-Specific Dependencies
-
-- Use feature flags for platform-specific dependencies
-- Provide fallback implementations for unsupported platforms
-- Test on multiple architectures in CI/CD pipeline
-
-### 3. Endianness Considerations
-
-- Use explicit byte order conversion when dealing with binary data
-- Prefer `to_le_bytes()`, `from_le_bytes()` for consistent little-endian format
-- Use `byteorder` crate for complex binary format handling
-
-### 4. SIMD and Performance Optimizations
-
-- Use portable SIMD libraries like `wide` or `packed_simd`
-- Provide fallback implementations for non-SIMD architectures
-- Use runtime feature detection when appropriate
-
-## Security Guidelines
-
-### 1. Memory Safety
-
-- Disable `unsafe` code (workspace.lints.rust.unsafe_code = "deny")
-- Use `rustls` instead of `openssl`
-
-### 2. Authentication and Authorization
-
-```rust
-// Use IAM system for permission checks
-let identity = iam.authenticate(&access_key, &secret_key).await?;
-iam.authorize(&identity, &action, &resource).await?;
-```
-
-## Configuration Management Guidelines
-
-### 1. Environment Variables
-
-- Use `RUSTFS_` prefix
-- Support both configuration files and environment variables
-- Provide reasonable default values
-
-### 2. Configuration Structure
-
-```rust
-#[derive(Debug, Deserialize, Clone)]
-pub struct Config {
-    pub address: String,
-    pub volumes: String,
-    #[serde(default)]
-    pub console_enable: bool,
-}
-```
-
-## Dependency Management Guidelines
-
-### 1. Workspace Dependencies
-
-- Manage versions uniformly at workspace level
-- Use `workspace = true` to inherit configuration
-
-### 2. Feature Flags
-
-```rust
-[features]
-default = ["file"]
-gpu = ["dep:nvml-wrapper"]
-kafka = ["dep:rdkafka"]
-```
-
-## Deployment and Operations Guidelines
-
-### 1. Containerization
-
-- Provide Dockerfile and docker-compose configuration
-- Support multi-stage builds to optimize image size
-
-### 2. Observability
-
-- Integrate OpenTelemetry for distributed tracing
-- Support Prometheus metrics collection
-- Provide Grafana dashboards
-
-### 3. Health Checks
-
-```rust
-// Implement health check endpoint
-async fn health_check() -> Result<HealthStatus> {
-    // Check component status
-}
-```
-
-## Code Review Checklist
-
-### 1. **Code Formatting and Quality (MANDATORY)**
-
-- [ ] **Code is properly formatted** (`cargo fmt --all --check` passes)
-- [ ] **All clippy warnings are resolved** (`cargo clippy --all-targets --all-features -- -D warnings` passes)
-- [ ] **Code compiles successfully** (`cargo check --all-targets` passes)
-- [ ] **Pre-commit hooks are working** and all checks pass
-- [ ] **No formatting-related changes** mixed with functional changes (separate commits)
-
-### 2. Functionality
-
-- [ ] Are all error cases properly handled?
-- [ ] Is there appropriate logging?
-- [ ] Is there necessary test coverage?
-
-### 3. Performance
-
-- [ ] Are unnecessary memory allocations avoided?
-- [ ] Are async operations used correctly?
-- [ ] Are there potential deadlock risks?
-
-### 4. Security
-
-- [ ] Are input parameters properly validated?
-- [ ] Are there appropriate permission checks?
-- [ ] Is information leakage avoided?
-
-### 5. Cross-Platform Compatibility
-
-- [ ] Does the code work on different CPU architectures (x86_64, aarch64)?
-- [ ] Are platform-specific features properly gated with conditional compilation?
-- [ ] Is byte order handling correct for binary data?
-- [ ] Are there appropriate fallback implementations for unsupported platforms?
-
-### 6. Code Commits and Documentation
-
-- [ ] Does it comply with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)?
-- [ ] Are commit messages concise and under 72 characters for the title line?
-- [ ] Commit titles should be concise and in English, avoid Chinese
-- [ ] Is PR description provided in copyable markdown format for easy copying?
-
-## Common Patterns and Best Practices
-
-### 1. Resource Management
-
-```rust
-// Use RAII pattern for resource management
-pub struct ResourceGuard {
-    resource: Resource,
-}
-
-impl Drop for ResourceGuard {
-    fn drop(&mut self) {
-        // Clean up resources
-    }
-}
-```
-
-### 2. Dependency Injection
-
-```rust
-// Use dependency injection pattern
-pub struct Service {
-    config: Arc<Config>,
-    storage: Arc<dyn StorageAPI>,
-}
-```
-
-### 3. Graceful Shutdown
-
-```rust
-// Implement graceful shutdown
-async fn shutdown_gracefully(shutdown_rx: &mut Receiver<()>) {
-    tokio::select! {
-        _ = shutdown_rx.recv() => {
-            info!("Received shutdown signal");
-            // Perform cleanup operations
-        }
-        _ = tokio::time::sleep(SHUTDOWN_TIMEOUT) => {
-            warn!("Shutdown timeout reached");
-        }
-    }
-}
-```
-
-## Domain-Specific Guidelines
-
-### 1. Storage Operations
-
-- All storage operations must support erasure coding
-- Implement read/write quorum mechanisms
-- Support data integrity verification
-
-### 2. Network Communication
-
-- Use gRPC for internal service communication
-- HTTP/HTTPS support for S3-compatible API
-- Implement connection pooling and retry mechanisms
-
-### 3. Metadata Management
-
-- Use FlatBuffers for serialization
-- Support version control and migration
-- Implement metadata caching
-
-## Branch Management and Development Workflow
-
-### Branch Management
-
-- **🚨 CRITICAL: NEVER modify code directly on main or master branch - THIS IS ABSOLUTELY FORBIDDEN 🚨**
-- **⚠️ ANY DIRECT COMMITS TO MASTER/MAIN WILL BE REJECTED AND MUST BE REVERTED IMMEDIATELY ⚠️**
-- **🔒 ALL CHANGES MUST GO THROUGH PULL REQUESTS - NO DIRECT COMMITS TO MAIN UNDER ANY CIRCUMSTANCES 🔒**
-- **Always work on feature branches - NO EXCEPTIONS**
-- Always check the .rules.md file before starting to ensure you understand the project guidelines
-- **MANDATORY workflow for ALL changes:**
-   1. `git checkout main` (switch to main branch)
-   2. `git pull` (get latest changes)
-   3. `git checkout -b feat/your-feature-name` (create and switch to feature branch)
-   4. Make your changes ONLY on the feature branch
-   5. Test thoroughly before committing
-   6. Commit and push to the feature branch
-   7. **Create a pull request for code review - THIS IS THE ONLY WAY TO MERGE TO MAIN**
-   8. **Wait for PR approval before merging - NEVER merge your own PRs without review**
-- Use descriptive branch names following the pattern: `feat/feature-name`, `fix/issue-name`, `refactor/component-name`, etc.
-- **Double-check current branch before ANY commit: `git branch` to ensure you're NOT on main/master**
-- **Pull Request Requirements:**
-  - All changes must be submitted via PR regardless of size or urgency
-  - PRs must include comprehensive description and testing information
-  - PRs must pass all CI/CD checks before merging
-  - PRs require at least one approval from code reviewers
-  - Even hotfixes and emergency changes must go through PR process
-- **Enforcement:**
-  - Main branch should be protected with branch protection rules
-  - Direct pushes to main should be blocked by repository settings
-  - Any accidental direct commits to main must be immediately reverted via PR
-
-### Development Workflow
-
-## 🎯 **Core Development Principles**
-
-- **🔴 Every change must be precise - don't modify unless you're confident**
-  - Carefully analyze code logic and ensure complete understanding before making changes
-  - When uncertain, prefer asking users or consulting documentation over blind modifications
-  - Use small iterative steps, modify only necessary parts at a time
-  - Evaluate impact scope before changes to ensure no new issues are introduced
-
-- **🚀 GitHub PR creation prioritizes gh command usage**
-  - Prefer using `gh pr create` command to create Pull Requests
-  - Avoid having users manually create PRs through web interface
-  - Provide clear and professional PR titles and descriptions
-  - Using `gh` commands ensures better integration and automation
-
-## 📝 **Code Quality Requirements**
-
-- Use English for all code comments, documentation, and variable names
-- Write meaningful and descriptive names for variables, functions, and methods
-- Avoid meaningless test content like "debug 111" or placeholder values
-- Before each change, carefully read the existing code to ensure you understand the code structure and implementation, do not break existing logic implementation, do not introduce new issues
-- Ensure each change provides sufficient test cases to guarantee code correctness
-- Do not arbitrarily modify numbers and constants in test cases, carefully analyze their meaning to ensure test case correctness
-- When writing or modifying tests, check existing test cases to ensure they have scientific naming and rigorous logic testing, if not compliant, modify test cases to ensure scientific and rigorous testing
-- **Before committing any changes, run `cargo clippy --all-targets --all-features -- -D warnings` to ensure all code passes Clippy checks**
-- After each development completion, first git add . then git commit -m "feat: feature description" or "fix: issue description", ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
-- **Keep commit messages concise and under 72 characters** for the title line, use body for detailed explanations if needed
-- After each development completion, first git push to remote repository
-- After each change completion, summarize the changes, do not create summary files, provide a brief change description, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
-- Provide change descriptions needed for PR in the conversation, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
-- **Always provide PR descriptions in English** after completing any changes, including:
-  - Clear and concise title following Conventional Commits format
-  - Detailed description of what was changed and why
-  - List of key changes and improvements
-  - Any breaking changes or migration notes if applicable
-  - Testing information and verification steps
-- **Provide PR descriptions in copyable markdown format** enclosed in code blocks for easy one-click copying
-
-## 🚫 AI Documentation Generation Restrictions
-
-### Forbidden Summary Documents
-
-- **Strictly forbidden to create any form of AI-generated summary documents**
-- **Do not create documents containing large amounts of emoji, detailed formatting tables and typical AI style**
-- **Do not generate the following types of documents in the project:**
-  - Benchmark summary documents (BENCHMARK*.md)
-  - Implementation comparison analysis documents (IMPLEMENTATION_COMPARISON*.md)
-  - Performance analysis report documents
-  - Architecture summary documents
-  - Feature comparison documents
-  - Any documents with large amounts of emoji and formatted content
-- **If documentation is needed, only create when explicitly requested by the user, and maintain a concise and practical style**
-- **Documentation should focus on actually needed information, avoiding excessive formatting and decorative content**
-- **Any discovered AI-generated summary documents should be immediately deleted**
-
-### Allowed Documentation Types
-
-- README.md (project introduction, keep concise)
-- Technical documentation (only create when explicitly needed)
-- User manual (only create when explicitly needed)
-- API documentation (generated from code)
-- Changelog (CHANGELOG.md)
-
-These rules should serve as guiding principles when developing the RustFS project, ensuring code quality, performance, and maintainability.

.vscode/launch.json

@@ -1,9 +1,31 @@
 {
-    // 使用 IntelliSense 了解相关属性。 
-    // 悬停以查看现有属性的描述。
-    // 欲了解更多信息，请访问: https://go.microsoft.com/fwlink/?linkid=830387
     "version": "0.2.0",
     "configurations": [
+          {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug(only) executable 'rustfs'",
+            "env": {
+              "RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
+              "RUSTFS_SKIP_BACKGROUND_TASK": "on"
+              //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
+              // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
+              // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
+            },
+            "program": "${workspaceFolder}/target/debug/rustfs",
+            "args": [
+              "--access-key",
+              "rustfsadmin",
+              "--secret-key",
+              "rustfsadmin",
+              "--address",
+              "0.0.0.0:9010",
+              "--server-domains",
+              "127.0.0.1:9010",
+              "./target/volume/test{1...4}"
+            ],
+            "cwd": "${workspaceFolder}"
+        },
         {
             "type": "lldb",
             "request": "launch",
@@ -22,6 +44,7 @@
             "env": {
                 "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
                 "RUSTFS_SKIP_BACKGROUND_TASK": "on",
+                //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
                 // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
                 // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" 
             },
@@ -66,12 +89,8 @@
                     "test",
                     "--no-run",
                     "--lib",
-                    "--package=ecstore"
+                    "--package=rustfs-ecstore"
-                ],
+                ]
-                "filter": {
-                    "name": "ecstore",
-                    "kind": "lib"
-                }
             },
             "args": [],
             "cwd": "${workspaceFolder}"
@@ -80,15 +99,58 @@
             "name": "Debug executable target/debug/rustfs",
             "type":  "lldb",
             "request": "launch",
-            "program": "${workspaceFolder}/target/debug/rustfs",
+            "cargo": {
+                "args": [
+                    "run",
+                    "--bin",
+                    "rustfs",
+                    "-j",
+                    "1",
+                    "--profile",
+                    "dev"
+                ]
+            },
             "args": [],
             "cwd": "${workspaceFolder}",
             //"stopAtEntry": false,
             //"preLaunchTask": "cargo build",
+            "env": {
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                //"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
+                // "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
+                // "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint
+                // "RUSTFS_COMPRESS_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+                "RUST_LOG":"rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
+            },
             "sourceLanguages": [
                 "rust"
             ],
         },
+        {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug test_lifecycle_transition_basic",
+            "cargo": {
+                "args": [
+                    "test",
+                    "-p",
+                    "rustfs-scanner",
+                    "--test",
+                    "lifecycle_integration_test",
+                    "serial_tests::test_lifecycle_transition_basic",
+                    "-j",
+                    "1"
+                ]
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}"
+        },
         {
             "name": "Debug executable target/debug/test",
             "type":  "lldb",
@@ -108,6 +170,89 @@
             "sourceLanguages": [
                 "rust"
             ],
-        }
+        },
+        {
+            "name": "Debug executable target/debug/rustfs with sse",
+            "type":  "lldb",
+            "request": "launch",
+            "program": "${workspaceFolder}/target/debug/rustfs",
+            "args": [],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "env": {
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                "RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+                // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
+                // "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
+                // "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint
+                // "RUSTFS_COMPRESS_ENABLE": "true",
+
+                // 1. simple sse test key (no kms system)
+                // "__RUSTFS_SSE_SIMPLE_CMK": "2dfNXGHlsEflGVCxb+5DIdGEl1sIvtwX+QfmYasi5QM=",
+
+                // 2. kms local backend test key
+                // "RUSTFS_KMS_ENABLE": "true",
+                // "RUSTFS_KMS_BACKEND": "local",
+                // "RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
+                // "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
+                // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+
+                // 3. kms vault backend test key
+                // "RUSTFS_KMS_ENABLE": "true",
+                // "RUSTFS_KMS_BACKEND": "vault-kv2",
+                // "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
+                // "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
+                // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+
+                // 4. kms vault transit backend test key
+                "RUSTFS_KMS_ENABLE": "true",
+                "RUSTFS_KMS_BACKEND": "vault-transit",
+                "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
+                "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
+                "RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
+                "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+
+            },
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
+        {
+            "name": "E2E test executable target/debug/rustfs",
+            "type":  "lldb",
+            "request": "launch",
+            "program": "${workspaceFolder}/target/debug/rustfs",
+            "args": [],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "env": {
+                "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
+                "RUST_BACKTRACE": "full",
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                "RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
+                "RUSTFS_REGION": "us-east-1",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+
+                "RUSTFS_KMS_ENABLE": "true",
+                "RUSTFS_KMS_BACKEND": "local",
+                "RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
+                "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
+                "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+            },
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
     ]
 }

AGENTS.md

@@ -1,24 +1,113 @@
-# Repository Guidelines
+# RustFS Agent Instructions (Global)
 
-## Communication Rules
+This root file keeps repository-wide rules only.
-- Respond to the user in Chinese; use English in all other contexts.
+Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
 
-## Project Structure & Module Organization
+## Rule Precedence
-The workspace root hosts shared dependencies in `Cargo.toml`. The service binary lives under `rustfs/src/main.rs`, while reusable crates sit in `crates/` (`crypto`, `iam`, `kms`, and `e2e_test`). Local fixtures for standalone flows reside in `test_standalone/`, deployment manifests are under `deploy/`, Docker assets sit at the root, and automation lives in `scripts/`. Skim each crate’s README or module docs before contributing changes.
 
-## Build, Test, and Development Commands
+1. System/developer instructions.
-Run `cargo check --all-targets` for fast validation. Build release binaries via `cargo build --release` or the pipeline-aligned `make build`. Use `./build-rustfs.sh --dev` for iterative development and `./build-rustfs.sh --platform <target>` for cross-compiles. Prefer `make pre-commit` before pushing to cover formatting, clippy, checks, and tests.
+2. Current user/task instructions.
-Always ensure `cargo fmt --all --check`, `cargo test --workspace --exclude e2e_test`, and `cargo clippy --all-targets --all-features -- -D warnings` complete successfully after each code change to keep the tree healthy and warning-free.
+3. The nearest `AGENTS.md` in the current path.
+4. This file (global defaults).
 
-## Coding Style & Naming Conventions
+If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
-Formatting follows the repo `rustfmt.toml` (130-column width). Use `snake_case` for items, `PascalCase` for types, and `SCREAMING_SNAKE_CASE` for constants. Avoid `unwrap()` or `expect()` outside tests; bubble errors with `Result` and crate-specific `thiserror` types. Keep async code non-blocking and offload CPU-heavy work with `tokio::task::spawn_blocking` when necessary.
 
-## Testing Guidelines
+## Execution Discipline
-Co-locate unit tests with their modules and give behavior-led names such as `handles_expired_token`. Integration suites belong in each crate’s `tests/` directory, while exhaustive end-to-end scenarios live in `crates/e2e_test/`. Run `cargo test --workspace --exclude e2e_test` during iteration, `cargo nextest run --all --exclude e2e_test` when available, and finish with `cargo test --all` before requesting review. Use `NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY=` for KMS e2e tests.
-When fixing bugs or adding features, include regression tests that capture the new behavior so future changes cannot silently break it.
 
-## Commit & Pull Request Guidelines
+- Read the relevant existing code, tests, and local guidance before changing behavior.
-Work on feature branches (e.g., `feat/...`) after syncing `main`. Follow Conventional Commits under 72 characters (e.g., `feat: add kms key rotation`). Each commit must compile, format cleanly, and pass `make pre-commit`. Open PRs with a concise summary, note verification commands, link relevant issues, and wait for reviewer approval.
+- State assumptions when they affect the implementation or verification path.
+- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
+- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
+- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
+- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification.
 
-## Security & Configuration Tips
+## Communication and Language
-Do not commit secrets or cloud credentials; prefer environment variables or vault tooling. Review IAM- and KMS-related changes with a second maintainer. Confirm proxy settings before running sensitive tests to avoid leaking traffic outside localhost.
+
+- Respond in the same language used by the requester.
+- Keep source code, comments, commit messages, and PR title/body in English.
+- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
+
+## Change Style for Existing Logic
+
+- Prefer direct, local code over extracting one-off helpers.
+- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
+- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
+- Prefer editing existing code over rewriting files or reshaping unrelated logic.
+- Modify only what is required and remove only artifacts introduced by your own changes.
+- Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
+- Do not refactor existing code only to make it easier to unit test.
+- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
+- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
+- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
+
+## Constant and String Usage
+
+- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
+- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
+- If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
+- When changing existing behavior, keep naming and format consistency by aligning with established project constants.
+
+## Sources of Truth
+
+- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
+- Local quality commands: `Makefile` and `.config/make/`
+- CI quality gates: `.github/workflows/ci.yml`
+- PR template: `.github/pull_request_template.md`
+
+Avoid duplicating long crate lists or command matrices in instruction files.
+Reference the source files above instead.
+
+## Verification Before PR
+
+Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
+
+For code changes, run and pass the following before opening a PR:
+
+```bash
+make pre-commit
+```
+
+Before pushing code changes, make sure formatting is clean:
+
+- Run `cargo fmt --all`.
+- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
+
+If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
+Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any installed git pre-commit hooks (for example, from `make setup-hooks`) may still run on commit unless explicitly skipped.
+After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
+Do not open a PR with code changes when the required checks fail.
+
+## Git and PR Baseline
+
+- Use feature branches based on the latest `main`.
+- Follow Conventional Commits, with subject length <= 72 characters.
+- Keep PR title and description in English.
+- Use `.github/pull_request_template.md` and keep all section headings.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands in the PR description.
+- When using `gh pr create`/`gh pr edit`, use `--body-file` instead of inline `--body` for multiline markdown.
+- Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment.
+- After fixing code review comments or CI findings, always mark corresponding review
+  comments/threads as resolved before returning to the user.
+- In handling review comments, confirm the underlying issue before changing code.
+  If a suggested change is not appropriate for behavior or risk, reply with a
+  concise rationale instead of blindly applying it.
+
+## Security Baseline
+
+- Never commit secrets, credentials, or key material.
+- Use environment variables or vault tooling for sensitive configuration.
+- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
+
+## Scoped Guidance in This Repository
+
+- `.github/AGENTS.md`
+- `crates/AGENTS.md`
+- `crates/config/AGENTS.md`
+- `crates/ecstore/AGENTS.md`
+- `crates/e2e_test/AGENTS.md`
+- `crates/iam/AGENTS.md`
+- `crates/kms/AGENTS.md`
+- `crates/policy/AGENTS.md`
+- `rustfs/src/admin/AGENTS.md`
+- `rustfs/src/storage/AGENTS.md`

ARCHITECTURE.md

@@ -0,0 +1,376 @@
+# ARCHITECTURE.md
+
+> Last updated: 2026-04-13 · Revision: 1 (draft)
+>
+> This document describes the high-level architecture of RustFS.
+> If you want to familiarize yourself with the code base, you are in the right place!
+>
+> See also [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow.
+
+## Bird's Eye View
+
+RustFS is a high-performance, S3-compatible distributed object storage system written
+in Rust. It uses erasure coding for data durability, supports multi-tenancy through
+IAM/STS, and provides a web-based admin console.
+
+A running RustFS node exposes:
+
+- **S3 API** (port 9000) — the primary data path for object CRUD
+- **Admin API** (port 9000, `/minio/` prefix) — cluster management, IAM, metrics
+- **Console** (port 9001) — web UI backed by the Admin API
+- **Inter-node RPC** (gRPC/tonic) — cluster communication for distributed mode
+
+The core data flow for a PUT request looks like:
+
+```
+HTTP request
+  → server (TLS, auth, routing, compression)
+    → app/object_usecase (validation, policy, lifecycle)
+      → storage/ecfs (erasure coding, encryption, checksums)
+        → ecstore (disk pool selection, data distribution)
+          → rio (reader pipeline: encrypt → compress → hash → write)
+            → io-core (zero-copy I/O, buffer pool, direct I/O)
+              → local disk / remote disk via RPC
+```
+
+## Code Map
+
+The repository is a Cargo workspace with a flat `crates/` layout:
+
+```
+rustfs/                      # Workspace root (virtual manifest)
+├── rustfs/                  # Main binary + library crate (75K lines)
+│   └── src/
+│       ├── main.rs          # Entry point, startup sequence
+│       ├── lib.rs           # Module tree root
+│       ├── server/          # HTTP server, TLS, routing, middleware
+│       ├── admin/           # Admin API handlers and console
+│       ├── app/             # Use-case layer (object, bucket, multipart)
+│       ├── storage/         # Storage engine interface and implementation
+│       ├── auth.rs          # S3 request authentication
+│       ├── config/          # CLI args, config parsing, workload profiles
+│       └── ...
+├── crates/                  # 39 library crates
+│   ├── ecstore/             # Erasure-coded storage engine (⚠️ 87K lines)
+│   ├── rio/                 # Reader I/O pipeline (encrypt, compress, hash)
+│   ├── io-core/             # Zero-copy I/O, scheduling, buffer pool
+│   ├── io-metrics/          # I/O metrics collection
+│   ├── common/              # Shared runtime state, globals, data usage types
+│   ├── config/              # Configuration types and parsing
+│   ├── utils/               # Pure utility functions
+│   ├── ...                  # (see "Crate Reference" below)
+│   └── e2e_test/            # End-to-end integration tests
+└── docs/                    # Design documents and analysis
+```
+
+### Main Crate Layers (`rustfs/src/`)
+
+The main crate is organized in layers, top to bottom:
+
+| Layer | Directory | Responsibility |
+|-------|-----------|----------------|
+| **Server** | `server/` | HTTP listener, TLS, CORS, compression, middleware, graceful shutdown |
+| **Admin** | `admin/` | Admin API routing, 30+ handler modules, web console |
+| **App** | `app/` | Use-case orchestration: object_usecase, bucket_usecase, multipart_usecase |
+| **Storage** | `storage/` | S3 API translation, erasure-coded FS, SSE encryption, RPC, concurrency |
+| **Auth** | `auth.rs` | S3 signature verification, credential validation |
+| **Config** | `config/` | CLI parsing, config struct, workload profiles |
+
+A request flows **downward** through the layers. No layer should reach upward
+(e.g., storage must not import from admin).
+
+### Crate Reference
+
+Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
+
+```
+Depth 0 — LEAF (no internal deps):
+  appauth, checksums, config, credentials, crypto, io-metrics,
+  madmin, s3-common, workers, zip
+
+Depth 1:
+  io-core (→ io-metrics)
+  policy (→ config, credentials, crypto)
+  utils (→ config)                        ⚠️ inverted: utils should be leaf
+
+Depth 2:
+  concurrency, filemeta, keystone, kms, lock, obs,
+  signer, targets, trusted-proxies
+
+Depth 3:
+  common (→ filemeta, madmin)             ⚠️ inverted: common should be leaf
+
+Depth 4:
+  object-capacity, protos, rio
+
+Depth 5 — CORE:
+  ecstore (16 internal deps, 11 dependents — the architectural heart)
+
+Depth 6:
+  audit, heal, iam, metrics, notify, s3select-api, scanner
+
+Depth 7:
+  object-io, protocols, s3select-query
+
+Depth 8 — TOP:
+  rustfs (35 internal deps — the binary, depends on almost everything)
+```
+
+#### By Domain
+
+**Core Infrastructure:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `config` | 3.3K | Configuration types and environment parsing |
+| `utils` | 8.7K | Pure utilities (paths, compression, network, retry) |
+| `common` | 4.4K | Shared runtime state, globals, data usage types, metrics |
+| `madmin` | 5.5K | Admin API request/response types |
+
+**I/O Pipeline:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `io-core` | 6.5K | Zero-copy I/O, buffer pool, direct I/O, scheduling, backpressure |
+| `io-metrics` | 4.5K | I/O operation metrics and counters |
+| `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
+| `object-io` | 2.4K | High-level object read/write using rio + ecstore |
+| `concurrency` | 1.8K | Concurrency control wrappers over io-core |
+
+**Storage Engine:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `ecstore` | 87K | ⚠️ Erasure-coded storage: disks, pools, buckets, replication, lifecycle |
+| `filemeta` | 10K | File/object metadata types and versioning |
+| `checksums` | 732 | Checksum computation |
+| `lock` | 7.1K | Distributed lock manager |
+| `heal` | 5.9K | Data healing / bitrot repair |
+| `scanner` | 5.4K | Background data usage scanner |
+| `object-capacity` | 2.5K | Capacity tracking and management |
+
+**Security & Auth:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `crypto` | 1.6K | Encryption primitives |
+| `credentials` | 713 | Credential types (access key / secret key) |
+| `signer` | 1.4K | S3 v4 request signing |
+| `iam` | 9.0K | Identity and access management |
+| `policy` | 8.8K | Policy engine (S3 bucket/IAM policies) |
+| `kms` | 8.1K | Key management service integration |
+| `keystone` | 1.9K | OpenStack Keystone auth |
+| `appauth` | 143 | Application-level auth tokens |
+
+**Protocol & API:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `protos` | 5.7K | Protobuf/gRPC definitions for inter-node RPC |
+| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
+| `s3-common` | 738 | Shared S3 types |
+| `s3select-api` | 1.9K | S3 Select interface |
+| `s3select-query` | 3.6K | S3 Select query engine |
+
+**Observability:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `metrics` | 8.4K | Prometheus metric collectors |
+| `io-metrics` | 4.5K | I/O-specific metrics |
+| `obs` | 5.6K | OpenTelemetry tracing and telemetry |
+| `audit` | 2.4K | Audit logging |
+
+**Events:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `notify` | 5.5K | Event notification system |
+| `targets` | 3.2K | Notification targets (Kafka, AMQP, webhook, etc.) |
+
+**Other:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
+| `zip` | 986 | ZIP archive support for bulk downloads |
+| `workers` | 136 | Simple worker abstraction |
+
+## Architecture Invariants
+
+> These are rules that the codebase should follow. Some are currently violated
+> (marked with ⚠️). Documenting them here makes the violations explicit and
+> trackable.
+
+1. **Layers flow downward.** Server → Admin/App → Storage → ecstore → rio/io-core.
+   No upward imports.
+
+2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
+   `io-metrics`, `madmin`, `s3-common` should depend only on external crates.
+   - ⚠️ VIOLATED: `utils` depends on `config`, `common` depends on `filemeta` and `madmin`.
+
+3. **Each type has exactly one definition.** Types shared across crates must be defined
+   in one crate and re-exported or imported by others.
+   - ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
+     `BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
+
+4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
+   storage-level abstractions (objects, buckets, disks, pools).
+
+5. **The `rustfs` binary crate is the only place that wires everything together.**
+   Individual crates should be testable in isolation.
+
+6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
+   not bare `Error`).
+   - ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
+      `heal` use `anyhow` in library code.
+
+## Known Structural Issues
+
+> This section documents known problems in the current architecture.
+> It exists so the team can track and address them deliberately.
+
+### Critical
+
+- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
+  but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
+  types instead of importing them.
+
+- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
+  bucket management, erasure coding, replication, lifecycle, RPC, and configuration
+  — all in one crate. It should be decomposed along its existing subdirectories.
+
+### High
+
+- **Dependency inversions.** `utils → config` and `common → filemeta/madmin` break
+  the layering model. These need to be untangled.
+
+- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
+  concurrency, and rustfs/storage. Should be defined once with builder/composition.
+
+### Medium
+
+- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
+  mixed naming (bare `Error` vs descriptive names).
+
+- **Ambiguous common vs utils boundary.** Both described as "utilities and data
+  structures." Need clear ownership rules.
+
+## Cross-Cutting Concerns
+
+### Error Handling
+
+The project convention is `thiserror` for typed errors with descriptive names.
+See `AGENTS.md`: "Prefer thiserror for library-facing error types."
+
+```rust
+// GOOD
+#[derive(Debug, thiserror::Error)]
+pub enum StorageError {
+    #[error("disk not found: {0}")]
+    DiskNotFound(String),
+}
+
+// AVOID
+pub enum Error { ... }        // too generic
+anyhow::Result<T>             // in library code (OK in tests/CLI)
+```
+
+### Logging & Tracing
+
+- Use `tracing` crate (`info!`, `warn!`, `error!`, `debug!`, `trace!`)
+- Structured fields: `tracing::info!(bucket = %name, "created bucket")`
+- Spans for request-scoped context
+
+### Metrics
+
+- Prometheus-style metrics via `rustfs-obs` runtime and schema
+- I/O-specific counters via `rustfs-io-metrics`
+- Registration happens at crate level, collection/reporting in `rustfs-obs`
+
+### Testing
+
+- Unit tests: `#[cfg(test)] mod tests` in the same file
+- Integration tests: inside respective crates (not top-level `tests/`)
+- E2E tests: `crates/e2e_test/` — tests against a running server
+- Run all: `make test` or `cargo nextest run`
+
+## Startup Sequence
+
+The binary (`main.rs`) boots in this order:
+
+1. Environment variable compatibility (`MINIO_*` → `RUSTFS_*`)
+2. Tokio runtime construction
+3. CLI argument parsing
+4. License, observability, TLS, trusted proxies initialization
+5. Config parsing, server address resolution
+6. Credentials, endpoints, local disks, lock client initialization
+7. Capacity management initialization
+8. HTTP server start (S3 API + optional console)
+9. ECStore initialization (erasure coding storage engine)
+10. Global config, background replication, KMS
+11. Optional: FTP/FTPS/WebDAV servers
+12. Event notifier, audit system, deadlock detector
+13. Bucket metadata, IAM, Keystone, OIDC
+14. Scanner and heal manager
+15. Metrics system, mark `FullReady`
+16. Wait for shutdown signal → graceful shutdown
+
+## Dependency Diagram (Simplified)
+
+```
+                            ┌─────────┐
+                            │  rustfs │  (binary + lib, 75K lines)
+                            │  main   │
+                            └────┬────┘
+                                 │
+                 ┌───────────────┼───────────────┐
+                 │               │               │
+            ┌────▼────┐    ┌────▼────┐    ┌─────▼─────┐
+            │ server  │    │  admin  │    │    app     │
+            │ (HTTP)  │    │(console)│    │(use-cases) │
+            └────┬────┘    └────┬────┘    └─────┬─────┘
+                 │               │               │
+                 └───────────────┼───────────────┘
+                                 │
+                          ┌──────▼──────┐
+                          │   storage   │
+                          │ (ecfs, SSE, │
+                          │  RPC, ACL)  │
+                          └──────┬──────┘
+                                 │
+              ┌──────────────────┼──────────────────┐
+              │                  │                  │
+        ┌─────▼─────┐    ┌──────▼──────┐    ┌──────▼──────┐
+        │  ecstore   │    │     rio     │    │   io-core   │
+        │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
+        └─────┬──────┘    └─────────────┘    └─────────────┘
+              │
+    ┌─────┬──┼──┬─────┬──────┐
+    │     │  │  │     │      │
+ common utils config policy filemeta ...
+```
+
+## How to Navigate
+
+- **"Where does S3 PutObject go?"**
+  `server/` routes → `app/object_usecase` validates → `storage/ecfs` encodes →
+  `ecstore` distributes → `rio` encrypts/compresses → `io-core` writes
+
+- **"Where are bucket policies enforced?"**
+  `app/bucket_usecase` calls into `crates/policy/`
+
+- **"Where is replication configured?"**
+  `admin/handlers/replication.rs` and `admin/handlers/site_replication.rs` for API,
+  `ecstore/src/bucket/replication/` for engine
+
+- **"Where do I add a new admin endpoint?"**
+  Add handler in `admin/handlers/`, register in `admin/router.rs`
+
+- **"Where do I add a new metric?"**
+  Define descriptor/collector in `crates/obs/src/metrics/`, expose via `/minio/v2/metrics`
+
+---
+
+*Inspired by [matklad's ARCHITECTURE.md](https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html)
+and [rust-analyzer's architecture.md](https://github.com/rust-analyzer/rust-analyzer/blob/master/docs/book/src/contributing/architecture.md).*

CHANGELOG.md

@@ -0,0 +1,79 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Fixed
+- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
+
+### Added
+- **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
+  - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
+  - Task-local storage for async-safe credential passing between middleware and auth handlers
+  - Automatic detection of Keystone credentials (access keys prefixed with `keystone:`)
+  - Role-based permission mapping (admin/reseller_admin roles grant owner permissions)
+  - Token caching for high-performance validation with configurable cache size and TTL
+  - Dual authentication support: Keystone and standard AWS Signature v4 work simultaneously
+  - Immediate 401 response for invalid tokens (no fallback to local auth)
+  - XML-formatted error responses compatible with S3 API
+  - Comprehensive integration documentation with manual testing guide
+  - **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection
+
+### Changed
+- **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
+- **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
+- **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
+
+### Technical Details
+- Middleware is self-contained in `rustfs-keystone` crate following the trusted-proxies pattern for integration-specific middleware
+- Uses `BoxBody` pattern for Hyper 1.x compatibility
+- Task-local storage provides request-scoped credential passing without modifying HTTP request/response types
+- Integration preserves existing S3 authentication flow while adding Keystone support
+- Zero breaking changes to existing functionality
+- No new top-level directories in main binary crate (middleware lives in integration crate)
+
+### Documentation
+- Updated `crates/keystone/README.md` with complete integration architecture and workflow
+- Added detailed manual testing guide with 10 test scenarios
+- Updated main `README.md` to list Keystone authentication as available feature
+- Added troubleshooting section for common integration issues
+
+### Configuration
+New environment variables:
+- `RUSTFS_KEYSTONE_ENABLE` - Enable/disable Keystone authentication (default: false)
+- `RUSTFS_KEYSTONE_AUTH_URL` - Keystone API endpoint URL
+- `RUSTFS_KEYSTONE_VERSION` - Keystone API version (v3)
+- `RUSTFS_KEYSTONE_ADMIN_USER` - Admin username for privileged operations
+- `RUSTFS_KEYSTONE_ADMIN_PASSWORD` - Admin password
+- `RUSTFS_KEYSTONE_ADMIN_PROJECT` - Admin project name
+- `RUSTFS_KEYSTONE_ADMIN_DOMAIN` - Admin domain name (default: Default)
+- `RUSTFS_KEYSTONE_CACHE_SIZE` - Token cache size (default: 10000)
+- `RUSTFS_KEYSTONE_CACHE_TTL` - Token cache TTL in seconds (default: 300)
+- `RUSTFS_KEYSTONE_VERIFY_SSL` - Verify SSL certificates (default: true)
+
+### Files Modified
+- `crates/keystone/src/middleware.rs` - Created Keystone authentication middleware (self-contained in keystone crate)
+- `crates/keystone/src/lib.rs` - Exported middleware module and KEYSTONE_CREDENTIALS
+- `crates/keystone/Cargo.toml` - Added Tower/HTTP dependencies for middleware functionality
+- `rustfs/src/server/http.rs` - Integrated KeystoneAuthLayer from rustfs-keystone crate
+- `rustfs/src/auth.rs` - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
+- `crates/keystone/README.md` - Comprehensive integration documentation
+- `README.md` - Added Keystone as available feature
+
+### Testing
+- 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
+- 10 integration tests in rustfs-keystone crate (task-local storage, middleware layer, scope isolation)
+- 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
+- **Total: 32 tests** passing with zero compilation errors
+- Manual testing guide provided for end-to-end validation
+- All tests passing with `cargo test --all --exclude e2e_test`
+
+---
+
+## Previous Releases
+
+See [GitHub Releases](https://github.com/rustfs/rustfs/releases) for previous version history.

CLA.md

@@ -1,88 +1,60 @@
-RustFS Individual Contributor License Agreement
+# RustFS Individual Contributor License Agreement
 
-Thank you for your interest in contributing documentation and related software code to a project hosted or managed by
+Thank you for your interest in contributing to RustFS. In order to clarify the intellectual property license granted with Contributions from any person or entity, RustFS, Inc. ("RustFS") must have a Contributor License Agreement ("CLA") on file that has been signed by each Contributor, indicating agreement to the license terms below. This license is for your protection as a Contributor as well as the protection of RustFS and its users; it does not change your rights to use your own Contributions for any other purpose.
-RustFS. In order to clarify the intellectual property license granted with Contributions from any person or entity,
-RustFS must have a Contributor License Agreement ("CLA") on file that has been signed by each Contributor, indicating
-agreement to the license terms below. This version of the Contributor License Agreement allows an individual to submit
-Contributions to the applicable project. If you are making a submission on behalf of a legal entity, then you should
-sign the separate Corporate Contributor License Agreement.
 
-You accept and agree to the following terms and conditions for Your present and future Contributions submitted to
+You accept and agree to the following terms and conditions for Your present and future Contributions submitted to RustFS. Except for the license granted herein to RustFS and recipients of software distributed by RustFS, You reserve all right, title, and interest in and to Your Contributions.
-RustFS. You hereby irrevocably assign and transfer to RustFS all right, title, and interest in and to Your
-Contributions, including all copyrights and other intellectual property rights therein.
 
-Definitions
+**You understand and agree that You will not receive any royalty or other compensation for Your Contributions.**
 
-“You” (or “Your”) shall mean the copyright owner or legal entity authorized by the copyright owner that is making this
+## 1. Definitions
-Agreement with RustFS. For legal entities, the entity making a Contribution and all other entities that control, are
-controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes
-of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such
-entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares,
-or (iii) beneficial ownership of such entity.
 
-“Contribution” shall mean any original work of authorship, including any modifications or additions to an existing work,
+*   **"You"** (or **"Your"**) shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with RustFS. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
-that is intentionally submitted by You to RustFS for inclusion in, or documentation of, any of the products or projects
-owned or managed by RustFS (the "Work"), including without limitation any Work described in Schedule A. For the purposes
-of this definition, "submitted" means any form of electronic or written communication sent to RustFS or its
-representatives, including but not limited to communication on electronic mailing lists, source code control systems,
-and issue tracking systems that are managed by, or on behalf of, RustFS for the purpose of discussing and improving the
-Work.
 
-Assignment of Copyright
+*   **"Contribution"** shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to RustFS for inclusion in, or documentation of, any of the products or projects owned or managed by RustFS (the "Work"). For the purposes of this definition, "submitted" means any form of electronic or written communication sent to RustFS or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, RustFS for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
 
-Subject to the terms and conditions of this Agreement, You hereby irrevocably assign and transfer to RustFS all right,
+## 2. Grant of Copyright License
-title, and interest in and to Your Contributions, including all copyrights and other intellectual property rights
-therein, for the entire term of such rights, including all renewals and extensions. You agree to execute all documents
-and take all actions as may be reasonably necessary to vest in RustFS the ownership of Your Contributions and to assist
-RustFS in perfecting, maintaining, and enforcing its rights in Your Contributions.
 
-Grant of Patent License
+Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works under any license, including proprietary or commercial licenses and open-source licenses, at RustFS's sole discretion.
 
-Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of documentation and
+## 3. Grant of Patent License
-software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as
-stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the
-Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your
-Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was
-submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or
-counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes
-direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for
-that Contribution or Work shall terminate as of the date such litigation is filed.
 
-You represent that you are legally entitled to grant the above assignment and license.
+Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
 
-You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of
+## 4. Representations
-others). You represent that Your Contribution submissions include complete details of any third-party license or other
-restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which
-are associated with any part of Your Contributions.
 
-You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You
+You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to RustFS, or that your employer has executed a separate Corporate CLA with RustFS.
-may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You
-provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
-including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR
-A PARTICULAR PURPOSE.
 
-Should You wish to submit work that is not Your original creation, You may submit it to RustFS separately from any
+You represent that each of Your Contributions is Your original creation. You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.
-Contribution, identifying the complete details of its source and of any license or other restriction (including, but not
-limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously
-marking the work as "Submitted on behalf of a third-party: [named here]”.
 
-You agree to notify RustFS of any facts or circumstances of which you become aware that would make these representations
+## 5. Support and Warranty
-inaccurate in any respect.
 
-Modification of CLA
+You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of **TITLE**, **NON-INFRINGEMENT**, **MERCHANTABILITY**, or **FITNESS FOR A PARTICULAR PURPOSE**.
 
-RustFS reserves the right to update or modify this CLA in the future. Any updates or modifications to this CLA shall
+## 6. Third-Party Work
-apply only to Contributions made after the effective date of the revised CLA. Contributions made prior to the update
-shall remain governed by the version of the CLA that was in effect at the time of submission. It is not necessary for
-all Contributors to re-sign the CLA when the CLA is updated or modified.
 
-Governing Law and Dispute Resolution
+Should You wish to submit work that is not Your original creation, You may submit it to RustFS separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
 
-This Agreement will be governed by and construed in accordance with the laws of the People's Republic of China excluding
+## 7. Governing Law and Jurisdiction
-that body of laws known as conflict of laws. The parties expressly agree that the United Nations Convention on Contracts
-for the International Sale of Goods will not apply. Any legal action or proceeding arising under this Agreement will be
-brought exclusively in the courts located in Beijing, China, and the parties hereby irrevocably consent to the personal
-jurisdiction and venue therein.
 
-For your reading convenience, this Agreement is written in parallel English and Chinese sections. To the extent there is
+This Agreement shall be governed by and construed in accordance with the laws of the State of Delaware, United States of America, without regard to its conflict of laws principles. The parties expressly agree that the United Nations Convention on Contracts for the International Sale of Goods will not apply. Any legal action or proceeding arising under or in connection with this Agreement shall be brought exclusively in the state or federal courts located in the State of Delaware, United States of America, and the parties hereby irrevocably consent to the personal jurisdiction and venue therein.
-a conflict between the English and Chinese sections, the English sections shall govern.
+
+## 8. Severability
+
+If any provision of this Agreement is found to be invalid or unenforceable, the remaining provisions will continue in full force and effect.
+
+## 9. Entire Agreement and Assignment
+
+This Agreement constitutes the entire agreement between the parties with respect to the subject matter hereof and supersedes all prior agreements, understandings, negotiations, and discussions, whether written or oral. RustFS may assign its rights and obligations under this Agreement to any third party. You may not assign Your rights or obligations under this Agreement without the prior written consent of RustFS.
+
+---
+
+**Please read the terms of this Agreement carefully. By submitting a Contribution to RustFS, You agree to be bound by the terms and conditions of this Agreement.**
+
+| | |
+|---|---|
+| **Full Name** | __________________________________ |
+| **GitHub Username** | __________________________________ |
+| **Email Address** | __________________________________ |
+| **Date** | __________________________________ |
+
+*(Electronic signature or acknowledgement via GitHub commit/Pull Request constitutes valid acceptance of this Agreement).*

CLAUDE.md

@@ -1,275 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Project Overview
-
-RustFS is a high-performance distributed object storage software built with Rust, providing S3-compatible APIs and
-advanced features like data lakes, AI, and big data support. It's designed as an alternative to MinIO with better
-performance and a more business-friendly Apache 2.0 license.
-
-## Build Commands
-
-### Primary Build Commands
-
-- `cargo build --release` - Build the main RustFS binary
-- `./build-rustfs.sh` - Recommended build script that handles console resources and cross-platform compilation
-- `./build-rustfs.sh --dev` - Development build with debug symbols
-- `make build` or `just build` - Use Make/Just for standardized builds
-
-### Platform-Specific Builds
-
-- `./build-rustfs.sh --platform x86_64-unknown-linux-musl` - Build for musl target
-- `./build-rustfs.sh --platform aarch64-unknown-linux-gnu` - Build for ARM64
-- `make build-musl` or `just build-musl` - Build musl variant
-- `make build-cross-all` - Build all supported architectures
-
-### Testing Commands
-
-- `cargo test --workspace --exclude e2e_test` - Run unit tests (excluding e2e tests)
-- `cargo nextest run --all --exclude e2e_test` - Use nextest if available (faster)
-- `cargo test --all --doc` - Run documentation tests
-- `make test` or `just test` - Run full test suite
-- `make pre-commit` - Run all quality checks (fmt, clippy, check, test)
-
-### End-to-End Testing
-
-- `cargo test --package e2e_test` - Run all e2e tests
-- `./scripts/run_e2e_tests.sh` - Run e2e tests via script
-- `./scripts/run_scanner_benchmarks.sh` - Run scanner performance benchmarks
-
-### KMS-Specific Testing (with proxy bypass)
-
--
-`NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1` -
-Run complete KMS end-to-end test
--
-`NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test kms:: -- --nocapture --test-threads=1` -
-Run all KMS tests
-- `cargo test --package e2e_test test_local_kms_key_isolation -- --nocapture --test-threads=1` - Test KMS key isolation
-- `cargo test --package e2e_test test_local_kms_large_file -- --nocapture --test-threads=1` - Test KMS with large files
-
-### Code Quality
-
-- `cargo fmt --all` - Format code
-- `cargo clippy --all-targets --all-features -- -D warnings` - Lint code
-- `make pre-commit` or `just pre-commit` - Run all quality checks (fmt, clippy, check, test)
-
-### Quick Development Commands
-
-- `make help` or `just help` - Show all available commands with descriptions
-- `make help-build` - Show detailed build options and cross-compilation help
-- `make help-docker` - Show comprehensive Docker build and deployment options
-- `./scripts/dev_deploy.sh <IP>` - Deploy development build to remote server
-- `./scripts/run.sh` - Start local development server
-- `./scripts/probe.sh` - Health check and connectivity testing
-
-### Docker Build Commands
-
-- `make docker-buildx` - Build multi-architecture production images
-- `make docker-dev-local` - Build development image for local use
-- `./docker-buildx.sh --push` - Build and push production images
-
-## Architecture Overview
-
-### Core Components
-
-**Main Binary (`rustfs/`):**
-
-- Entry point at `rustfs/src/main.rs`
-- Core modules: admin, auth, config, server, storage, license management, profiling
-- HTTP server with S3-compatible APIs
-- Service state management and graceful shutdown
-- Parallel service initialization with DNS resolver, bucket metadata, and IAM
-
-**Key Crates (`crates/`):**
-
-- `ecstore` - Erasure coding storage implementation (core storage layer)
-- `iam` - Identity and Access Management
-- `kms` - Key Management Service for encryption and key handling
-- `madmin` - Management dashboard and admin API interface
-- `s3select-api` & `s3select-query` - S3 Select API and query engine
-- `config` - Configuration management with notify features
-- `crypto` - Cryptography and security features
-- `lock` - Distributed locking implementation
-- `filemeta` - File metadata management
-- `rio` - Rust I/O utilities and abstractions
-- `common` - Shared utilities and data structures
-- `protos` - Protocol buffer definitions
-- `audit-logger` - Audit logging for file operations
-- `notify` - Event notification system
-- `obs` - Observability utilities
-- `workers` - Worker thread pools and task scheduling
-- `appauth` - Application authentication and authorization
-- `ahm` - Asynchronous Hash Map for concurrent data structures
-- `mcp` - MCP server for S3 operations
-- `signer` - Client request signing utilities
-- `checksums` - Client checksum calculation utilities
-- `utils` - General utility functions and helpers
-- `zip` - ZIP file handling and compression
-- `targets` - Target-specific configurations and utilities
-
-### Build System
-
-- Cargo workspace with 25+ crates (including new KMS functionality)
-- Custom `build-rustfs.sh` script for advanced build options
-- Multi-architecture Docker builds via `docker-buildx.sh`
-- Both Make and Just task runners supported with comprehensive help
-- Cross-compilation support for multiple Linux targets
-- Automated CI/CD with GitHub Actions for testing, building, and Docker publishing
-- Performance benchmarking and audit workflows
-
-### Key Dependencies
-
-- `axum` - HTTP framework for S3 API server
-- `tokio` - Async runtime
-- `s3s` - S3 protocol implementation library
-- `datafusion` - For S3 Select query processing
-- `hyper`/`hyper-util` - HTTP client/server utilities
-- `rustls` - TLS implementation
-- `serde`/`serde_json` - Serialization
-- `tracing` - Structured logging and observability
-- `pprof` - Performance profiling with flamegraph support
-- `tikv-jemallocator` - Memory allocator for Linux GNU builds
-
-### Development Workflow
-
-- Console resources are embedded during build via `rust-embed`
-- Protocol buffers generated via custom `gproto` binary
-- E2E tests in separate crate (`e2e_test`) with comprehensive KMS testing
-- Shadow build for version/metadata embedding
-- Support for both GNU and musl libc targets
-- Development scripts in `scripts/` directory for common tasks
-- Git hooks setup available via `make setup-hooks` or `just setup-hooks`
-
-### Performance & Observability
-
-- Performance profiling available with `pprof` integration (disabled on Windows)
-- Profiling enabled via environment variables in production
-- Built-in observability with OpenTelemetry integration
-- Background services (scanner, heal) can be controlled via environment variables:
-    - `RUSTFS_ENABLE_SCANNER` (default: true)
-    - `RUSTFS_ENABLE_HEAL` (default: true)
-
-### Service Architecture
-
-- Service state management with graceful shutdown handling
-- Parallel initialization of core systems (DNS, bucket metadata, IAM)
-- Event notification system with MQTT and webhook support
-- Auto-heal and data scanner for storage integrity
-- Jemalloc allocator for Linux GNU targets for better performance
-
-## Environment Variables
-
-- `RUSTFS_ENABLE_SCANNER` - Enable/disable background data scanner (default: true)
-- `RUSTFS_ENABLE_HEAL` - Enable/disable auto-heal functionality (default: true)
-- Various profiling and observability controls
-- Build-time variables for Docker builds (RELEASE, REGISTRY, etc.)
-- Test environment configurations in `scripts/dev_rustfs.env`
-
-### KMS Environment Variables
-
-- `NO_PROXY=127.0.0.1,localhost` - Required for KMS E2E tests to bypass proxy
-- `HTTP_PROXY=` `HTTPS_PROXY=` `http_proxy=` `https_proxy=` - Clear proxy settings for local KMS testing
-
-## KMS (Key Management Service) Architecture
-
-### KMS Implementation Status
-
-- **Full KMS Integration:** Complete implementation with Local and Vault backends
-- **Automatic Configuration:** KMS auto-configures on startup with `--kms-enable` flag
-- **Encryption Support:** Full S3-compatible server-side encryption (SSE-S3, SSE-KMS, SSE-C)
-- **Admin API:** Complete KMS management via HTTP admin endpoints
-- **Production Ready:** Comprehensive testing including large files and key isolation
-
-### KMS Configuration
-
-- **Local Backend:** `--kms-backend local --kms-key-dir <path> --kms-default-key-id <id>`
-- **Vault Backend:** `--kms-backend vault --kms-vault-endpoint <url> --kms-vault-key-name <name>`
-- **Auto-startup:** KMS automatically initializes when `--kms-enable` is provided
-- **Manual Configuration:** Also supports dynamic configuration via admin API
-
-### S3 Encryption Support
-
-- **SSE-S3:** Server-side encryption with S3-managed keys (`ServerSideEncryption: AES256`)
-- **SSE-KMS:** Server-side encryption with KMS-managed keys (`ServerSideEncryption: aws:kms`)
-- **SSE-C:** Server-side encryption with customer-provided keys
-- **Response Headers:** All encryption types return correct `server_side_encryption` headers in PUT/GET responses
-
-### KMS Testing Architecture
-
-- **Comprehensive E2E Tests:** Located in `crates/e2e_test/src/kms/`
-- **Test Environments:** Automated test environment setup with temporary directories
-- **Encryption Coverage:** Tests all three encryption types (SSE-S3, SSE-KMS, SSE-C)
-- **API Coverage:** Tests all KMS admin APIs (CreateKey, DescribeKey, ListKeys, etc.)
-- **Edge Cases:** Key isolation, large file handling, error scenarios
-
-### Key Files for KMS
-
-- `crates/kms/` - Core KMS implementation with Local/Vault backends
-- `rustfs/src/main.rs` - KMS auto-initialization in `init_kms_system()`
-- `rustfs/src/storage/ecfs.rs` - SSE encryption/decryption in PUT/GET operations
-- `rustfs/src/admin/handlers/kms*.rs` - KMS admin endpoints
-- `crates/e2e_test/src/kms/` - Comprehensive KMS test suite
-- `crates/rio/src/encrypt_reader.rs` - Streaming encryption for large files
-
-## Code Style and Safety Requirements
-
-- **Language Requirements:**
-    - Communicate with me in Chinese, but **only English can be used in code files**
-    - Code comments, function names, variable names, and all text in source files must be in English only
-    - No Chinese characters, emojis, or non-ASCII characters are allowed in any source code files
-    - This includes comments, strings, documentation, and any other text within code files
-- **Safety-Critical Rules:**
-    - `unsafe_code = "deny"` enforced at workspace level
-    - Never use `unwrap()`, `expect()`, or panic-inducing code except in tests
-    - Avoid blocking I/O operations in async contexts
-    - Use proper error handling with `Result<T, E>` and `Option<T>`
-    - Follow Rust's ownership and borrowing rules strictly
-- **Performance Guidelines:**
-    - Use `cargo clippy --all-targets --all-features -- -D warnings` to catch issues
-    - Prefer `anyhow` for error handling in applications, `thiserror` for libraries
-    - Use appropriate async runtimes and avoid blocking calls
-- **Testing Standards:**
-    - All new features must include comprehensive tests
-    - Use `#[cfg(test)]` for test-only code that may use panic macros
-    - E2E tests should cover KMS integration scenarios
-
-## Common Development Tasks
-
-### Running KMS Tests Locally
-
-1. **Clear proxy settings:** KMS tests require direct localhost connections
-2. **Use serial execution:** `--test-threads=1` prevents port conflicts
-3. **Enable output:** `--nocapture` shows detailed test logs
-4. **Full command:**
-   `NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1`
-
-### KMS Development Workflow
-
-1. **Code changes:** Modify KMS-related code in `crates/kms/` or `rustfs/src/`
-2. **Compile:** Always run `cargo build` after changes
-3. **Test specific functionality:** Use targeted test commands for faster iteration
-4. **Full validation:** Run complete end-to-end tests before commits
-
-### Debugging KMS Issues
-
-- **Server startup:** Check that KMS auto-initializes with debug logs
-- **Encryption failures:** Verify SSE headers are correctly set in both PUT and GET responses
-- **Test failures:** Use `--nocapture` to see detailed error messages
-- **Key management:** Test admin API endpoints with proper authentication
-
-## Important Reminders
-
-- **Always compile after code changes:** Use `cargo build` to catch errors early
-- **Don't bypass tests:** All functionality must be properly tested, not worked around
-- **Use proper error handling:** Never use `unwrap()` or `expect()` in production code (except tests)
-- **Follow S3 compatibility:** Ensure all encryption types return correct HTTP response headers
-
-# important-instruction-reminders
-
-Do what has been asked; nothing more, nothing less.
-NEVER create files unless they're absolutely necessary for achieving your goal.
-ALWAYS prefer editing an existing file to creating a new one.
-NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly
-requested by the User.

CONTRIBUTING.md

@@ -2,6 +2,8 @@
 
 ## 📋 Code Quality Requirements
 
+For instructions on setting up and running the local development environment, please see [Development Guide](docs/DEVELOPMENT.md).
+
 ### 🔧 Code Formatting Rules
 
 **MANDATORY**: All code must be properly formatted before committing. This project enforces strict formatting standards to maintain code consistency and readability.
@@ -184,6 +186,39 @@ cargo clippy --all-targets --all-features -- -D warnings
 cargo clippy --fix --all-targets --all-features
 ```
 
+## 📝 Pull Request Guidelines
+
+### Language Requirements
+
+**All Pull Request titles and descriptions MUST be written in English.**
+
+This ensures:
+- Consistency across all contributions
+- Accessibility for international contributors
+- Better integration with automated tools and CI/CD systems
+- Clear communication in a globally understood language
+
+#### PR Description Requirements
+
+When creating a Pull Request, ensure:
+
+1. **Title**: Use English and follow Conventional Commits format (e.g., `fix: improve s3-tests readiness detection`)
+2. **Description**: Write in English, following the PR template format
+3. **Code Comments**: Must be in English (as per coding standards)
+4. **Commit Messages**: Must be in English (as per commit guidelines)
+
+#### PR Template
+
+Always use the PR template (`.github/pull_request_template.md`) and fill in all sections:
+- Type of Change
+- Related Issues
+- Summary of Changes
+- Checklist
+- Impact
+- Additional Notes
+
+**Note**: While you may communicate with reviewers in Chinese during discussions, the PR itself (title, description, and all formal documentation) must be in English.
+
 ---
 
 Following these guidelines ensures high code quality and smooth collaboration across the RustFS project! 🚀

Cargo.toml

@@ -17,76 +17,97 @@ members = [
     "rustfs", # Core file system implementation
     "crates/appauth", # Application authentication and authorization
     "crates/audit", # Audit target management system with multi-target fan-out
+    "crates/checksums", # client checksums
     "crates/common", # Shared utilities and data structures
     "crates/config", # Configuration management
+    "crates/credentials", # Credential management system
     "crates/crypto", # Cryptography and security features
     "crates/ecstore", # Erasure coding storage implementation
     "crates/e2e_test", # End-to-end test suite
     "crates/filemeta", # File metadata management
+    "crates/heal", # Erasure set and object healing
     "crates/iam", # Identity and Access Management
+    "crates/keystone", # OpenStack Keystone integration
+    "crates/kms", # Key Management Service
     "crates/lock", # Distributed locking implementation
     "crates/madmin", # Management dashboard and admin API interface
     "crates/notify", # Notification system for events
     "crates/obs", # Observability utilities
+    "crates/object-capacity", # Capacity scan and refresh core
     "crates/policy", # Policy management
+    "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
-    "crates/targets", # Target-specific configurations and utilities
+    "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
+    "crates/s3-common", # Common utilities and data structures for S3 compatibility
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
+    "crates/scanner", # Scanner for data integrity checks and health monitoring
     "crates/signer", # client signer
-    "crates/checksums", # client checksums
+    "crates/targets", # Target-specific configurations and utilities
+    "crates/trusted-proxies", # Trusted proxies management
     "crates/utils", # Utility functions and helpers
     "crates/workers", # Worker thread pools and task scheduling
+    "crates/io-metrics", # Zero-copy metrics collection for performance analysis
+    "crates/io-core", # Zero-copy core reader and writer implementations
     "crates/zip", # ZIP file handling and compression
-    "crates/ahm", # Asynchronous Hash Map for concurrent data structures
-    "crates/mcp", # MCP server for S3 operations
-    "crates/kms", # Key Management Service
 ]
-resolver = "2"
+resolver = "3"
 
 [workspace.package]
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.85"
+rust-version = "1.93.0"
 version = "0.0.5"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
 categories = ["web-programming", "development-tools", "filesystem", "network-programming"]
+authors = ["RustFS Team"]
 
 [workspace.lints.rust]
 unsafe_code = "deny"
 
 [workspace.lints.clippy]
 all = "warn"
+needless_collect = "warn"
+redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
 rustfs = { path = "./rustfs", version = "0.0.5" }
-rustfs-ahm = { path = "crates/ahm", version = "0.0.5" }
+rustfs-heal = { path = "crates/heal", version = "0.0.5" }
 rustfs-appauth = { path = "crates/appauth", version = "0.0.5" }
 rustfs-audit = { path = "crates/audit", version = "0.0.5" }
 rustfs-checksums = { path = "crates/checksums", version = "0.0.5" }
 rustfs-common = { path = "crates/common", version = "0.0.5" }
 rustfs-config = { path = "./crates/config", version = "0.0.5" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "0.0.5" }
+rustfs-credentials = { path = "crates/credentials", version = "0.0.5" }
 rustfs-crypto = { path = "crates/crypto", version = "0.0.5" }
 rustfs-ecstore = { path = "crates/ecstore", version = "0.0.5" }
 rustfs-filemeta = { path = "crates/filemeta", version = "0.0.5" }
 rustfs-iam = { path = "crates/iam", version = "0.0.5" }
+rustfs-keystone = { path = "crates/keystone", version = "0.0.5" }
 rustfs-kms = { path = "crates/kms", version = "0.0.5" }
 rustfs-lock = { path = "crates/lock", version = "0.0.5" }
 rustfs-madmin = { path = "crates/madmin", version = "0.0.5" }
-rustfs-mcp = { path = "crates/mcp", version = "0.0.5" }
 rustfs-notify = { path = "crates/notify", version = "0.0.5" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "0.0.5" }
+rustfs-io-core = { path = "crates/io-core", version = "0.0.5" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "0.0.5" }
 rustfs-obs = { path = "crates/obs", version = "0.0.5" }
 rustfs-policy = { path = "crates/policy", version = "0.0.5" }
 rustfs-protos = { path = "crates/protos", version = "0.0.5" }
+rustfs-protocols = { path = "crates/protocols", version = "0.0.5" }
 rustfs-rio = { path = "crates/rio", version = "0.0.5" }
+rustfs-s3-common = { path = "crates/s3-common", version = "0.0.5" }
 rustfs-s3select-api = { path = "crates/s3select-api", version = "0.0.5" }
 rustfs-s3select-query = { path = "crates/s3select-query", version = "0.0.5" }
+rustfs-scanner = { path = "crates/scanner", version = "0.0.5" }
 rustfs-signer = { path = "crates/signer", version = "0.0.5" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "0.0.5" }
 rustfs-targets = { path = "crates/targets", version = "0.0.5" }
 rustfs-utils = { path = "crates/utils", version = "0.0.5" }
 rustfs-workers = { path = "crates/workers", version = "0.0.5" }
@@ -94,193 +115,208 @@ rustfs-zip = { path = "./crates/zip", version = "0.0.5" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-async-compression = { version = "0.4.19" }
+async-compression = { version = "0.4.41" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
-axum = "0.8.6"
+async-nats = "0.47.0"
-axum-extra = "0.12.1"
+axum = "0.8.9"
-axum-server = { version = "0.7.2", features = ["tls-rustls-no-provider"], default-features = false }
+futures = "0.3.32"
-futures = "0.3.31"
+futures-core = "0.3.32"
-futures-core = "0.3.31"
+futures-util = "0.3.32"
-futures-util = "0.3.31"
+pollster = "0.4.0"
-hyper = { version = "1.7.0", features = ["http2", "http1", "server"] }
+pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
-hyper-rustls = { version = "0.27.7", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "ring", "webpki-roots"] }
+hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
-hyper-util = { version = "0.1.17", features = ["tokio", "server-auto", "server-graceful"] }
+hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
-http = "1.3.1"
+hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
+http = "1.4.0"
 http-body = "1.0.1"
-reqwest = { version = "0.12.24", default-features = false, features = ["rustls-tls-webpki-roots", "charset", "http2", "system-proxy", "stream", "json", "blocking"] }
+http-body-util = "0.1.3"
-socket2 = "0.6.1"
+reqwest = { version = "0.13.2", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
-tokio = { version = "1.48.0", features = ["fs", "rt-multi-thread"] }
+socket2 = { version = "0.6.3", features = ["all"] }
-tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "ring"] }
+tokio = { version = "1.52.1", features = ["fs", "rt-multi-thread"] }
-tokio-stream = { version = "0.1.17" }
+tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
-tokio-test = "0.4.4"
+tokio-stream = { version = "0.1.18" }
-tokio-util = { version = "0.7.17", features = ["io", "compat"] }
+tokio-test = "0.4.5"
-tonic = { version = "0.14.2", features = ["gzip"] }
+tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic-prost = { version = "0.14.2" }
+tonic = { version = "0.14.5", features = ["gzip"] }
-tonic-prost-build = { version = "0.14.2" }
+tonic-prost = { version = "0.14.5" }
-tower = { version = "0.5.2", features = ["timeout"] }
+tonic-prost-build = { version = "0.14.5" }
-tower-http = { version = "0.6.6", features = ["cors"] }
+tower = { version = "0.5.3", features = ["timeout"] }
+tower-http = { version = "0.6.8", features = ["cors"] }
 
 # Serialization and Data Formats
-bytes = { version = "1.10.1", features = ["serde"] }
+bytes = { version = "1.11.1", features = ["serde"] }
-bytesize = "2.2.0"
+bytesize = "2.3.1"
 byteorder = "1.5.0"
-flatbuffers = "25.9.23"
+flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
-prost = "0.14.1"
+prost = "0.14.3"
-quick-xml = "0.38.3"
+quick-xml = "0.39.2"
-rmcp = { version = "0.8.5" }
+rmp = { version = "0.8.15" }
-rmp = { version = "0.8.14" }
+rmp-serde = { version = "1.3.1" }
-rmp-serde = { version = "1.3.0" }
 serde = { version = "1.0.228", features = ["derive"] }
-serde_json = { version = "1.0.145", features = ["raw_value"] }
+serde_json = { version = "1.0.149", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
-schemars = "1.1.0"
 
 # Cryptography and Security
-aes-gcm = { version = "0.11.0-rc.2", features = ["rand_core"] }
+aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
-argon2 = { version = "0.6.0-rc.2", features = ["std"] }
+argon2 = { version = "0.6.0-rc.8" }
-blake3 = { version = "1.8.2" }
+blake2 = "0.11.0-rc.6"
-chacha20poly1305 = { version = "0.11.0-rc.2" }
+chacha20poly1305 = { version = "0.11.0-rc.3" }
-crc-fast = "1.3.0"
+crc-fast = "1.9.0"
-crc32c = "0.6.8"
+hmac = { version = "0.13.0" }
-crc32fast = "1.5.0"
+jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
-crc64fast-nvme = "1.2.0"
+openidconnect = { version = "4.0", default-features = false }
-hmac = { version = "0.13.0-rc.3" }
+pbkdf2 = "0.13.0-rc.10"
-jsonwebtoken = { version = "10.2.0", features = ["rust_crypto"] }
+rsa = { version = "0.10.0-rc.17" }
-pbkdf2 = "0.13.0-rc.2"
+rustls = { version = "0.23.38", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
-rsa = { version = "0.10.0-rc.10" }
+rustls-pki-types = "1.14.0"
-rustls = { version = "0.23.35", features = ["ring", "logging", "std", "tls12"], default-features = false }
+sha1 = "0.11.0"
-rustls-pemfile = "2.2.0"
+sha2 = "0.11.0"
-rustls-pki-types = "1.13.0"
+subtle = "2.6"
-sha1 = "0.11.0-rc.3"
-sha2 = "0.11.0-rc.3"
 zeroize = { version = "1.8.2", features = ["derive"] }
 
 # Time and Date
-chrono = { version = "0.4.42", features = ["serde"] }
+chrono = { version = "0.4.44", features = ["serde"] }
 humantime = "2.3.0"
-time = { version = "0.3.44", features = ["std", "parsing", "formatting", "macros", "serde"] }
+jiff = { version = "0.2.23", features = ["serde"] }
+time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Utilities and Tools
-anyhow = "1.0.100"
+anyhow = "1.0.102"
-arc-swap = "1.7.1"
+arc-swap = "1.9.1"
-astral-tokio-tar = "0.5.6"
+astral-tokio-tar = "0.6.0"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.10" }
+aws-config = { version = "1.8.15" }
-aws-credential-types = { version = "1.2.9" }
+aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.112.0", default-features = false, features = ["sigv4a", "rustls", "rt-tokio"] }
+aws-sdk-s3 = { version = "1.129.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-types = { version = "1.3.4" }
+aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-types = { version = "1.4.7" }
+backtrace = "0.3.76"
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
 cfg-if = "1.0.4"
-clap = { version = "4.5.51", features = ["derive", "env"] }
+clap = { version = "4.6.1", features = ["derive", "env"] }
-const-str = { version = "0.7.0", features = ["std", "proc"] }
+const-str = { version = "1.1.0", features = ["std", "proc"] }
-convert_case = "0.9.0"
+convert_case = "0.11.0"
-criterion = { version = "0.7", features = ["html_reports"] }
+criterion = { version = "0.8", features = ["html_reports"] }
 crossbeam-queue = "0.3.12"
-datafusion = "50.3.0"
+crossbeam-channel = "0.5.15"
+crossbeam-deque = "0.8.6"
+crossbeam-utils = "0.8.21"
+datafusion = "53.1.0"
 derive_builder = "0.20.2"
 enumset = "1.1.10"
 faster-hex = "0.10.0"
-flate2 = "1.1.5"
+flate2 = "1.1.9"
-flexi_logger = { version = "0.31.7", features = ["trc", "dont_minimize_extra_stacks", "compress", "kv", "json"] }
 glob = "0.3.3"
-google-cloud-storage = "1.2.0"
+google-cloud-storage = "1.11.0"
-google-cloud-auth = "1.1.0"
+google-cloud-auth = "1.9.0"
-hashbrown = { version = "0.16.0", features = ["serde", "rayon"] }
+hashbrown = { version = "0.17.0", features = ["serde", "rayon"] }
-heed = { version = "0.22.0" }
+hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
-libc = "0.2.177"
+libc = "0.2.185"
 libsystemd = "0.7.2"
-local-ip-address = "0.6.5"
+local-ip-address = "0.6.11"
+memmap2 = "0.9.10"
 lz4 = "1.28.1"
-matchit = "0.9.0"
+matchit = "0.9.2"
-md-5 = "0.11.0-rc.3"
+md-5 = "0.11.0"
 md5 = "0.8.0"
-metrics = "0.24.2"
-metrics-exporter-opentelemetry = "0.1.2"
 mime_guess = "2.0.5"
-moka = { version = "0.12.11", features = ["future"] }
+moka = { version = "0.12.15", features = ["future"] }
 netif = "0.1.6"
-nix = { version = "0.30.1", features = ["fs"] }
-nu-ansi-term = "0.50.3"
 num_cpus = { version = "1.17.0" }
-nvml-wrapper = "0.11.0"
+nvml-wrapper = "0.12.1"
-object_store = "0.12.4"
+object_store = "0.13.2"
-once_cell = "1.21.3"
 parking_lot = "0.12.5"
 path-absolutize = "3.1.1"
 path-clean = "1.0.1"
-pin-project-lite = "0.2.16"
+percent-encoding = "2.3.2"
+pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
-rand = { version = "0.10.0-rc.5", features = ["serde"] }
+rand = { version = "0.10.1", features = ["serde"] }
-rayon = "1.11.0"
+ratelimit = "0.10.1"
-reed-solomon-simd = { version = "3.1.0" }
+rayon = "1.12.0"
-regex = { version = "1.12.2" }
+reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
-rumqttc = { version = "0.25.0" }
+reed-solomon-simd = "3.1.0"
-rust-embed = { version = "8.9.0" }
+regex = { version = "1.12.3" }
-rustc-hash = { version = "2.1.1" }
+rumqttc = { package = "rumqttc-next", version = "0.30.0", features = ["websocket"] }
-s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "1ab064b", version = "0.12.0-rc.3", features = ["minio"] }
+rustix = { version = "1.1.4", features = ["fs"] }
-serial_test = "3.2.0"
+rust-embed = { version = "8.11.0" }
-shadow-rs = { version = "1.4.0", default-features = false }
+rustc-hash = { version = "2.1.2" }
-siphasher = "1.0.1"
+s3s = { git = "https://github.com/rustfs/s3s", rev = "a3b16608df35aaeed8fff08b4988d03f4ca9445b", features = ["minio"] }
+serial_test = "3.4.0"
+shadow-rs = { version = "1.7.1", default-features = false }
+siphasher = "1.0.2"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
-snafu = "0.8.9"
+snafu = "0.9.0"
 snap = "1.1.1"
-starshard = { version = "0.5.0", features = ["rayon", "async", "serde"] }
+starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
-strum = { version = "0.27.2", features = ["derive"] }
+strum = { version = "0.28.0", features = ["derive"] }
-sysctl = "0.7.1"
+sysinfo = "0.38.4"
-sysinfo = "0.37.2"
 temp-env = "0.3.6"
-tempfile = "3.23.0"
+tempfile = "3.27.0"
 test-case = "3.3.1"
-thiserror = "2.0.17"
+thiserror = "2.0.18"
-tracing = { version = "0.1.41" }
+tracing = { version = "0.1.44" }
-tracing-appender = "0.2.3"
+tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
-tracing-opentelemetry = "0.32.0"
+tracing-opentelemetry = "0.32.1"
-tracing-subscriber = { version = "0.3.20", features = ["env-filter", "time"] }
+tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
-url = "2.5.7"
+url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.18.1", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.23.1", features = ["v4", "fast-rng", "macro-diagnostics"] }
-vaultrs = { version = "0.7.4" }
+vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
-wildmatch = { version = "2.6.0", features = ["serde"] }
+wildmatch = { version = "2.6.1", features = ["serde"] }
-winapi = { version = "0.3.9" }
+windows = { version = "0.62.2" }
 xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
-zip = "6.0.0"
+zip = "8.5.1"
 zstd = "0.13.3"
 
 # Observability and Metrics
+metrics = "0.24.3"
+dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.31.0" }
 opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
-opentelemetry-otlp = { version = "0.31.0", features = ["http-proto", "zstd-http"] }
+opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rustls"] }
 opentelemetry_sdk = { version = "0.31.0" }
 opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.31.0" }
+pyroscope = { version = "2.0.0", features = ["backend-pprof-rs"] }
+
+# FTP and SFTP
+libunftp = { version = "0.23.0", features = ["experimental"] }
+unftp-core = "0.1.0"
+suppaftp = { version = "8.0.2", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+rcgen = "0.14.7"
+
+# WebDAV
+dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
+mimalloc = "0.1"
 # Use tikv-jemallocator as memory allocator and enable performance analysis
 tikv-jemallocator = { version = "0.6", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms", "background_threads"] }
 # Used to control and obtain statistics for jemalloc at runtime
 tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profiling"] }
 # Used to generate pprof-compatible memory profiling data and support symbolization and flame graphs
-jemalloc_pprof = { version = "0.8.1", features = ["symbolize", "flamegraph"] }
+jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
-pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
+# pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
-mimalloc = "0.1"
+# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
-
+pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]
-ignored = ["rustfs", "rustfs-mcp", "tokio-test"]
+ignored = ["rustfs"]
 
 [profile.release]
 opt-level = 3

Dockerfile

@@ -1,4 +1,18 @@
-FROM alpine:3.22 AS build
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM alpine:3.23.4 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -40,7 +54,7 @@ RUN set -eux; \
     rm -rf rustfs.zip /build/.tmp || true
 
 
-FROM alpine:3.22
+FROM alpine:3.23.4
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -58,7 +72,8 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-RUN apk add --no-cache ca-certificates coreutils curl
+RUN apk update && \
+    apk add --no-cache ca-certificates coreutils curl
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs
@@ -66,28 +81,22 @@ COPY entrypoint.sh /entrypoint.sh
 
 RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
-RUN addgroup -g 1000 -S rustfs && \
+RUN addgroup -g 10001 -S rustfs && \
-    adduser -u 1000 -G rustfs -S rustfs -D && \
+    adduser -u 10001 -G rustfs -S rustfs -D && \
     mkdir -p /data /logs && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 
-ENV RUSTFS_ADDRESS=":9000" \
+ENV RUSTFS_CORS_ALLOWED_ORIGINS="*" \
-    RUSTFS_CONSOLE_ADDRESS=":9001" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_EXTERNAL_ADDRESS="" \
-    RUSTFS_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
-    RUST_LOG="warn" \
+    RUSTFS_OBS_LOGGER_LEVEL=warn \
-    RUSTFS_OBS_LOG_DIRECTORY="/logs" \
+    RUSTFS_OBS_LOG_DIRECTORY=/logs \
-    RUSTFS_SINKS_FILE_PATH="/logs"
+    RUSTFS_OBS_ENVIRONMENT=production
-
+    
 EXPOSE 9000 9001
 
-VOLUME ["/data", "/logs"]
+VOLUME ["/data"]
 
 USER rustfs
 

Dockerfile.decommission-local

@@ -0,0 +1,65 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM rust:1.91-trixie
+
+RUN set -eux; \
+    export DEBIAN_FRONTEND=noninteractive; \
+    apt-get update; \
+    apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    curl \
+    git \
+    pkg-config \
+    libssl-dev \
+    lld \
+    protobuf-compiler \
+    flatbuffers-compiler; \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /usr/src/rustfs
+
+COPY . .
+
+RUN ./scripts/static.sh
+RUN cargo run --bin gproto
+RUN cargo build --release --locked --bin rustfs
+
+RUN set -eux; \
+    groupadd -g 10001 rustfs; \
+    useradd -u 10001 -g rustfs -M -s /usr/sbin/nologin rustfs
+
+WORKDIR /app
+
+RUN set -eux; \
+    mkdir -p /data /logs; \
+    chown -R rustfs:rustfs /data /logs /app; \
+    chmod 0750 /data /logs
+
+COPY entrypoint.sh /entrypoint.sh
+RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && chmod +x /entrypoint.sh
+
+ENV RUSTFS_VOLUMES="/data" \
+    RUST_LOG="warn" \
+    RUSTFS_OBS_LOG_DIRECTORY="/logs" \
+    RUSTFS_USERNAME="rustfs" \
+    RUSTFS_GROUPNAME="rustfs" \
+    RUSTFS_UID="10001" \
+    RUSTFS_GID="10001"
+
+EXPOSE 9000 9001
+
+ENTRYPOINT ["/entrypoint.sh"]
+CMD ["/usr/bin/rustfs"]

Dockerfile.glibc

@@ -0,0 +1,115 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM ubuntu:24.04 AS build
+
+ARG TARGETARCH
+ARG RELEASE=latest
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    curl \
+    unzip \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /build
+
+RUN set -eux; \
+    case "$TARGETARCH" in \
+      amd64)  ARCH_SUBSTR="x86_64-gnu"  ;; \
+      arm64)  ARCH_SUBSTR="aarch64-gnu" ;; \
+      *) echo "Unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
+    esac; \
+    \
+    if [ "$RELEASE" = "latest" ]; then \
+      TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
+              | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+    else \
+      TAG="$RELEASE"; \
+    fi; \
+    \
+    URL="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" \
+           | grep -o "\"browser_download_url\": \"[^\"]*${ARCH_SUBSTR}[^\"]*\\.zip\"" \
+           | cut -d'"' -f4 | head -n 1)"; \
+    \
+    if [ -z "$URL" ]; then echo "Failed to locate release asset for $ARCH_SUBSTR at tag $TAG" >&2; exit 1; fi; \
+    \
+    curl -fL "$URL" -o rustfs.zip; \
+    unzip -q rustfs.zip -d /build; \
+    \
+    if [ ! -x /build/rustfs ]; then \
+      BIN_PATH="$(unzip -Z -1 rustfs.zip | grep -E '(^|/)rustfs$' | head -n 1 || true)"; \
+      if [ -n "$BIN_PATH" ]; then \
+        mkdir -p /build/.tmp && unzip -q rustfs.zip "$BIN_PATH" -d /build/.tmp && \
+        mv "/build/.tmp/$BIN_PATH" /build/rustfs; \
+      fi; \
+    fi; \
+    [ -x /build/rustfs ] || { echo "rustfs binary not found in asset" >&2; exit 1; }; \
+    chmod +x /build/rustfs; \
+    rm -rf rustfs.zip /build/.tmp || true
+
+FROM ubuntu:24.04
+
+ARG RELEASE=latest
+ARG BUILD_DATE
+ARG VCS_REF
+
+LABEL name="RustFS" \
+      vendor="RustFS Team" \
+      maintainer="RustFS Team <dev@rustfs.com>" \
+      version="v${RELEASE#v}" \
+      release="${RELEASE}" \
+      build-date="${BUILD_DATE}" \
+      vcs-ref="${VCS_REF}" \
+      summary="High-performance distributed object storage system (glibc)" \
+      url="https://rustfs.com" \
+      license="Apache-2.0"
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=build /build/rustfs /usr/bin/rustfs
+COPY entrypoint.sh /entrypoint.sh
+
+RUN chmod +x /usr/bin/rustfs /entrypoint.sh
+
+RUN groupadd -g 10001 rustfs && \
+    useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
+    mkdir -p /data /logs && \
+    chown -R rustfs:rustfs /data /logs && \
+    chmod 0750 /data /logs
+
+ENV RUSTFS_ADDRESS=":9000" \
+    RUSTFS_CONSOLE_ADDRESS=":9001" \
+    RUSTFS_ACCESS_KEY="rustfsadmin" \
+    RUSTFS_SECRET_KEY="rustfsadmin" \
+    RUSTFS_CONSOLE_ENABLE="true" \
+    RUSTFS_CORS_ALLOWED_ORIGINS="*" \
+    RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
+    RUSTFS_VOLUMES="/data" \
+    RUSTFS_OBS_LOGGER_LEVEL=warn \
+    RUSTFS_OBS_LOG_DIRECTORY=/logs \
+    RUSTFS_OBS_ENVIRONMENT=production
+    
+EXPOSE 9000 9001
+
+VOLUME ["/data"]
+
+USER rustfs
+
+ENTRYPOINT ["/entrypoint.sh"]
+
+CMD ["rustfs"]

Dockerfile.source

@@ -1,4 +1,18 @@
 # syntax=docker/dockerfile:1.6
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 # Multi-stage Dockerfile for RustFS - LOCAL DEVELOPMENT ONLY
 #
 # IMPORTANT: This Dockerfile builds RustFS from source for local development and testing.
@@ -16,7 +30,7 @@ ARG BUILDPLATFORM
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.88-bookworm AS builder
+FROM rust:1.91-trixie AS builder
 
 # Re-declare args after FROM
 ARG TARGETPLATFORM
@@ -39,7 +53,9 @@ RUN set -eux; \
     libssl-dev \
     lld \
     protobuf-compiler \
-    flatbuffers-compiler; \
+    flatbuffers-compiler \
+    gcc-aarch64-linux-gnu \
+    gcc-x86-64-linux-gnu; \
     rm -rf /var/lib/apt/lists/*
 
 # Optional: cross toolchain for aarch64 (only when targeting linux/arm64)
@@ -51,18 +67,18 @@ RUN set -eux; \
     rm -rf /var/lib/apt/lists/*; \
     fi
 
-# Add Rust targets based on TARGETPLATFORM
+# Add Rust targets for both arches (to support cross-builds on multi-arch runners)
 RUN set -eux; \
-    case "${TARGETPLATFORM:-linux/amd64}" in \
+    rustup target add x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu; \
-    linux/amd64) rustup target add x86_64-unknown-linux-gnu ;; \
+    rustup component add rust-std-x86_64-unknown-linux-gnu rust-std-aarch64-unknown-linux-gnu
-    linux/arm64) rustup target add aarch64-unknown-linux-gnu ;; \
-    *) echo "Unsupported TARGETPLATFORM=${TARGETPLATFORM}" >&2; exit 1 ;; \
-    esac
 
 # Cross-compilation environment (used only when targeting aarch64)
 ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc
 ENV CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
 ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
+ENV CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
+ENV CC_x86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc
+ENV CXX_x86_64_unknown_linux_gnu=x86_64-linux-gnu-g++
 
 WORKDIR /usr/src/rustfs
 
@@ -72,7 +88,6 @@ COPY Cargo.toml Cargo.lock ./
 # 2) workspace member manifests (adjust if workspace layout changes)
 COPY rustfs/Cargo.toml rustfs/Cargo.toml
 COPY crates/*/Cargo.toml crates/
-COPY cli/rustfs-gui/Cargo.toml cli/rustfs-gui/Cargo.toml
 
 # Pre-fetch dependencies for better caching
 RUN --mount=type=cache,target=/usr/local/cargo/registry \
@@ -82,6 +97,10 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
 # 3) copy full sources (this is the main cache invalidation point)
 COPY . .
 
+# Generate static files
+
+RUN ./scripts/static.sh
+
 # Cargo build configuration for lean release artifacts
 ENV CARGO_NET_GIT_FETCH_WITH_CLI=true \
     CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse \
@@ -117,6 +136,45 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
     ;; \
     esac
 
+# -----------------------------
+# Development stage (keeps toolchain)
+# -----------------------------
+FROM builder AS dev
+
+ARG BUILD_DATE
+ARG VCS_REF
+
+LABEL name="RustFS (dev-source)" \
+    maintainer="RustFS Team" \
+    build-date="${BUILD_DATE}" \
+    vcs-ref="${VCS_REF}" \
+    description="RustFS - local development with Rust toolchain."
+
+# Install runtime dependencies that might be missing in partial builder
+# (builder already has build-essential, lld, etc.)
+WORKDIR /app
+
+ENV CARGO_INCREMENTAL=1
+
+# Ensure we have the same default env vars available
+ENV RUSTFS_VOLUMES="/data" \
+    RUST_LOG="warn" \
+    RUSTFS_OBS_LOG_DIRECTORY="/logs" \
+    RUSTFS_USERNAME="rustfs" \
+    RUSTFS_GROUPNAME="rustfs" \
+    RUSTFS_UID="10001" \
+    RUSTFS_GID="10001"
+
+# Note: We don't COPY source here because we expect it to be mounted at /app
+# We rely on cargo run to build and run
+EXPOSE 9000 9001
+
+COPY entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
+
+ENTRYPOINT ["/entrypoint.sh"]
+CMD ["cargo", "run", "--bin", "rustfs", "--"]
+
 # -----------------------------
 # Runtime stage (Ubuntu minimal)
 # -----------------------------
@@ -143,8 +201,8 @@ RUN set -eux; \
 
 # Create a conventional runtime user/group (final switch happens in entrypoint via chroot --userspec)
 RUN set -eux; \
-    groupadd -g 1000 rustfs; \
+    groupadd -g 10001 rustfs; \
-    useradd -u 1000 -g rustfs -M -s /usr/sbin/nologin rustfs
+    useradd -u 10001 -g rustfs -M -s /usr/sbin/nologin rustfs
 
 WORKDIR /app
 
@@ -160,21 +218,15 @@ COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
 # Default environment (override in docker run/compose as needed)
-ENV RUSTFS_ADDRESS=":9000" \
+ENV RUSTFS_VOLUMES="/data" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_VOLUMES="/data" \
     RUST_LOG="warn" \
-    RUSTFS_OBS_LOG_DIRECTORY="/logs" \
-    RUSTFS_SINKS_FILE_PATH="/logs" \
     RUSTFS_USERNAME="rustfs" \
     RUSTFS_GROUPNAME="rustfs" \
-    RUSTFS_UID="1000" \
+    RUSTFS_UID="10001" \
-    RUSTFS_GID="1000"
+    RUSTFS_GID="10001"
 
 EXPOSE 9000
-VOLUME ["/data", "/logs"]
+VOLUME ["/data"]
 
 # Keep root here; entrypoint will drop privileges using chroot --userspec
 ENTRYPOINT ["/entrypoint.sh"]

Makefile

@@ -2,375 +2,80 @@
 # Remote development requires VSCode with Dev Containers, Remote SSH, Remote Explorer
 # https://code.visualstudio.com/docs/remote/containers
 ###########
+
+.PHONY: SHELL
+
+# Makefile global config
+# Use config.mak to override any of the following variables.
+# Do not make changes here.
+
+.DEFAULT_GOAL := help
+.EXPORT_ALL_VARIABLES:
+.ONESHELL:
+.SILENT:
+
+NUM_CORES := $(shell nproc 2>/dev/null || sysctl -n hw.ncpu)
+
+MAKEFLAGS += -j$(NUM_CORES) -l$(NUM_CORES)
+MAKEFLAGS += --silent
+
+SHELL := $(shell which bash)
+.SHELLFLAGS = -eu -o pipefail -c
+
 DOCKER_CLI ?= docker
 IMAGE_NAME ?= rustfs:v1.0.0
 CONTAINER_NAME ?= rustfs-dev
 # Docker build configurations
 DOCKERFILE_PRODUCTION = Dockerfile
 DOCKERFILE_SOURCE = Dockerfile.source
-
-# Code quality and formatting targets
-.PHONY: fmt
-fmt:
-	@echo "🔧 Formatting code..."
-	cargo fmt --all
-
-.PHONY: fmt-check
-fmt-check:
-	@echo "📝 Checking code formatting..."
-	cargo fmt --all --check
-
-.PHONY: clippy
-clippy:
-	@echo "🔍 Running clippy checks..."
-	cargo clippy --fix --allow-dirty
-	cargo clippy --all-targets --all-features -- -D warnings
-
-.PHONY: check
-check:
-	@echo "🔨 Running compilation check..."
-	cargo check --all-targets
-
-.PHONY: test
-test:
-	@echo "🧪 Running tests..."
-	@if command -v cargo-nextest >/dev/null 2>&1; then \
-		cargo nextest run --all --exclude e2e_test; \
-	else \
-		echo "ℹ️ cargo-nextest not found; falling back to 'cargo test'"; \
-		cargo test --workspace --exclude e2e_test -- --nocapture; \
-	fi
-	cargo test --all --doc
-
-.PHONY: pre-commit
-pre-commit: fmt clippy check test
-	@echo "✅ All pre-commit checks passed!"
-
-.PHONY: setup-hooks
-setup-hooks:
-	@echo "🔧 Setting up git hooks..."
-	chmod +x .git/hooks/pre-commit
-	@echo "✅ Git hooks setup complete!"
-
-.PHONY: e2e-server
-e2e-server:
-	sh $(shell pwd)/scripts/run.sh
-
-.PHONY: probe-e2e
-probe-e2e:
-	sh $(shell pwd)/scripts/probe.sh
-
-# Native build using build-rustfs.sh script
-.PHONY: build
-build:
-	@echo "🔨 Building RustFS using build-rustfs.sh script..."
-	./build-rustfs.sh
-
-.PHONY: build-dev
-build-dev:
-	@echo "🔨 Building RustFS in development mode..."
-	./build-rustfs.sh --dev
-
-# Docker-based build (alternative approach)
-# Usage: make BUILD_OS=ubuntu22.04 build-docker
-# Output: target/ubuntu22.04/release/rustfs
 BUILD_OS ?= rockylinux9.3
-.PHONY: build-docker
-build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
-build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
-build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
-build-docker:
-	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
-	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
-	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
 
-.PHONY: build-musl
+# Makefile colors config
-build-musl:
+bold := $(shell tput bold)
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
+normal := $(shell tput sgr0)
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+errorTitle := $(shell tput setab 1 && tput bold && echo '\n')
-	./build-rustfs.sh --platform x86_64-unknown-linux-musl
+recommendation := $(shell tput setab 4)
+underline := $(shell tput smul)
+reset := $(shell tput -Txterm sgr0)
+black := $(shell tput setaf 0)
+red := $(shell tput setaf 1)
+green := $(shell tput setaf 2)
+yellow := $(shell tput setaf 3)
+blue := $(shell tput setaf 4)
+magenta := $(shell tput setaf 5)
+cyan := $(shell tput setaf 6)
+white := $(shell tput setaf 7)
 
-.PHONY: build-gnu
+define HEADER
-build-gnu:
+How to use me:
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
+	# To get help for each target
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	${bold}make help${reset}
-	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
 
-.PHONY: build-musl-arm64
+	# To run and execute a target
-build-musl-arm64:
+	${bold}make ${cyan}<target>${reset}
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform aarch64-unknown-linux-musl
 
-.PHONY: build-gnu-arm64
+	💡 For more help use 'make help', 'make help-build' or 'make help-docker'
-build-gnu-arm64:
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform aarch64-unknown-linux-gnu
 
-.PHONY: deploy-dev
+	🦀 RustFS Makefile Help:
-deploy-dev: build-musl
-	@echo "🚀 Deploying to dev server: $${IP}"
-	./scripts/dev_deploy.sh $${IP}
 
-# ========================================================================================
+	📋 Main Command Categories:
-# Docker Multi-Architecture Builds (Primary Methods)
+	 	make help-build                          # Show build-related help
-# ========================================================================================
+	 	make help-docker                         # Show Docker-related help
 
-# Production builds using docker-buildx.sh (for CI/CD and production)
+	🔧 Code Quality:
-.PHONY: docker-buildx
+		make fmt                                 # Format code
-docker-buildx:
+		make clippy                              # Run clippy checks
-	@echo "🏗️ Building multi-architecture production Docker images with buildx..."
+		make test                                # Run tests
-	./docker-buildx.sh
+		make pre-commit                          # Run all pre-commit checks
 
-.PHONY: docker-buildx-push
+	🚀 Quick Start:
-docker-buildx-push:
+		make build                               # Build RustFS binary
-	@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
+		make docker-dev-local                    # Build development Docker image (local)
-	./docker-buildx.sh --push
+		make dev-env-start                       # Start development environment
-
-.PHONY: docker-buildx-version
-docker-buildx-version:
-	@if [ -z "$(VERSION)" ]; then \
-		echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
-		exit 1; \
-	fi
-	@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
-	./docker-buildx.sh --release $(VERSION)
-
-.PHONY: docker-buildx-push-version
-docker-buildx-push-version:
-	@if [ -z "$(VERSION)" ]; then \
-		echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
-		exit 1; \
-	fi
-	@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
-	./docker-buildx.sh --release $(VERSION) --push
-
-# Development/Source builds using direct buildx commands
-.PHONY: docker-dev
-docker-dev:
-	@echo "🏗️ Building multi-architecture development Docker images with buildx..."
-	@echo "💡 This builds from source code and is intended for local development and testing"
-	@echo "⚠️  Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
-	$(DOCKER_CLI) buildx build \
-		--platform linux/amd64,linux/arm64 \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:source-latest \
-		--tag rustfs:dev-latest \
-		.
-
-.PHONY: docker-dev-local
-docker-dev-local:
-	@echo "🏗️ Building single-architecture development Docker image for local use..."
-	@echo "💡 This builds from source code for the current platform and loads locally"
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:source-latest \
-		--tag rustfs:dev-latest \
-		--load \
-		.
-
-.PHONY: docker-dev-push
-docker-dev-push:
-	@if [ -z "$(REGISTRY)" ]; then \
-		echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
-		exit 1; \
-	fi
-	@echo "🚀 Building and pushing multi-architecture development Docker images..."
-	@echo "💡 Pushing to registry: $(REGISTRY)"
-	$(DOCKER_CLI) buildx build \
-		--platform linux/amd64,linux/arm64 \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag $(REGISTRY)/rustfs:source-latest \
-		--tag $(REGISTRY)/rustfs:dev-latest \
-		--push \
-		.
 
 
+endef
+export HEADER
 
-# Local production builds using direct buildx (alternative to docker-buildx.sh)
+-include $(addsuffix /*.mak, $(shell find .config/make -type d))
-.PHONY: docker-buildx-production-local
-docker-buildx-production-local:
-	@echo "🏗️ Building single-architecture production Docker image locally..."
-	@echo "💡 Alternative to docker-buildx.sh for local testing"
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_PRODUCTION) \
-		--tag rustfs:production-latest \
-		--tag rustfs:latest \
-		--load \
-		--build-arg RELEASE=latest \
-		.
 
-# ========================================================================================
-# Single Architecture Docker Builds (Traditional)
-# ========================================================================================
-
-.PHONY: docker-build-production
-docker-build-production:
-	@echo "🏗️ Building single-architecture production Docker image..."
-	@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
-	$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
-
-.PHONY: docker-build-source
-docker-build-source:
-	@echo "🏗️ Building single-architecture source Docker image..."
-	@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
-	DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
-		--build-arg BUILDKIT_INLINE_CACHE=1 \
-		-f $(DOCKERFILE_SOURCE) -t rustfs:source .
-
-# ========================================================================================
-# Development Environment
-# ========================================================================================
-
-.PHONY: dev-env-start
-dev-env-start:
-	@echo "🚀 Starting development environment..."
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:dev \
-		--load \
-		.
-	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
-		-p 9010:9010 -p 9000:9000 \
-		-v $(shell pwd):/workspace \
-		-it rustfs:dev
-
-.PHONY: dev-env-stop
-dev-env-stop:
-	@echo "🛑 Stopping development environment..."
-	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
-
-.PHONY: dev-env-restart
-dev-env-restart: dev-env-stop dev-env-start
-
-
-
-# ========================================================================================
-# Build Utilities
-# ========================================================================================
-
-.PHONY: docker-inspect-multiarch
-docker-inspect-multiarch:
-	@if [ -z "$(IMAGE)" ]; then \
-		echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
-		exit 1; \
-	fi
-	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
-	docker buildx imagetools inspect $(IMAGE)
-
-.PHONY: build-cross-all
-build-cross-all:
-	@echo "🔧 Building all target architectures..."
-	@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
-	@echo "🔨 Generating protobuf code..."
-	cargo run --bin gproto || true
-	@echo "🔨 Building x86_64-unknown-linux-gnu..."
-	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
-	@echo "🔨 Building aarch64-unknown-linux-gnu..."
-	./build-rustfs.sh --platform aarch64-unknown-linux-gnu
-	@echo "🔨 Building x86_64-unknown-linux-musl..."
-	./build-rustfs.sh --platform x86_64-unknown-linux-musl
-	@echo "🔨 Building aarch64-unknown-linux-musl..."
-	./build-rustfs.sh --platform aarch64-unknown-linux-musl
-	@echo "✅ All architectures built successfully!"
-
-# ========================================================================================
-# Help and Documentation
-# ========================================================================================
-
-.PHONY: help-build
-help-build:
-	@echo "🔨 RustFS Build Help:"
-	@echo ""
-	@echo "🚀 Local Build (Recommended):"
-	@echo "  make build                               # Build RustFS binary (includes console by default)"
-	@echo "  make build-dev                           # Development mode build"
-	@echo "  make build-musl                          # Build x86_64 musl version"
-	@echo "  make build-gnu                           # Build x86_64 GNU version"
-	@echo "  make build-musl-arm64                    # Build aarch64 musl version"
-	@echo "  make build-gnu-arm64                     # Build aarch64 GNU version"
-	@echo ""
-	@echo "🐳 Docker Build:"
-	@echo "  make build-docker                        # Build using Docker container"
-	@echo "  make build-docker BUILD_OS=ubuntu22.04   # Specify build system"
-	@echo ""
-	@echo "🏗️ Cross-architecture Build:"
-	@echo "  make build-cross-all                     # Build binaries for all architectures"
-	@echo ""
-	@echo "🔧 Direct usage of build-rustfs.sh script:"
-	@echo "  ./build-rustfs.sh --help                 # View script help"
-	@echo "  ./build-rustfs.sh --no-console           # Build without console resources"
-	@echo "  ./build-rustfs.sh --force-console-update # Force update console resources"
-	@echo "  ./build-rustfs.sh --dev                  # Development mode build"
-	@echo "  ./build-rustfs.sh --sign                 # Sign binary files"
-	@echo "  ./build-rustfs.sh --platform x86_64-unknown-linux-gnu   # Specify target platform"
-	@echo "  ./build-rustfs.sh --skip-verification    # Skip binary verification"
-	@echo ""
-	@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
-
-.PHONY: help-docker
-help-docker:
-	@echo "🐳 Docker Multi-architecture Build Help:"
-	@echo ""
-	@echo "🚀 Production Image Build (Recommended to use docker-buildx.sh):"
-	@echo "  make docker-buildx                       # Build production multi-arch image (no push)"
-	@echo "  make docker-buildx-push                  # Build and push production multi-arch image"
-	@echo "  make docker-buildx-version VERSION=v1.0.0        # Build specific version"
-	@echo "  make docker-buildx-push-version VERSION=v1.0.0   # Build and push specific version"
-	@echo ""
-	@echo "🔧 Development/Source Image Build (Local development testing):"
-	@echo "  make docker-dev                          # Build dev multi-arch image (cannot load locally)"
-	@echo "  make docker-dev-local                    # Build dev single-arch image (local load)"
-	@echo "  make docker-dev-push REGISTRY=xxx       # Build and push dev image"
-	@echo ""
-	@echo "🏗️ Local Production Image Build (Alternative):"
-	@echo "  make docker-buildx-production-local      # Build production single-arch image locally"
-	@echo ""
-	@echo "📦 Single-architecture Build (Traditional way):"
-	@echo "  make docker-build-production             # Build single-arch production image"
-	@echo "  make docker-build-source                 # Build single-arch source image"
-	@echo ""
-	@echo "🚀 Development Environment Management:"
-	@echo "  make dev-env-start                       # Start development container environment"
-	@echo "  make dev-env-stop                        # Stop development container environment"
-	@echo "  make dev-env-restart                     # Restart development container environment"
-	@echo ""
-	@echo "🔧 Auxiliary Tools:"
-	@echo "  make build-cross-all                     # Build binaries for all architectures"
-	@echo "  make docker-inspect-multiarch IMAGE=xxx  # Check image architecture support"
-	@echo ""
-	@echo "📋 Environment Variables:"
-	@echo "  REGISTRY          Image registry address (required for push)"
-	@echo "  DOCKERHUB_USERNAME    Docker Hub username"
-	@echo "  DOCKERHUB_TOKEN       Docker Hub access token"
-	@echo "  GITHUB_TOKEN          GitHub access token"
-	@echo ""
-	@echo "💡 Suggestions:"
-	@echo "  - Production use: Use docker-buildx* commands (based on precompiled binaries)"
-	@echo "  - Local development: Use docker-dev* commands (build from source)"
-	@echo "  - Development environment: Use dev-env-* commands to manage dev containers"
-
-.PHONY: help
-help:
-	@echo "🦀 RustFS Makefile Help:"
-	@echo ""
-	@echo "📋 Main Command Categories:"
-	@echo "  make help-build                          # Show build-related help"
-	@echo "  make help-docker                         # Show Docker-related help"
-	@echo ""
-	@echo "🔧 Code Quality:"
-	@echo "  make fmt                                 # Format code"
-	@echo "  make clippy                              # Run clippy checks"
-	@echo "  make test                                # Run tests"
-	@echo "  make pre-commit                          # Run all pre-commit checks"
-	@echo ""
-	@echo "🚀 Quick Start:"
-	@echo "  make build                               # Build RustFS binary"
-	@echo "  make docker-dev-local                    # Build development Docker image (local)"
-	@echo "  make dev-env-start                       # Start development environment"
-	@echo ""
-	@echo "💡 For more help use 'make help-build' or 'make help-docker'"

README.md

@@ -1,6 +1,6 @@
-[![RustFS](https://rustfs.com/images/rustfs-github.png)](https://rustfs.com)
+[![RustFS](https://repository-images.githubusercontent.com/722597620/0fa936a2-8164-4f53-867f-def4beb64b21)](https://rustfs.com)
 
-<p align="center">RustFS is a high-performance distributed object storage software built using Rust</p>
+<p align="center">RustFS is a high-performance, distributed object storage system built in Rust.</p>
 
 <p align="center">
   <a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
@@ -11,7 +11,12 @@
 </p>
 
 <p align="center">
-  <a href="https://docs.rustfs.com/introduction.html">Getting Started</a>
+<a href="https://trendshift.io/repositories/14181" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14181" alt="rustfs%2Frustfs | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a> 
+<a href="https://runacap.com/ross-index/q4-2025/" target="_blank" rel="noopener"><img style="width: 260px; height: 55px" src="https://runacap.com/wp-content/uploads/2026/01/ROSS_badge_white_Q4_2025.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q4 2025 | Runa Capital" height="55" /></a>
+</p>
+
+<p align="center">
+  <a href="https://docs.rustfs.com/installation/">Getting Started</a>
   · <a href="https://docs.rustfs.com/">Docs</a>
   · <a href="https://github.com/rustfs/rustfs/issues">Bug reports</a>
   · <a href="https://github.com/rustfs/rustfs/discussions">Discussions</a>
@@ -19,7 +24,6 @@
 
 <p align="center">
 English | <a href="https://github.com/rustfs/rustfs/blob/main/README_ZH.md">简体中文</a> |
-  <!-- Keep these links. Translations will automatically update with the README. -->
   <a href="https://readme-i18n.com/rustfs/rustfs?lang=de">Deutsch</a> |
   <a href="https://readme-i18n.com/rustfs/rustfs?lang=es">Español</a> |
   <a href="https://readme-i18n.com/rustfs/rustfs?lang=fr">français</a> |
@@ -29,145 +33,245 @@ English | <a href="https://github.com/rustfs/rustfs/blob/main/README_ZH.md">简
   <a href="https://readme-i18n.com/rustfs/rustfs?lang=ru">Русский</a>
 </p>
 
-RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages
+RustFS is a high-performance, distributed object storage system built in Rust—one of the most loved programming languages worldwide. RustFS combines the simplicity of MinIO with the memory safety and raw performance of Rust. It offers full S3 compatibility, is completely open-source, and is optimized for data lakes, AI, and big data workloads.
-worldwide. Along with MinIO, it shares a range of advantages such as simplicity, S3 compatibility, open-source nature,
-support for data lakes, AI, and big data. Furthermore, it has a better and more user-friendly open-source license in
-comparison to other storage systems, being constructed under the Apache license. As Rust serves as its foundation,
-RustFS provides faster speed and safer distributed features for high-performance object storage.
 
-> ⚠️ **RustFS is under rapid development. Do NOT use in production environments!**
+Unlike other storage systems, RustFS is released under the permissible Apache 2.0 license, avoiding the restrictions of AGPL. With Rust as its foundation, RustFS delivers superior speed and secure distributed features for next-generation object storage.
 
-## Features
+## Feature & Status
 
-- **High Performance**: Built with Rust, ensuring speed and efficiency.
+- **High Performance**: Built with Rust to ensure maximum speed and resource efficiency.
-- **Distributed Architecture**: Scalable and fault-tolerant design for large-scale deployments.
+- **Distributed Architecture**: Scalable and fault-tolerant design suitable for large-scale deployments.
-- **S3 Compatibility**: Seamless integration with existing S3-compatible applications.
+- **S3 Compatibility**: Seamless integration with existing S3-compatible applications and tools.
-- **Data Lake Support**: Optimized for big data and AI workloads.
+- **OpenStack Swift API**: Native support for Swift protocol with Keystone authentication.
-- **Open Source**: Licensed under Apache 2.0, encouraging community contributions and transparency.
+- **OpenStack Keystone Integration**: Native support for OpenStack Keystone authentication with X-Auth-Token headers.
-- **User-Friendly**: Designed with simplicity in mind, making it easy to deploy and manage.
+- **Data Lake Support**: Optimized for high-throughput big data and AI workloads.
+- **Open Source**: Licensed under Apache 2.0, encouraging unrestricted community contributions and commercial usage.
+- **User-Friendly**: Designed with simplicity in mind for easy deployment and management.
 
-## RustFS vs MinIO
+| Feature                 | Status       | Feature                  | Status           |
+| :---------------------- | :----------- | :----------------------- | :--------------- |
+| **S3 Core Features**    | ✅ Available | **Bitrot Protection**    | ✅ Available     |
+| **Upload / Download**   | ✅ Available | **Single Node Mode**     | ✅ Available     |
+| **Versioning**          | ✅ Available | **Bucket Replication**   | ✅ Available     |
+| **Logging**             | ✅ Available | **Lifecycle Management** | 🚧 Under Testing |
+| **Event Notifications** | ✅ Available | **Distributed Mode**     | 🚧 Under Testing |
+| **K8s Helm Charts**     | ✅ Available | **RustFS KMS**           | 🚧 Under Testing |
+| **Keystone Auth**       | ✅ Available | **Multi-Tenancy**        | ✅ Available     |
+| **Swift API**           | ✅ Available | **Swift Metadata Ops**   | 🚧 Partial       |
 
-Stress test server parameters
+## RustFS vs MinIO Performance
 
-| Type    | parameter | Remark                                                   |
+**Stress Test Environment:**
-|---------|-----------|----------------------------------------------------------|
+
-| CPU     | 2 Core    | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz |   |
+| Type    | Parameter | Remark                                                   |
-| Memory  | 4GB       |                                                          |
+| ------- | --------- | -------------------------------------------------------- |
-| Network | 15Gbp     |                                                          |
+| CPU     | 2 Core    | Intel Xeon (Sapphire Rapids) Platinum 8475B, 2.7/3.2 GHz |
-| Driver  | 40GB x 4  | IOPS 3800 / Driver                                       |
+| Memory  | 4GB       |                                                          |
+| Network | 15Gbps    |                                                          |
+| Drive   | 40GB x 4  | IOPS 3800 / Drive                                        |
 
 <https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a>
 
-### RustFS vs Other object storage
+### RustFS vs Other Object Storage
 
-| RustFS                                                                          | Other object storage                                                                                                    |
+| Feature                | RustFS                                                                                                                                                | Other Object Storage                                                                     |
-|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
+| :--------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------- |
-| Powerful Console                                                                | Simple and useless Console                                                                                              |
+| **Console Experience** | **Powerful Console**<br>Comprehensive management interface.                                                                                           | **Basic / Limited Console**<br>Often overly simple or lacking critical features.         |
-| Developed based on Rust language, memory is safer                               | Developed in Go or C, with potential issues like memory GC/leaks                                                        |
+| **Language & Safety**  | **Rust-based**<br>Memory safety by design.                                                                                                            | **Go or C-based**<br>Potential for memory GC pauses or leaks.                            |
-| Does not report logs to third-party countries                                   | Reporting logs to other third countries may violate national security laws                                              |
+| **Data Sovereignty**   | **No Telemetry / Full Compliance**<br>Guards against unauthorized cross-border data egress. Compliant with GDPR (EU/UK), CCPA (US), and APPI (Japan). | **Potential Risk**<br>Possible legal exposure and unwanted data telemetry.               |
-| Licensed under Apache, more business-friendly                                   | AGPL V3 License and other License, polluted open source and License traps, infringement of intellectual property rights |
+| **Licensing**          | **Permissive Apache 2.0**<br>Business-friendly, no "poison pill" clauses.                                                                             | **Restrictive AGPL v3**<br>Risk of license traps and intellectual property pollution.    |
-| Comprehensive S3 support, works with domestic and international cloud providers | Full support for S3, but no local cloud vendor support                                                                  |
+| **Compatibility**      | **100% S3 Compatible**<br>Works with any cloud provider or client, anywhere.                                                                          | **Variable Compatibility**<br>May lack support for local cloud vendors or specific APIs. |
-| Rust-based development, strong support for secure and innovative devices        | Poor support for edge gateways and secure innovative devices                                                            |
+| **Edge & IoT**         | **Strong Edge Support**<br>Ideal for secure, innovative edge devices.                                                                                 | **Weak Edge Support**<br>Often too heavy for edge gateways.                              |
-| Stable commercial prices, free community support                                | High pricing, with costs up to $250,000 for 1PiB                                                                        |
+| **Risk Profile**       | **Enterprise Risk Mitigation**<br>Clear IP rights and safe for commercial use.                                                                        | **Legal Risks**<br>Intellectual property ambiguity and usage restrictions.               |
-| No risk                                                                         | Intellectual property risks and risks of prohibited uses                                                                |
+
+## Staying ahead
+
+Star RustFS on GitHub and be instantly notified of new releases.
+
+<img src="https://github.com/user-attachments/assets/7ee40bb4-3e46-4eac-b0d0-5fbeb85ff8f3" />
 
 ## Quickstart
 
 To get started with RustFS, follow these steps:
 
-1. **One-click installation script (Option 1)​​**
+### 1. One-click Installation (Option 1)
 
-  ```bash
+```bash
-  curl -O  https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
+curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
-  ```
+```
 
-2. **Docker Quick Start (Option 2)​​**
+### 2. Docker Quick Start (Option 2)
 
-  ```bash
+The RustFS container runs as a non-root user `rustfs` (UID `10001`). If you run Docker with `-v` to mount a host directory, please ensure the host directory owner is set to `10001`, otherwise you will encounter permission denied errors.
-   # create data and logs directories
-   mkdir -p data logs
 
-   # using latest alpha version
+```bash
-   docker run -d -p 9000:9000 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:alpha
+# Create data and logs directories
+mkdir -p data logs
 
-   # Specific version
+# Change the owner of these directories
-   docker run -d -p 9000:9000 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0.alpha.45
+chown -R 10001:10001 data logs
-   ```
 
-For docker installation, you can also run the container with docker compose. With the `docker-compose.yml` file under
+# Using latest version
-root directory, running the command:
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
-  ```
+# Using specific version
-  docker compose --profile observability up -d
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-alpha.76
-  ```
+```
 
-**NOTE**: You should be better to have a look for `docker-compose.yaml` file. Because, several services contains in the
+If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
-file. Grafan,prometheus,jaeger containers will be launched using docker compose file, which is helpful for rustfs
-observability. If you want to start redis as well as nginx container, you can specify the corresponding profiles.
 
-3. **Build from Source (Option 3) - Advanced Users**
+```bash
+podman run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
+```
 
-   For developers who want to build RustFS Docker images from source with multi-architecture support:
+You can also use Docker Compose. Using the `docker-compose.yml` file in the root directory:
 
-   ```bash
+```bash
-   # Build multi-architecture images locally
+docker compose --profile observability up -d
-   ./docker-buildx.sh --build-arg RELEASE=latest
+```
 
-   # Build and push to registry
+Similarly, you can run the command with podman
-   ./docker-buildx.sh --push
 
-   # Build specific version
+```bash
-   ./docker-buildx.sh --release v1.0.0 --push
+podman compose --profile observability up -d
+```
 
-   # Build for custom registry
+**NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
-   ./docker-buildx.sh --registry your-registry.com --namespace yourname --push
-   ```
 
-   The `docker-buildx.sh` script supports:
+### 3\. Build from Source (Option 3) - Advanced Users
-    - **Multi-architecture builds**: `linux/amd64`, `linux/arm64`
-    - **Automatic version detection**: Uses git tags or commit hashes
-    - **Registry flexibility**: Supports Docker Hub, GitHub Container Registry, etc.
-    - **Build optimization**: Includes caching and parallel builds
 
-   You can also use Make targets for convenience:
+For developers who want to build RustFS Docker images from source with multi-architecture support:
 
-   ```bash
+```bash
-   make docker-buildx                    # Build locally
+# Build multi-architecture images locally
-   make docker-buildx-push               # Build and push
+./docker-buildx.sh --build-arg RELEASE=latest
-   make docker-buildx-version VERSION=v1.0.0  # Build specific version
-   make help-docker                      # Show all Docker-related commands
-   ```
 
-   > **Heads-up (macOS cross-compilation)**: macOS keeps the default `ulimit -n` at 256, so `cargo zigbuild` or `./build-rustfs.sh --platform ...` may fail with `ProcessFdQuotaExceeded` when targeting Linux. The build script now tries to raise the limit automatically, but if you still see the warning, run `ulimit -n 4096` (or higher) in your shell before building.
+# Build and push to registry
+./docker-buildx.sh --push
 
-4. **Build with helm chart(Option 4) - Cloud Native environment**
+# Build specific version
+./docker-buildx.sh --release v1.0.0 --push
 
-   Following the instructions on [helm chart README](./helm/README.md) to install RustFS on kubernetes cluster.
+# Build for custom registry
+./docker-buildx.sh --registry your-registry.com --namespace yourname --push
+```
 
-5. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console,
+The `docker-buildx.sh` script supports:
-   default username and password is `rustfsadmin` .
-6. **Create a Bucket**: Use the console to create a new bucket for your objects.
-7. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your
-   RustFS instance.
 
-**NOTE**: If you want to access RustFS instance with `https`, you can refer
+- **Multi-architecture builds**: `linux/amd64`, `linux/arm64`
-to [TLS configuration docs](https://docs.rustfs.com/integration/tls-configured.html).
+- **Automatic version detection**: Uses git tags or commit hashes
+- **Registry flexibility**: Supports Docker Hub, GitHub Container Registry, etc.
+- **Build optimization**: Includes caching and parallel builds
+
+You can also use Make targets for convenience:
+
+```bash
+make docker-buildx                    # Build locally
+make docker-buildx-push               # Build and push
+make docker-buildx-version VERSION=v1.0.0  # Build specific version
+make help-docker                      # Show all Docker-related commands
+```
+
+> **Heads-up (macOS cross-compilation)**: macOS keeps the default `ulimit -n` at 256, so `cargo zigbuild` or `./build-rustfs.sh --platform ...` may fail with `ProcessFdQuotaExceeded` when targeting Linux. The build script attempts to raise the limit automatically, but if you still see the warning, run `ulimit -n 4096` (or higher) in your shell before building.
+
+### 4\. Build with Helm Chart (Option 4) - Cloud Native
+
+Follow the instructions in the [Helm Chart README](https://charts.rustfs.com/) to install RustFS on a Kubernetes cluster.
+
+### 5\. Nix Flake (Option 5)
+
+If you have [Nix with flakes enabled](https://nixos.wiki/wiki/Flakes#Enable_flakes):
+
+```bash
+# Run directly without installing
+nix run github:rustfs/rustfs
+
+# Build the binary
+nix build github:rustfs/rustfs
+./result/bin/rustfs --help
+
+# Or from a local checkout
+nix build
+nix run
+```
+
+### 6\. X-CMD (Option 6)
+
+If you are an [x-cmd](https://www.x-cmd.com/install/rustfs) user:
+
+```bash
+# Run directly without installing
+x rustfs
+
+# Download the binary and install it to the global environment
+x env use rustfs
+rustfs --help
+```
+
+---
+
+### Accessing RustFS
+
+1. **Access the Console**: Open your web browser and navigate to `http://localhost:9001` to access the RustFS console.
+    - Default credentials: `rustfsadmin` / `rustfsadmin`
+2. **Create a Bucket**: Use the console to create a new bucket for your objects.
+3. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.
+
+**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured.html).
+
+### OIDC Roles Claim (Microsoft Entra ID)
+
+RustFS supports mapping an OIDC claim containing role values into the existing
+authorization pipeline. The `roles_claim` setting is **optional**: when unset or
+empty, only the `groups` claim contributes to authorization (same as older
+RustFS releases). For Microsoft Entra ID app roles, set `roles_claim=roles` so
+both console admin checks and bucket IAM policies can evaluate those roles.
+
+Example environment configuration (opt-in roles claim):
+
+```bash
+RUSTFS_IDENTITY_OPENID_ENABLE=on
+RUSTFS_IDENTITY_OPENID_CONFIG_URL="https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration"
+RUSTFS_IDENTITY_OPENID_CLIENT_ID="<client-id>"
+RUSTFS_IDENTITY_OPENID_CLIENT_SECRET="<client-secret>"
+RUSTFS_IDENTITY_OPENID_SCOPES="openid,profile,email"
+RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM="groups"
+RUSTFS_IDENTITY_OPENID_ROLES_CLAIM="roles"
+```
+
+Policy condition example (evaluate app roles directly with `jwt:roles`; when
+`roles_claim` is configured, RustFS also merges those values into `jwt:groups`
+for backward compatibility with older policies):
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["admin:*"],
+      "Resource": ["arn:aws:s3:::*"],
+      "Condition": {
+        "ForAnyValue:StringEquals": {
+          "jwt:roles": ["RustFS.ConsoleAdmin"]
+        }
+      }
+    }
+  ]
+}
+```
 
 ## Documentation
 
-For detailed documentation, including configuration options, API references, and advanced usage, please visit
+For detailed documentation, including configuration options, API references, and advanced usage, please visit our [Documentation](https://docs.rustfs.com).
-our [Documentation](https://docs.rustfs.com).
 
 ## Getting Help
 
-If you have any questions or need assistance, you can:
+If you have any questions or need assistance:
 
 - Check the [FAQ](https://github.com/rustfs/rustfs/discussions/categories/q-a) for common issues and solutions.
-- Join our [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) to ask questions and share your
+- Join our [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) to ask questions and share your experiences.
-  experiences.
+- Open an issue on our [GitHub Issues](https://github.com/rustfs/rustfs/issues) page for bug reports or feature requests.
-- Open an issue on our [GitHub Issues](https://github.com/rustfs/rustfs/issues) page for bug reports or feature
-  requests.
 
 ## Links
 
@@ -178,28 +282,19 @@ If you have any questions or need assistance, you can:
 ## Contact
 
 - **Bugs**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Business**: <hello@rustfs.com>
+- **Business**: [hello@rustfs.com](mailto:hello@rustfs.com)
-- **Jobs**: <jobs@rustfs.com>
+- **Jobs**: [jobs@rustfs.com](mailto:jobs@rustfs.com)
 - **General Discussion**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
 - **Contributing**: [CONTRIBUTING.md](CONTRIBUTING.md)
 
 ## Contributors
 
-RustFS is a community-driven project, and we appreciate all contributions. Check out
+RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
-the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped
-make RustFS better.
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-  <img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors"/>
+<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
 </a>
 
-## Github Trending Top
-
-🚀 RustFS is beloved by open-source enthusiasts and enterprise users worldwide, often appearing on the GitHub Trending
-top charts.
-
-<a href="https://trendshift.io/repositories/14181" target="_blank"><img src="https://raw.githubusercontent.com/rustfs/rustfs/refs/heads/main/docs/rustfs-trending.jpg" alt="rustfs%2Frustfs | Trendshift" /></a>
-
 ## Star History
 
 [![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)

README_ZH.md

@@ -1,182 +1,246 @@
-[![RustFS](https://rustfs.com/images/rustfs-github.png)](https://rustfs.com)
+[![RustFS](https://github.com/user-attachments/assets/1b5afcd6-a2c3-47ff-8bc3-ce882b0ddca7)](https://rustfs.com.cn)
 
-<p align="center">RustFS 是一个使用 Rust 构建的高性能分布式对象存储软件</p >
+<p align="center">RustFS 是一个基于 Rust 构建的高性能分布式对象存储系统。</p>
 
 <p align="center">
   <a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
-  <a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="Build and Push Docker Images" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
+  <a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="构建并推送 Docker 镜像" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
-  <img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
+  <img alt="GitHub 提交活跃度" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
-  <img alt="Github Last Commit" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
+  <img alt="Github 最新提交" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
   <a href="https://hellogithub.com/repository/rustfs/rustfs" target="_blank"><img src="https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=b95bcb72bdc340b68f16fdf6790b7d5b&claim_uid=MsbvjYeLDKAH457&theme=small" alt="Featured｜HelloGitHub" /></a>
-</p >
+</p>
+
 
 <p align="center">
-  <a href="https://docs.rustfs.com/zh/introduction.html">快速开始</a >
+<a href="https://trendshift.io/repositories/14181" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14181" alt="rustfs%2Frustfs | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a> 
-  · <a href="https://docs.rustfs.com/zh/">文档</a >
+<a href="https://runacap.com/ross-index/q4-2025/" target="_blank" rel="noopener"><img style="width: 260px; height: 55px" src="https://runacap.com/wp-content/uploads/2026/01/ROSS_badge_white_Q4_2025.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q4 2025 | Runa Capital" height="55" /></a>
-  · <a href="https://github.com/rustfs/rustfs/issues">问题报告</a >
+</p>
-  · <a href="https://github.com/rustfs/rustfs/discussions">讨论</a >
-</p >
 
 <p align="center">
-<a href="https://github.com/rustfs/rustfs/blob/main/README.md">English</a > | 简体中文
+  <a href="https://docs.rustfs.com/installation/">快速开始</a>
-</p >
+  · <a href="https://docs.rustfs.com/">文档</a>
+  · <a href="https://github.com/rustfs/rustfs/issues">报告 Bug</a>
+  · <a href="https://github.com/rustfs/rustfs/discussions">社区讨论</a>
+</p>
 
-RustFS 是一个使用 Rust（全球最受欢迎的编程语言之一）构建的高性能分布式对象存储软件。与 MinIO 一样，它具有简单性、S3
+<p align="center">
-兼容性、开源特性以及对数据湖、AI 和大数据的支持等一系列优势。此外，与其他存储系统相比，它采用 Apache
+  <a href="https://github.com/rustfs/rustfs/blob/main/README.md">English</a> | 简体中文 |
-许可证构建，拥有更好、更用户友好的开源许可证。由于以 Rust 为基础，RustFS 为高性能对象存储提供了更快的速度和更安全的分布式功能。
+  <a href="https://readme-i18n.com/rustfs/rustfs?lang=de">Deutsch</a> |
+  <a href="https://readme-i18n.com/rustfs/rustfs?lang=es">Español</a> |
+  <a href="https://readme-i18n.com/rustfs/rustfs?lang=fr">français</a> |
+  <a href="https://readme-i18n.com/rustfs/rustfs?lang=ja">日本語</a> |
+  <a href="https://readme-i18n.com/rustfs/rustfs?lang=ko">한국어</a> |
+  <a href="https://readme-i18n.com/rustfs/rustfs?lang=pt">Portuguese</a> |
+  <a href="https://readme-i18n.com/rustfs/rustfs?lang=ru">Русский</a>
+</p>
 
-## 特性
+RustFS 是一个基于 Rust 构建的高性能分布式对象存储系统。Rust 是全球最受开发者喜爱的编程语言之一，RustFS 完美结合了 MinIO 的简洁性与 Rust 的内存安全及高性能优势。它提供完整的 S3 兼容性，完全开源，并专为数据湖、人工智能（AI）和大数据负载进行了优化。
 
-- **高性能**：使用 Rust 构建，确保速度和效率。
+与其他存储系统不同，RustFS 采用更宽松、商业友好的 Apache 2.0 许可证，避免了 AGPL 协议的限制。以 Rust 为基石，RustFS 为下一代对象存储提供了更快的速度和更安全的分布式特性。
+
+## 特征和功能状态
+
+- **高性能**：基于 Rust 构建，确保极致的速度和资源效率。
 - **分布式架构**：可扩展且容错的设计，适用于大规模部署。
-- **S3 兼容性**：与现有 S3 兼容应用程序无缝集成。
+- **S3 兼容性**：与现有的 S3 兼容应用和工具无缝集成。
-- **数据湖支持**：针对大数据和 AI 工作负载进行了优化。
+- **数据湖支持**：专为高吞吐量的大数据和 AI 工作负载优化。
-- **开源**：采用 Apache 2.0 许可证，鼓励社区贡献和透明度。
+- **完全开源**：采用 Apache 2.0 许可证，鼓励社区贡献和商业使用。
-- **用户友好**：设计简单，易于部署和管理。
+- **简单易用**：设计简洁，易于部署和管理。
 
-## RustFS vs MinIO
+| 功能               | 状态    | 功能                    | 状态      |
+| :----------------- | :------ | :---------------------- | :-------- |
+| **S3 核心功能**    | ✅ 可用 | **Bitrot (防数据腐烂)** | ✅ 可用   |
+| **上传 / 下载**    | ✅ 可用 | **单机模式**            | ✅ 可用   |
+| **版本控制**       | ✅ 可用 | **存储桶复制**          | ✅ 可用   |
+| **日志功能**       | ✅ 可用 | **生命周期管理**        | 🚧 测试中 |
+| **事件通知**       | ✅ 可用 | **分布式模式**          | 🚧 测试中 |
+| **K8s Helm Chart** | ✅ 可用 | **OPA (策略引擎)**      | 🚧 测试中 |
 
-压力测试服务器参数
+## RustFS vs MinIO 性能对比
 
-| 类型  | 参数       | 备注                                                       |
+**压力测试环境参数：**
-|-----|----------|----------------------------------------------------------|
+
-| CPU | 2 核心     | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz |   |
+| 类型 | 参数     | 备注                                                      |
-| 内存  | 4GB      |                                                          |
+| ---- | -------- | --------------------------------------------------------- |
-| 网络  | 15Gbp    |                                                          |
+| CPU  | 2 核     | Intel Xeon (Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz |
-| 驱动器 | 40GB x 4 | IOPS 3800 / 驱动器                                          |
+| 内存 | 4GB      |                                                           |
+| 网络 | 15Gbps   |                                                           |
+| 硬盘 | 40GB x 4 | IOPS 3800 / Drive                                         |
 
 <https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a>
 
 ### RustFS vs 其他对象存储
 
-| RustFS                   | 其他对象存储                              |
+| 特性           | RustFS                                                                                                              | 其他对象存储                                                             |
-|--------------------------|-------------------------------------|
+| :------------- | :------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------- |
-| 强大的控制台                   | 简单且无用的控制台                           |
+| **控制台体验** | **功能强大的控制台**<br>提供全面的管理界面。                                                                        | **基础/简陋的控制台**<br>通常功能过于简单或缺失关键特性。                |
-| 基于 Rust 语言开发，内存更安全       | 使用 Go 或 C 开发，存在内存 GC/泄漏等潜在问题        |
+| **语言与安全** | **基于 Rust 开发**<br>天生的内存安全。                                                                              | **基于 Go 或 C 开发**<br>存在内存 GC 停顿或内存泄漏的潜在风险。          |
-| 不向第三方国家报告日志              | 向其他第三方国家报告日志可能违反国家安全法律              |
+| **数据主权**   | **无遥测 / 完全合规**<br>防止未经授权的数据跨境传输。完全符合 GDPR (欧盟/英国)、CCPA (美国) 和 APPI (日本) 等法规。 | **潜在风险**<br>可能存在法律风险和隐蔽的数据遥测（Telemetry）。          |
-| 采用 Apache 许可证，对商业更友好     | AGPL V3 许可证等其他许可证，污染开源和许可证陷阱，侵犯知识产权 |
+| **开源协议**   | **宽松的 Apache 2.0**<br>商业友好，无“毒丸”条款。                                                                   | **受限的 AGPL v3**<br>存在许可证陷阱和知识产权污染的风险。               |
-| 全面的 S3 支持，适用于国内外云提供商     | 完全支持 S3，但不支持本地云厂商                   |
+| **兼容性**     | **100% S3 兼容**<br>适用于任何云提供商和客户端，随处运行。                                                          | **兼容性不一**<br>虽然支持 S3，但可能缺乏对本地云厂商或特定 API 的支持。 |
-| 基于 Rust 开发，对安全和创新设备有强大支持 | 对边缘网关和安全创新设备支持较差                    |
+| **边缘与 IoT** | **强大的边缘支持**<br>非常适合安全、创新的边缘设备。                                                                | **边缘支持较弱**<br>对于边缘网关来说通常过于沉重。                       |
-| 稳定的商业价格，免费社区支持           | 高昂的定价，1PiB 成本高达 $250,000            |
+| **成本**       | **稳定且免费**<br>免费社区支持，稳定的商业定价。                                                                    | **高昂成本**<br>1PiB 的成本可能高达 250,000 美元。                       |
-| 无风险                      | 知识产权风险和禁止使用的风险                      |
+| **风险控制**   | **企业级风险规避**<br>清晰的知识产权，商业使用安全无忧。                                                            | **法律风险**<br>知识产权归属模糊及使用限制风险。                         |
+
+## 保持领先
+
+在 GitHub 上为 RustFS 点赞，即可第一时间收到新版本发布通知。
+
+<img src="https://github.com/user-attachments/assets/7ee40bb4-3e46-4eac-b0d0-5fbeb85ff8f3" />
 
 ## 快速开始
 
-要开始使用 RustFS，请按照以下步骤操作：
+请按照以下步骤快速上手 RustFS：
 
-1. **一键脚本快速启动 (方案一)**
+### 1. 一键安装脚本 (选项 1)
 
-   ```bash
+```bash
-   curl -O  https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
+curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
-   ```
+```
 
-2. **Docker 快速启动（方案二）**
+### 2\. Docker 快速启动 (选项 2)
 
-  ```bash
+RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用 Docker 的 `-v` 参数挂载宿主机目录，请务必确保宿主机目录的所有者已更改为 `10001`，否则会遇到权限拒绝错误。
-   docker run -d -p 9000:9000  -v /data:/data rustfs/rustfs
-   ```
 
-对于使用 Docker 安装来讲，你还可以使用 `docker compose` 来启动 rustfs 实例。在仓库的根目录下面有一个 `docker-compose.yml`
+```bash
-文件。运行如下命令即可：
+ # 创建数据和日志目录
+ mkdir -p data logs
 
-  ```
+ # 更改这两个目录的所有者
-  docker compose --profile observability up -d
+ chown -R 10001:10001 data logs
-  ```
 
-**注意**：在使用 `docker compose` 之前，你应该仔细阅读一下 `docker-compose.yaml`，因为该文件中包含多个服务，除了 rustfs
+ # 使用最新版本运行
-以外，还有 grafana、prometheus、jaeger 等，这些是为 rustfs 可观测性服务的，还有 redis 和 nginx。你想启动哪些容器，就需要用
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
-`--profile` 参数指定相应的 profile。
 
-3. **从源码构建（方案三）- 高级用户**
+ # 使用指定版本运行
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0.alpha.68
+```
 
-   面向希望从源码构建支持多架构 Docker 镜像的开发者：
+您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：
 
-   ```bash
+```bash
-   # 本地构建多架构镜像
+docker compose --profile observability up -d
-   ./docker-buildx.sh --build-arg RELEASE=latest
+```
 
-   # 构建并推送至镜像仓库
+**注意**: 我们建议您在运行前查看 `docker-compose.yml` 文件。该文件定义了包括 Grafana、Prometheus 和 Jaeger 在内的多个服务，有助于 RustFS 的可观测性监控。如果您还想启动 Redis 或 Nginx 容器，可以指定相应的 profile。
-   ./docker-buildx.sh --push
 
-   # 构建指定版本
+### 3\. 源码编译 (选项 3) - 进阶用户
-   ./docker-buildx.sh --release v1.0.0 --push
 
-   # 构建并推送到自定义镜像仓库
+适用于希望从源码构建支持多架构 RustFS Docker 镜像的开发者：
-   ./docker-buildx.sh --registry your-registry.com --namespace yourname --push
-   ```
 
-   `docker-buildx.sh` 脚本支持：
+```bash
-    - **多架构构建**：`linux/amd64`、`linux/arm64`
+# 在本地构建多架构镜像
-    - **自动版本检测**：可使用 git 标签或提交哈希
+./docker-buildx.sh --build-arg RELEASE=latest
-    - **仓库灵活性**：支持 Docker Hub、GitHub Container Registry 等
-    - **构建优化**：包含缓存和并行构建
 
-   你也可以使用 Makefile 提供的目标命令以提升便捷性：
+# 构建并推送到仓库
+./docker-buildx.sh --push
 
-  ```bash
+# 构建指定版本
-  make docker-buildx                    # 本地构建
+./docker-buildx.sh --release v1.0.0 --push
-  make docker-buildx-push               # 构建并推送
-  make docker-buildx-version VERSION=v1.0.0  # 构建指定版本
-  make help-docker                      # 显示全部 Docker 相关命令
-  ```
 
-   > **提示（macOS 交叉编译）**：macOS 默认的 `ulimit -n` 只有 256，使用 `cargo zigbuild` 或 `./build-rustfs.sh --platform ...` 编译 Linux 目标时容易触发 `ProcessFdQuotaExceeded` 链接错误。脚本会尝试自动提升该限制，如仍提示失败，请在构建前手动执行 `ulimit -n 4096`（或更大的值）。
+# 构建并推送到自定义仓库
+./docker-buildx.sh --registry your-registry.com --namespace yourname --push
+```
 
-4. **使用 Helm Chart 部署（方案四）- 云原生环境**
+`docker-buildx.sh` 脚本支持：
+- **多架构构建**: `linux/amd64`, `linux/arm64`
+- **自动版本检测**: 使用 git tags 或 commit hash
+- **灵活的仓库支持**: 支持 Docker Hub, GitHub Container Registry 等
+- **构建优化**: 包含缓存和并行构建
 
-   按照 [helm chart 说明文档](./helm/README.md) 的指引，在 Kubernetes 集群中安装 RustFS。
+为了方便起见，您也可以使用 Make 命令：
 
-5. **访问控制台**：打开 Web 浏览器并导航到 `http://localhost:9000` 以访问 RustFS 控制台，默认的用户名和密码是
+```bash
-   `rustfsadmin` 。
+make docker-buildx                    # 本地构建
-6. **创建存储桶**：使用控制台为您的对象创建新的存储桶。
+make docker-buildx-push               # 构建并推送
-7. **上传对象**：您可以直接通过控制台上传文件，或使用 S3 兼容的 API 与您的 RustFS 实例交互。
+make docker-buildx-version VERSION=v1.0.0  # 构建指定版本
+make help-docker                      # 显示所有 Docker 相关命令
+```
 
-**注意**：如果你想通过 `https` 来访问 RustFS 实例，请参考 [TLS 配置文档](https://docs.rustfs.com/zh/integration/tls-configured.html)
+> **注意 (macOS 交叉编译)**: macOS 默认的 `ulimit -n` 限制为 256，因此在使用 `cargo zigbuild` 或 `./build-rustfs.sh --platform ...` 交叉编译 Linux 版本时，可能会因 `ProcessFdQuotaExceeded` 失败。构建脚本会尝试自动提高限制，但如果您仍然看到警告，请在构建前在终端运行 `ulimit -n 4096` (或更高)。
+
+### 4\. 使用 Helm Chart 安装 (选项 4) - 云原生环境
+
+请按照 [Helm Chart README](https://charts.rustfs.com) 上的说明在 Kubernetes 集群上安装 RustFS。
+
+### 5\. Nix Flake (Option 5)
+
+如果你已经 启用了 [Nix Flakes 功能](https://nixos.wiki/wiki/Flakes#Enable_flakes):
+
+```bash
+# 直接运行，无需安装
+nix run github:rustfs/rustfs
+
+# 编译二进制文件
+nix build github:rustfs/rustfs
+./result/bin/rustfs --help
+
+# 或者从本地检出的代码库运行/编译
+nix build
+nix run
+```
+
+### 6\. X-CMD (Option 6)
+
+如果你是 [x-cmd](https://www.x-cmd.com/install/rustfs) 用户：
+
+```bash
+# 直接运行，无需安装
+x rustfs
+
+# 下载二进制文件并安装到全局环境中
+x env use rustfs
+rustfs --help
+```
+
+---
+
+### 访问 RustFS
+
+1. **访问控制台**: 打开浏览器并访问 `http://localhost:9001` 进入 RustFS 控制台。
+   - 默认账号/密码: `rustfsadmin` / `rustfsadmin`
+2. **创建存储桶**: 使用控制台为您​​的对象创建一个新的存储桶 (Bucket)。
+3. **上传对象**: 您可以直接通过控制台上传文件，或使用 S3 兼容的 API/客户端与您的 RustFS 实例进行交互。
+
+**注意**: 如果您希望通过 `https` 访问 RustFS 实例，请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured.html)。
 
 ## 文档
 
-有关详细文档，包括配置选项、API 参考和高级用法，请访问我们的[文档](https://docs.rustfs.com)。
+有关详细文档，包括配置选项、API 参考和高级用法，请访问我们的 [官方文档](https://docs.rustfs.com)。
 
 ## 获取帮助
 
-如果您有任何问题或需要帮助，您可以：
+如果您有任何问题或需要帮助：
 
-- 查看[常见问题解答](https://github.com/rustfs/rustfs/discussions/categories/q-a)以获取常见问题和解决方案。
+- 查看 [FAQ](https://github.com/rustfs/rustfs/discussions/categories/q-a) 寻找常见问题和解决方案。
-- 加入我们的 [GitHub 讨论](https://github.com/rustfs/rustfs/discussions)来提问和分享您的经验。
+- 加入我们的 [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) 提问并分享您的经验。
-- 在我们的 [GitHub Issues](https://github.com/rustfs/rustfs/issues) 页面上开启问题，报告错误或功能请求。
+- 在我们的 [GitHub Issues](https://github.com/rustfs/rustfs/issues) 页面提交 Bug 报告或功能请求。
 
 ## 链接
 
-- [文档](https://docs.rustfs.com) - 您应该阅读的手册
+- [官方文档](https://docs.rustfs.com) - 必读手册
-- [更新日志](https://docs.rustfs.com/changelog) - 我们破坏和修复的内容
+- [更新日志](https://github.com/rustfs/rustfs/releases) - 版本变更记录
-- [GitHub 讨论](https://github.com/rustfs/rustfs/discussions) - 社区所在地
+- [社区讨论](https://github.com/rustfs/rustfs/discussions) - 社区交流地
 
-## 联系
+## 联系方式
 
-- **错误报告**：[GitHub Issues](https://github.com/rustfs/rustfs/issues)
+- **Bug 反馈**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **商务合作**：<hello@rustfs.com>
+- **商务合作**: [hello@rustfs.com](mailto:hello@rustfs.com)
-- **招聘**：<jobs@rustfs.com>
+- **工作机会**: [jobs@rustfs.com](mailto:jobs@rustfs.com)
-- **一般讨论**：[GitHub 讨论](https://github.com/rustfs/rustfs/discussions)
+- **一般讨论**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-- **贡献**：[CONTRIBUTING.md](CONTRIBUTING.md)
+- **贡献指南**: [CONTRIBUTING.md](CONTRIBUTING.md)
 
 ## 贡献者
 
-RustFS 是一个社区驱动的项目，我们感谢所有的贡献。查看[贡献者](https://github.com/rustfs/rustfs/graphs/contributors)页面，了解帮助
+RustFS 是一个社区驱动的项目，我们感谢所有的贡献。请查看 [贡献者](https://github.com/rustfs/rustfs/graphs/contributors) 页面，看看那些让 RustFS 变得更好的了不起的人们。
-RustFS 变得更好的杰出人员。
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-  <img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="贡献者"/>
+<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
-</a >
+</a>
 
-## Github 全球推荐榜
+## Star 历史
 
-🚀 RustFS 受到了全世界开源爱好者和企业用户的喜欢，多次登顶 Github Trending 全球榜。
+[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
-
-<a href="https://trendshift.io/repositories/14181" target="_blank"><img src="https://raw.githubusercontent.com/rustfs/rustfs/refs/heads/main/docs/rustfs-trending.jpg" alt="rustfs%2Frustfs | Trendshift" /></a>
-
-## Star 历史图
-
-[![Star 历史图](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
 
 ## 许可证
 

SECURITY.md

@@ -1,18 +1,40 @@
 # Security Policy
 
+## Security Philosophy
+
+At RustFS, we take security seriously. We believe that **transparency leads to better security**. The more open our code is, the more eyes are on it, and the faster we can identify and resolve potential issues.
+
+We highly value the contributions of the security community and welcome anyone to audit our code. Your efforts help us make RustFS safer for everyone.
+
 ## Supported Versions
 
-Use this section to tell people about which versions of your project are
+To help us focus our security efforts, please refer to the table below to see which versions of RustFS are currently supported with security updates.
-currently being supported with security updates.
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 1.x.x   | :white_check_mark: |
+| Latest  | :white_check_mark: |
+| < 1.0   | :x:                |
 
 ## Reporting a Vulnerability
 
-Use this section to tell people how to report a vulnerability.
+If you discover a security vulnerability in RustFS, we appreciate your help in disclosing it to us responsibly.
 
-Tell them where to go, how often they can expect to get an update on a
+**Please do not open a public GitHub issue for security vulnerabilities.** Publicly disclosing a vulnerability can put the entire community at risk before a fix is available.
-reported vulnerability, what to expect if the vulnerability is accepted or
+
-declined, etc.
+### How to Report
+
+1. https://github.com/rustfs/rustfs/security/advisories/new
+2. Please email us directly at: **security@rustfs.com**
+
+In your email, please include:
+1.  **Description**: A detailed description of the vulnerability.
+2.  **Steps to Reproduce**: Steps or a script to reproduce the issue.
+3.  **Impact**: The potential impact of the vulnerability.
+
+### Our Response Process
+
+1.  **Acknowledgment**: We will acknowledge your email within 48 hours.
+2.  **Assessment**: We will investigate the issue and determine its severity.
+3.  **Fix & Disclosure**: We will work on a patch. Once the patch is released, we will publicly announce the vulnerability and acknowledge your contribution (unless you prefer to remain anonymous).
+
+Thank you for helping keep RustFS and its users safe!

_typos.toml

@@ -36,6 +36,15 @@ clen = "clen"
 datas = "datas"
 bre = "bre"
 abd = "abd"
+mak = "mak"
+gae = "gae"
+GAE = "GAE"
+thr = "thr"
+# s3-tests original test names (cannot be changed)
+nonexisted = "nonexisted"
+consts = "consts"
+# Swift API - company/product names
+Hashi = "Hashi"  # HashiCorp
 
 [files]
-extend-exclude = []
+extend-exclude = []

build-rustfs.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 # RustFS Binary Build Script
 # This script compiles RustFS binaries for different platforms and architectures
@@ -31,7 +31,7 @@ detect_platform() {
                     echo "armv7-unknown-linux-gnueabihf"
                     ;;
                 "loongarch64")
-                    echo "loongarch64-unknown-linux-musl"
+                    echo "loongarch64-unknown-linux-gnu"
                     ;;
                 *)
                     echo "unknown-platform"
@@ -126,6 +126,7 @@ usage() {
     echo "                              Supported platforms:"
     echo "                                x86_64-unknown-linux-gnu"
     echo "                                aarch64-unknown-linux-gnu"
+    echo "                                loongarch64-unknown-linux-gnu"
     echo "                                armv7-unknown-linux-gnueabihf"
     echo "                                x86_64-unknown-linux-musl"
     echo "                                aarch64-unknown-linux-musl"
@@ -212,7 +213,7 @@ setup_rust_environment() {
     # Set up environment variables for musl targets
     if [[ "$PLATFORM" == *"musl"* ]]; then
         print_message $YELLOW "Setting up environment for musl target..."
-        export RUSTFLAGS="-C target-feature=-crt-static"
+        export RUSTFLAGS="'--cfg tokio_unstable -C target-feature=-crt-static'"
 
         # For cargo-zigbuild, set up additional environment variables
         if command -v cargo-zigbuild &> /dev/null; then
@@ -429,7 +430,7 @@ build_binary() {
         fi
     else
         # Native compilation
-        build_cmd="RUSTFLAGS=-Clink-arg=-lm cargo build"
+        build_cmd="RUSTFLAGS='--cfg tokio_unstable -Clink-arg=-lm' cargo build"
     fi
 
     if [ "$BUILD_TYPE" = "release" ]; then

crates/AGENTS.md

@@ -0,0 +1,20 @@
+# Crates Instructions
+
+Applies to all paths under `crates/`.
+
+## Library Design
+
+- Treat crate code as reusable library code by default.
+- Prefer `thiserror` for library-facing error types.
+- Do not use `unwrap()`, `expect()`, or panic-driven control flow outside tests.
+
+## Testing
+
+- Keep unit tests close to the module they test.
+- Keep integration tests under each crate's `tests/` directory.
+- Add regression tests for bug fixes and behavior changes.
+
+## Async and Performance
+
+- Keep async paths non-blocking.
+- Move CPU-heavy operations out of async hot paths with `tokio::task::spawn_blocking` when appropriate.

crates/ahm/Cargo.toml

@@ -1,43 +0,0 @@
-[package]
-name = "rustfs-ahm"
-version.workspace = true
-edition.workspace = true
-authors = ["RustFS Team"]
-license.workspace = true
-description = "RustFS AHM (Automatic Health Management) Scanner"
-repository.workspace = true
-rust-version.workspace = true
-homepage.workspace = true
-documentation = "https://docs.rs/rustfs-ahm/latest/rustfs_ahm/"
-keywords = ["RustFS", "AHM", "health-management", "scanner", "Minio"]
-categories = ["web-programming", "development-tools", "filesystem"]
-
-[dependencies]
-rustfs-ecstore = { workspace = true }
-rustfs-common = { workspace = true }
-rustfs-filemeta = { workspace = true }
-rustfs-madmin = { workspace = true }
-tokio = { workspace = true, features = ["full"] }
-tokio-util = { workspace = true }
-tracing = { workspace = true }
-serde = { workspace = true, features = ["derive"] }
-time = { workspace = true }
-serde_json = { workspace = true }
-thiserror = { workspace = true }
-uuid = { workspace = true, features = ["v4", "serde"] }
-anyhow = { workspace = true }
-async-trait = { workspace = true }
-futures = { workspace = true }
-s3s = { workspace = true }
-chrono = { workspace = true }
-rand = { workspace = true }
-reqwest = { workspace = true }
-tempfile = { workspace = true }
-walkdir = { workspace = true }
-
-[dev-dependencies]
-serde_json = { workspace = true }
-serial_test = { workspace = true }
-tracing-subscriber = { workspace = true }
-tempfile = { workspace = true }
-heed = { workspace = true }

crates/ahm/src/heal/manager.rs

@@ -1,440 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use crate::heal::{
-    progress::{HealProgress, HealStatistics},
-    storage::HealStorageAPI,
-    task::{HealOptions, HealPriority, HealRequest, HealTask, HealTaskStatus, HealType},
-};
-use crate::{Error, Result};
-use rustfs_ecstore::disk::DiskAPI;
-use rustfs_ecstore::disk::error::DiskError;
-use rustfs_ecstore::global::GLOBAL_LOCAL_DISK_MAP;
-use std::{
-    collections::{HashMap, VecDeque},
-    sync::Arc,
-    time::{Duration, SystemTime},
-};
-use tokio::{
-    sync::{Mutex, RwLock},
-    time::interval,
-};
-use tokio_util::sync::CancellationToken;
-use tracing::{error, info, warn};
-
-/// Heal config
-#[derive(Debug, Clone)]
-pub struct HealConfig {
-    /// Whether to enable auto heal
-    pub enable_auto_heal: bool,
-    /// Heal interval
-    pub heal_interval: Duration,
-    /// Maximum concurrent heal tasks
-    pub max_concurrent_heals: usize,
-    /// Task timeout
-    pub task_timeout: Duration,
-    /// Queue size
-    pub queue_size: usize,
-}
-
-impl Default for HealConfig {
-    fn default() -> Self {
-        Self {
-            enable_auto_heal: true,
-            heal_interval: Duration::from_secs(10), // 10 seconds
-            max_concurrent_heals: 4,
-            task_timeout: Duration::from_secs(300), // 5 minutes
-            queue_size: 1000,
-        }
-    }
-}
-
-/// Heal state
-#[derive(Debug, Default)]
-pub struct HealState {
-    /// Whether running
-    pub is_running: bool,
-    /// Current heal cycle
-    pub current_cycle: u64,
-    /// Last heal time
-    pub last_heal_time: Option<SystemTime>,
-    /// Total healed objects
-    pub total_healed_objects: u64,
-    /// Total heal failures
-    pub total_heal_failures: u64,
-    /// Current active heal tasks
-    pub active_heal_count: usize,
-}
-
-/// Heal manager
-pub struct HealManager {
-    /// Heal config
-    config: Arc<RwLock<HealConfig>>,
-    /// Heal state
-    state: Arc<RwLock<HealState>>,
-    /// Active heal tasks
-    active_heals: Arc<Mutex<HashMap<String, Arc<HealTask>>>>,
-    /// Heal queue
-    heal_queue: Arc<Mutex<VecDeque<HealRequest>>>,
-    /// Storage layer interface
-    storage: Arc<dyn HealStorageAPI>,
-    /// Cancel token
-    cancel_token: CancellationToken,
-    /// Statistics
-    statistics: Arc<RwLock<HealStatistics>>,
-}
-
-impl HealManager {
-    /// Create new HealManager
-    pub fn new(storage: Arc<dyn HealStorageAPI>, config: Option<HealConfig>) -> Self {
-        let config = config.unwrap_or_default();
-        Self {
-            config: Arc::new(RwLock::new(config)),
-            state: Arc::new(RwLock::new(HealState::default())),
-            active_heals: Arc::new(Mutex::new(HashMap::new())),
-            heal_queue: Arc::new(Mutex::new(VecDeque::new())),
-            storage,
-            cancel_token: CancellationToken::new(),
-            statistics: Arc::new(RwLock::new(HealStatistics::new())),
-        }
-    }
-
-    /// Start HealManager
-    pub async fn start(&self) -> Result<()> {
-        let mut state = self.state.write().await;
-        if state.is_running {
-            warn!("HealManager is already running");
-            return Ok(());
-        }
-        state.is_running = true;
-        drop(state);
-
-        info!("Starting HealManager");
-
-        // start scheduler
-        self.start_scheduler().await?;
-
-        // start auto disk scanner
-        self.start_auto_disk_scanner().await?;
-
-        info!("HealManager started successfully");
-        Ok(())
-    }
-
-    /// Stop HealManager
-    pub async fn stop(&self) -> Result<()> {
-        info!("Stopping HealManager");
-
-        // cancel all tasks
-        self.cancel_token.cancel();
-
-        // wait for all tasks to complete
-        let mut active_heals = self.active_heals.lock().await;
-        for task in active_heals.values() {
-            if let Err(e) = task.cancel().await {
-                warn!("Failed to cancel task {}: {}", task.id, e);
-            }
-        }
-        active_heals.clear();
-
-        // update state
-        let mut state = self.state.write().await;
-        state.is_running = false;
-
-        info!("HealManager stopped successfully");
-        Ok(())
-    }
-
-    /// Submit heal request
-    pub async fn submit_heal_request(&self, request: HealRequest) -> Result<String> {
-        let config = self.config.read().await;
-        let mut queue = self.heal_queue.lock().await;
-
-        if queue.len() >= config.queue_size {
-            return Err(Error::ConfigurationError {
-                message: "Heal queue is full".to_string(),
-            });
-        }
-
-        let request_id = request.id.clone();
-        queue.push_back(request);
-        drop(queue);
-
-        info!("Submitted heal request: {}", request_id);
-        Ok(request_id)
-    }
-
-    /// Get task status
-    pub async fn get_task_status(&self, task_id: &str) -> Result<HealTaskStatus> {
-        let active_heals = self.active_heals.lock().await;
-        if let Some(task) = active_heals.get(task_id) {
-            Ok(task.get_status().await)
-        } else {
-            Err(Error::TaskNotFound {
-                task_id: task_id.to_string(),
-            })
-        }
-    }
-
-    /// Get task progress
-    pub async fn get_active_tasks_count(&self) -> usize {
-        self.active_heals.lock().await.len()
-    }
-
-    pub async fn get_task_progress(&self, task_id: &str) -> Result<HealProgress> {
-        let active_heals = self.active_heals.lock().await;
-        if let Some(task) = active_heals.get(task_id) {
-            Ok(task.get_progress().await)
-        } else {
-            Err(Error::TaskNotFound {
-                task_id: task_id.to_string(),
-            })
-        }
-    }
-
-    /// Cancel task
-    pub async fn cancel_task(&self, task_id: &str) -> Result<()> {
-        let mut active_heals = self.active_heals.lock().await;
-        if let Some(task) = active_heals.get(task_id) {
-            task.cancel().await?;
-            active_heals.remove(task_id);
-            info!("Cancelled heal task: {}", task_id);
-            Ok(())
-        } else {
-            Err(Error::TaskNotFound {
-                task_id: task_id.to_string(),
-            })
-        }
-    }
-
-    /// Get statistics
-    pub async fn get_statistics(&self) -> HealStatistics {
-        self.statistics.read().await.clone()
-    }
-
-    /// Get active task count
-    pub async fn get_active_task_count(&self) -> usize {
-        let active_heals = self.active_heals.lock().await;
-        active_heals.len()
-    }
-
-    /// Get queue length
-    pub async fn get_queue_length(&self) -> usize {
-        let queue = self.heal_queue.lock().await;
-        queue.len()
-    }
-
-    /// Start scheduler
-    async fn start_scheduler(&self) -> Result<()> {
-        let config = self.config.clone();
-        let heal_queue = self.heal_queue.clone();
-        let active_heals = self.active_heals.clone();
-        let cancel_token = self.cancel_token.clone();
-        let statistics = self.statistics.clone();
-        let storage = self.storage.clone();
-
-        tokio::spawn(async move {
-            let mut interval = interval(config.read().await.heal_interval);
-
-            loop {
-                tokio::select! {
-                    _ = cancel_token.cancelled() => {
-                        info!("Heal scheduler received shutdown signal");
-                        break;
-                    }
-                    _ = interval.tick() => {
-                        Self::process_heal_queue(&heal_queue, &active_heals, &config, &statistics, &storage).await;
-                    }
-                }
-            }
-        });
-
-        Ok(())
-    }
-
-    /// Start background task to auto scan local disks and enqueue erasure set heal requests
-    async fn start_auto_disk_scanner(&self) -> Result<()> {
-        let config = self.config.clone();
-        let heal_queue = self.heal_queue.clone();
-        let active_heals = self.active_heals.clone();
-        let cancel_token = self.cancel_token.clone();
-        let storage = self.storage.clone();
-
-        tokio::spawn(async move {
-            let mut interval = interval(config.read().await.heal_interval);
-
-            loop {
-                tokio::select! {
-                    _ = cancel_token.cancelled() => {
-                        info!("Auto disk scanner received shutdown signal");
-                        break;
-                    }
-                    _ = interval.tick() => {
-                        // Build list of endpoints that need healing
-                        let mut endpoints = Vec::new();
-                        for (_, disk_opt) in GLOBAL_LOCAL_DISK_MAP.read().await.iter() {
-                            if let Some(disk) = disk_opt {
-                                // detect unformatted disk via get_disk_id()
-                                if let Err(err) = disk.get_disk_id().await {
-                                    if err == DiskError::UnformattedDisk {
-                                        endpoints.push(disk.endpoint());
-                                        continue;
-                                    }
-                                }
-                            }
-                        }
-
-                        if endpoints.is_empty() {
-                            continue;
-                        }
-
-                        // Get bucket list for erasure set healing
-                        let buckets = match storage.list_buckets().await {
-                            Ok(buckets) => buckets.iter().map(|b| b.name.clone()).collect::<Vec<String>>(),
-                            Err(e) => {
-                                error!("Failed to get bucket list for auto healing: {}", e);
-                                continue;
-                            }
-                        };
-
-                        // Create erasure set heal requests for each endpoint
-                        for ep in endpoints {
-                            let Some(set_disk_id) =
-                                crate::heal::utils::format_set_disk_id_from_i32(ep.pool_idx, ep.set_idx)
-                            else {
-                                warn!("Skipping endpoint {} without valid pool/set index", ep);
-                                continue;
-                            };
-                            // skip if already queued or healing
-                            // Use consistent lock order: queue first, then active_heals to avoid deadlock
-                            let mut skip = false;
-                            {
-                                let queue = heal_queue.lock().await;
-                                if queue.iter().any(|req| {
-                                    matches!(
-                                        &req.heal_type,
-                                        crate::heal::task::HealType::ErasureSet { set_disk_id: queued_id, .. }
-                                        if queued_id == &set_disk_id
-                                    )
-                                }) {
-                                    skip = true;
-                                }
-                            }
-                            if !skip {
-                                let active = active_heals.lock().await;
-                                if active.values().any(|task| {
-                                    matches!(
-                                        &task.heal_type,
-                                        crate::heal::task::HealType::ErasureSet { set_disk_id: active_id, .. }
-                                        if active_id == &set_disk_id
-                                    )
-                                }) {
-                                    skip = true;
-                                }
-                            }
-
-                            if skip {
-                                continue;
-                            }
-
-                            // enqueue erasure set heal request for this disk
-                            let req = HealRequest::new(
-                                HealType::ErasureSet {
-                                    buckets: buckets.clone(),
-                                    set_disk_id: set_disk_id.clone(),
-                                },
-                                HealOptions::default(),
-                                HealPriority::Normal,
-                            );
-                            let mut queue = heal_queue.lock().await;
-                            queue.push_back(req);
-                            info!("Enqueued auto erasure set heal for endpoint: {} (set_disk_id: {})", ep, set_disk_id);
-                        }
-                    }
-                }
-            }
-        });
-        Ok(())
-    }
-
-    /// Process heal queue
-    async fn process_heal_queue(
-        heal_queue: &Arc<Mutex<VecDeque<HealRequest>>>,
-        active_heals: &Arc<Mutex<HashMap<String, Arc<HealTask>>>>,
-        config: &Arc<RwLock<HealConfig>>,
-        statistics: &Arc<RwLock<HealStatistics>>,
-        storage: &Arc<dyn HealStorageAPI>,
-    ) {
-        let config = config.read().await;
-        let mut active_heals_guard = active_heals.lock().await;
-
-        // check if new heal tasks can be started
-        if active_heals_guard.len() >= config.max_concurrent_heals {
-            return;
-        }
-
-        let mut queue = heal_queue.lock().await;
-        if let Some(request) = queue.pop_front() {
-            let task = Arc::new(HealTask::from_request(request, storage.clone()));
-            let task_id = task.id.clone();
-            active_heals_guard.insert(task_id.clone(), task.clone());
-            drop(active_heals_guard);
-            let active_heals_clone = active_heals.clone();
-            let statistics_clone = statistics.clone();
-
-            // start heal task
-            tokio::spawn(async move {
-                info!("Starting heal task: {}", task_id);
-                let result = task.execute().await;
-                match result {
-                    Ok(_) => {
-                        info!("Heal task completed successfully: {}", task_id);
-                    }
-                    Err(e) => {
-                        error!("Heal task failed: {} - {}", task_id, e);
-                    }
-                }
-                let mut active_heals_guard = active_heals_clone.lock().await;
-                if let Some(completed_task) = active_heals_guard.remove(&task_id) {
-                    // update statistics
-                    let mut stats = statistics_clone.write().await;
-                    match completed_task.get_status().await {
-                        HealTaskStatus::Completed => {
-                            stats.update_task_completion(true);
-                        }
-                        _ => {
-                            stats.update_task_completion(false);
-                        }
-                    }
-                    stats.update_running_tasks(active_heals_guard.len() as u64);
-                }
-            });
-
-            // update statistics
-            let mut stats = statistics.write().await;
-            stats.total_tasks += 1;
-        }
-    }
-}
-
-impl std::fmt::Debug for HealManager {
-    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        f.debug_struct("HealManager")
-            .field("config", &"<config>")
-            .field("state", &"<state>")
-            .field("active_heals_count", &"<active_heals>")
-            .field("queue_length", &"<queue>")
-            .finish()
-    }
-}

crates/ahm/src/scanner/checkpoint.rs

@@ -1,326 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use crate::scanner::node_scanner::ScanProgress;
-use crate::{Error, Result};
-use serde::{Deserialize, Serialize};
-use std::{
-    path::{Path, PathBuf},
-    time::{Duration, SystemTime},
-};
-use tokio::sync::RwLock;
-use tracing::{debug, error, info, warn};
-
-#[derive(Debug, Serialize, Deserialize, Clone)]
-pub struct CheckpointData {
-    pub version: u32,
-    pub timestamp: SystemTime,
-    pub progress: ScanProgress,
-    pub node_id: String,
-    pub checksum: u64,
-}
-
-impl CheckpointData {
-    pub fn new(progress: ScanProgress, node_id: String) -> Self {
-        let mut checkpoint = Self {
-            version: 1,
-            timestamp: SystemTime::now(),
-            progress,
-            node_id,
-            checksum: 0,
-        };
-
-        checkpoint.checksum = checkpoint.calculate_checksum();
-        checkpoint
-    }
-
-    fn calculate_checksum(&self) -> u64 {
-        use std::collections::hash_map::DefaultHasher;
-        use std::hash::{Hash, Hasher};
-
-        let mut hasher = DefaultHasher::new();
-        self.version.hash(&mut hasher);
-        self.node_id.hash(&mut hasher);
-        self.progress.current_cycle.hash(&mut hasher);
-        self.progress.current_disk_index.hash(&mut hasher);
-
-        if let Some(ref bucket) = self.progress.current_bucket {
-            bucket.hash(&mut hasher);
-        }
-
-        if let Some(ref key) = self.progress.last_scan_key {
-            key.hash(&mut hasher);
-        }
-
-        hasher.finish()
-    }
-
-    pub fn verify_integrity(&self) -> bool {
-        let calculated_checksum = self.calculate_checksum();
-        self.checksum == calculated_checksum
-    }
-}
-
-pub struct CheckpointManager {
-    checkpoint_file: PathBuf,
-    backup_file: PathBuf,
-    temp_file: PathBuf,
-    save_interval: Duration,
-    last_save: RwLock<SystemTime>,
-    node_id: String,
-}
-
-impl CheckpointManager {
-    pub fn new(node_id: &str, data_dir: &Path) -> Self {
-        if !data_dir.exists() {
-            if let Err(e) = std::fs::create_dir_all(data_dir) {
-                error!("create data dir failed {:?}: {}", data_dir, e);
-            }
-        }
-
-        let checkpoint_file = data_dir.join(format!("scanner_checkpoint_{node_id}.json"));
-        let backup_file = data_dir.join(format!("scanner_checkpoint_{node_id}.backup"));
-        let temp_file = data_dir.join(format!("scanner_checkpoint_{node_id}.tmp"));
-
-        Self {
-            checkpoint_file,
-            backup_file,
-            temp_file,
-            save_interval: Duration::from_secs(30), // 30s
-            last_save: RwLock::new(SystemTime::UNIX_EPOCH),
-            node_id: node_id.to_string(),
-        }
-    }
-
-    pub async fn save_checkpoint(&self, progress: &ScanProgress) -> Result<()> {
-        let now = SystemTime::now();
-        let last_save = *self.last_save.read().await;
-
-        if now.duration_since(last_save).unwrap_or(Duration::ZERO) < self.save_interval {
-            return Ok(());
-        }
-
-        let checkpoint_data = CheckpointData::new(progress.clone(), self.node_id.clone());
-
-        let json_data = serde_json::to_string_pretty(&checkpoint_data)
-            .map_err(|e| Error::Serialization(format!("serialize checkpoint failed: {e}")))?;
-
-        tokio::fs::write(&self.temp_file, json_data)
-            .await
-            .map_err(|e| Error::IO(format!("write temp checkpoint file failed: {e}")))?;
-
-        if self.checkpoint_file.exists() {
-            tokio::fs::copy(&self.checkpoint_file, &self.backup_file)
-                .await
-                .map_err(|e| Error::IO(format!("backup checkpoint file failed: {e}")))?;
-        }
-
-        tokio::fs::rename(&self.temp_file, &self.checkpoint_file)
-            .await
-            .map_err(|e| Error::IO(format!("replace checkpoint file failed: {e}")))?;
-
-        *self.last_save.write().await = now;
-
-        debug!(
-            "save checkpoint to {:?}, cycle: {}, disk index: {}",
-            self.checkpoint_file, checkpoint_data.progress.current_cycle, checkpoint_data.progress.current_disk_index
-        );
-
-        Ok(())
-    }
-
-    pub async fn load_checkpoint(&self) -> Result<Option<ScanProgress>> {
-        // first try main checkpoint file
-        match self.load_checkpoint_from_file(&self.checkpoint_file).await {
-            Ok(checkpoint) => {
-                info!(
-                    "restore scan progress from main checkpoint file: cycle={}, disk index={}, last scan key={:?}",
-                    checkpoint.current_cycle, checkpoint.current_disk_index, checkpoint.last_scan_key
-                );
-                Ok(Some(checkpoint))
-            }
-            Err(e) => {
-                warn!("main checkpoint file is corrupted or not exists: {}", e);
-
-                // try backup file
-                match self.load_checkpoint_from_file(&self.backup_file).await {
-                    Ok(checkpoint) => {
-                        warn!(
-                            "restore scan progress from backup file: cycle={}, disk index={}",
-                            checkpoint.current_cycle, checkpoint.current_disk_index
-                        );
-
-                        // copy backup file to main checkpoint file
-                        if let Err(copy_err) = tokio::fs::copy(&self.backup_file, &self.checkpoint_file).await {
-                            warn!("restore main checkpoint file failed: {}", copy_err);
-                        }
-
-                        Ok(Some(checkpoint))
-                    }
-                    Err(backup_e) => {
-                        warn!("backup file is corrupted or not exists: {}", backup_e);
-                        info!("cannot restore scan progress, will start fresh scan");
-                        Ok(None)
-                    }
-                }
-            }
-        }
-    }
-
-    /// load checkpoint from file
-    async fn load_checkpoint_from_file(&self, file_path: &Path) -> Result<ScanProgress> {
-        if !file_path.exists() {
-            return Err(Error::NotFound(format!("checkpoint file not exists: {file_path:?}")));
-        }
-
-        // read file content
-        let content = tokio::fs::read_to_string(file_path)
-            .await
-            .map_err(|e| Error::IO(format!("read checkpoint file failed: {e}")))?;
-
-        // deserialize
-        let checkpoint_data: CheckpointData =
-            serde_json::from_str(&content).map_err(|e| Error::Serialization(format!("deserialize checkpoint failed: {e}")))?;
-
-        // validate checkpoint data
-        self.validate_checkpoint(&checkpoint_data)?;
-
-        Ok(checkpoint_data.progress)
-    }
-
-    /// validate checkpoint data
-    fn validate_checkpoint(&self, checkpoint: &CheckpointData) -> Result<()> {
-        // validate data integrity
-        if !checkpoint.verify_integrity() {
-            return Err(Error::InvalidCheckpoint(
-                "checkpoint data verification failed, may be corrupted".to_string(),
-            ));
-        }
-
-        // validate node id match
-        if checkpoint.node_id != self.node_id {
-            return Err(Error::InvalidCheckpoint(format!(
-                "checkpoint node id not match: expected {}, actual {}",
-                self.node_id, checkpoint.node_id
-            )));
-        }
-
-        let now = SystemTime::now();
-        let checkpoint_age = now.duration_since(checkpoint.timestamp).unwrap_or(Duration::MAX);
-
-        // checkpoint is too old (more than 24 hours), may be data expired
-        if checkpoint_age > Duration::from_secs(24 * 3600) {
-            return Err(Error::InvalidCheckpoint(format!("checkpoint data is too old: {checkpoint_age:?}")));
-        }
-
-        // validate version compatibility
-        if checkpoint.version > 1 {
-            return Err(Error::InvalidCheckpoint(format!(
-                "unsupported checkpoint version: {}",
-                checkpoint.version
-            )));
-        }
-
-        Ok(())
-    }
-
-    /// clean checkpoint file
-    ///
-    /// called when scanner stops or resets
-    pub async fn cleanup_checkpoint(&self) -> Result<()> {
-        // delete main file
-        if self.checkpoint_file.exists() {
-            tokio::fs::remove_file(&self.checkpoint_file)
-                .await
-                .map_err(|e| Error::IO(format!("delete main checkpoint file failed: {e}")))?;
-        }
-
-        // delete backup file
-        if self.backup_file.exists() {
-            tokio::fs::remove_file(&self.backup_file)
-                .await
-                .map_err(|e| Error::IO(format!("delete backup checkpoint file failed: {e}")))?;
-        }
-
-        // delete temp file
-        if self.temp_file.exists() {
-            tokio::fs::remove_file(&self.temp_file)
-                .await
-                .map_err(|e| Error::IO(format!("delete temp checkpoint file failed: {e}")))?;
-        }
-
-        info!("cleaned up all checkpoint files");
-        Ok(())
-    }
-
-    /// get checkpoint file info
-    pub async fn get_checkpoint_info(&self) -> Result<Option<CheckpointInfo>> {
-        if !self.checkpoint_file.exists() {
-            return Ok(None);
-        }
-
-        let metadata = tokio::fs::metadata(&self.checkpoint_file)
-            .await
-            .map_err(|e| Error::IO(format!("get checkpoint file metadata failed: {e}")))?;
-
-        let content = tokio::fs::read_to_string(&self.checkpoint_file)
-            .await
-            .map_err(|e| Error::IO(format!("read checkpoint file failed: {e}")))?;
-
-        let checkpoint_data: CheckpointData =
-            serde_json::from_str(&content).map_err(|e| Error::Serialization(format!("deserialize checkpoint failed: {e}")))?;
-
-        Ok(Some(CheckpointInfo {
-            file_size: metadata.len(),
-            last_modified: metadata.modified().unwrap_or(SystemTime::UNIX_EPOCH),
-            checkpoint_timestamp: checkpoint_data.timestamp,
-            current_cycle: checkpoint_data.progress.current_cycle,
-            current_disk_index: checkpoint_data.progress.current_disk_index,
-            completed_disks_count: checkpoint_data.progress.completed_disks.len(),
-            is_valid: checkpoint_data.verify_integrity(),
-        }))
-    }
-
-    /// force save checkpoint (ignore time interval limit)
-    pub async fn force_save_checkpoint(&self, progress: &ScanProgress) -> Result<()> {
-        // temporarily reset last save time, force save
-        *self.last_save.write().await = SystemTime::UNIX_EPOCH;
-        self.save_checkpoint(progress).await
-    }
-
-    /// set save interval
-    pub async fn set_save_interval(&mut self, interval: Duration) {
-        self.save_interval = interval;
-        info!("checkpoint save interval set to: {:?}", interval);
-    }
-}
-
-/// checkpoint info
-#[derive(Debug, Clone)]
-pub struct CheckpointInfo {
-    /// file size
-    pub file_size: u64,
-    /// file last modified time
-    pub last_modified: SystemTime,
-    /// checkpoint creation time
-    pub checkpoint_timestamp: SystemTime,
-    /// current scan cycle
-    pub current_cycle: u64,
-    /// current disk index
-    pub current_disk_index: usize,
-    /// completed disks count
-    pub completed_disks_count: usize,
-    /// checkpoint is valid
-    pub is_valid: bool,
-}

crates/ahm/src/scanner/histogram.rs

@@ -1,305 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use serde::{Deserialize, Serialize};
-use std::{
-    collections::HashMap,
-    sync::atomic::{AtomicU64, Ordering},
-    time::{Duration, SystemTime},
-};
-use tracing::info;
-
-/// Scanner metrics
-#[derive(Debug, Clone, Default, Serialize, Deserialize)]
-pub struct ScannerMetrics {
-    /// Total objects scanned since server start
-    pub objects_scanned: u64,
-    /// Total object versions scanned since server start
-    pub versions_scanned: u64,
-    /// Total directories scanned since server start
-    pub directories_scanned: u64,
-    /// Total bucket scans started since server start
-    pub bucket_scans_started: u64,
-    /// Total bucket scans finished since server start
-    pub bucket_scans_finished: u64,
-    /// Total objects with health issues found
-    pub objects_with_issues: u64,
-    /// Total heal tasks queued
-    pub heal_tasks_queued: u64,
-    /// Total heal tasks completed
-    pub heal_tasks_completed: u64,
-    /// Total heal tasks failed
-    pub heal_tasks_failed: u64,
-    /// Total healthy objects found
-    pub healthy_objects: u64,
-    /// Total corrupted objects found
-    pub corrupted_objects: u64,
-    /// Last scan activity time
-    pub last_activity: Option<SystemTime>,
-    /// Current scan cycle
-    pub current_cycle: u64,
-    /// Total scan cycles completed
-    pub total_cycles: u64,
-    /// Current scan duration
-    pub current_scan_duration: Option<Duration>,
-    /// Average scan duration
-    pub avg_scan_duration: Duration,
-    /// Objects scanned per second
-    pub objects_per_second: f64,
-    /// Buckets scanned per second
-    pub buckets_per_second: f64,
-    /// Storage metrics by bucket
-    pub bucket_metrics: HashMap<String, BucketMetrics>,
-    /// Disk metrics
-    pub disk_metrics: HashMap<String, DiskMetrics>,
-}
-
-/// Bucket-specific metrics
-#[derive(Debug, Clone, Default, Serialize, Deserialize)]
-pub struct BucketMetrics {
-    /// Bucket name
-    pub bucket: String,
-    /// Total objects in bucket
-    pub total_objects: u64,
-    /// Total size of objects in bucket (bytes)
-    pub total_size: u64,
-    /// Objects with health issues
-    pub objects_with_issues: u64,
-    /// Last scan time
-    pub last_scan_time: Option<SystemTime>,
-    /// Scan duration
-    pub scan_duration: Option<Duration>,
-    /// Heal tasks queued for this bucket
-    pub heal_tasks_queued: u64,
-    /// Heal tasks completed for this bucket
-    pub heal_tasks_completed: u64,
-    /// Heal tasks failed for this bucket
-    pub heal_tasks_failed: u64,
-}
-
-/// Disk-specific metrics
-#[derive(Debug, Clone, Default, Serialize, Deserialize)]
-pub struct DiskMetrics {
-    /// Disk path
-    pub disk_path: String,
-    /// Total disk space (bytes)
-    pub total_space: u64,
-    /// Used disk space (bytes)
-    pub used_space: u64,
-    /// Free disk space (bytes)
-    pub free_space: u64,
-    /// Objects scanned on this disk
-    pub objects_scanned: u64,
-    /// Objects with issues on this disk
-    pub objects_with_issues: u64,
-    /// Last scan time
-    pub last_scan_time: Option<SystemTime>,
-    /// Whether disk is online
-    pub is_online: bool,
-    /// Whether disk is being scanned
-    pub is_scanning: bool,
-}
-
-/// Thread-safe metrics collector
-pub struct MetricsCollector {
-    /// Atomic counters for real-time metrics
-    objects_scanned: AtomicU64,
-    versions_scanned: AtomicU64,
-    directories_scanned: AtomicU64,
-    bucket_scans_started: AtomicU64,
-    bucket_scans_finished: AtomicU64,
-    objects_with_issues: AtomicU64,
-    heal_tasks_queued: AtomicU64,
-    heal_tasks_completed: AtomicU64,
-    heal_tasks_failed: AtomicU64,
-    current_cycle: AtomicU64,
-    total_cycles: AtomicU64,
-    healthy_objects: AtomicU64,
-    corrupted_objects: AtomicU64,
-}
-
-impl MetricsCollector {
-    /// Create a new metrics collector
-    pub fn new() -> Self {
-        Self {
-            objects_scanned: AtomicU64::new(0),
-            versions_scanned: AtomicU64::new(0),
-            directories_scanned: AtomicU64::new(0),
-            bucket_scans_started: AtomicU64::new(0),
-            bucket_scans_finished: AtomicU64::new(0),
-            objects_with_issues: AtomicU64::new(0),
-            heal_tasks_queued: AtomicU64::new(0),
-            heal_tasks_completed: AtomicU64::new(0),
-            heal_tasks_failed: AtomicU64::new(0),
-            current_cycle: AtomicU64::new(0),
-            total_cycles: AtomicU64::new(0),
-            healthy_objects: AtomicU64::new(0),
-            corrupted_objects: AtomicU64::new(0),
-        }
-    }
-
-    /// Increment objects scanned count
-    pub fn increment_objects_scanned(&self, count: u64) {
-        self.objects_scanned.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Increment versions scanned count
-    pub fn increment_versions_scanned(&self, count: u64) {
-        self.versions_scanned.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Increment directories scanned count
-    pub fn increment_directories_scanned(&self, count: u64) {
-        self.directories_scanned.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Increment bucket scans started count
-    pub fn increment_bucket_scans_started(&self, count: u64) {
-        self.bucket_scans_started.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Increment bucket scans finished count
-    pub fn increment_bucket_scans_finished(&self, count: u64) {
-        self.bucket_scans_finished.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Increment objects with issues count
-    pub fn increment_objects_with_issues(&self, count: u64) {
-        self.objects_with_issues.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Increment heal tasks queued count
-    pub fn increment_heal_tasks_queued(&self, count: u64) {
-        self.heal_tasks_queued.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Increment heal tasks completed count
-    pub fn increment_heal_tasks_completed(&self, count: u64) {
-        self.heal_tasks_completed.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Increment heal tasks failed count
-    pub fn increment_heal_tasks_failed(&self, count: u64) {
-        self.heal_tasks_failed.fetch_add(count, Ordering::Relaxed);
-    }
-
-    /// Set current cycle
-    pub fn set_current_cycle(&self, cycle: u64) {
-        self.current_cycle.store(cycle, Ordering::Relaxed);
-    }
-
-    /// Increment total cycles
-    pub fn increment_total_cycles(&self) {
-        self.total_cycles.fetch_add(1, Ordering::Relaxed);
-    }
-
-    /// Increment healthy objects count
-    pub fn increment_healthy_objects(&self) {
-        self.healthy_objects.fetch_add(1, Ordering::Relaxed);
-    }
-
-    /// Increment corrupted objects count
-    pub fn increment_corrupted_objects(&self) {
-        self.corrupted_objects.fetch_add(1, Ordering::Relaxed);
-    }
-
-    /// Get current metrics snapshot
-    pub fn get_metrics(&self) -> ScannerMetrics {
-        ScannerMetrics {
-            objects_scanned: self.objects_scanned.load(Ordering::Relaxed),
-            versions_scanned: self.versions_scanned.load(Ordering::Relaxed),
-            directories_scanned: self.directories_scanned.load(Ordering::Relaxed),
-            bucket_scans_started: self.bucket_scans_started.load(Ordering::Relaxed),
-            bucket_scans_finished: self.bucket_scans_finished.load(Ordering::Relaxed),
-            objects_with_issues: self.objects_with_issues.load(Ordering::Relaxed),
-            heal_tasks_queued: self.heal_tasks_queued.load(Ordering::Relaxed),
-            heal_tasks_completed: self.heal_tasks_completed.load(Ordering::Relaxed),
-            heal_tasks_failed: self.heal_tasks_failed.load(Ordering::Relaxed),
-            healthy_objects: self.healthy_objects.load(Ordering::Relaxed),
-            corrupted_objects: self.corrupted_objects.load(Ordering::Relaxed),
-            last_activity: Some(SystemTime::now()),
-            current_cycle: self.current_cycle.load(Ordering::Relaxed),
-            total_cycles: self.total_cycles.load(Ordering::Relaxed),
-            current_scan_duration: None,       // Will be set by scanner
-            avg_scan_duration: Duration::ZERO, // Will be calculated
-            objects_per_second: 0.0,           // Will be calculated
-            buckets_per_second: 0.0,           // Will be calculated
-            bucket_metrics: HashMap::new(),    // Will be populated by scanner
-            disk_metrics: HashMap::new(),      // Will be populated by scanner
-        }
-    }
-
-    /// Reset all metrics
-    pub fn reset(&self) {
-        self.objects_scanned.store(0, Ordering::Relaxed);
-        self.versions_scanned.store(0, Ordering::Relaxed);
-        self.directories_scanned.store(0, Ordering::Relaxed);
-        self.bucket_scans_started.store(0, Ordering::Relaxed);
-        self.bucket_scans_finished.store(0, Ordering::Relaxed);
-        self.objects_with_issues.store(0, Ordering::Relaxed);
-        self.heal_tasks_queued.store(0, Ordering::Relaxed);
-        self.heal_tasks_completed.store(0, Ordering::Relaxed);
-        self.heal_tasks_failed.store(0, Ordering::Relaxed);
-        self.current_cycle.store(0, Ordering::Relaxed);
-        self.total_cycles.store(0, Ordering::Relaxed);
-        self.healthy_objects.store(0, Ordering::Relaxed);
-        self.corrupted_objects.store(0, Ordering::Relaxed);
-
-        info!("Scanner metrics reset");
-    }
-}
-
-impl Default for MetricsCollector {
-    fn default() -> Self {
-        Self::new()
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn test_metrics_collector_creation() {
-        let collector = MetricsCollector::new();
-        let metrics = collector.get_metrics();
-        assert_eq!(metrics.objects_scanned, 0);
-        assert_eq!(metrics.versions_scanned, 0);
-    }
-
-    #[test]
-    fn test_metrics_increment() {
-        let collector = MetricsCollector::new();
-
-        collector.increment_objects_scanned(10);
-        collector.increment_versions_scanned(5);
-        collector.increment_objects_with_issues(2);
-
-        let metrics = collector.get_metrics();
-        assert_eq!(metrics.objects_scanned, 10);
-        assert_eq!(metrics.versions_scanned, 5);
-        assert_eq!(metrics.objects_with_issues, 2);
-    }
-
-    #[test]
-    fn test_metrics_reset() {
-        let collector = MetricsCollector::new();
-
-        collector.increment_objects_scanned(10);
-        collector.reset();
-
-        let metrics = collector.get_metrics();
-        assert_eq!(metrics.objects_scanned, 0);
-    }
-}

crates/ahm/src/scanner/io_monitor.rs

@@ -1,555 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use crate::Result;
-use crate::scanner::LoadLevel;
-use serde::{Deserialize, Serialize};
-use std::{
-    collections::VecDeque,
-    sync::{
-        Arc,
-        atomic::{AtomicU64, Ordering},
-    },
-    time::{Duration, SystemTime},
-};
-use tokio::sync::RwLock;
-use tokio_util::sync::CancellationToken;
-use tracing::{debug, error, info, warn};
-
-/// IO monitor config   
-#[derive(Debug, Clone)]
-pub struct IOMonitorConfig {
-    /// monitor interval
-    pub monitor_interval: Duration,
-    /// history data retention time
-    pub history_retention: Duration,
-    /// load evaluation window size
-    pub load_window_size: usize,
-    /// whether to enable actual system monitoring
-    pub enable_system_monitoring: bool,
-    /// disk path list (for monitoring specific disks)
-    pub disk_paths: Vec<String>,
-}
-
-impl Default for IOMonitorConfig {
-    fn default() -> Self {
-        Self {
-            monitor_interval: Duration::from_secs(1),    // 1 second monitor interval
-            history_retention: Duration::from_secs(300), // keep 5 minutes history
-            load_window_size: 30,                        // 30 sample points sliding window
-            enable_system_monitoring: false,             // default use simulated data
-            disk_paths: Vec::new(),
-        }
-    }
-}
-
-/// IO monitor metrics
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct IOMetrics {
-    /// timestamp
-    pub timestamp: SystemTime,
-    /// disk IOPS (read + write)
-    pub iops: u64,
-    /// read IOPS
-    pub read_iops: u64,
-    /// write IOPS
-    pub write_iops: u64,
-    /// disk queue depth
-    pub queue_depth: u64,
-    /// average latency (milliseconds)
-    pub avg_latency: u64,
-    /// read latency (milliseconds)
-    pub read_latency: u64,
-    /// write latency (milliseconds)
-    pub write_latency: u64,
-    /// CPU usage (0-100)
-    pub cpu_usage: u8,
-    /// memory usage (0-100)
-    pub memory_usage: u8,
-    /// disk usage (0-100)
-    pub disk_utilization: u8,
-    /// network IO (Mbps)
-    pub network_io: u64,
-}
-
-impl Default for IOMetrics {
-    fn default() -> Self {
-        Self {
-            timestamp: SystemTime::now(),
-            iops: 0,
-            read_iops: 0,
-            write_iops: 0,
-            queue_depth: 0,
-            avg_latency: 0,
-            read_latency: 0,
-            write_latency: 0,
-            cpu_usage: 0,
-            memory_usage: 0,
-            disk_utilization: 0,
-            network_io: 0,
-        }
-    }
-}
-
-/// load level stats
-#[derive(Debug, Clone, Default)]
-pub struct LoadLevelStats {
-    /// low load duration (seconds)
-    pub low_load_duration: u64,
-    /// medium load duration (seconds)
-    pub medium_load_duration: u64,
-    /// high load duration (seconds)
-    pub high_load_duration: u64,
-    /// critical load duration (seconds)
-    pub critical_load_duration: u64,
-    /// load transitions
-    pub load_transitions: u64,
-}
-
-/// advanced IO monitor
-pub struct AdvancedIOMonitor {
-    /// config
-    config: Arc<RwLock<IOMonitorConfig>>,
-    /// current metrics
-    current_metrics: Arc<RwLock<IOMetrics>>,
-    /// history metrics (sliding window)
-    history_metrics: Arc<RwLock<VecDeque<IOMetrics>>>,
-    /// current load level
-    current_load_level: Arc<RwLock<LoadLevel>>,
-    /// load level history
-    load_level_history: Arc<RwLock<VecDeque<(SystemTime, LoadLevel)>>>,
-    /// load level stats
-    load_stats: Arc<RwLock<LoadLevelStats>>,
-    /// business IO metrics (updated by external)
-    business_metrics: Arc<BusinessIOMetrics>,
-    /// cancel token
-    cancel_token: CancellationToken,
-}
-
-/// business IO metrics
-pub struct BusinessIOMetrics {
-    /// business request latency (milliseconds)
-    pub request_latency: AtomicU64,
-    /// business request QPS
-    pub request_qps: AtomicU64,
-    /// business error rate (0-10000, 0.00%-100.00%)
-    pub error_rate: AtomicU64,
-    /// active connections
-    pub active_connections: AtomicU64,
-    /// last update time
-    pub last_update: Arc<RwLock<SystemTime>>,
-}
-
-impl Default for BusinessIOMetrics {
-    fn default() -> Self {
-        Self {
-            request_latency: AtomicU64::new(0),
-            request_qps: AtomicU64::new(0),
-            error_rate: AtomicU64::new(0),
-            active_connections: AtomicU64::new(0),
-            last_update: Arc::new(RwLock::new(SystemTime::UNIX_EPOCH)),
-        }
-    }
-}
-
-impl AdvancedIOMonitor {
-    /// create new advanced IO monitor
-    pub fn new(config: IOMonitorConfig) -> Self {
-        Self {
-            config: Arc::new(RwLock::new(config)),
-            current_metrics: Arc::new(RwLock::new(IOMetrics::default())),
-            history_metrics: Arc::new(RwLock::new(VecDeque::new())),
-            current_load_level: Arc::new(RwLock::new(LoadLevel::Low)),
-            load_level_history: Arc::new(RwLock::new(VecDeque::new())),
-            load_stats: Arc::new(RwLock::new(LoadLevelStats::default())),
-            business_metrics: Arc::new(BusinessIOMetrics::default()),
-            cancel_token: CancellationToken::new(),
-        }
-    }
-
-    /// start monitoring
-    pub async fn start(&self) -> Result<()> {
-        info!("start advanced IO monitor");
-
-        let monitor = self.clone_for_background();
-        tokio::spawn(async move {
-            if let Err(e) = monitor.monitoring_loop().await {
-                error!("IO monitoring loop failed: {}", e);
-            }
-        });
-
-        Ok(())
-    }
-
-    /// stop monitoring
-    pub async fn stop(&self) {
-        info!("stop IO monitor");
-        self.cancel_token.cancel();
-    }
-
-    /// monitoring loop
-    async fn monitoring_loop(&self) -> Result<()> {
-        let mut interval = {
-            let config = self.config.read().await;
-            tokio::time::interval(config.monitor_interval)
-        };
-
-        let mut last_load_level = LoadLevel::Low;
-        let mut load_level_start_time = SystemTime::now();
-
-        loop {
-            tokio::select! {
-                _ = self.cancel_token.cancelled() => {
-                    info!("IO monitoring loop cancelled");
-                    break;
-                }
-                _ = interval.tick() => {
-                    // collect system metrics
-                    let metrics = self.collect_system_metrics().await;
-
-                    // update current metrics
-                    *self.current_metrics.write().await = metrics.clone();
-
-                    // update history metrics
-                    self.update_metrics_history(metrics.clone()).await;
-
-                    // calculate load level
-                    let new_load_level = self.calculate_load_level(&metrics).await;
-
-                    // check if load level changed
-                    if new_load_level != last_load_level {
-                        self.handle_load_level_change(last_load_level, new_load_level, load_level_start_time).await;
-                        last_load_level = new_load_level;
-                        load_level_start_time = SystemTime::now();
-                    }
-
-                    // update current load level
-                    *self.current_load_level.write().await = new_load_level;
-
-                    debug!("IO monitor updated: IOPS={}, queue depth={}, latency={}ms, load level={:?}",
-                           metrics.iops, metrics.queue_depth, metrics.avg_latency, new_load_level);
-                }
-            }
-        }
-
-        Ok(())
-    }
-
-    /// collect system metrics
-    async fn collect_system_metrics(&self) -> IOMetrics {
-        let config = self.config.read().await;
-
-        if config.enable_system_monitoring {
-            // actual system monitoring implementation
-            self.collect_real_system_metrics().await
-        } else {
-            // simulated data
-            self.generate_simulated_metrics().await
-        }
-    }
-
-    /// collect real system metrics (need to be implemented according to specific system)
-    async fn collect_real_system_metrics(&self) -> IOMetrics {
-        // TODO: implement actual system metrics collection
-        // can use procfs, sysfs or other system API
-
-        let metrics = IOMetrics {
-            timestamp: SystemTime::now(),
-            ..Default::default()
-        };
-
-        // example: read /proc/diskstats
-        if let Ok(diskstats) = tokio::fs::read_to_string("/proc/diskstats").await {
-            // parse disk stats info
-            // here need to implement specific parsing logic
-            debug!("read disk stats info: {} bytes", diskstats.len());
-        }
-
-        // example: read /proc/stat to get CPU info
-        if let Ok(stat) = tokio::fs::read_to_string("/proc/stat").await {
-            // parse CPU stats info
-            debug!("read CPU stats info: {} bytes", stat.len());
-        }
-
-        // example: read /proc/meminfo to get memory info
-        if let Ok(meminfo) = tokio::fs::read_to_string("/proc/meminfo").await {
-            // parse memory stats info
-            debug!("read memory stats info: {} bytes", meminfo.len());
-        }
-
-        metrics
-    }
-
-    /// generate simulated metrics (for testing and development)
-    async fn generate_simulated_metrics(&self) -> IOMetrics {
-        use rand::Rng;
-        let mut rng = rand::rng();
-
-        // get business metrics impact
-        let business_latency = self.business_metrics.request_latency.load(Ordering::Relaxed);
-        let business_qps = self.business_metrics.request_qps.load(Ordering::Relaxed);
-
-        // generate simulated system metrics based on business load
-        let base_iops = 100 + (business_qps / 10);
-        let base_latency = 5 + (business_latency / 10);
-
-        IOMetrics {
-            timestamp: SystemTime::now(),
-            iops: base_iops + rng.random_range(0..50),
-            read_iops: (base_iops * 6 / 10) + rng.random_range(0..20),
-            write_iops: (base_iops * 4 / 10) + rng.random_range(0..20),
-            queue_depth: rng.random_range(1..20),
-            avg_latency: base_latency + rng.random_range(0..10),
-            read_latency: base_latency + rng.random_range(0..5),
-            write_latency: base_latency + rng.random_range(0..15),
-            cpu_usage: rng.random_range(10..70),
-            memory_usage: rng.random_range(30..80),
-            disk_utilization: rng.random_range(20..90),
-            network_io: rng.random_range(10..1000),
-        }
-    }
-
-    /// update metrics history
-    async fn update_metrics_history(&self, metrics: IOMetrics) {
-        let mut history = self.history_metrics.write().await;
-        let config = self.config.read().await;
-
-        // add new metrics
-        history.push_back(metrics);
-
-        // clean expired data
-        let retention_cutoff = SystemTime::now() - config.history_retention;
-        while let Some(front) = history.front() {
-            if front.timestamp < retention_cutoff {
-                history.pop_front();
-            } else {
-                break;
-            }
-        }
-
-        // limit window size
-        while history.len() > config.load_window_size {
-            history.pop_front();
-        }
-    }
-
-    /// calculate load level
-    async fn calculate_load_level(&self, metrics: &IOMetrics) -> LoadLevel {
-        // multi-dimensional load evaluation algorithm
-        let mut load_score = 0u32;
-
-        // IOPS load evaluation (weight: 25%)
-        let iops_score = match metrics.iops {
-            0..=200 => 0,
-            201..=500 => 15,
-            501..=1000 => 25,
-            _ => 35,
-        };
-        load_score += iops_score;
-
-        // latency load evaluation (weight: 30%)
-        let latency_score = match metrics.avg_latency {
-            0..=10 => 0,
-            11..=50 => 20,
-            51..=100 => 30,
-            _ => 40,
-        };
-        load_score += latency_score;
-
-        // queue depth evaluation (weight: 20%)
-        let queue_score = match metrics.queue_depth {
-            0..=5 => 0,
-            6..=15 => 10,
-            16..=30 => 20,
-            _ => 25,
-        };
-        load_score += queue_score;
-
-        // CPU usage evaluation (weight: 15%)
-        let cpu_score = match metrics.cpu_usage {
-            0..=30 => 0,
-            31..=60 => 8,
-            61..=80 => 12,
-            _ => 15,
-        };
-        load_score += cpu_score;
-
-        // disk usage evaluation (weight: 10%)
-        let disk_score = match metrics.disk_utilization {
-            0..=50 => 0,
-            51..=75 => 5,
-            76..=90 => 8,
-            _ => 10,
-        };
-        load_score += disk_score;
-
-        // business metrics impact
-        let business_latency = self.business_metrics.request_latency.load(Ordering::Relaxed);
-        let business_error_rate = self.business_metrics.error_rate.load(Ordering::Relaxed);
-
-        if business_latency > 100 {
-            load_score += 20; // business latency too high
-        }
-        if business_error_rate > 100 {
-            // > 1%
-            load_score += 15; // business error rate too high
-        }
-
-        // history trend analysis
-        let trend_score = self.calculate_trend_score().await;
-        load_score += trend_score;
-
-        // determine load level based on total score
-        match load_score {
-            0..=30 => LoadLevel::Low,
-            31..=60 => LoadLevel::Medium,
-            61..=90 => LoadLevel::High,
-            _ => LoadLevel::Critical,
-        }
-    }
-
-    /// calculate trend score
-    async fn calculate_trend_score(&self) -> u32 {
-        let history = self.history_metrics.read().await;
-
-        if history.len() < 5 {
-            return 0; // data insufficient, cannot analyze trend
-        }
-
-        // analyze trend of last 5 samples
-        let recent: Vec<_> = history.iter().rev().take(5).collect();
-
-        // check IOPS rising trend
-        let mut iops_trend = 0;
-        for i in 1..recent.len() {
-            if recent[i - 1].iops > recent[i].iops {
-                iops_trend += 1;
-            }
-        }
-
-        // check latency rising trend
-        let mut latency_trend = 0;
-        for i in 1..recent.len() {
-            if recent[i - 1].avg_latency > recent[i].avg_latency {
-                latency_trend += 1;
-            }
-        }
-
-        // if IOPS and latency are both rising, increase load score
-        if iops_trend >= 3 && latency_trend >= 3 {
-            15 // obvious rising trend
-        } else if iops_trend >= 2 || latency_trend >= 2 {
-            5 // slight rising trend
-        } else {
-            0 // no obvious trend
-        }
-    }
-
-    /// handle load level change
-    async fn handle_load_level_change(&self, old_level: LoadLevel, new_level: LoadLevel, start_time: SystemTime) {
-        let duration = SystemTime::now().duration_since(start_time).unwrap_or(Duration::ZERO);
-
-        // update stats
-        {
-            let mut stats = self.load_stats.write().await;
-            match old_level {
-                LoadLevel::Low => stats.low_load_duration += duration.as_secs(),
-                LoadLevel::Medium => stats.medium_load_duration += duration.as_secs(),
-                LoadLevel::High => stats.high_load_duration += duration.as_secs(),
-                LoadLevel::Critical => stats.critical_load_duration += duration.as_secs(),
-            }
-            stats.load_transitions += 1;
-        }
-
-        // update history
-        {
-            let mut history = self.load_level_history.write().await;
-            history.push_back((SystemTime::now(), new_level));
-
-            // keep history record in reasonable range
-            while history.len() > 100 {
-                history.pop_front();
-            }
-        }
-
-        info!("load level changed: {:?} -> {:?}, duration: {:?}", old_level, new_level, duration);
-
-        // if enter critical load state, record warning
-        if new_level == LoadLevel::Critical {
-            warn!("system entered critical load state, Scanner will pause running");
-        }
-    }
-
-    /// get current load level
-    pub async fn get_business_load_level(&self) -> LoadLevel {
-        *self.current_load_level.read().await
-    }
-
-    /// get current metrics
-    pub async fn get_current_metrics(&self) -> IOMetrics {
-        self.current_metrics.read().await.clone()
-    }
-
-    /// get history metrics
-    pub async fn get_history_metrics(&self) -> Vec<IOMetrics> {
-        self.history_metrics.read().await.iter().cloned().collect()
-    }
-
-    /// get load stats
-    pub async fn get_load_stats(&self) -> LoadLevelStats {
-        self.load_stats.read().await.clone()
-    }
-
-    /// update business IO metrics
-    pub async fn update_business_metrics(&self, latency: u64, qps: u64, error_rate: u64, connections: u64) {
-        self.business_metrics.request_latency.store(latency, Ordering::Relaxed);
-        self.business_metrics.request_qps.store(qps, Ordering::Relaxed);
-        self.business_metrics.error_rate.store(error_rate, Ordering::Relaxed);
-        self.business_metrics.active_connections.store(connections, Ordering::Relaxed);
-
-        *self.business_metrics.last_update.write().await = SystemTime::now();
-
-        debug!(
-            "update business metrics: latency={}ms, QPS={}, error rate={}‰, connections={}",
-            latency, qps, error_rate, connections
-        );
-    }
-
-    /// clone for background task
-    fn clone_for_background(&self) -> Self {
-        Self {
-            config: self.config.clone(),
-            current_metrics: self.current_metrics.clone(),
-            history_metrics: self.history_metrics.clone(),
-            current_load_level: self.current_load_level.clone(),
-            load_level_history: self.load_level_history.clone(),
-            load_stats: self.load_stats.clone(),
-            business_metrics: self.business_metrics.clone(),
-            cancel_token: self.cancel_token.clone(),
-        }
-    }
-
-    /// reset stats
-    pub async fn reset_stats(&self) {
-        *self.load_stats.write().await = LoadLevelStats::default();
-        self.load_level_history.write().await.clear();
-        self.history_metrics.write().await.clear();
-        info!("IO monitor stats reset");
-    }
-
-    /// get load level history
-    pub async fn get_load_level_history(&self) -> Vec<(SystemTime, LoadLevel)> {
-        self.load_level_history.read().await.iter().cloned().collect()
-    }
-}

crates/ahm/src/scanner/io_throttler.rs

@@ -1,499 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use crate::scanner::LoadLevel;
-use std::{
-    sync::{
-        Arc,
-        atomic::{AtomicU8, AtomicU64, Ordering},
-    },
-    time::{Duration, SystemTime},
-};
-use tokio::sync::RwLock;
-use tracing::{debug, info, warn};
-
-/// IO throttler config
-#[derive(Debug, Clone)]
-pub struct IOThrottlerConfig {
-    /// max IOPS limit
-    pub max_iops: u64,
-    /// business priority baseline (percentage)
-    pub base_business_priority: u8,
-    /// scanner minimum delay (milliseconds)
-    pub min_scan_delay: u64,
-    /// scanner maximum delay (milliseconds)
-    pub max_scan_delay: u64,
-    /// whether enable dynamic adjustment
-    pub enable_dynamic_adjustment: bool,
-    /// adjustment response time (seconds)
-    pub adjustment_response_time: u64,
-}
-
-impl Default for IOThrottlerConfig {
-    fn default() -> Self {
-        Self {
-            max_iops: 1000,             // default max 1000 IOPS
-            base_business_priority: 95, // business priority 95%
-            min_scan_delay: 5000,       // minimum 5s delay
-            max_scan_delay: 60000,      // maximum 60s delay
-            enable_dynamic_adjustment: true,
-            adjustment_response_time: 5, // 5 seconds response time
-        }
-    }
-}
-
-/// resource allocation strategy
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum ResourceAllocationStrategy {
-    /// business priority strategy
-    BusinessFirst,
-    /// balanced strategy
-    Balanced,
-    /// maintenance priority strategy (only used in special cases)
-    MaintenanceFirst,
-}
-
-/// throttle decision
-#[derive(Debug, Clone)]
-pub struct ThrottleDecision {
-    /// whether should pause scanning
-    pub should_pause: bool,
-    /// suggested scanning delay
-    pub suggested_delay: Duration,
-    /// resource allocation suggestion
-    pub resource_allocation: ResourceAllocation,
-    /// decision reason
-    pub reason: String,
-}
-
-/// resource allocation
-#[derive(Debug, Clone)]
-pub struct ResourceAllocation {
-    /// business IO allocation percentage (0-100)
-    pub business_percentage: u8,
-    /// scanner IO allocation percentage (0-100)
-    pub scanner_percentage: u8,
-    /// allocation strategy
-    pub strategy: ResourceAllocationStrategy,
-}
-
-/// enhanced IO throttler
-///
-/// dynamically adjust the resource usage of the scanner based on real-time system load and business demand,
-/// ensure business IO gets priority protection.
-pub struct AdvancedIOThrottler {
-    /// config
-    config: Arc<RwLock<IOThrottlerConfig>>,
-    /// current IOPS usage (reserved field)
-    #[allow(dead_code)]
-    current_iops: Arc<AtomicU64>,
-    /// business priority weight (0-100)
-    business_priority: Arc<AtomicU8>,
-    /// scanning operation delay (milliseconds)
-    scan_delay: Arc<AtomicU64>,
-    /// resource allocation strategy
-    allocation_strategy: Arc<RwLock<ResourceAllocationStrategy>>,
-    /// throttle history record
-    throttle_history: Arc<RwLock<Vec<ThrottleRecord>>>,
-    /// last adjustment time (reserved field)
-    #[allow(dead_code)]
-    last_adjustment: Arc<RwLock<SystemTime>>,
-}
-
-/// throttle record
-#[derive(Debug, Clone)]
-pub struct ThrottleRecord {
-    /// timestamp
-    pub timestamp: SystemTime,
-    /// load level
-    pub load_level: LoadLevel,
-    /// decision
-    pub decision: ThrottleDecision,
-    /// system metrics snapshot
-    pub metrics_snapshot: MetricsSnapshot,
-}
-
-/// metrics snapshot
-#[derive(Debug, Clone)]
-pub struct MetricsSnapshot {
-    /// IOPS
-    pub iops: u64,
-    /// latency
-    pub latency: u64,
-    /// CPU usage
-    pub cpu_usage: u8,
-    /// memory usage
-    pub memory_usage: u8,
-}
-
-impl AdvancedIOThrottler {
-    /// create new advanced IO throttler
-    pub fn new(config: IOThrottlerConfig) -> Self {
-        Self {
-            config: Arc::new(RwLock::new(config)),
-            current_iops: Arc::new(AtomicU64::new(0)),
-            business_priority: Arc::new(AtomicU8::new(95)),
-            scan_delay: Arc::new(AtomicU64::new(5000)),
-            allocation_strategy: Arc::new(RwLock::new(ResourceAllocationStrategy::BusinessFirst)),
-            throttle_history: Arc::new(RwLock::new(Vec::new())),
-            last_adjustment: Arc::new(RwLock::new(SystemTime::UNIX_EPOCH)),
-        }
-    }
-
-    /// adjust scanning delay based on load level
-    pub async fn adjust_for_load_level(&self, load_level: LoadLevel) -> Duration {
-        let config = self.config.read().await;
-
-        let delay_ms = match load_level {
-            LoadLevel::Low => {
-                // low load: use minimum delay
-                self.scan_delay.store(config.min_scan_delay, Ordering::Relaxed);
-                self.business_priority
-                    .store(config.base_business_priority.saturating_sub(5), Ordering::Relaxed);
-                config.min_scan_delay
-            }
-            LoadLevel::Medium => {
-                // medium load: increase delay moderately
-                let delay = config.min_scan_delay * 5; // 500ms
-                self.scan_delay.store(delay, Ordering::Relaxed);
-                self.business_priority.store(config.base_business_priority, Ordering::Relaxed);
-                delay
-            }
-            LoadLevel::High => {
-                // high load: increase delay significantly
-                let delay = config.min_scan_delay * 10; // 50s
-                self.scan_delay.store(delay, Ordering::Relaxed);
-                self.business_priority
-                    .store(config.base_business_priority.saturating_add(3), Ordering::Relaxed);
-                delay
-            }
-            LoadLevel::Critical => {
-                // critical load: maximum delay or pause
-                let delay = config.max_scan_delay; // 60s
-                self.scan_delay.store(delay, Ordering::Relaxed);
-                self.business_priority.store(99, Ordering::Relaxed);
-                delay
-            }
-        };
-
-        let duration = Duration::from_millis(delay_ms);
-
-        debug!("Adjust scanning delay based on load level {:?}: {:?}", load_level, duration);
-
-        duration
-    }
-
-    /// create throttle decision
-    pub async fn make_throttle_decision(&self, load_level: LoadLevel, metrics: Option<MetricsSnapshot>) -> ThrottleDecision {
-        let _config = self.config.read().await;
-
-        let should_pause = matches!(load_level, LoadLevel::Critical);
-
-        let suggested_delay = self.adjust_for_load_level(load_level).await;
-
-        let resource_allocation = self.calculate_resource_allocation(load_level).await;
-
-        let reason = match load_level {
-            LoadLevel::Low => "system load is low, scanner can run normally".to_string(),
-            LoadLevel::Medium => "system load is moderate, scanner is running at reduced speed".to_string(),
-            LoadLevel::High => "system load is high, scanner is running at significantly reduced speed".to_string(),
-            LoadLevel::Critical => "system load is too high, scanner is paused".to_string(),
-        };
-
-        let decision = ThrottleDecision {
-            should_pause,
-            suggested_delay,
-            resource_allocation,
-            reason,
-        };
-
-        // record decision history
-        if let Some(snapshot) = metrics {
-            self.record_throttle_decision(load_level, decision.clone(), snapshot).await;
-        }
-
-        decision
-    }
-
-    /// calculate resource allocation
-    async fn calculate_resource_allocation(&self, load_level: LoadLevel) -> ResourceAllocation {
-        let strategy = *self.allocation_strategy.read().await;
-
-        let (business_pct, scanner_pct) = match (strategy, load_level) {
-            (ResourceAllocationStrategy::BusinessFirst, LoadLevel::Low) => (90, 10),
-            (ResourceAllocationStrategy::BusinessFirst, LoadLevel::Medium) => (95, 5),
-            (ResourceAllocationStrategy::BusinessFirst, LoadLevel::High) => (98, 2),
-            (ResourceAllocationStrategy::BusinessFirst, LoadLevel::Critical) => (99, 1),
-
-            (ResourceAllocationStrategy::Balanced, LoadLevel::Low) => (80, 20),
-            (ResourceAllocationStrategy::Balanced, LoadLevel::Medium) => (85, 15),
-            (ResourceAllocationStrategy::Balanced, LoadLevel::High) => (90, 10),
-            (ResourceAllocationStrategy::Balanced, LoadLevel::Critical) => (95, 5),
-
-            (ResourceAllocationStrategy::MaintenanceFirst, _) => (70, 30), // special maintenance mode
-        };
-
-        ResourceAllocation {
-            business_percentage: business_pct,
-            scanner_percentage: scanner_pct,
-            strategy,
-        }
-    }
-
-    /// check whether should pause scanning
-    pub async fn should_pause_scanning(&self, load_level: LoadLevel) -> bool {
-        match load_level {
-            LoadLevel::Critical => {
-                warn!("System load reached critical level, pausing scanner");
-                true
-            }
-            _ => false,
-        }
-    }
-
-    /// record throttle decision
-    async fn record_throttle_decision(&self, load_level: LoadLevel, decision: ThrottleDecision, metrics: MetricsSnapshot) {
-        let record = ThrottleRecord {
-            timestamp: SystemTime::now(),
-            load_level,
-            decision,
-            metrics_snapshot: metrics,
-        };
-
-        let mut history = self.throttle_history.write().await;
-        history.push(record);
-
-        // keep history record in reasonable range (last 1000 records)
-        while history.len() > 1000 {
-            history.remove(0);
-        }
-    }
-
-    /// set resource allocation strategy
-    pub async fn set_allocation_strategy(&self, strategy: ResourceAllocationStrategy) {
-        *self.allocation_strategy.write().await = strategy;
-        info!("Set resource allocation strategy: {:?}", strategy);
-    }
-
-    /// get current resource allocation
-    pub async fn get_current_allocation(&self) -> ResourceAllocation {
-        let current_load = LoadLevel::Low; // need to get from external
-        self.calculate_resource_allocation(current_load).await
-    }
-
-    /// get throttle history
-    pub async fn get_throttle_history(&self) -> Vec<ThrottleRecord> {
-        self.throttle_history.read().await.clone()
-    }
-
-    /// get throttle stats
-    pub async fn get_throttle_stats(&self) -> ThrottleStats {
-        let history = self.throttle_history.read().await;
-
-        let total_decisions = history.len();
-        let pause_decisions = history.iter().filter(|r| r.decision.should_pause).count();
-
-        let mut delay_sum = Duration::ZERO;
-        for record in history.iter() {
-            delay_sum += record.decision.suggested_delay;
-        }
-
-        let avg_delay = if total_decisions > 0 {
-            delay_sum / total_decisions as u32
-        } else {
-            Duration::ZERO
-        };
-
-        // count by load level
-        let low_count = history.iter().filter(|r| r.load_level == LoadLevel::Low).count();
-        let medium_count = history.iter().filter(|r| r.load_level == LoadLevel::Medium).count();
-        let high_count = history.iter().filter(|r| r.load_level == LoadLevel::High).count();
-        let critical_count = history.iter().filter(|r| r.load_level == LoadLevel::Critical).count();
-
-        ThrottleStats {
-            total_decisions,
-            pause_decisions,
-            average_delay: avg_delay,
-            load_level_distribution: LoadLevelDistribution {
-                low_count,
-                medium_count,
-                high_count,
-                critical_count,
-            },
-        }
-    }
-
-    /// reset throttle history
-    pub async fn reset_history(&self) {
-        self.throttle_history.write().await.clear();
-        info!("Reset throttle history");
-    }
-
-    /// update config
-    pub async fn update_config(&self, new_config: IOThrottlerConfig) {
-        *self.config.write().await = new_config;
-        info!("Updated IO throttler configuration");
-    }
-
-    /// get current scanning delay
-    pub fn get_current_scan_delay(&self) -> Duration {
-        let delay_ms = self.scan_delay.load(Ordering::Relaxed);
-        Duration::from_millis(delay_ms)
-    }
-
-    /// get current business priority
-    pub fn get_current_business_priority(&self) -> u8 {
-        self.business_priority.load(Ordering::Relaxed)
-    }
-
-    /// simulate business load pressure test
-    pub async fn simulate_business_pressure(&self, duration: Duration) -> SimulationResult {
-        info!("Start simulating business load pressure test, duration: {:?}", duration);
-
-        let start_time = SystemTime::now();
-        let mut simulation_records = Vec::new();
-
-        // simulate different load level changes
-        let load_levels = [
-            LoadLevel::Low,
-            LoadLevel::Medium,
-            LoadLevel::High,
-            LoadLevel::Critical,
-            LoadLevel::High,
-            LoadLevel::Medium,
-            LoadLevel::Low,
-        ];
-
-        let step_duration = duration / load_levels.len() as u32;
-
-        for (i, &load_level) in load_levels.iter().enumerate() {
-            let _step_start = SystemTime::now();
-
-            // simulate metrics for this load level
-            let metrics = MetricsSnapshot {
-                iops: match load_level {
-                    LoadLevel::Low => 200,
-                    LoadLevel::Medium => 500,
-                    LoadLevel::High => 800,
-                    LoadLevel::Critical => 1200,
-                },
-                latency: match load_level {
-                    LoadLevel::Low => 10,
-                    LoadLevel::Medium => 25,
-                    LoadLevel::High => 60,
-                    LoadLevel::Critical => 150,
-                },
-                cpu_usage: match load_level {
-                    LoadLevel::Low => 30,
-                    LoadLevel::Medium => 50,
-                    LoadLevel::High => 75,
-                    LoadLevel::Critical => 95,
-                },
-                memory_usage: match load_level {
-                    LoadLevel::Low => 40,
-                    LoadLevel::Medium => 60,
-                    LoadLevel::High => 80,
-                    LoadLevel::Critical => 90,
-                },
-            };
-
-            let decision = self.make_throttle_decision(load_level, Some(metrics.clone())).await;
-
-            simulation_records.push(SimulationRecord {
-                step: i + 1,
-                load_level,
-                metrics,
-                decision: decision.clone(),
-                step_duration,
-            });
-
-            info!(
-                "simulate step {}: load={:?}, delay={:?}, pause={}",
-                i + 1,
-                load_level,
-                decision.suggested_delay,
-                decision.should_pause
-            );
-
-            // wait for step duration
-            tokio::time::sleep(step_duration).await;
-        }
-
-        let total_duration = SystemTime::now().duration_since(start_time).unwrap_or(Duration::ZERO);
-
-        SimulationResult {
-            total_duration,
-            simulation_records,
-            final_stats: self.get_throttle_stats().await,
-        }
-    }
-}
-
-/// throttle stats
-#[derive(Debug, Clone)]
-pub struct ThrottleStats {
-    /// total decisions
-    pub total_decisions: usize,
-    /// pause decisions
-    pub pause_decisions: usize,
-    /// average delay
-    pub average_delay: Duration,
-    /// load level distribution
-    pub load_level_distribution: LoadLevelDistribution,
-}
-
-/// load level distribution
-#[derive(Debug, Clone)]
-pub struct LoadLevelDistribution {
-    /// low load count
-    pub low_count: usize,
-    /// medium load count
-    pub medium_count: usize,
-    /// high load count
-    pub high_count: usize,
-    /// critical load count
-    pub critical_count: usize,
-}
-
-/// simulation result
-#[derive(Debug, Clone)]
-pub struct SimulationResult {
-    /// total duration
-    pub total_duration: Duration,
-    /// simulation records
-    pub simulation_records: Vec<SimulationRecord>,
-    /// final stats
-    pub final_stats: ThrottleStats,
-}
-
-/// simulation record
-#[derive(Debug, Clone)]
-pub struct SimulationRecord {
-    /// step number
-    pub step: usize,
-    /// load level
-    pub load_level: LoadLevel,
-    /// metrics snapshot
-    pub metrics: MetricsSnapshot,
-    /// throttle decision
-    pub decision: ThrottleDecision,
-    /// step duration
-    pub step_duration: Duration,
-}
-
-impl Default for AdvancedIOThrottler {
-    fn default() -> Self {
-        Self::new(IOThrottlerConfig::default())
-    }
-}

crates/ahm/src/scanner/lifecycle.rs

@@ -1,269 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use crate::Result;
-use rustfs_common::data_usage::SizeSummary;
-use rustfs_common::metrics::IlmAction;
-use rustfs_ecstore::bucket::{
-    lifecycle::{
-        bucket_lifecycle_audit::LcEventSrc,
-        bucket_lifecycle_ops::{GLOBAL_ExpiryState, apply_lifecycle_action, eval_action_from_lifecycle},
-        lifecycle,
-        lifecycle::Lifecycle,
-    },
-    metadata_sys::get_object_lock_config,
-    object_lock::objectlock_sys::{BucketObjectLockSys, enforce_retention_for_deletion},
-    versioning::VersioningApi,
-    versioning_sys::BucketVersioningSys,
-};
-use rustfs_ecstore::store_api::{ObjectInfo, ObjectToDelete};
-use rustfs_filemeta::FileInfo;
-use s3s::dto::{BucketLifecycleConfiguration as LifecycleConfig, VersioningConfiguration};
-use std::sync::{
-    Arc,
-    atomic::{AtomicU64, Ordering},
-};
-use time::OffsetDateTime;
-use tracing::info;
-
-static SCANNER_EXCESS_OBJECT_VERSIONS: AtomicU64 = AtomicU64::new(100);
-static SCANNER_EXCESS_OBJECT_VERSIONS_TOTAL_SIZE: AtomicU64 = AtomicU64::new(1024 * 1024 * 1024 * 1024); // 1 TB
-
-#[derive(Clone)]
-pub struct ScannerItem {
-    pub bucket: String,
-    pub object_name: String,
-    pub lifecycle: Option<Arc<LifecycleConfig>>,
-    pub versioning: Option<Arc<VersioningConfiguration>>,
-}
-
-impl ScannerItem {
-    pub fn new(
-        bucket: String,
-        lifecycle: Option<Arc<LifecycleConfig>>,
-        versioning: Option<Arc<VersioningConfiguration>>,
-    ) -> Self {
-        Self {
-            bucket,
-            object_name: "".to_string(),
-            lifecycle,
-            versioning,
-        }
-    }
-
-    pub async fn apply_versions_actions(&self, fivs: &[FileInfo]) -> Result<Vec<ObjectInfo>> {
-        let obj_infos = self.apply_newer_noncurrent_version_limit(fivs).await?;
-        if obj_infos.len() >= SCANNER_EXCESS_OBJECT_VERSIONS.load(Ordering::SeqCst) as usize {
-            // todo
-        }
-
-        let mut cumulative_size = 0;
-        for obj_info in obj_infos.iter() {
-            cumulative_size += obj_info.size;
-        }
-
-        if cumulative_size >= SCANNER_EXCESS_OBJECT_VERSIONS_TOTAL_SIZE.load(Ordering::SeqCst) as i64 {
-            //todo
-        }
-
-        Ok(obj_infos)
-    }
-
-    pub async fn apply_newer_noncurrent_version_limit(&self, fivs: &[FileInfo]) -> Result<Vec<ObjectInfo>> {
-        let lock_enabled = if let Some(rcfg) = BucketObjectLockSys::get(&self.bucket).await {
-            rcfg.mode.is_some()
-        } else {
-            false
-        };
-        let _vcfg = BucketVersioningSys::get(&self.bucket).await?;
-
-        let versioned = match BucketVersioningSys::get(&self.bucket).await {
-            Ok(vcfg) => vcfg.versioned(&self.object_name),
-            Err(_) => false,
-        };
-        let mut object_infos = Vec::with_capacity(fivs.len());
-
-        if self.lifecycle.is_none() {
-            for info in fivs.iter() {
-                object_infos.push(ObjectInfo::from_file_info(info, &self.bucket, &self.object_name, versioned));
-            }
-            return Ok(object_infos);
-        }
-
-        let event = self
-            .lifecycle
-            .as_ref()
-            .expect("lifecycle err.")
-            .clone()
-            .noncurrent_versions_expiration_limit(&lifecycle::ObjectOpts {
-                name: self.object_name.clone(),
-                ..Default::default()
-            })
-            .await;
-        let lim = event.newer_noncurrent_versions;
-        if lim == 0 || fivs.len() <= lim + 1 {
-            for fi in fivs.iter() {
-                object_infos.push(ObjectInfo::from_file_info(fi, &self.bucket, &self.object_name, versioned));
-            }
-            return Ok(object_infos);
-        }
-
-        let overflow_versions = &fivs[lim + 1..];
-        for fi in fivs[..lim + 1].iter() {
-            object_infos.push(ObjectInfo::from_file_info(fi, &self.bucket, &self.object_name, versioned));
-        }
-
-        let mut to_del = Vec::<ObjectToDelete>::with_capacity(overflow_versions.len());
-        for fi in overflow_versions.iter() {
-            let obj = ObjectInfo::from_file_info(fi, &self.bucket, &self.object_name, versioned);
-            if lock_enabled && enforce_retention_for_deletion(&obj) {
-                //if enforce_retention_for_deletion(&obj) {
-                /*if self.debug {
-                    if obj.version_id.is_some() {
-                        info!("lifecycle: {} v({}) is locked, not deleting\n", obj.name, obj.version_id.expect("err"));
-                    } else {
-                        info!("lifecycle: {} is locked, not deleting\n", obj.name);
-                    }
-                }*/
-                object_infos.push(obj);
-                continue;
-            }
-
-            if OffsetDateTime::now_utc().unix_timestamp()
-                < lifecycle::expected_expiry_time(obj.successor_mod_time.expect("err"), event.noncurrent_days as i32)
-                    .unix_timestamp()
-            {
-                object_infos.push(obj);
-                continue;
-            }
-
-            to_del.push(ObjectToDelete {
-                object_name: obj.name,
-                version_id: obj.version_id,
-                ..Default::default()
-            });
-        }
-
-        if !to_del.is_empty() {
-            let mut expiry_state = GLOBAL_ExpiryState.write().await;
-            expiry_state.enqueue_by_newer_noncurrent(&self.bucket, to_del, event).await;
-        }
-
-        Ok(object_infos)
-    }
-
-    pub async fn apply_actions(&mut self, oi: &ObjectInfo, _size_s: &mut SizeSummary) -> (bool, i64) {
-        let (action, _size) = self.apply_lifecycle(oi).await;
-
-        info!(
-            "apply_actions {} {} {:?} {:?}",
-            oi.bucket.clone(),
-            oi.name.clone(),
-            oi.version_id.clone(),
-            oi.user_defined.clone()
-        );
-
-        // Create a mutable clone if you need to modify fields
-        /*let mut oi = oi.clone();
-        oi.replication_status = ReplicationStatusType::from(
-            oi.user_defined
-                .get("x-amz-bucket-replication-status")
-                .unwrap_or(&"PENDING".to_string()),
-        );
-        info!("apply status is: {:?}", oi.replication_status);
-        self.heal_replication(&oi, _size_s).await;*/
-
-        if action.delete_all() {
-            return (true, 0);
-        }
-
-        (false, oi.size)
-    }
-
-    async fn apply_lifecycle(&mut self, oi: &ObjectInfo) -> (IlmAction, i64) {
-        let size = oi.size;
-        if self.lifecycle.is_none() {
-            info!("apply_lifecycle: No lifecycle config for object: {}", oi.name);
-            return (IlmAction::NoneAction, size);
-        }
-
-        info!("apply_lifecycle: Lifecycle config exists for object: {}", oi.name);
-
-        let (olcfg, rcfg) = if self.bucket != ".minio.sys" {
-            (
-                get_object_lock_config(&self.bucket).await.ok(),
-                None, // FIXME: replication config
-            )
-        } else {
-            (None, None)
-        };
-
-        info!("apply_lifecycle: Evaluating lifecycle for object: {}", oi.name);
-
-        let lifecycle = match self.lifecycle.as_ref() {
-            Some(lc) => lc,
-            None => {
-                info!("No lifecycle configuration found for object: {}", oi.name);
-                return (IlmAction::NoneAction, 0);
-            }
-        };
-
-        let lc_evt = eval_action_from_lifecycle(
-            lifecycle,
-            olcfg
-                .as_ref()
-                .and_then(|(c, _)| c.rule.as_ref().and_then(|r| r.default_retention.clone())),
-            rcfg.clone(),
-            oi, // Pass oi directly
-        )
-        .await;
-
-        info!("lifecycle: {} Initial scan: {} (action: {:?})", oi.name, lc_evt.action, lc_evt.action);
-
-        let mut new_size = size;
-        match lc_evt.action {
-            IlmAction::DeleteVersionAction | IlmAction::DeleteAllVersionsAction | IlmAction::DelMarkerDeleteAllVersionsAction => {
-                info!("apply_lifecycle: Object {} marked for version deletion, new_size=0", oi.name);
-                new_size = 0;
-            }
-            IlmAction::DeleteAction => {
-                info!("apply_lifecycle: Object {} marked for deletion", oi.name);
-                if let Some(vcfg) = &self.versioning {
-                    if !vcfg.enabled() {
-                        info!("apply_lifecycle: Versioning disabled, setting new_size=0");
-                        new_size = 0;
-                    }
-                } else {
-                    info!("apply_lifecycle: No versioning config, setting new_size=0");
-                    new_size = 0;
-                }
-            }
-            IlmAction::NoneAction => {
-                info!("apply_lifecycle: No action for object {}", oi.name);
-            }
-            _ => {
-                info!("apply_lifecycle: Other action {:?} for object {}", lc_evt.action, oi.name);
-            }
-        }
-
-        if lc_evt.action != IlmAction::NoneAction {
-            info!("apply_lifecycle: Applying lifecycle action {:?} for object {}", lc_evt.action, oi.name);
-            apply_lifecycle_action(&lc_evt, &LcEventSrc::Scanner, oi).await;
-        } else {
-            info!("apply_lifecycle: Skipping lifecycle action for object {} as no action is needed", oi.name);
-        }
-
-        (lc_evt.action, new_size)
-    }
-}