feat(admin): add persisted OIDC config APIs (#2267)

Co-authored-by: heihutu <heihutu@gmail.com>

.agents/skills/code-change-verification/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: code-change-verification
+description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
+---
+
+# Code Change Verification
+
+Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
+
+## Quick Start
+
+1. Read the scope: commit, PR, patch, or file list.
+2. Map each changed area by risk and user impact.
+3. Inspect each risky change in context.
+4. Report findings first, ordered by severity.
+5. Close with residual risks and verification recommendations.
+
+## Core Workflow
+
+### 1) Scope and assumptions
+- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
+- If context is missing, state assumptions before deeper analysis.
+- Focus only on requested scope; avoid reviewing unrelated files.
+
+### 2) Risk map
+- Prioritize in this order:
+  - Data correctness and user-visible behavior
+  - API/contract compatibility
+  - Security and authz/authn boundaries
+  - Concurrency and lifecycle correctness
+  - Performance and resource usage
+- Give higher priority to stateful paths, migration logic, defaults, and error handling.
+
+### 3) Evidence-based inspection
+- Read each modified hunk with neighboring context.
+- Trace call paths and call-site expectations.
+- Check for:
+  - invariant breaks and missing guards
+  - unchecked assumptions and null/empty/error-path handling
+  - stale tests, fixtures, and configs
+  - hidden coupling to shared helpers/constants/features
+- If a point is uncertain, mark it as an open question instead of guessing.
+
+### 4) Findings-first output
+- Order findings by severity:
+  - P0: critical failure, security breach, or data loss risk
+  - P1: high-impact regression
+  - P2: medium risk correctness gap
+  - P3: low risk/quality debt
+- For each finding include:
+  - Severity
+  - `path:line` reference
+  - concise issue statement
+  - impact and likely failure mode
+  - specific fix or mitigation
+  - validation step to confirm
+- If no issues exist, explicitly state `No findings` and why.
+
+### 5) Close
+- Report assumptions and unknowns.
+- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
+
+## Output Template
+
+1. Findings
+2. No findings (if applicable)
+3. Assumptions / Unknowns
+4. Recommended verification steps
+
+## Finding Template
+
+- `[P1] Missing timeout for downstream call`
+  - Location: `path/to/file.rs:123`
+  - Issue: ...
+  - Impact: ...
+  - Fix suggestion: ...
+  - Validation: ...
+

.agents/skills/code-change-verification/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Code Change Verification"
+  short_description: "Prioritize risks and verify code changes before merge."
+  default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

.agents/skills/pr-creation-checker/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: pr-creation-checker
+description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
+---
+
+# PR Creation Checker
+
+Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
+
+## Read sources of truth first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Use `Makefile` and `.config/make/` for local quality commands.
+- Use `.github/workflows/ci.yml` for CI expectations.
+- Do not restate long command matrices or template sections from memory when the files exist.
+
+## Workflow
+
+1. Collect PR context
+- Confirm base branch, current branch, change goal, and scope.
+- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
+- Confirm whether the branch includes only intended changes.
+
+2. Inspect change scope
+- Review the diff and summarize what changed.
+- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
+- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+
+3. Verify readiness requirements
+- Require `make pre-commit` before marking the PR ready.
+- If `make` is unavailable, use the equivalent commands from `.config/make/`.
+- Add scope-specific verification commands when the changed area needs more than the baseline.
+- If required checks fail, stop and return `BLOCKED`.
+
+4. Draft PR metadata
+- Write the PR title in English using Conventional Commits and keep it within 72 characters.
+- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
+- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
+- Keep the PR body in English.
+- Use the exact section headings from `.github/pull_request_template.md`.
+- Fill non-applicable sections with `N/A`.
+- Include verification commands in the PR description.
+- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
+- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
+
+5. Prepare reviewer context
+- Summarize why the change exists.
+- Summarize what was verified.
+- Call out risks, rollout notes, config impact, and rollback notes when applicable.
+- Mention assumptions or missing context instead of guessing.
+
+6. Prepare CLI-safe output
+- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
+- Return a ready-to-save PR body plus a short title.
+- If not ready, return blockers first and list the minimum steps needed to unblock.
+
+## Output format
+
+### Status
+- `READY` or `BLOCKED`
+
+### Title
+- `<type>(<scope>): <summary>`
+
+### PR Body
+- Reproduce the repository template headings exactly.
+- Fill every section.
+- Omit local absolute paths unless explicitly required.
+
+### Verification
+- List each command run.
+- State pass/fail.
+
+### Risks
+- List breaking changes, config changes, migration impact, or `N/A`.
+
+## Blocker rules
+
+- Return `BLOCKED` if `make pre-commit` has not passed.
+- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
+- Return `BLOCKED` if required template sections are missing.
+- Return `BLOCKED` if the title/body is not in English.
+- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
+
+## Reference
+
+- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "PR Creation Checker"
+  short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
+  default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -0,0 +1,14 @@
+# PR Readiness Checklist
+
+- Confirm the branch is based on current `main`.
+- Confirm the diff matches the stated scope.
+- Confirm no secrets, logs, temp files, or unrelated refactors are included.
+- Confirm `make pre-commit` passed, or document why it could not run.
+- Confirm extra verification commands are listed for risky changes.
+- Confirm the PR title uses Conventional Commits and stays within 72 characters.
+- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+- Confirm the PR body is in English.
+- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
+- Confirm non-applicable sections are filled with `N/A`.
+- Confirm the PR body does not include local absolute paths unless explicitly required.
+- Confirm multiline GitHub CLI commands use `--body-file`.

.agents/skills/test-coverage-improver/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: test-coverage-improver
+description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
+---
+
+# Test Coverage Improver
+
+Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
+
+## Usage assumptions
+- Focus scope is either changed lines/files, a module, or the whole repository.
+- Coverage artifact must be generated or provided in a supported format.
+- If required context is missing, call out assumptions explicitly before proposing work.
+
+## Workflow
+
+1. Define scope and baseline
+   - Confirm target language, framework, and branch.
+   - Confirm whether the scope is changed files only or full-repo.
+
+2. Produce coverage snapshot
+   - Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
+   - JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
+   - Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
+   - Collect total, per-file, and changed-line coverage.
+
+3. Rank highest-risk gaps
+   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
+   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
+   - Keep shortlist to 5–8 gaps.
+   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
+
+4. Propose high-impact tests
+   - For each shortlisted gap, output:
+     - Intent and expected behavior.
+     - Normal, edge, and failure scenarios.
+     - Assertions and side effects to verify.
+     - Setup needs (fixtures, mocks, integration dependencies).
+     - Estimated effort (`S/M/L`).
+
+5. Close with validation plan
+   - State which gaps remain after proposals.
+   - Provide concrete verification command and acceptance threshold.
+   - List assumptions or blockers (environment, fixtures, flaky dependencies).
+
+## Output template
+
+### Coverage Snapshot
+- total / branch coverage
+- changed-file coverage
+- top missing regions by size
+
+### Top Gaps (ranked)
+- `path:line-range` | risk score | why critical
+
+### Test Proposals
+- `path:line-range`
+  - Test name
+  - scenarios
+  - assertions
+  - effort
+
+### Validation Plan
+- command
+- pass criteria
+- remaining risk

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Test Coverage Improver"
+  short_description: "Find top uncovered risk areas and propose high-impact tests."
+  default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -0,0 +1,25 @@
+# Coverage Gap Prioritization Guide
+
+Use this rubric for each uncovered area.
+
+Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
+
+- Criticality:
+  - 5: authz/authn, data-loss, payment/consistency path
+  - 4: state mutation, cache invalidation, scheduling
+  - 3: error handling + fallbacks in user-visible flows
+  - 2: parsing/format conversion paths
+  - 1: logging-only or low-impact utilities
+
+- CoverageDebt:
+  - 0: 0–5 uncovered lines
+  - 1: 6–20 uncovered lines
+  - 2: 21–40 uncovered lines
+  - 3: 41+ uncovered lines
+
+- Volatility:
+  - 1: stable legacy code with few recent edits
+  - 2: changed in last 2 releases
+  - 3: touched in last 30 days or currently in active PR
+
+Sort by score descending, then by business impact.

.docker/compose/docker-compose.observability.yaml

@@ -104,7 +104,7 @@ services:
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/loki-config.yaml:/etc/loki/local-config.yaml:ro
+      - ../../.docker/observability/loki.yaml:/etc/loki/local-config.yaml:ro
       - loki-data:/loki
     ports:
       - "3100:3100"

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -0,0 +1,270 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  rustfs:
+    security_opt:
+      - "no-new-privileges:true"
+    image: rustfs/rustfs:latest
+    container_name: rustfs-server
+    ports:
+      - "9000:9000" # S3 API port
+      - "9001:9001" # Console port
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+    volumes:
+      - rustfs-data:/data/rustfs
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+    depends_on:
+      otel-collector:
+        condition: service_started
+
+  rustfs-init:
+    image: alpine
+    container_name: rustfs-init
+    volumes:
+      - rustfs-data:/data
+    networks:
+      - otel-network
+    command: >
+      sh -c "
+        chown -R 10001:10001 /data &&
+        echo 'Volume Permissions fixed' &&
+        exit 0
+      "
+    restart: no
+
+  # --- Tracing ---
+
+  tempo:
+    image: grafana/tempo:latest
+    container_name: tempo
+    command: [ "-config.file=/etc/tempo.yaml" ]
+    volumes:
+      - ./tempo.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200" # tempo
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - redpanda
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  redpanda:
+    image: redpandadata/redpanda:latest # for tempo ingest
+    container_name: redpanda
+    ports:
+      - "9092:9092"
+    networks:
+      - otel-network
+    restart: unless-stopped
+    command: >
+      redpanda start --overprovisioned
+      --mode=dev-container
+      --kafka-addr=PLAINTEXT://0.0.0.0:9092
+      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
+
+  jaeger:
+    image: jaegertracing/jaeger:latest
+    container_name: jaeger
+    environment:
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
+      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - ./jaeger.yaml:/etc/jaeger/config.yml
+      - jaeger-data:/badger
+    ports:
+      - "16686:16686" # Web UI
+      - "14269:14269" # Admin/Metrics
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  # --- Metrics ---
+
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: prometheus
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-data:/prometheus
+    ports:
+      - "9090:9090"
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--web.enable-otlp-receiver" # Enable OTLP
+      - "--web.enable-remote-write-receiver" # Enable remote write
+      - "--enable-feature=promql-experimental-functions" # Enable info()
+      - "--storage.tsdb.retention.time=30d"
+    restart: unless-stopped
+    networks:
+      - otel-network
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Logging ---
+
+  loki:
+    image: grafana/loki:latest
+    container_name: loki
+    volumes:
+      - ./loki.yaml:/etc/loki/loki.yaml:ro
+      - loki-data:/loki
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/loki.yaml
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  # --- Collection ---
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    ports:
+      - "1888:1888" # pprof
+      - "8888:8888" # Prometheus metrics for Collector
+      - "8889:8889" # Prometheus metrics for application indicators
+      - "13133:13133" # health check
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
+      - "55679:55679" # zpages
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Profiles ---
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    container_name: pyroscope
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - otel-network
+    restart: unless-stopped
+
+  # --- Visualization ---
+
+  grafana:
+    image: grafana/grafana:latest
+    container_name: grafana
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+      - GF_SECURITY_ADMIN_USER=admin
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/etc/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+    healthcheck:
+      test:
+        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  rustfs-data:
+  tempo-data:
+  jaeger-data:
+  prometheus-data:
+  loki-data:
+  grafana-data:
+
+networks:
+  otel-network:
+    driver: bridge
+    name: "network_otel"
+    ipam:
+      config:
+        - subnet: 172.28.0.0/16
+    driver_opts:
+      com.docker.network.enable_ipv6: "true"

.docker/observability/docker-compose-tempo-ha-override.yml

@@ -0,0 +1,62 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Docker Compose override file for High Availability Tempo setup
+#
+# Usage:
+# docker-compose -f docker-compose-example-for-rustfs.yml \
+#                -f docker-compose-tempo-ha-override.yml up
+
+services:
+  # Override Tempo to use high-availability configuration
+  tempo:
+    volumes:
+      - ./tempo-ha.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200"   # Tempo HTTP
+      - "4317:4317"   # OTLP gRPC
+      - "4318:4318"   # OTLP HTTP
+      - "7946:7946"   # Memberlist
+      - "14250:14250" # Jaeger gRPC
+      - "14268:14268" # Jaeger Thrift HTTP
+      - "9411:9411"   # Zipkin
+    environment:
+      - TEMPO_MEMBERLIST_BIND_PORT=7946
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    depends_on:
+      - redpanda
+
+volumes:
+  tempo-data:
+    driver: local
+    driver_opts:
+      type: tmpfs
+      device: tmpfs
+      o: "size=4g"  # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
+
+# Network configuration remains the same
+# networks:
+#   otel-network:
+#     driver: bridge
+#     name: "network_otel"
+#     ipam:
+#       config:
+#         - subnet: 172.28.0.0/16
+

.docker/observability/docker-compose.yml

@@ -13,100 +13,64 @@
 # limitations under the License.
 
 services:
-  rustfs:
-    security_opt:
-      - "no-new-privileges:true"
-    image: rustfs/rustfs:latest
-    container_name: rustfs-server
-    ports:
-      - "9000:9000" # S3 API port
-      - "9001:9001" # Console port
-    environment:
-      - RUSTFS_VOLUMES=/data/rustfs
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
-      - RUSTFS_OBS_LOGGER_LEVEL=info
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-    volumes:
-      - rustfs-data:/data/rustfs
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test:
-        [
-          "CMD",
-          "sh",
-          "-c",
-          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
-        ]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 40s
-    depends_on:
-      otel-collector:
-        condition: service_started
-
-  rustfs-init:
-    image: alpine
-    container_name: rustfs-init
-    volumes:
-      - rustfs-data:/data
-    networks:
-      - otel-network
-    command: >
-      sh -c "
-        chown -R 10001:10001 /data &&
-        echo 'Volume Permissions fixed' &&
-        exit 0
-      "
-    restart: no
 
   # --- Tracing ---
-
   tempo:
-    image: grafana/tempo:latest
+    image: grafana/tempo:2.10.3
     container_name: tempo
-    command: ["-config.file=/etc/tempo.yaml"]
+    depends_on:
+      - redpanda
+    command: [ "-config.file=/etc/tempo.yaml" ]
     volumes:
       - ./tempo.yaml:/etc/tempo.yaml:ro
       - tempo-data:/var/tempo
     ports:
       - "3200:3200" # tempo
-      - "4317"      # otlp grpc
-      - "4318"      # otlp http
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+      - "7946" # memberlist
     networks:
       - otel-network
     restart: unless-stopped
-    depends_on:
-      - redpanda
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3200/ready"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
       interval: 10s
       timeout: 5s
       retries: 3
       start_period: 15s
-
   redpanda:
-    image: redpandadata/redpanda:latest # for tempo ingest
-    container_name: redpanda
+    image: redpandadata/redpanda:latest
     ports:
-      - "9092:9092"
-    networks:
-      - otel-network
-    restart: unless-stopped
+      - "9092:9092" # Kafka API for clients
     command: >
       redpanda start --overprovisioned
       --mode=dev-container
       --kafka-addr=PLAINTEXT://0.0.0.0:9092
       --advertise-kafka-addr=PLAINTEXT://redpanda:9092
 
+  redpanda-console:
+    image: docker.redpanda.com/redpandadata/console:latest
+    environment:
+      - CONFIG_FILEPATH=/etc/redpanda/redpanda-console-config.yaml
+    volumes:
+      - ./redpanda-console.yaml:/etc/redpanda/redpanda-console-config.yaml
+    ports:
+      - "8080:8080"
+    depends_on:
+      - redpanda
+
+  vulture:
+    image: grafana/tempo-vulture:latest
+    restart: always
+    command:
+      [
+        "-prometheus-listen-address=:8080",
+        "-tempo-query-url=http://tempo:3200",
+        "-tempo-push-url=http://tempo:4317",
+      ]
+    depends_on:
+      - tempo
+
   jaeger:
     image: jaegertracing/jaeger:latest
     container_name: jaeger
@@ -124,12 +88,12 @@ services:
       - "14269:14269" # Admin/Metrics
       - "4317" # otlp grpc
       - "4318" # otlp http
-    command: ["--config", "/etc/jaeger/config.yml"]
+    command: [ "--config", "/etc/jaeger/config.yml" ]
     networks:
       - otel-network
     restart: unless-stopped
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:14269"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
       interval: 10s
       timeout: 5s
       retries: 3
@@ -155,7 +119,7 @@ services:
     networks:
       - otel-network
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
       interval: 10s
       timeout: 5s
       retries: 3
@@ -175,7 +139,7 @@ services:
       - otel-network
     restart: unless-stopped
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3100/ready"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
       interval: 15s
       timeout: 10s
       retries: 5
@@ -188,12 +152,12 @@ services:
     volumes:
       - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
     ports:
-      - "1888:1888"   # pprof
-      - "8888:8888"   # Prometheus metrics for Collector
-      - "8889:8889"   # Prometheus metrics for application indicators
+      - "1888:1888" # pprof
+      - "8888:8888" # Prometheus metrics for Collector
+      - "8889:8889" # Prometheus metrics for application indicators
       - "13133:13133" # health check
-      - "4317:4317"   # OTLP gRPC
-      - "4318:4318"   # OTLP HTTP
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
       - "55679:55679" # zpages
     networks:
       - otel-network
@@ -204,11 +168,24 @@ services:
       - prometheus
       - loki
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:13133"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
       interval: 10s
       timeout: 5s
       retries: 3
 
+  # --- Profiles ---
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    container_name: pyroscope
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - otel-network
+    restart: unless-stopped
+
   # --- Visualization ---
 
   grafana:
@@ -231,13 +208,13 @@ services:
       - tempo
       - loki
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health"]
+      test:
+        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
       interval: 10s
       timeout: 5s
       retries: 3
 
 volumes:
-  rustfs-data:
   tempo-data:
   jaeger-data:
   prometheus-data:

.docker/observability/grafana/provisioning/dashboards/dashboard.yml

@@ -1,3 +1,17 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 apiVersion: 1
 
 providers:

.docker/observability/grafana/provisioning/datasources/datasources.yaml

@@ -1,3 +1,17 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 apiVersion: 1
 
 datasources:
@@ -76,3 +90,9 @@ datasources:
         spanEndTimeShift: '-1s'
         filterByTraceID: true
         filterBySpanID: false
+
+  - name: Pyroscope
+    type: grafana-pyroscope-datasource
+    url: http://pyroscope:4040
+    jsonData:
+      minStep: '15s'

.docker/observability/prometheus.yml

@@ -50,6 +50,11 @@ scrape_configs:
     static_configs:
       - targets: [ 'localhost:9090' ]
 
+  - job_name: 'vulture'
+    static_configs:
+      - targets:
+          - 'vulture:8080'
+
 otlp:
   promote_resource_attributes:
     - service.instance.id

.docker/observability/redpanda-console.yaml

@@ -0,0 +1,3 @@
+kafka:
+  brokers:
+    - redpanda:9092

.docker/observability/tempo-ha.yaml

@@ -0,0 +1,286 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
+# Features:
+# - Distributed architecture with multiple components
+# - Kafka-based ingestion for fault tolerance
+# - Replication factor of 3 for data resilience
+# - Query frontend for load balancing
+# - Metrics generation from traces
+# - WAL for durability
+
+partition_ring_live_store: true
+stream_over_http_enabled: true
+
+server:
+  http_listen_port: 3200
+  http_server_read_timeout: 30s
+  http_server_write_timeout: 30s
+  grpc_server_max_recv_msg_size: 4194304  # 4MB
+  grpc_server_max_send_msg_size: 4194304
+  log_level: info
+  log_format: json
+
+# Memberlist configuration for distributed mode
+memberlist:
+  node_name: tempo
+  bind_port: 7946
+  join_members:
+    - tempo:7946
+  retransmit_factor: 4
+  node_timeout: 15s
+  retransmit_interval: 300ms
+  dead_node_reclaim_time: 30s
+
+# Distributor configuration - receives traces and routes to ingesters
+distributor:
+  ingester_write_path_enabled: true
+  kafka_write_path_enabled: true
+  rate_limit_bytes: 10MB
+  rate_limit_enabled: true
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:4317"
+          max_concurrent_streams: 0
+          max_receive_message_size: 4194304
+        http:
+          endpoint: "0.0.0.0:4318"
+          cors:
+            allowed_origins:
+              - "*"
+            max_age: 86400
+    jaeger:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:14250"
+        thrift_http:
+          endpoint: "0.0.0.0:14268"
+    zipkin:
+      endpoint: "0.0.0.0:9411"
+  ring:
+    kvstore:
+      store: memberlist
+    heartbeat_timeout: 5s
+    replication_factor: 3
+    heartbeat_interval: 5s
+
+# Ingester configuration - stores traces and querying
+ingester:
+  lifecycler:
+    address: tempo
+    ring:
+      kvstore:
+        store: memberlist
+      replication_factor: 3
+      max_cache_freshness_per_sec: 10s
+      heartbeat_interval: 5s
+      heartbeat_timeout: 5s
+    num_tokens: 128
+    tokens_file_path: /var/tempo/tokens.json
+    claim_on_rollout: true
+  trace_idle_period: 20s
+  max_block_bytes: 10_000_000
+  max_block_duration: 10m
+  chunk_size_bytes: 1_000_000
+  chunk_encoding: snappy
+  wal:
+    checkpoint_duration: 5s
+    max_wal_blocks: 4
+  metrics:
+    enabled: true
+    level: block
+    target_info_duration: 15m
+
+# WAL configuration for data durability
+wal:
+  checkpoint_duration: 5s
+  flush_on_shutdown: true
+  path: /var/tempo/wal
+
+# Kafka ingestion configuration - for high throughput scenarios
+ingest:
+  enabled: true
+  kafka:
+    brokers: [ redpanda:9092 ]
+    topic: tempo-ingest
+    encoding: protobuf
+    consumer_group: tempo-ingest-consumer
+    session_timeout: 10s
+    rebalance_timeout: 1m
+    partition: auto
+    verbosity: 2
+
+# Query frontend configuration - distributed querying
+query_frontend:
+  compression: gzip
+  downstream_url: http://localhost:3200
+  log_queries_longer_than: 5s
+  cache_uncompressed_bytes: 100MB
+  max_outstanding_requests_per_tenant: 100
+  max_query_length: 48h
+  max_query_lookback: 30d
+  default_result_cache_ttl: 1m
+  result_cache:
+    cache:
+      enable_fifocache: true
+      default_validity: 1m
+  rf1_after: "1999-01-01T00:00:00Z"
+  mcp_server:
+    enabled: true
+
+# Querier configuration - queries traces
+querier:
+  frontend_worker:
+    frontend_address: localhost:3200
+    grpc_client_config:
+      max_recv_msg_size: 104857600
+  max_concurrent_queries: 20
+  max_metric_bytes_per_trace: 1MB
+
+# Query scheduler configuration - for distributed querying
+query_scheduler:
+  use_scheduler_ring: false
+
+# Metrics generator configuration - generates metrics from traces
+metrics_generator:
+  enabled: true
+  registry:
+    enabled: true
+    external_labels:
+      source: tempo
+      cluster: rustfs-docker-ha
+      environment: production
+  storage:
+    path: /var/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+        resource_to_telemetry_conversion:
+          enabled: true
+  processor:
+    batch:
+      timeout: 10s
+      send_batch_size: 1024
+    memory_limiter:
+      check_interval: 5s
+      limit_mib: 512
+      spike_limit_mib: 128
+  processors:
+    - span-metrics
+    - local-blocks
+    - service-graphs
+  generate_native_histograms: both
+
+# Backend worker configuration
+backend_worker:
+  backend_scheduler_addr: localhost:3200
+  compaction:
+    block_retention: 24h
+    compacted_block_retention: 1h
+  ring:
+    kvstore:
+      store: memberlist
+
+# Backend scheduler configuration
+backend_scheduler:
+  enabled: true
+  provider:
+    compaction:
+      compaction:
+        block_retention: 24h
+        compacted_block_retention: 1h
+        concurrency: 25
+        v2_out_path: /var/tempo/blocks/compaction
+
+# Storage configuration - local backend with proper retention
+storage:
+  trace:
+    backend: local
+    wal:
+      path: /var/tempo/wal
+      checkpoint_duration: 5s
+      flush_on_shutdown: true
+    local:
+      path: /var/tempo/blocks
+      bloom_filter_false_positive: 0.05
+      bloom_shift: 4
+      index:
+        downsample_bytes: 1000000
+        page_size_bytes: 0
+        cache_size_bytes: 0
+    pool:
+      max_workers: 400
+      queue_depth: 10000
+
+# Compactor configuration - manages block compaction
+compactor:
+  compaction:
+    block_retention: 168h  # 7 days
+    compacted_block_retention: 1h
+    concurrency: 25
+    v2_out_path: /var/tempo/blocks/compaction
+    shard_count: 32
+    max_block_bytes: 107374182400  # 100GB
+    max_compaction_objects: 6000000
+    max_time_per_tenant: 5m
+    block_size_bytes: 107374182400
+  ring:
+    kvstore:
+      store: memberlist
+    heartbeat_interval: 5s
+    heartbeat_timeout: 5s
+
+# Limits configuration - rate limiting and quotas
+limits:
+  max_traces_per_user: 10000
+  max_bytes_per_trace: 10485760  # 10MB
+  max_search_bytes_per_trace: 0
+  forgiving_oversize_traces: true
+  rate_limit_bytes: 10MB
+  rate_limit_enabled: true
+  ingestion_burst_size_bytes: 20MB
+  ingestion_rate_limit_bytes: 10MB
+  max_bytes_per_second: 10485760
+  metrics_generator_max_active_series: 10000
+  metrics_generator_max_churned_series: 10000
+  metrics_generator_forta_out_of_order_ttl: 5m
+
+# Override configuration
+overrides:
+  defaults:
+    metrics_generator:
+      processors:
+        - span-metrics
+        - local-blocks
+        - service-graphs
+      generate_native_histograms: both
+      max_active_series: 10000
+      max_churned_series: 10000
+
+# Usage reporting configuration
+usage_report:
+  reporting_enabled: false
+
+# Tracing configuration for debugging
+tracing:
+  enabled: true
+  jaeger:
+    sampler:
+      name: probabilistic
+      param: 0.1
+    reporter_log_spans: false
+

.docker/observability/tempo.yaml

@@ -19,16 +19,20 @@ server:
   http_listen_port: 3200
   log_level: info
 
+memberlist:
+  node_name: tempo
+  bind_port: 7946
+  join_members:
+    - tempo:7946
+
 distributor:
-  ingester_write_path_enabled: false
-  kafka_write_path_enabled: true
   receivers:
     otlp:
       protocols:
         grpc:
-          endpoint: "tempo:4317"
+          endpoint: "0.0.0.0:4317"
         http:
-          endpoint: "tempo:4318"
+          endpoint: "0.0.0.0:4318"
   #log_received_spans:
   #    enabled: true
   #  log_discarded_spans:
@@ -71,14 +75,14 @@ storage:
   trace:
     backend: local
     wal:
-      path: /var/tempo/wal
+      path: /var/tempo/wal # where to store the wal locally
     local:
-      path: /var/tempo/blocks
+      path: /var/tempo/blocks # where to store the traces locally
 
 overrides:
   defaults:
     metrics_generator:
-      processors: ["span-metrics", "service-graphs", "local-blocks"]
+      processors: [ "span-metrics", "service-graphs", "local-blocks" ]
       generate_native_histograms: both
 
 ingest:
@@ -92,3 +96,4 @@ block_builder:
 
 usage_report:
   reporting_enabled: false
+

.github/AGENTS.md

@@ -0,0 +1,30 @@
+# GitHub Workflow Instructions
+
+Applies to `.github/` and repository pull-request operations.
+
+## Pull Requests
+
+- PR titles and descriptions must be in English.
+- Use `.github/pull_request_template.md` for every PR body.
+- Keep all template section headings.
+- Use `N/A` for non-applicable sections.
+- Include verification commands in the PR details.
+- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
+- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
+- Recommended pattern:
+  - `cat > /tmp/pr_body.md <<'EOF'`
+  - `...markdown...`
+  - `EOF`
+  - `gh pr create ... --body-file /tmp/pr_body.md`
+
+## CI Alignment
+
+When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`.
+
+Current `test-and-lint` gate includes:
+
+- `cargo nextest run --all --exclude e2e_test`
+- `cargo test --all --doc`
+- `cargo fmt --all --check`
+- `cargo clippy --all-targets --all-features -- -D warnings`
+- `./scripts/check_layer_dependencies.sh`

.github/dependabot.yml

@@ -26,11 +26,19 @@ updates:
       day: "monday"
       timezone: "Asia/Shanghai"
       time: "08:00"
+    assignees:
+      - "heihutu"
+    reviewers:
+      - "houseme"
+      - "overtrue"
+      - "majinghe"
     ignore:
       - dependency-name: "object_store"
         versions: [ "0.13.x" ]
       - dependency-name: "libunftp"
         versions: [ "0.23.x" ]
+      - dependency-name: "ratelimit"
+        versions: [ "1.x" ]
     groups:
       s3s:
         update-types:

.github/workflows/build.yml

@@ -23,6 +23,7 @@
 #
 # Manual Parameters:
 # - build_docker: Build and push Docker images (default: true)
+# - platforms: Comma-separated platform IDs or 'all' (default: all)
 
 name: Build and Release
 
@@ -44,22 +45,6 @@ on:
       - "**/*.svg"
       - ".gitignore"
       - ".dockerignore"
-  pull_request:
-    branches: [ main ]
-    paths-ignore:
-      - "**.md"
-      - "**.txt"
-      - ".github/**"
-      - "docs/**"
-      - "deploy/**"
-      - "scripts/dev_*.sh"
-      - "LICENSE*"
-      - "README*"
-      - "**/*.png"
-      - "**/*.jpg"
-      - "**/*.svg"
-      - ".gitignore"
-      - ".dockerignore"
   schedule:
     - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
   workflow_dispatch:
@@ -69,6 +54,11 @@ on:
         required: false
         default: true
         type: boolean
+      platforms:
+        description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
+        required: false
+        default: "all"
+        type: string
 
 permissions:
   contents: read
@@ -154,63 +144,70 @@ jobs:
           echo "  - Is prerelease: $is_prerelease"
 
   # Build RustFS binaries
+  prepare-platform-matrix:
+    name: Prepare Platform Matrix
+    runs-on: ubicloud-standard-2
+    outputs:
+      matrix: ${{ steps.select.outputs.matrix }}
+      selected: ${{ steps.select.outputs.selected }}
+    steps:
+      - name: Select target platforms
+        id: select
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
+          selected="$(echo "${selected}" | tr -d '[:space:]')"
+          if [[ -z "${selected}" ]]; then
+            selected="all"
+          fi
+
+          all='{"include":[
+            {"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
+          ]}'
+
+          if [[ "${selected}" == "all" ]]; then
+            matrix="$(jq -c . <<<"${all}")"
+          else
+            unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | ($all.include | map(.target_id)) as $known
+              | [$req[] | select(( $known | index(.) ) == null)]
+            ')"
+            if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
+              echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
+              echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
+              exit 1
+            fi
+
+            matrix="$(jq -c --arg selected "${selected}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | .include |= map(select(.target_id as $id | ($req | index($id))))
+            ' <<<"${all}")"
+          fi
+
+          echo "selected=${selected}" >> "$GITHUB_OUTPUT"
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
+          echo "Selected platforms: ${selected}"
+
   build-rustfs:
     name: Build RustFS
-    needs: [ build-check ]
-    if: needs.build-check.outputs.should_build == 'true'
+    needs: [ build-check, prepare-platform-matrix ]
+    if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
     timeout-minutes: 60
     env:
       RUSTFLAGS: ${{ matrix.rustflags }}
     strategy:
       fail-fast: false
-      matrix:
-        include:
-          # Linux builds
-          # Use x86-64-v2 (SSE4.2 baseline) instead of native to ensure distributed
-          # binaries run on older x86_64 CPUs (e.g. Intel Celeron/Atom, Synology NAS).
-          # See: https://github.com/rustfs/rustfs/issues/1838
-          - os: ubicloud-standard-2
-            target: x86_64-unknown-linux-musl
-            cross: false
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: aarch64-unknown-linux-musl
-            cross: true
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: x86_64-unknown-linux-gnu
-            cross: false
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: aarch64-unknown-linux-gnu
-            cross: true
-            platform: linux
-            rustflags: ''
-          # macOS builds
-          - os: macos-latest
-            target: aarch64-apple-darwin
-            cross: false
-            platform: macos
-            rustflags: ''
-          - os: macos-latest
-            target: x86_64-apple-darwin
-            cross: false
-            platform: macos
-            rustflags: ''
-          # Windows builds (temporarily disabled)
-          - os: windows-latest
-            target: x86_64-pc-windows-msvc
-            cross: false
-            platform: windows
-            rustflags: ''
-          #- os: windows-latest
-          #  target: aarch64-pc-windows-msvc
-          #  cross: true
-          #  platform: windows
+      matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6

.github/workflows/ci.yml

@@ -91,6 +91,8 @@ jobs:
 
   typos:
     name: Typos
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-2
     steps:
       - uses: actions/checkout@v6
@@ -116,9 +118,6 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Install cargo-nextest
-        uses: taiki-e/install-action@nextest
-
       - name: Run tests
         run: |
           cargo nextest run --all --exclude e2e_test
@@ -133,9 +132,40 @@ jobs:
       - name: Check layered dependencies
         run: ./scripts/check_layer_dependencies.sh
 
+  build-rustfs-debug-binary:
+    name: Build RustFS Debug Binary
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-rustfs-debug-binary-${{ hashFiles('**/Cargo.lock') }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build debug binary
+        run: |
+          touch rustfs/build.rs
+          cargo build -p rustfs --bins --jobs 2
+
+      - name: Upload debug binary
+        uses: actions/upload-artifact@v6
+        with:
+          name: rustfs-debug-binary
+          path: target/debug/rustfs
+          if-no-files-found: error
+          retention-days: 1
+
   e2e-tests:
     name: End-to-End Tests
-    needs: skip-check
+    needs: [ skip-check, build-rustfs-debug-binary ]
     if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-2
     timeout-minutes: 30
@@ -148,13 +178,17 @@ jobs:
           rm -rf /tmp/rustfs
           rm -f /tmp/rustfs.log
 
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
         with:
-          rust-version: stable
-          cache-shared-key: ci-e2e-${{ hashFiles('**/Cargo.lock') }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Setup Rust toolchain for s3s-e2e installation
+        uses: dtolnay/rust-toolchain@stable
 
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@v2
@@ -163,12 +197,6 @@ jobs:
           git: https://github.com/s3s-project/s3s.git
           rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
 
-      - name: Build debug binary
-        run: |
-          touch rustfs/build.rs
-          # Limit concurrency to prevent OOM
-          cargo build -p rustfs --bins --jobs 2
-
       - name: Run end-to-end tests
         run: |
           s3s-e2e --version
@@ -184,7 +212,7 @@ jobs:
 
   s3-implemented-tests:
     name: S3 Implemented Tests
-    needs: skip-check
+    needs: [ skip-check, build-rustfs-debug-binary ]
     if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-4
     timeout-minutes: 60
@@ -192,18 +220,14 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v6
 
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
         with:
-          rust-version: stable
-          cache-shared-key: ci-s3tests-${{ hashFiles('**/Cargo.lock') }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          name: rustfs-debug-binary
+          path: target/debug
 
-      - name: Build debug binary
-        run: |
-          touch rustfs/build.rs
-          cargo build -p rustfs --bins --jobs 2
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
 
       - name: Run implemented s3-tests
         run: |

.github/workflows/docker.yml

@@ -66,6 +66,7 @@ env:
   CARGO_TERM_COLOR: always
   REGISTRY_DOCKERHUB: rustfs/rustfs
   REGISTRY_GHCR: ghcr.io/${{ github.repository }}
+  REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
   DOCKER_PLATFORMS: linux/amd64,linux/arm64
 
 jobs:
@@ -287,12 +288,19 @@ jobs:
           username: ${{ env.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
-      # - name: Login to GitHub Container Registry
-      #   uses: docker/login-action@v3
-      #   with:
-      #     registry: ghcr.io
-      #     username: ${{ github.actor }}
-      #     password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ secrets.GHCR_USERNAME }}
+          password: ${{ secrets.GHCR_PASSWORD }}
+
+      - name: Login to Quay.io
+        uses: docker/login-action@v3
+        with:
+          registry: quay.io
+          username: ${{ secrets.QUAY_USERNAME }}
+          password: ${{ secrets.QUAY_PASSWORD }}
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -338,14 +346,15 @@ jobs:
 
           # Generate tags based on build type
           # Only support release and prerelease builds (no development builds)
-          TAGS="${{ env.REGISTRY_DOCKERHUB }}:${VERSION}${VARIANT_SUFFIX}"
+          TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
+          TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
 
           # Add channel tags for prereleases and latest for stable
           if [[ "$CREATE_LATEST" == "true" ]]; then
             # TODO: Temporary change - the current alpha version will also create the latest tag
             # After the version is stabilized, the logic here remains unchanged, but the upstream CREATE_LATEST setting needs to be restored.
             # Stable release (and temporary alpha versions)
-            TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX}"
+            TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
           elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
             # Prerelease channel tags (alpha, beta, rc)
             if [[ "$VERSION" == *"alpha"* ]]; then
@@ -357,7 +366,7 @@ jobs:
             fi
 
             if [[ -n "$CHANNEL" ]]; then
-              TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX}"
+              TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
             fi
           fi
 

.github/workflows/e2e-s3tests.yml

@@ -50,6 +50,11 @@ env:
 
   RUST_LOG: info
   PLATFORM: linux/amd64
+  BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
+  cancel-in-progress: true
 
 defaults:
   run:
@@ -57,12 +62,25 @@ defaults:
 
 jobs:
   s3tests-single:
-    if: github.event.inputs.test-mode == 'single'
+    if: github.event.inputs['test-mode'] == 'single'
     runs-on: ubicloud-standard-2
     timeout-minutes: 120
     steps:
       - uses: actions/checkout@v6
 
+      - name: Cache pip downloads
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-e2e-s3tests-
+
+      - name: Install Python tools
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl tox
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
       - name: Enable buildx
         uses: docker/setup-buildx-action@v3
 
@@ -70,8 +88,8 @@ jobs:
         run: |
           DOCKER_BUILDKIT=1 docker buildx build --load \
             --platform ${PLATFORM} \
-            --cache-from type=gha \
-            --cache-to type=gha,mode=max \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
             -t rustfs-ci \
             -f Dockerfile.source .
 
@@ -121,9 +139,6 @@ jobs:
 
       - name: Provision s3-tests alt user (required by suite)
         run: |
-          python3 -m pip install --user --upgrade pip awscurl
-          export PATH="$HOME/.local/bin:$PATH"
-
           # Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well.
           awscurl \
             --service s3 \
@@ -156,8 +171,6 @@ jobs:
 
       - name: Prepare s3-tests
         run: |
-          python3 -m pip install --user --upgrade pip tox
-          export PATH="$HOME/.local/bin:$PATH"
           git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
 
       - name: Run ceph s3-tests (debug friendly)
@@ -211,12 +224,25 @@ jobs:
           path: artifacts/**
 
   s3tests-multi:
-    if: github.event_name == 'workflow_dispatch' && github.event.inputs.test-mode == 'multi'
+    if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi'
     runs-on: ubicloud-standard-2
     timeout-minutes: 150
     steps:
       - uses: actions/checkout@v6
 
+      - name: Cache pip downloads
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-e2e-s3tests-
+
+      - name: Install Python tools
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl tox
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
       - name: Enable buildx
         uses: docker/setup-buildx-action@v3
 
@@ -224,8 +250,8 @@ jobs:
         run: |
           DOCKER_BUILDKIT=1 docker buildx build --load \
             --platform ${PLATFORM} \
-            --cache-from type=gha \
-            --cache-to type=gha,mode=max \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
             -t rustfs-ci \
             -f Dockerfile.source .
 
@@ -337,9 +363,6 @@ jobs:
 
       - name: Provision s3-tests alt user (required by suite)
         run: |
-          python3 -m pip install --user --upgrade pip awscurl
-          export PATH="$HOME/.local/bin:$PATH"
-
           awscurl \
             --service s3 \
             --region "${S3_REGION}" \
@@ -368,8 +391,6 @@ jobs:
 
       - name: Prepare s3-tests
         run: |
-          python3 -m pip install --user --upgrade pip tox
-          export PATH="$HOME/.local/bin:$PATH"
           git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
 
       - name: Run ceph s3-tests (multi, debug friendly)

.github/workflows/nix-flake-update.yml

@@ -31,11 +31,11 @@ jobs:
   update-flake:
     name: Update flake.lock
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 45
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
-
+        
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@v3
 
@@ -46,14 +46,18 @@ jobs:
         id: update
         uses: DeterminateSystems/update-flake-lock@main
         with:
+          git-author-name: heihutu
+          git-author-email: heihutu@gmail.com
+          git-committer-name: heihutu
+          git-committer-email: heihutu@gmail.com
           pr-title: "chore(deps): update flake.lock"
           pr-labels: |
             dependencies
             nix
             automated
           commit-msg: "chore(deps): update flake.lock"
-          pr-reviewers: houseme, heihutu
-          token: ${{ secrets.GITHUB_TOKEN }}
+          pr-reviewers: houseme, overtrue, majinghe
+          token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
 
       - name: Log PR details
         if: steps.update.outputs.pull-request-number

.github/workflows/nix.yml

@@ -43,7 +43,7 @@ jobs:
   nix-validation:
     name: Nix Build & Check
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 45
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -57,7 +57,7 @@ jobs:
 
       - name: Setup Magic Nix Cache
         uses: DeterminateSystems/magic-nix-cache-action@v13
-        
+
       - name: Setup Flake Checker
         uses: DeterminateSystems/flake-checker-action@v12
 

.gitignore

@@ -16,6 +16,7 @@ vendor
 cli/rustfs-gui/embedded-rustfs/rustfs
 *.log
 deploy/certs/*
+deploy/data/*
 *jsonl
 .env
 .rustfs.sys
@@ -45,3 +46,7 @@ docs
 # nix stuff
 result*
 *.gz
+rustfs-webdav.code-workspace
+
+.aiexclude
+*.bak

AGENTS.md

@@ -1,103 +1,67 @@
-# Repository Guidelines
+# RustFS Agent Instructions (Global)
 
-This file provides guidance for AI agents and developers working with code in this repository.
+This root file keeps repository-wide rules only.
+Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
 
-## Project Overview
+## Rule Precedence
 
-RustFS is a high-performance distributed object storage software built with Rust, providing S3-compatible APIs and advanced features like data lakes, AI, and big data support. It's designed as an alternative to MinIO with better performance and a more business-friendly Apache 2.0 license.
+1. System/developer instructions.
+2. This file (global defaults).
+3. The nearest `AGENTS.md` in the current path (more specific scope wins).
 
-## ⚠️ Pre-Commit Checklist (MANDATORY)
+If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
+
+## Communication and Language
+
+- Respond in the same language used by the requester.
+- Keep source code, comments, commit messages, and PR title/body in English.
+
+## Sources of Truth
+
+- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
+- Local quality commands: `Makefile` and `.config/make/`
+- CI quality gates: `.github/workflows/ci.yml`
+- PR template: `.github/pull_request_template.md`
+
+Avoid duplicating long crate lists or command matrices in instruction files.
+Reference the source files above instead.
+
+## Mandatory Before Commit
+
+Run and pass:
 
-**Before EVERY commit, you MUST run and pass ALL of the following:**
 ```bash
-cargo fmt --all --check    # Code formatting
-cargo clippy --all-targets --all-features -- -D warnings  # Lints
-cargo test --workspace --exclude e2e_test  # Unit tests
+make pre-commit
 ```
-Or simply run `make pre-commit` which covers all checks. **DO NOT commit if any check fails.**
 
-## Communication Rules
+If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
+Do not commit when required checks fail.
 
-- Respond to the user in the same language used by the user.
-- Code and documentation must be written in English only.
-- **Pull Request titles and descriptions must be written in English** to ensure consistency and accessibility for all contributors.
+## Git and PR Baseline
 
-## Project Structure
+- Use feature branches based on the latest `main`.
+- Follow Conventional Commits, with subject length <= 72 characters.
+- Keep PR title and description in English.
+- Use `.github/pull_request_template.md` and keep all section headings.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands in the PR description.
+- When using `gh pr create`/`gh pr edit`, use `--body-file` instead of inline `--body` for multiline markdown.
 
-The workspace root hosts shared dependencies in `Cargo.toml`. The service binary lives under `rustfs/src/main.rs`, while reusable crates sit in `crates/` (including `ecstore`, `iam`, `kms`, `madmin`, `s3select-api`, `s3select-query`, `config`, `crypto`, `lock`, `filemeta`, `rio`, `common`, `protos`, `audit-logger`, `notify`, `obs`, `workers`, `appauth`, `ahm`, `mcp`, `signer`, `checksums`, `utils`, `zip`, `targets`, and `e2e_test`). Deployment manifests are under `deploy/`, Docker assets sit at the root, and automation lives in `scripts/`.
+## Security Baseline
 
-### Core Architecture
+- Never commit secrets, credentials, or key material.
+- Use environment variables or vault tooling for sensitive configuration.
+- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
-- **Main Binary (`rustfs/`):** Entry point at `rustfs/src/main.rs`, includes admin, auth, config, server, storage, license management, and profiling modules.
-- **Key Crates:** Cargo workspace with 25+ crates supporting S3-compatible APIs, erasure coding storage, IAM, KMS, S3 Select, and observability.
-- **Build System:** Custom `build-rustfs.sh` script, multi-architecture Docker builds, Make/Just task runners, cross-compilation support.
+## Scoped Guidance in This Repository
 
-## Build, Test, and Development Commands
-
-### Quick Commands
-- `cargo check --all-targets` - Fast validation
-- `cargo build --release` or `make build` - Release build
-- `./build-rustfs.sh --dev` - Development build with debug symbols
-- `make pre-commit` - Run all quality checks (fmt, clippy, check, test)
-
-### Testing
-- `cargo test --workspace --exclude e2e_test` - Unit tests
-- `cargo test --package e2e_test` - E2E tests
-- For KMS tests: `NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1`
-
-### Cross-Platform Builds
-- `./build-rustfs.sh --platform x86_64-unknown-linux-musl` - Build for musl
-- `./build-rustfs.sh --platform aarch64-unknown-linux-gnu` - Build for ARM64
-- `make build-cross-all` - Build all supported architectures
-
-## Coding Style & Safety Requirements
-
-- **Formatting:** Follow `rustfmt.toml` (130-column width). Use `snake_case` for items, `PascalCase` for types, `SCREAMING_SNAKE_CASE` for constants.
-- **Safety:** `unsafe_code = "deny"` enforced at workspace level. Never use `unwrap()`, `expect()`, or panic-inducing code except in tests.
-- **Error Handling:** Prefer `anyhow` for applications, `thiserror` for libraries. Use proper error handling with `Result<T, E>` and `Option<T>`.
-- **Async:** Keep async code non-blocking. Offload CPU-heavy work with `tokio::task::spawn_blocking` when necessary.
-- **Language:** Code comments, function names, variable names, and all text in source files must be in English only.
-
-## Testing Guidelines
-
-Co-locate unit tests with their modules and give behavior-led names. Integration suites belong in each crate's `tests/` directory, while exhaustive end-to-end scenarios live in `crates/e2e_test/`. When fixing bugs or adding features, include regression tests that capture the new behavior.
-
-## KMS (Key Management Service)
-
-- **Implementation:** Complete with Local and Vault backends, auto-configures on startup with `--kms-enable` flag.
-- **Encryption:** Full S3-compatible server-side encryption (SSE-S3, SSE-KMS, SSE-C).
-- **Testing:** Comprehensive E2E tests in `crates/e2e_test/src/kms/`. Requires proxy bypass for local testing.
-- **Key Files:** `crates/kms/`, `rustfs/src/storage/ecfs.rs`, `rustfs/src/admin/handlers/kms*.rs`
-
-## Environment Variables
-
-- Global configuration environment variables must use flat `RUSTFS_*` names (no module segments), such as `RUSTFS_REGION`, `RUSTFS_ADDRESS`, `RUSTFS_VOLUMES`, and `RUSTFS_LICENSE`.
-- `RUSTFS_SCANNER_ENABLED` - Enable/disable background data scanner (default: true)
-- `RUSTFS_HEAL_ENABLED` - Enable/disable auto-heal functionality (default: true)
-- Deprecated aliases (for pre-beta compatibility) are documented in [the config module README](crates/config/README.md#environment-variable-naming-conventions) and must log warnings when used.
-- For KMS tests: `NO_PROXY=127.0.0.1,localhost` and clear proxy environment variables
-
-## Commit & Pull Request Guidelines
-
-Work on feature branches (e.g., `feat/...`) after syncing `main`. Follow Conventional Commits under 72 characters. Each commit must compile, format cleanly, and pass `make pre-commit`.
-
-**Pull Request Requirements:**
-- PR titles and descriptions **MUST be written in English**
-- Open PRs with a concise summary, note verification commands, link relevant issues
-- Follow the PR template format and fill in all required sections
-- Wait for reviewer approval before merging
-
-## Security & Configuration Tips
-
-Do not commit secrets or cloud credentials; prefer environment variables or vault tooling. Review IAM- and KMS-related changes with a second maintainer. Confirm proxy settings before running sensitive tests to avoid leaking traffic outside localhost.
-
-## Important Reminders
-
-- Always compile after code changes: Use `cargo build` to catch errors early
-- Don't bypass tests: All functionality must be properly tested, not worked around
-- Use proper error handling: Never use `unwrap()` or `expect()` in production code (except tests)
-- Do what has been asked; nothing more, nothing less
-- NEVER create files unless they're absolutely necessary for achieving your goal
-- ALWAYS prefer editing an existing file to creating a new one
-- NEVER proactively create documentation files (*.md) or README files unless explicitly requested
-- NEVER commit PR description files (e.g., PR_DESCRIPTION.md): These are temporary reference files for creating pull requests and should remain local only
+- `.github/AGENTS.md`
+- `crates/AGENTS.md`
+- `crates/config/AGENTS.md`
+- `crates/ecstore/AGENTS.md`
+- `crates/e2e_test/AGENTS.md`
+- `crates/iam/AGENTS.md`
+- `crates/kms/AGENTS.md`
+- `crates/policy/AGENTS.md`
+- `rustfs/src/admin/AGENTS.md`
+- `rustfs/src/storage/AGENTS.md`

CHANGELOG.md

@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
+
 ### Added
 - **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
   - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate

Cargo.toml

@@ -39,6 +39,7 @@ members = [
     "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
+    "crates/s3-common", # Common utilities and data structures for S3 compatibility
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
     "crates/scanner", # Scanner for data integrity checks and health monitoring
@@ -49,7 +50,7 @@ members = [
     "crates/workers", # Worker thread pools and task scheduling
     "crates/zip", # ZIP file handling and compression
 ]
-resolver = "2"
+resolver = "3"
 
 [workspace.package]
 edition = "2024"
@@ -94,6 +95,7 @@ rustfs-obs = { path = "crates/obs", version = "0.0.5" }
 rustfs-policy = { path = "crates/policy", version = "0.0.5" }
 rustfs-protos = { path = "crates/protos", version = "0.0.5" }
 rustfs-rio = { path = "crates/rio", version = "0.0.5" }
+rustfs-s3-common = { path = "crates/s3-common", version = "0.0.5" }
 rustfs-s3select-api = { path = "crates/s3select-api", version = "0.0.5" }
 rustfs-s3select-query = { path = "crates/s3select-query", version = "0.0.5" }
 rustfs-scanner = { path = "crates/scanner", version = "0.0.5" }
@@ -122,8 +124,8 @@ http = "1.4.0"
 http-body = "1.0.1"
 http-body-util = "0.1.3"
 reqwest = { version = "0.13.2", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
-socket2 = { version = "0.6.2", features = ["all"] }
-tokio = { version = "1.49.0", features = ["fs", "rt-multi-thread"] }
+socket2 = { version = "0.6.3", features = ["all"] }
+tokio = { version = "1.50.0", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
@@ -142,7 +144,7 @@ flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
 prost = "0.14.3"
 quick-xml = "0.39.2"
-rmcp = { version = "0.17.0" }
+rmcp = { version = "1.2.0" }
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
@@ -152,15 +154,15 @@ schemars = "1.2.1"
 
 # Cryptography and Security
 aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
-argon2 = { version = "0.6.0-rc.7" }
-blake3 = { version = "1.8.3", features = ["rayon", "mmap"] }
+argon2 = { version = "0.6.0-rc.8" }
+blake2 = "0.11.0-rc.5"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
 crc-fast = "1.9.0"
 hmac = { version = "0.13.0-rc.5" }
 jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
 openidconnect = { version = "4.0", default-features = false }
 pbkdf2 = "0.13.0-rc.9"
-rsa = { version = "0.10.0-rc.16" }
+rsa = { version = "0.10.0-rc.17" }
 rustls = { version = "0.23.37", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
 rustls-pki-types = "1.14.0"
 sha1 = "0.11.0-rc.5"
@@ -171,44 +173,48 @@ zeroize = { version = "1.8.2", features = ["derive"] }
 # Time and Date
 chrono = { version = "0.4.44", features = ["serde"] }
 humantime = "2.3.0"
-jiff = { version = "0.2.22", features = ["serde"] }
+jiff = { version = "0.2.23", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Utilities and Tools
 anyhow = "1.0.102"
-arc-swap = "1.8.2"
-astral-tokio-tar = "0.5.6"
+arc-swap = "1.9.0"
+astral-tokio-tar = "0.6.0"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.14" }
-aws-credential-types = { version = "1.2.13" }
-aws-sdk-s3 = { version = "1.124.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-http-client = { version = "1.1.11", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.5" }
+aws-config = { version = "1.8.15" }
+aws-credential-types = { version = "1.2.14" }
+aws-sdk-s3 = { version = "1.127.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-types = { version = "1.4.7" }
 backtrace = "0.3.76"
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
 cfg-if = "1.0.4"
-clap = { version = "4.5.60", features = ["derive", "env"] }
+clap = { version = "4.6.0", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
 criterion = { version = "0.8", features = ["html_reports"] }
 crossbeam-queue = "0.3.12"
-datafusion = "52.2.0"
+crossbeam-channel = "0.5.15"
+crossbeam-deque = "0.8.6"
+crossbeam-utils = "0.8.21"
+datafusion = "52.4.0"
 derive_builder = "0.20.2"
 enumset = "1.1.10"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.8.0"
-google-cloud-auth = "1.6.0"
+google-cloud-storage = "1.9.0"
+google-cloud-auth = "1.7.0"
 hashbrown = { version = "0.16.1", features = ["serde", "rayon"] }
+hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
-libc = "0.2.182"
+libc = "0.2.183"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.10"
 lz4 = "1.28.1"
@@ -216,7 +222,7 @@ matchit = "0.9.1"
 md-5 = "0.11.0-rc.5"
 md5 = "0.8.0"
 mime_guess = "2.0.5"
-moka = { version = "0.12.14", features = ["future"] }
+moka = { version = "0.12.15", features = ["future"] }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.0"
@@ -224,57 +230,60 @@ object_store = "0.12.5"
 parking_lot = "0.12.5"
 path-absolutize = "3.1.1"
 path-clean = "1.0.1"
+percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.0", features = ["serde"] }
-ratelimit = "0.10.0"
+ratelimit = "0.10.1"
 rayon = "1.11.0"
-reed-solomon-simd = { version = "3.1.0" }
+reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
+reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
 rumqttc = { version = "0.25.1" }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.1" }
-s3s = { version = "0.13.0", features = ["minio"] }
+s3s = { git = "https://github.com/rustfs/s3s", rev = "d9556e3c0036bd3f2b330966009cbaa5aebf19a3", features = ["minio"] }
 serial_test = "3.4.0"
-shadow-rs = { version = "1.7.0", default-features = false }
+shadow-rs = { version = "1.7.1", default-features = false }
 siphasher = "1.0.2"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
-snafu = "0.8.9"
+snafu = "0.9.0"
 snap = "1.1.1"
 starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.38.3"
+sysinfo = "0.38.4"
 temp-env = "0.3.6"
-tempfile = "3.26.0"
+tempfile = "3.27.0"
 test-case = "3.3.1"
 thiserror = "2.0.18"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.4"
 tracing-error = "0.2.1"
 tracing-opentelemetry = "0.32.1"
-tracing-subscriber = { version = "0.3.22", features = ["env-filter", "time"] }
+tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.21.0", features = ["v4", "fast-rng", "macro-diagnostics"] }
-vaultrs = { version = "0.7.4" }
+uuid = { version = "1.22.0", features = ["v4", "fast-rng", "macro-diagnostics"] }
+vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
 windows = { version = "0.62.2" }
 xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
-zip = "8.1.0"
+zip = "8.4.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
 metrics = "0.24.3"
 opentelemetry = { version = "0.31.0" }
 opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
-opentelemetry-otlp = { version = "0.31.0", features = ["gzip-http", "reqwest-rustls"] }
+opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rustls"] }
 opentelemetry_sdk = { version = "0.31.0" }
 opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.31.0" }
+pyroscope = { version = "2.0.0", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
@@ -282,6 +291,9 @@ unftp-core = "0.1.0"
 suppaftp = { version = "8.0.2", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.7"
 
+# WebDAV
+dav-server = "0.11.0"
+
 # Performance Analysis and Memory Profiling
 mimalloc = "0.1"
 # Use tikv-jemallocator as memory allocator and enable performance analysis
@@ -291,7 +303,9 @@ tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profilin
 # Used to generate pprof-compatible memory profiling data and support symbolization and flame graphs
 jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
-pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
+# pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
+# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
+pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs", "rustfs-mcp"]

Dockerfile

@@ -72,7 +72,8 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-RUN apk add --no-cache ca-certificates coreutils curl
+RUN apk update && \
+    apk add --no-cache ca-certificates coreutils curl "zlib>=1.3.2-r0"
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs
@@ -86,12 +87,7 @@ RUN addgroup -g 10001 -S rustfs && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_CONSOLE_ADDRESS=":9001" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_CORS_ALLOWED_ORIGINS="*" \
+ENV RUSTFS_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \

Dockerfile.decommission-local

@@ -0,0 +1,65 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM rust:1.91-trixie
+
+RUN set -eux; \
+    export DEBIAN_FRONTEND=noninteractive; \
+    apt-get update; \
+    apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    curl \
+    git \
+    pkg-config \
+    libssl-dev \
+    lld \
+    protobuf-compiler \
+    flatbuffers-compiler; \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /usr/src/rustfs
+
+COPY . .
+
+RUN ./scripts/static.sh
+RUN cargo run --bin gproto
+RUN cargo build --release --locked --bin rustfs
+
+RUN set -eux; \
+    groupadd -g 10001 rustfs; \
+    useradd -u 10001 -g rustfs -M -s /usr/sbin/nologin rustfs
+
+WORKDIR /app
+
+RUN set -eux; \
+    mkdir -p /data /logs; \
+    chown -R rustfs:rustfs /data /logs /app; \
+    chmod 0750 /data /logs
+
+COPY entrypoint.sh /entrypoint.sh
+RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && chmod +x /entrypoint.sh
+
+ENV RUSTFS_VOLUMES="/data" \
+    RUST_LOG="warn" \
+    RUSTFS_OBS_LOG_DIRECTORY="/logs" \
+    RUSTFS_USERNAME="rustfs" \
+    RUSTFS_GROUPNAME="rustfs" \
+    RUSTFS_UID="10001" \
+    RUSTFS_GID="10001"
+
+EXPOSE 9000 9001
+
+ENTRYPOINT ["/entrypoint.sh"]
+CMD ["/usr/bin/rustfs"]

Dockerfile.source

@@ -157,11 +157,7 @@ WORKDIR /app
 ENV CARGO_INCREMENTAL=1
 
 # Ensure we have the same default env vars available
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_VOLUMES="/data" \
+ENV RUSTFS_VOLUMES="/data" \
     RUST_LOG="warn" \
     RUSTFS_OBS_LOG_DIRECTORY="/logs" \
     RUSTFS_USERNAME="rustfs" \
@@ -222,11 +218,7 @@ COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
 # Default environment (override in docker run/compose as needed)
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_VOLUMES="/data" \
+ENV RUSTFS_VOLUMES="/data" \
     RUST_LOG="warn" \
     RUSTFS_USERNAME="rustfs" \
     RUSTFS_GROUPNAME="rustfs" \

README.md

@@ -1,4 +1,4 @@
-[![RustFS](https://github.com/user-attachments/assets/3ba82e75-2f2d-4415-a4aa-1e4ffe9f22fd)](https://rustfs.com)
+[![RustFS](https://repository-images.githubusercontent.com/722597620/0fa936a2-8164-4f53-867f-def4beb64b21)](https://rustfs.com)
 
 <p align="center">RustFS is a high-performance, distributed object storage system built in Rust.</p>
 
@@ -42,6 +42,7 @@ Unlike other storage systems, RustFS is released under the permissible Apache 2.
 - **High Performance**: Built with Rust to ensure maximum speed and resource efficiency.
 - **Distributed Architecture**: Scalable and fault-tolerant design suitable for large-scale deployments.
 - **S3 Compatibility**: Seamless integration with existing S3-compatible applications and tools.
+- **OpenStack Swift API**: Native support for Swift protocol with Keystone authentication.
 - **OpenStack Keystone Integration**: Native support for OpenStack Keystone authentication with X-Auth-Token headers.
 - **Data Lake Support**: Optimized for high-throughput big data and AI workloads.
 - **Open Source**: Licensed under Apache 2.0, encouraging unrestricted community contributions and commercial usage.
@@ -56,6 +57,7 @@ Unlike other storage systems, RustFS is released under the permissible Apache 2.
 | **Event Notifications** | ✅ Available | **Distributed Mode**     | 🚧 Under Testing |
 | **K8s Helm Charts**     | ✅ Available | **RustFS KMS**           | 🚧 Under Testing |
 | **Keystone Auth**       | ✅ Available | **Multi-Tenancy**        | ✅ Available     |
+| **Swift API**           | ✅ Available | **Swift Metadata Ops**   | 🚧 Partial       |
 
 ## RustFS vs MinIO Performance
 

README_ZH.md

@@ -100,7 +100,7 @@ curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
 
 ### 2\. Docker 快速启动 (选项 2)
 
-RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用 Docker 的 `-v` 参数挂载宿主机目录，请务必确保宿主机目录的所有者已更改为 `1000`，否则会遇到权限拒绝错误。
+RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用 Docker 的 `-v` 参数挂载宿主机目录，请务必确保宿主机目录的所有者已更改为 `10001`，否则会遇到权限拒绝错误。
 
 ```bash
  # 创建数据和日志目录

_typos.toml

@@ -42,6 +42,8 @@ GAE = "GAE"
 # s3-tests original test names (cannot be changed)
 nonexisted = "nonexisted"
 consts = "consts"
+# Swift API - company/product names
+Hashi = "Hashi"  # HashiCorp
 
 [files]
 extend-exclude = []

crates/AGENTS.md

@@ -0,0 +1,20 @@
+# Crates Instructions
+
+Applies to all paths under `crates/`.
+
+## Library Design
+
+- Treat crate code as reusable library code by default.
+- Prefer `thiserror` for library-facing error types.
+- Do not use `unwrap()`, `expect()`, or panic-driven control flow outside tests.
+
+## Testing
+
+- Keep unit tests close to the module they test.
+- Keep integration tests under each crate's `tests/` directory.
+- Add regression tests for bug fixes and behavior changes.
+
+## Async and Performance
+
+- Keep async paths non-blocking.
+- Move CPU-heavy operations out of async hot paths with `tokio::task::spawn_blocking` when appropriate.

crates/appauth/Cargo.toml

@@ -33,3 +33,6 @@ rand.workspace = true
 
 [lints]
 workspace = true
+
+[lib]
+doctest = false

crates/audit/Cargo.toml

@@ -29,6 +29,7 @@ categories = ["web-programming", "development-tools", "asynchronous", "api-bindi
 rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
+rustfs-s3-common = { workspace = true }
 async-trait = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }
@@ -46,3 +47,7 @@ rumqttc = { workspace = true }
 
 [lints]
 workspace = true
+
+[lib]
+test = false
+doctest = false

crates/audit/src/entity.rs

@@ -14,7 +14,7 @@
 
 use chrono::{DateTime, Utc};
 use hashbrown::HashMap;
-use rustfs_targets::EventName;
+use rustfs_s3_common::EventName;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
 

crates/audit/src/registry.rs

@@ -313,6 +313,12 @@ impl AuditRegistry {
                 }
             }
 
+            if &new_config == config {
+                info!("Audit target configuration unchanged, skip persisting server config");
+                info!(count = successful_targets.len(), "All target processing completed");
+                return Ok(successful_targets);
+            }
+
             let Some(store) = rustfs_ecstore::global::new_object_layer_fn() else {
                 return Err(AuditError::StorageNotAvailable(
                     "Failed to save target configuration: server storage not initialized".to_string(),

crates/audit/tests/performance_test.rs

@@ -159,7 +159,8 @@ async fn test_audit_log_dispatch_performance() {
 
     let start = Instant::now();
 
-    // Dispatch audit log (should be fast since no targets are configured)
+    // Dispatch audit log against an unstarted system state. Empty config keeps
+    // the audit system stopped, so dispatch should fail fast without targets.
     let result = system.dispatch(Arc::new(audit_entry)).await;
     let elapsed = start.elapsed();
 
@@ -168,8 +169,10 @@ async fn test_audit_log_dispatch_performance() {
     // Should be very fast (sub-millisecond for no targets)
     assert!(elapsed < Duration::from_millis(100), "Dispatch took too long: {elapsed:?}");
 
-    // Should succeed even with no targets
-    assert!(result.is_ok(), "Dispatch should succeed with no targets");
+    assert!(
+        matches!(result, Err(AuditError::NotInitialized(_))),
+        "Dispatch on a stopped system should return NotInitialized, got: {result:?}"
+    );
 
     // Clean up
     let _ = system.close().await;
@@ -186,11 +189,11 @@ async fn test_system_state_transitions() {
     let config = rustfs_ecstore::config::Config(std::collections::HashMap::new());
     let start_result = system.start(config).await;
 
-    // Should be running (or failed due to server storage)
+    // Empty config keeps the audit system stopped even when start() succeeds.
     let state = system.get_state().await;
     match start_result {
         Ok(_) => {
-            assert_eq!(state, rustfs_audit::system::AuditSystemState::Running);
+            assert_eq!(state, rustfs_audit::system::AuditSystemState::Stopped);
         }
         Err(_) => {
             // Expected in test environment due to server storage not being initialized

crates/audit/tests/system_integration_test.rs

@@ -29,27 +29,21 @@ async fn test_complete_audit_system_lifecycle() {
     assert_eq!(system.get_state().await, system::AuditSystemState::Stopped);
     assert!(!system.is_running().await);
 
-    // 2. Start with empty config (will fail due to no server storage in test)
+    // 2. Start with empty config. The current implementation returns Ok(())
+    // but keeps the system stopped when no audit targets are enabled.
     let config = Config(HashMap::new());
     let start_result = system.start(config).await;
 
-    // Should fail in test environment but state handling should work
+    // State handling should remain consistent for both empty-config success and
+    // storage-unavailable failure paths.
     match start_result {
         Err(AuditError::StorageNotAvailable(_)) => {
             // Expected in test environment
             assert_eq!(system.get_state().await, system::AuditSystemState::Stopped);
         }
         Ok(_) => {
-            // If it somehow succeeds, verify running state
-            assert_eq!(system.get_state().await, system::AuditSystemState::Running);
-            assert!(system.is_running().await);
-
-            // Test pause/resume
-            system.pause().await.expect("Should pause successfully");
-            assert_eq!(system.get_state().await, system::AuditSystemState::Paused);
-
-            system.resume().await.expect("Should resume successfully");
-            assert_eq!(system.get_state().await, system::AuditSystemState::Running);
+            assert_eq!(system.get_state().await, system::AuditSystemState::Stopped);
+            assert!(!system.is_running().await);
         }
         Err(e) => {
             panic!("Unexpected error: {e}");

crates/checksums/Cargo.toml

@@ -36,3 +36,6 @@ sha2 = { workspace = true }
 
 [dev-dependencies]
 pretty_assertions = { workspace = true }
+
+[lib]
+doctest = false

crates/common/Cargo.toml

@@ -41,3 +41,6 @@ rmp-serde = { workspace = true }
 async-trait = { workspace = true }
 s3s = { workspace = true }
 tracing = { workspace = true }
+
+[lib]
+doctest = false

crates/common/src/data_usage.rs

@@ -458,10 +458,7 @@ impl DataUsageEntry {
         self.size += other.size;
 
         if let Some(o_rep) = &other.replication_stats {
-            if self.replication_stats.is_none() {
-                self.replication_stats = Some(ReplicationAllStats::default());
-            }
-            let s_rep = self.replication_stats.as_mut().unwrap();
+            let s_rep = self.replication_stats.get_or_insert_with(ReplicationAllStats::default);
             s_rep.targets.clear();
             s_rep.replica_size += o_rep.replica_size;
             s_rep.replica_count += o_rep.replica_count;
@@ -586,7 +583,7 @@ impl DataUsageCache {
                     return Some(root);
                 }
                 let mut flat = self.flatten(&root);
-                if flat.replication_stats.is_some() && flat.replication_stats.as_ref().unwrap().empty() {
+                if flat.replication_stats.as_ref().is_some_and(|stats| stats.empty()) {
                     flat.replication_stats = None;
                 }
                 Some(flat)
@@ -679,7 +676,9 @@ impl DataUsageCache {
         leaves.sort_by(|a, b| a.objects.cmp(&b.objects));
 
         while remove > 0 && !leaves.is_empty() {
-            let e = leaves.first().unwrap();
+            let Some(e) = leaves.first() else {
+                break;
+            };
             let candidate = e.path.clone();
             if candidate == *path && !compact_self {
                 break;
@@ -703,12 +702,9 @@ impl DataUsageCache {
     }
 
     pub fn total_children_rec(&self, path: &str) -> usize {
-        let root = self.find(path);
-
-        if root.is_none() {
+        let Some(root) = self.find(path) else {
             return 0;
-        }
-        let root = root.unwrap();
+        };
         if root.children.is_empty() {
             return 0;
         }
@@ -721,31 +717,36 @@ impl DataUsageCache {
     }
 
     pub fn merge(&mut self, o: &DataUsageCache) {
-        let mut existing_root = self.root();
-        let other_root = o.root();
-        if existing_root.is_none() && other_root.is_none() {
-            return;
-        }
-        if other_root.is_none() {
-            return;
-        }
-        if existing_root.is_none() {
+        let Some(mut existing_root) = self.root() else {
+            if o.root().is_none() {
+                return;
+            }
             *self = o.clone();
             return;
-        }
-        if o.info.last_update.gt(&self.info.last_update) {
+        };
+
+        let Some(other_root) = o.root() else {
+            return;
+        };
+
+        if o.info.last_update > self.info.last_update {
             self.info.last_update = o.info.last_update;
         }
 
-        existing_root.as_mut().unwrap().merge(other_root.as_ref().unwrap());
-        self.cache.insert(hash_path(&self.info.name).key(), existing_root.unwrap());
-        let e_hash = self.root_hash();
-        for key in other_root.as_ref().unwrap().children.iter() {
-            let entry = &o.cache[key];
+        existing_root.merge(&other_root);
+        self.cache.insert(hash_path(&self.info.name).key(), existing_root);
+
+        let root_hash = self.root_hash();
+        for key in other_root.children.iter() {
+            let Some(entry) = o.cache.get(key) else {
+                continue;
+            };
             let flat = o.flatten(entry);
-            let mut existing = self.cache[key].clone();
-            existing.merge(&flat);
-            self.replace_hashed(&DataUsageHash(key.clone()), &Some(e_hash.clone()), &existing);
+            if let Some(existing) = self.cache.get_mut(key) {
+                existing.merge(&flat);
+            } else {
+                self.replace_hashed(&DataUsageHash(key.clone()), &Some(root_hash.clone()), &flat);
+            }
         }
     }
 
@@ -1141,10 +1142,12 @@ impl DataUsageInfo {
         self.buckets_count = self.buckets_usage.len() as u64;
 
         // Update last update time
-        if let Some(other_update) = other.last_update
-            && (self.last_update.is_none() || other_update > self.last_update.unwrap())
-        {
-            self.last_update = Some(other_update);
+        if let Some(other_update) = other.last_update {
+            match self.last_update {
+                None => self.last_update = Some(other_update),
+                Some(self_update) if other_update > self_update => self.last_update = Some(other_update),
+                _ => {}
+            }
         }
     }
 }
@@ -1285,4 +1288,59 @@ mod tests {
         assert_eq!(summary1.total_size, 300);
         assert_eq!(summary1.versions, 15);
     }
+
+    #[test]
+    fn test_data_usage_cache_merge_adds_missing_child() {
+        let mut base = DataUsageCache::default();
+        base.info.name = "bucket".to_string();
+        base.replace("bucket", "", DataUsageEntry::default());
+
+        let mut other = DataUsageCache::default();
+        other.info.name = "bucket".to_string();
+        let child = DataUsageEntry {
+            size: 42,
+            ..Default::default()
+        };
+        other.replace("bucket/child", "bucket", child);
+
+        base.merge(&other);
+
+        let root = base.find("bucket").expect("root bucket should exist");
+        assert_eq!(root.size, 0);
+        let child_entry = base.find("bucket/child").expect("merged child should be added");
+        assert_eq!(child_entry.size, 42);
+    }
+
+    #[test]
+    fn test_data_usage_cache_merge_accumulates_existing_child() {
+        let mut base = DataUsageCache::default();
+        base.info.name = "bucket".to_string();
+        base.replace(
+            "bucket/child",
+            "bucket",
+            DataUsageEntry {
+                size: 10,
+                objects: 1,
+                ..Default::default()
+            },
+        );
+
+        let mut other = DataUsageCache::default();
+        other.info.name = "bucket".to_string();
+        other.replace(
+            "bucket/child",
+            "bucket",
+            DataUsageEntry {
+                size: 20,
+                objects: 2,
+                ..Default::default()
+            },
+        );
+
+        base.merge(&other);
+
+        let child_entry = base.find("bucket/child").expect("child should remain after merge");
+        assert_eq!(child_entry.size, 30);
+        assert_eq!(child_entry.objects, 3);
+    }
 }

crates/common/src/heal_channel.rs

@@ -296,12 +296,13 @@ type HealResponseSender = broadcast::Sender<HealChannelResponse>;
 static GLOBAL_HEAL_RESPONSE_SENDER: OnceLock<HealResponseSender> = OnceLock::new();
 
 /// Initialize global heal channel
-pub fn init_heal_channel() -> HealChannelReceiver {
+pub fn init_heal_channel() -> Result<HealChannelReceiver, &'static str> {
     let (tx, rx) = mpsc::unbounded_channel();
-    GLOBAL_HEAL_CHANNEL_SENDER
-        .set(tx)
-        .expect("Heal channel sender already initialized");
-    rx
+    if GLOBAL_HEAL_CHANNEL_SENDER.set(tx).is_ok() {
+        Ok(rx)
+    } else {
+        Err("Heal channel sender already initialized")
+    }
 }
 
 /// Get global heal channel sender

crates/common/src/internode_metrics.rs

@@ -0,0 +1,170 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use metrics::{counter, gauge};
+use std::sync::{
+    Arc, LazyLock,
+    atomic::{AtomicU64, Ordering},
+};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct InternodeMetricsSnapshot {
+    pub sent_bytes_total: u64,
+    pub recv_bytes_total: u64,
+    pub outgoing_requests_total: u64,
+    pub incoming_requests_total: u64,
+    pub errors_total: u64,
+    pub dial_errors_total: u64,
+    pub dial_avg_time_nanos: u64,
+    pub last_dial_unix_millis: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct InternodeMetrics {
+    sent_bytes_total: AtomicU64,
+    recv_bytes_total: AtomicU64,
+    outgoing_requests_total: AtomicU64,
+    incoming_requests_total: AtomicU64,
+    errors_total: AtomicU64,
+    dial_errors_total: AtomicU64,
+    dial_total_time_nanos: AtomicU64,
+    dial_samples_total: AtomicU64,
+    last_dial_unix_millis: AtomicU64,
+}
+
+impl InternodeMetrics {
+    pub fn record_sent_bytes(&self, bytes: usize) {
+        let bytes = bytes as u64;
+        if bytes == 0 {
+            return;
+        }
+        self.sent_bytes_total.fetch_add(bytes, Ordering::Relaxed);
+        counter!("rustfs.internode.sent.bytes.total").increment(bytes);
+    }
+
+    pub fn record_recv_bytes(&self, bytes: usize) {
+        let bytes = bytes as u64;
+        if bytes == 0 {
+            return;
+        }
+        self.recv_bytes_total.fetch_add(bytes, Ordering::Relaxed);
+        counter!("rustfs.internode.recv.bytes.total").increment(bytes);
+    }
+
+    pub fn record_outgoing_request(&self) {
+        self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
+        counter!("rustfs.internode.requests.outgoing.total").increment(1);
+    }
+
+    pub fn record_incoming_request(&self) {
+        self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
+        counter!("rustfs.internode.requests.incoming.total").increment(1);
+    }
+
+    pub fn record_error(&self) {
+        self.errors_total.fetch_add(1, Ordering::Relaxed);
+        counter!("rustfs.internode.errors.total").increment(1);
+    }
+
+    pub fn record_dial_result(&self, duration: Duration, success: bool) {
+        let elapsed_nanos = duration.as_nanos().min(u128::from(u64::MAX)) as u64;
+        self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
+        let samples = self.dial_samples_total.fetch_add(1, Ordering::Relaxed) + 1;
+        let total = self.dial_total_time_nanos.load(Ordering::Relaxed);
+        gauge!("rustfs.internode.dial.avg_time.nanos").set(total as f64 / samples as f64);
+
+        if !success {
+            self.dial_errors_total.fetch_add(1, Ordering::Relaxed);
+            counter!("rustfs.internode.dial.errors.total").increment(1);
+        }
+
+        let now_ms = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_millis()
+            .min(u128::from(u64::MAX)) as u64;
+        self.last_dial_unix_millis.store(now_ms, Ordering::Relaxed);
+    }
+
+    pub fn snapshot(&self) -> InternodeMetricsSnapshot {
+        let dial_samples_total = self.dial_samples_total.load(Ordering::Relaxed);
+        let dial_total_time_nanos = self.dial_total_time_nanos.load(Ordering::Relaxed);
+        let dial_avg_time_nanos = if dial_samples_total == 0 {
+            0
+        } else {
+            dial_total_time_nanos / dial_samples_total
+        };
+
+        InternodeMetricsSnapshot {
+            sent_bytes_total: self.sent_bytes_total.load(Ordering::Relaxed),
+            recv_bytes_total: self.recv_bytes_total.load(Ordering::Relaxed),
+            outgoing_requests_total: self.outgoing_requests_total.load(Ordering::Relaxed),
+            incoming_requests_total: self.incoming_requests_total.load(Ordering::Relaxed),
+            errors_total: self.errors_total.load(Ordering::Relaxed),
+            dial_errors_total: self.dial_errors_total.load(Ordering::Relaxed),
+            dial_avg_time_nanos,
+            last_dial_unix_millis: self.last_dial_unix_millis.load(Ordering::Relaxed),
+        }
+    }
+
+    #[doc(hidden)]
+    pub fn reset_for_test(&self) {
+        self.sent_bytes_total.store(0, Ordering::Relaxed);
+        self.recv_bytes_total.store(0, Ordering::Relaxed);
+        self.outgoing_requests_total.store(0, Ordering::Relaxed);
+        self.incoming_requests_total.store(0, Ordering::Relaxed);
+        self.errors_total.store(0, Ordering::Relaxed);
+        self.dial_errors_total.store(0, Ordering::Relaxed);
+        self.dial_total_time_nanos.store(0, Ordering::Relaxed);
+        self.dial_samples_total.store(0, Ordering::Relaxed);
+        self.last_dial_unix_millis.store(0, Ordering::Relaxed);
+    }
+}
+
+pub fn global_internode_metrics() -> &'static Arc<InternodeMetrics> {
+    static GLOBAL_INTERNODE_METRICS: LazyLock<Arc<InternodeMetrics>> = LazyLock::new(|| Arc::new(InternodeMetrics::default()));
+    &GLOBAL_INTERNODE_METRICS
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn snapshot_reports_recorded_values() {
+        let metrics = global_internode_metrics();
+        metrics.reset_for_test();
+
+        metrics.record_sent_bytes(64);
+        metrics.record_recv_bytes(32);
+        metrics.record_outgoing_request();
+        metrics.record_incoming_request();
+        metrics.record_error();
+        metrics.record_dial_result(Duration::from_millis(9), true);
+        metrics.record_dial_result(Duration::from_millis(3), false);
+
+        let snapshot = metrics.snapshot();
+        assert_eq!(snapshot.sent_bytes_total, 64);
+        assert_eq!(snapshot.recv_bytes_total, 32);
+        assert_eq!(snapshot.outgoing_requests_total, 1);
+        assert_eq!(snapshot.incoming_requests_total, 1);
+        assert_eq!(snapshot.errors_total, 1);
+        assert_eq!(snapshot.dial_errors_total, 1);
+        assert_eq!(snapshot.dial_avg_time_nanos, 6_000_000);
+        assert!(snapshot.last_dial_unix_millis > 0);
+
+        metrics.reset_for_test();
+    }
+}

crates/common/src/last_minute.rs

@@ -155,10 +155,7 @@ impl LastMinuteLatency {
     }
 
     pub fn add(&mut self, t: &Duration) {
-        let sec = SystemTime::now()
-            .duration_since(UNIX_EPOCH)
-            .expect("Time went backwards")
-            .as_secs();
+        let sec = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs();
         self.forward_to(sec);
         let win_idx = sec % 60;
         self.totals[win_idx as usize].add(t);
@@ -174,10 +171,7 @@ impl LastMinuteLatency {
 
     pub fn get_total(&mut self) -> AccElem {
         let mut res = AccElem::default();
-        let sec = SystemTime::now()
-            .duration_since(UNIX_EPOCH)
-            .expect("Time went backwards")
-            .as_secs();
+        let sec = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs();
         self.forward_to(sec);
         for elem in self.totals.iter() {
             res.merge(elem);

crates/common/src/lib.rs

@@ -17,6 +17,7 @@ pub mod bucket_stats;
 pub mod data_usage;
 pub mod globals;
 pub mod heal_channel;
+pub mod internode_metrics;
 pub mod last_minute;
 pub mod metrics;
 mod readiness;

crates/common/src/readiness.rs

@@ -60,6 +60,23 @@ impl GlobalReadiness {
     pub fn is_ready(&self) -> bool {
         self.status.load(Ordering::SeqCst) == SystemStage::FullReady as u8
     }
+
+    /// Get the current system stage
+    /// # Returns
+    /// The current SystemStage of the service
+    pub fn current_stage(&self) -> SystemStage {
+        match self.status.load(Ordering::SeqCst) {
+            0 => SystemStage::Booting,
+            1 => SystemStage::StorageReady,
+            2 => SystemStage::IamReady,
+            3 => SystemStage::FullReady,
+            invalid => {
+                debug_assert!(false, "GlobalReadiness::current_stage: invalid status value {}", invalid);
+                // Fallback to the most conservative stage on invalid values
+                SystemStage::Booting
+            }
+        }
+    }
 }
 
 #[cfg(test)]

crates/config/AGENTS.md

@@ -0,0 +1,28 @@
+# Config Crate Instructions
+
+Applies to `crates/config/`.
+
+## Environment Variable Naming
+
+- Global configuration variables must use flat `RUSTFS_*` names.
+- Do not introduce module-segmented names such as `RUSTFS_CONFIG_*`.
+
+Canonical examples:
+
+- `RUSTFS_REGION`
+- `RUSTFS_ADDRESS`
+- `RUSTFS_VOLUMES`
+- `RUSTFS_LICENSE`
+- `RUSTFS_SCANNER_ENABLED`
+- `RUSTFS_HEAL_ENABLED`
+
+## Compatibility
+
+- Deprecated aliases must keep warning behavior.
+- Document aliases in `crates/config/README.md`.
+- Any alias change should include tests for both canonical and deprecated forms.
+
+## Source of Truth
+
+- Constants: `crates/config/src/constants/app.rs`
+- Naming conventions: `crates/config/README.md#environment-variable-naming-conventions`

crates/config/Cargo.toml

@@ -37,3 +37,6 @@ constants = ["dep:const-str"]
 notify = ["dep:const-str", "constants"]
 observability = ["constants"]
 opa = ["constants"]
+
+[lib]
+doctest = false

crates/config/README.md

@@ -49,6 +49,12 @@ Current guidance:
 - Deprecated example:
   - `RUSTFS_ENABLE_SCANNER` -> `RUSTFS_SCANNER_ENABLED`
   - `RUSTFS_ENABLE_HEAL` -> `RUSTFS_HEAL_ENABLED`
+  - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` -> `RUSTFS_SCANNER_START_DELAY_SECS`
+
+## Scanner environment aliases
+
+- `RUSTFS_SCANNER_START_DELAY_SECS` (canonical)
+- `RUSTFS_DATA_SCANNER_START_DELAY_SECS` (deprecated alias for compatibility)
 
 ## 📄 License
 

crates/config/src/constants/app.rs

@@ -131,6 +131,64 @@ pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
 /// Environment variable for server volumes.
 pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
 
+/// Environment variable for server access key.
+pub const ENV_RUSTFS_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";
+
+/// Environment variable for server access key file.
+pub const ENV_RUSTFS_ACCESS_KEY_FILE: &str = "RUSTFS_ACCESS_KEY_FILE";
+
+/// Environment variable for server root user.
+pub const ENV_RUSTFS_ROOT_USER: &str = "RUSTFS_ROOT_USER";
+
+/// Environment variable for server secret key.
+pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
+
+/// Environment variable for server secret key file.
+pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
+
+/// Environment variable for server root password.
+pub const ENV_RUSTFS_ROOT_PASSWORD: &str = "RUSTFS_ROOT_PASSWORD";
+
+/// Environment variable for server OBS endpoint.
+pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
+
+/// Environment variable for console server enable.
+pub const ENV_RUSTFS_CONSOLE_ENABLE: &str = "RUSTFS_CONSOLE_ENABLE";
+
+/// Environment variable for console server address.
+pub const ENV_RUSTFS_CONSOLE_ADDRESS: &str = "RUSTFS_CONSOLE_ADDRESS";
+
+/// Environment variable for server tls path.
+pub const ENV_RUSTFS_TLS_PATH: &str = "RUSTFS_TLS_PATH";
+
+/// Environment variable for server KMS enable.
+pub const ENV_RUSTFS_KMS_ENABLE: &str = "RUSTFS_KMS_ENABLE";
+
+/// Default KMS enable for server-side encryption
+/// This is the default value for enabling KMS encryption for server-side encryption.
+/// Default value: false
+pub const DEFAULT_KMS_ENABLE: bool = false;
+
+/// Environment variable for server KMS backend.
+pub const ENV_RUSTFS_KMS_BACKEND: &str = "RUSTFS_KMS_BACKEND";
+
+/// Default KMS backend for server-side encryption
+/// This is the default KMS backend for server-side encryption.
+/// Default value: local
+pub const DEFAULT_KMS_BACKEND: &str = "local";
+
+/// Environment variable for selecting the buffer profile used for adaptive buffer sizing.
+pub const ENV_RUSTFS_BUFFER_PROFILE: &str = "RUSTFS_BUFFER_PROFILE";
+
+/// Default buffer profile for adaptive buffer sizing
+/// This is the default buffer profile for adaptive buffer sizing.
+/// It is used to identify the workload profile for adaptive buffer sizing.
+/// Default value: GeneralPurpose
+pub const DEFAULT_BUFFER_PROFILE: &str = "GeneralPurpose";
+
+/// Default value for the server TLS path if `ENV_RUSTFS_TLS_PATH` is not set.
+pub const DEFAULT_RUSTFS_TLS_PATH: &str = "";
+
 /// Default port for rustfs
 /// This is the default port for rustfs.
 /// This is used to bind the server to a specific port.
@@ -221,6 +279,12 @@ pub const DEFAULT_OBS_METRICS_EXPORT_ENABLED: bool = true;
 /// Environment variable: RUSTFS_OBS_LOGS_EXPORT_ENABLED
 pub const DEFAULT_OBS_LOGS_EXPORT_ENABLED: bool = true;
 
+/// Default profiling export enabled
+/// It is used to enable or disable exporting profiles
+/// Default value: true
+/// Environment variable: RUSTFS_OBS_PROFILING_EXPORT_ENABLED
+pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = true;
+
 /// Default log local logging enabled for rustfs
 /// This is the default log local logging enabled for rustfs.
 /// It is used to enable or disable local logging of the application.

crates/config/src/constants/mod.rs

@@ -28,3 +28,4 @@ pub(crate) mod runtime;
 pub(crate) mod scanner;
 pub(crate) mod targets;
 pub(crate) mod tls;
+pub(crate) mod workload;

crates/config/src/constants/object.rs

@@ -167,3 +167,278 @@ pub const DEFAULT_OBJECT_CACHE_TTI_SECS: u64 = 120;
 ///
 /// Default is set to 5 hits.
 pub const DEFAULT_OBJECT_HOT_MIN_HITS_TO_EXTEND: usize = 5;
+
+/// Skip bitrot hash verification on GetObject reads.
+///
+/// When enabled, GetObject reads skip the per-shard hash
+/// computation and comparison, reducing CPU usage on the read path.
+/// The background scanner still performs full integrity verification.
+/// Does not affect writes, heals, or scanner operations.
+///
+/// Default is false (verify on every read, matching pre-existing behavior).
+pub const ENV_OBJECT_GET_SKIP_BITROT_VERIFY: &str = "RUSTFS_OBJECT_GET_SKIP_BITROT_VERIFY";
+
+/// Default: bitrot verification is enabled on GetObject reads (do not skip).
+pub const DEFAULT_OBJECT_GET_SKIP_BITROT_VERIFY: bool = false;
+
+// =============================================================================
+// Concurrent Request Fix - Timeout and Backpressure Configuration
+// =============================================================================
+
+/// Environment variable for GetObject request timeout in seconds.
+///
+/// When a GetObject request exceeds this duration, it will be cancelled
+/// and return a 504 Gateway Timeout error. This prevents requests from
+/// hanging indefinitely due to deadlocks or resource exhaustion.
+///
+/// Default: 30 seconds (can be overridden by `RUSTFS_OBJECT_GET_TIMEOUT`).
+/// Set to 0 to disable timeout (not recommended for production).
+pub const ENV_OBJECT_GET_TIMEOUT: &str = "RUSTFS_OBJECT_GET_TIMEOUT";
+
+/// Default GetObject request timeout in seconds.
+///
+/// This value balances between allowing large object transfers to complete
+/// and preventing indefinite hangs. For 20-26MB objects with concurrent
+/// range reads, 30 seconds should be sufficient under normal conditions.
+pub const DEFAULT_OBJECT_GET_TIMEOUT: u64 = 30;
+
+/// Environment variable for disk read operation timeout in seconds.
+///
+/// Individual disk read operations that exceed this duration will be
+/// cancelled and treated as failures. This helps detect slow or hung
+/// disks without waiting indefinitely.
+///
+/// Default: 10 seconds (can be overridden by `RUSTFS_OBJECT_DISK_READ_TIMEOUT`).
+pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT";
+
+/// Default disk read timeout in seconds.
+pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;
+
+/// Environment variable for duplex pipe buffer size in bytes.
+///
+/// The duplex pipe connects the disk read task to the HTTP response stream.
+/// A larger buffer reduces backpressure but increases memory usage.
+/// For large objects (20-26MB), a 4MB buffer provides good throughput.
+///
+/// Default: 4194304 (4 MB, can be overridden by `RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE`).
+/// Minimum recommended: 1048576 (1 MB).
+pub const ENV_OBJECT_DUPLEX_BUFFER_SIZE: &str = "RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE";
+
+/// Default duplex buffer size: 4 MB.
+///
+/// This is 4x larger than the original 1 MB buffer, providing better
+/// handling of large objects and reducing backpressure-related hangs.
+pub const DEFAULT_OBJECT_DUPLEX_BUFFER_SIZE: usize = 4 * 1024 * 1024;
+
+/// Environment variable for I/O buffer size in bytes.
+///
+/// This controls the buffer size used for individual I/O operations.
+/// A larger buffer improves throughput for sequential reads but may
+/// increase latency for small random reads.
+///
+/// Default: 131072 (128 KB, can be overridden by `RUSTFS_OBJECT_IO_BUFFER_SIZE`).
+pub const ENV_OBJECT_IO_BUFFER_SIZE: &str = "RUSTFS_OBJECT_IO_BUFFER_SIZE";
+
+/// Default I/O buffer size: 128 KB.
+pub const DEFAULT_OBJECT_IO_BUFFER_SIZE: usize = 128 * 1024;
+
+/// Environment variable to enable/disable lock optimization.
+///
+/// When enabled, read locks are released immediately after metadata
+/// is read, rather than being held for the entire data transfer.
+/// This significantly reduces lock contention under high concurrency.
+///
+/// Default: true (enabled, can be overridden by `RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE`).
+pub const ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE: &str = "RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE";
+
+/// Default: lock optimization is enabled.
+pub const DEFAULT_OBJECT_LOCK_OPTIMIZATION_ENABLE: bool = true;
+
+/// Environment variable to enable/disable priority-based I/O scheduling.
+///
+/// When enabled, smaller requests (< 1MB) are given higher priority
+/// than larger requests (> 10MB), preventing "starvation" of small
+/// requests by large ones.
+///
+/// Default: true (enabled, can be overridden by `RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE`).
+pub const ENV_OBJECT_PRIORITY_SCHEDULING_ENABLE: &str = "RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE";
+
+/// Default: priority scheduling is enabled.
+pub const DEFAULT_OBJECT_PRIORITY_SCHEDULING_ENABLE: bool = true;
+
+/// Environment variable to enable/disable deadlock detection.
+///
+/// When enabled, the system monitors active requests and detects
+/// potential deadlock situations (circular lock wait chains).
+/// This has some performance overhead and is intended for debugging.
+///
+/// Default: false (disabled, can be overridden by `RUSTFS_OBJECT_DEADLOCK_DETECTION_ENABLE`).
+pub const ENV_OBJECT_DEADLOCK_DETECTION_ENABLE: &str = "RUSTFS_OBJECT_DEADLOCK_DETECTION_ENABLE";
+
+/// Default: deadlock detection is disabled for performance.
+pub const DEFAULT_OBJECT_DEADLOCK_DETECTION_ENABLE: bool = false;
+
+/// Environment variable for deadlock detection check interval in seconds.
+///
+/// How often the deadlock detector analyzes the lock wait graph.
+/// More frequent checks detect deadlocks faster but use more CPU.
+///
+/// Default: 5 seconds (can be overridden by `RUSTFS_OBJECT_DEADLOCK_CHECK_INTERVAL`).
+pub const ENV_OBJECT_DEADLOCK_CHECK_INTERVAL: &str = "RUSTFS_OBJECT_DEADLOCK_CHECK_INTERVAL";
+
+/// Default deadlock check interval: 5 seconds.
+pub const DEFAULT_OBJECT_DEADLOCK_CHECK_INTERVAL: u64 = 5;
+
+/// Environment variable for deadlock detection hang threshold in seconds.
+///
+/// Requests that have been running longer than this threshold are
+/// considered "potentially hung" and included in deadlock analysis.
+///
+/// Default: 10 seconds (can be overridden by `RUSTFS_OBJECT_DEADLOCK_HANG_THRESHOLD`).
+pub const ENV_OBJECT_DEADLOCK_HANG_THRESHOLD: &str = "RUSTFS_OBJECT_DEADLOCK_HANG_THRESHOLD";
+
+/// Default hang threshold: 10 seconds.
+pub const DEFAULT_OBJECT_DEADLOCK_HANG_THRESHOLD: u64 = 10;
+
+/// Environment variable for backpressure high watermark percentage.
+///
+/// When buffer usage exceeds this percentage, the system enters
+/// "high watermark" state and may apply backpressure to producers.
+///
+/// Default: 80 (80%, can be overridden by `RUSTFS_OBJECT_BACKPRESSURE_HIGH_WATERMARK`).
+pub const ENV_OBJECT_BACKPRESSURE_HIGH_WATERMARK: &str = "RUSTFS_OBJECT_BACKPRESSURE_HIGH_WATERMARK";
+
+/// Default high watermark: 80%.
+pub const DEFAULT_OBJECT_BACKPRESSURE_HIGH_WATERMARK: u32 = 80;
+
+/// Environment variable for backpressure low watermark percentage.
+///
+/// When buffer usage drops below this percentage after being in
+/// high watermark state, backpressure is released.
+///
+/// Default: 50 (50%, can be overridden by `RUSTFS_OBJECT_BACKPRESSURE_LOW_WATERMARK`).
+pub const ENV_OBJECT_BACKPRESSURE_LOW_WATERMARK: &str = "RUSTFS_OBJECT_BACKPRESSURE_LOW_WATERMARK";
+
+/// Default low watermark: 50%.
+pub const DEFAULT_OBJECT_BACKPRESSURE_LOW_WATERMARK: u32 = 50;
+
+/// Environment variable for lock acquisition timeout in seconds.
+///
+/// When a lock cannot be acquired within this duration, the operation
+/// will fail with a timeout error. This prevents indefinite waiting
+/// for locks that may never be released due to deadlocks.
+///
+/// Default: 5 seconds (can be overridden by `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT`).
+pub const ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT: &str = "RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT";
+
+/// Default lock acquisition timeout: 5 seconds.
+pub const DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT: u64 = 5;
+
+// ============================================================================
+// I/O priority scheduling configuration
+// ============================================================================
+
+/// Environment variable for I/O high priority size threshold in bytes.
+///
+/// Requests smaller than this threshold are classified as high priority.
+/// High priority requests are processed first to prevent starvation of small requests.
+///
+/// Default: 1048576 (1 MB, can be overridden by `RUSTFS_OBJECT_IO_HIGH_PRIORITY_SIZE_THRESHOLD`).
+pub const ENV_OBJECT_IO_HIGH_PRIORITY_SIZE_THRESHOLD: &str = "RUSTFS_OBJECT_IO_HIGH_PRIORITY_SIZE_THRESHOLD";
+
+/// Default high priority size threshold: 1 MB.
+pub const DEFAULT_OBJECT_IO_HIGH_PRIORITY_SIZE_THRESHOLD: usize = 1024 * 1024;
+
+/// Environment variable for I/O low priority size threshold in bytes.
+///
+/// Requests larger than this threshold are classified as low priority.
+/// Low priority requests are processed last to avoid blocking small requests.
+///
+/// Default: 104857600 (100 MB, can be overridden by `RUSTFS_OBJECT_IO_LOW_PRIORITY_SIZE_THRESHOLD`).
+pub const ENV_OBJECT_IO_LOW_PRIORITY_SIZE_THRESHOLD: &str = "RUSTFS_OBJECT_IO_LOW_PRIORITY_SIZE_THRESHOLD";
+
+/// Default low priority size threshold: 100 MB.
+pub const DEFAULT_OBJECT_IO_LOW_PRIORITY_SIZE_THRESHOLD: usize = 100 * 1024 * 1024;
+
+/// Environment variable for high priority queue capacity.
+///
+/// Maximum number of requests that can be queued in the high priority queue.
+///
+/// Default: 32 (can be overridden by `RUSTFS_OBJECT_IO_QUEUE_HIGH_CAPACITY`).
+pub const ENV_OBJECT_IO_QUEUE_HIGH_CAPACITY: &str = "RUSTFS_OBJECT_IO_QUEUE_HIGH_CAPACITY";
+
+/// Default high priority queue capacity: 32.
+pub const DEFAULT_OBJECT_IO_QUEUE_HIGH_CAPACITY: usize = 32;
+
+/// Environment variable for normal priority queue capacity.
+///
+/// Maximum number of requests that can be queued in the normal priority queue.
+///
+/// Default: 64 (can be overridden by `RUSTFS_OBJECT_IO_QUEUE_NORMAL_CAPACITY`).
+pub const ENV_OBJECT_IO_QUEUE_NORMAL_CAPACITY: &str = "RUSTFS_OBJECT_IO_QUEUE_NORMAL_CAPACITY";
+
+/// Default normal priority queue capacity: 64.
+pub const DEFAULT_OBJECT_IO_QUEUE_NORMAL_CAPACITY: usize = 64;
+
+/// Environment variable for low priority queue capacity.
+///
+/// Maximum number of requests that can be queued in the low priority queue.
+///
+/// Default: 16 (can be overridden by `RUSTFS_OBJECT_IO_QUEUE_LOW_CAPACITY`).
+pub const ENV_OBJECT_IO_QUEUE_LOW_CAPACITY: &str = "RUSTFS_OBJECT_IO_QUEUE_LOW_CAPACITY";
+
+/// Default low priority queue capacity: 16.
+pub const DEFAULT_OBJECT_IO_QUEUE_LOW_CAPACITY: usize = 16;
+
+/// Environment variable for starvation prevention check interval in milliseconds.
+///
+/// How often the system checks for starving low-priority requests.
+/// When a low-priority request has been waiting longer than the starvation threshold,
+/// it is promoted to normal priority.
+///
+/// Default: 100 ms (can be overridden by `RUSTFS_OBJECT_IO_STARVATION_PREVENTION_INTERVAL`).
+pub const ENV_OBJECT_IO_STARVATION_PREVENTION_INTERVAL: &str = "RUSTFS_OBJECT_IO_STARVATION_PREVENTION_INTERVAL";
+
+/// Default starvation prevention interval: 100 ms.
+pub const DEFAULT_OBJECT_IO_STARVATION_PREVENTION_INTERVAL: u64 = 100;
+
+/// Environment variable for starvation threshold in seconds.
+///
+/// Maximum time a low-priority request can wait before being promoted to normal priority.
+/// This prevents indefinite starvation of low-priority requests.
+///
+/// Default: 5 seconds (can be overridden by `RUSTFS_OBJECT_IO_STARVATION_THRESHOLD_SECS`).
+pub const ENV_OBJECT_IO_STARVATION_THRESHOLD_SECS: &str = "RUSTFS_OBJECT_IO_STARVATION_THRESHOLD_SECS";
+
+/// Default starvation threshold: 5 seconds.
+pub const DEFAULT_OBJECT_IO_STARVATION_THRESHOLD_SECS: u64 = 5;
+
+/// Environment variable for load sampling window size.
+///
+/// Number of recent samples used to calculate I/O load metrics.
+///
+/// Default: 100 samples (can be overridden by `RUSTFS_OBJECT_IO_LOAD_SAMPLE_WINDOW`).
+pub const ENV_OBJECT_IO_LOAD_SAMPLE_WINDOW: &str = "RUSTFS_OBJECT_IO_LOAD_SAMPLE_WINDOW";
+
+/// Default load sampling window: 100 samples.
+pub const DEFAULT_OBJECT_IO_LOAD_SAMPLE_WINDOW: usize = 100;
+
+/// Environment variable for high load wait time threshold in milliseconds.
+///
+/// When average wait time exceeds this threshold, the system is considered to be under high load.
+///
+/// Default: 50 ms (can be overridden by `RUSTFS_OBJECT_IO_LOAD_HIGH_THRESHOLD_MS`).
+pub const ENV_OBJECT_IO_LOAD_HIGH_THRESHOLD_MS: &str = "RUSTFS_OBJECT_IO_LOAD_HIGH_THRESHOLD_MS";
+
+/// Default high load threshold: 50 ms.
+pub const DEFAULT_OBJECT_IO_LOAD_HIGH_THRESHOLD_MS: u64 = 50;
+
+/// Environment variable for low load wait time threshold in milliseconds.
+///
+/// When average wait time is below this threshold, the system is considered to be under low load.
+///
+/// Default: 10 ms (can be overridden by `RUSTFS_OBJECT_IO_LOAD_LOW_THRESHOLD_MS`).
+pub const ENV_OBJECT_IO_LOAD_LOW_THRESHOLD_MS: &str = "RUSTFS_OBJECT_IO_LOAD_LOW_THRESHOLD_MS";
+
+/// Default low load threshold: 10 ms.
+pub const DEFAULT_OBJECT_IO_LOAD_LOW_THRESHOLD_MS: u64 = 10;

crates/config/src/constants/protocols.rs

@@ -45,3 +45,15 @@ pub const ENV_FTPS_CERTS_DIR: &str = "RUSTFS_FTPS_CERTS_DIR";
 pub const ENV_FTPS_CA_FILE: &str = "RUSTFS_FTPS_CA_FILE";
 pub const ENV_FTPS_PASSIVE_PORTS: &str = "RUSTFS_FTPS_PASSIVE_PORTS";
 pub const ENV_FTPS_EXTERNAL_IP: &str = "RUSTFS_FTPS_EXTERNAL_IP";
+
+/// Default WebDAV server bind address
+pub const DEFAULT_WEBDAV_ADDRESS: &str = "0.0.0.0:8080";
+
+/// WebDAV environment variable names
+pub const ENV_WEBDAV_ENABLE: &str = "RUSTFS_WEBDAV_ENABLE";
+pub const ENV_WEBDAV_ADDRESS: &str = "RUSTFS_WEBDAV_ADDRESS";
+pub const ENV_WEBDAV_TLS_ENABLED: &str = "RUSTFS_WEBDAV_TLS_ENABLED";
+pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
+pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
+pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
+pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";

crates/config/src/constants/scanner.rs

@@ -12,17 +12,107 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-/// Environment variable name that specifies the data scanner start delay in seconds.
-/// - Purpose: Define the delay between data scanner operations.
+use std::time::Duration;
+
+/// Canonical environment variable name that specifies the scanner start delay in seconds.
+/// If set, this overrides the cycle interval derived from `RUSTFS_SCANNER_SPEED`.
 /// - Unit: seconds (u64).
-/// - Valid values: any positive integer.
-/// - Semantics: This delay controls how frequently the data scanner checks for and processes data; shorter delays lead to more responsive scanning but may increase system load.
-/// - Example: `export RUSTFS_DATA_SCANNER_START_DELAY_SECS=10`
-/// - Note: Choose an appropriate delay that balances scanning responsiveness with overall system performance.
+/// - Example: `export RUSTFS_SCANNER_START_DELAY_SECS=10`
+pub const ENV_SCANNER_START_DELAY_SECS: &str = "RUSTFS_SCANNER_START_DELAY_SECS";
+
+/// Deprecated compatibility alias for scanner start delay.
+/// Prefer `RUSTFS_SCANNER_START_DELAY_SECS`.
+#[deprecated(note = "Use RUSTFS_SCANNER_START_DELAY_SECS instead")]
 pub const ENV_DATA_SCANNER_START_DELAY_SECS: &str = "RUSTFS_DATA_SCANNER_START_DELAY_SECS";
 
-/// Default data scanner start delay in seconds if not specified in the environment variable.
-/// - Value: 10 seconds.
-/// - Rationale: This default interval provides a reasonable balance between scanning responsiveness and system load for most deployments.
-/// - Adjustments: Users may modify this value via the `RUSTFS_DATA_SCANNER_START_DELAY_SECS` environment variable based on their specific scanning requirements and system performance.
-pub const DEFAULT_DATA_SCANNER_START_DELAY_SECS: u64 = 60;
+/// Environment variable that selects the scanner speed preset.
+/// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
+/// Controls the sleep factor, maximum sleep duration, and cycle interval.
+/// - Example: `export RUSTFS_SCANNER_SPEED=slow`
+pub const ENV_SCANNER_SPEED: &str = "RUSTFS_SCANNER_SPEED";
+
+/// Default scanner speed preset.
+pub const DEFAULT_SCANNER_SPEED: &str = "default";
+
+/// Environment variable that controls whether the scanner sleeps between operations.
+/// When `true` (default), the scanner throttles itself. When `false`, it runs at full speed.
+/// - Example: `export RUSTFS_SCANNER_IDLE_MODE=false`
+pub const ENV_SCANNER_IDLE_MODE: &str = "RUSTFS_SCANNER_IDLE_MODE";
+
+/// Default scanner idle mode.
+pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
+
+/// Scanner speed preset controlling throttling behavior.
+///
+/// Each preset defines three parameters:
+/// - **sleep_factor**: Multiplier applied to elapsed work time to compute inter-object sleep.
+/// - **max_sleep**: Upper bound on any single throttle sleep.
+/// - **cycle_interval**: Base delay between scan cycles.
+///
+/// | Preset    | Factor | Max Sleep | Cycle Interval |
+/// |-----------|--------|-----------|----------------|
+/// | `fastest` | 0      | 0         | 1 second       |
+/// | `fast`    | 1x     | 100ms     | 1 minute       |
+/// | `default` | 2x     | 1 second  | 1 minute       |
+/// | `slow`    | 10x    | 15 seconds| 1 minute       |
+/// | `slowest` | 100x   | 15 seconds| 30 minutes     |
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum ScannerSpeed {
+    Fastest,
+    Fast,
+    #[default]
+    Default,
+    Slow,
+    Slowest,
+}
+
+impl ScannerSpeed {
+    pub fn sleep_factor(self) -> f64 {
+        match self {
+            Self::Fastest => 0.0,
+            Self::Fast => 1.0,
+            Self::Default => 2.0,
+            Self::Slow => 10.0,
+            Self::Slowest => 100.0,
+        }
+    }
+
+    pub fn max_sleep(self) -> Duration {
+        match self {
+            Self::Fastest => Duration::ZERO,
+            Self::Fast => Duration::from_millis(100),
+            Self::Default => Duration::from_secs(1),
+            Self::Slow | Self::Slowest => Duration::from_secs(15),
+        }
+    }
+
+    pub fn cycle_interval(self) -> Duration {
+        match self {
+            Self::Fastest => Duration::from_secs(1),
+            Self::Fast | Self::Default | Self::Slow => Duration::from_secs(60),
+            Self::Slowest => Duration::from_secs(30 * 60),
+        }
+    }
+
+    pub fn from_env_str(s: &str) -> Self {
+        match s.trim().to_ascii_lowercase().as_str() {
+            "fastest" => Self::Fastest,
+            "fast" => Self::Fast,
+            "slow" => Self::Slow,
+            "slowest" => Self::Slowest,
+            _ => Self::Default,
+        }
+    }
+}
+
+impl std::fmt::Display for ScannerSpeed {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::Fastest => write!(f, "fastest"),
+            Self::Fast => write!(f, "fast"),
+            Self::Default => write!(f, "default"),
+            Self::Slow => write!(f, "slow"),
+            Self::Slowest => write!(f, "slowest"),
+        }
+    }
+}

crates/config/src/constants/workload.rs

@@ -0,0 +1,70 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Workload profile buffer configuration constants.
+//!
+//! This module defines environment variable keys and default values for
+//! custom buffer profile configuration.
+
+use crate::{KI_B, MI_B};
+
+/// Environment variable for minimum buffer size
+/// Default: 64KB (65536 bytes)
+pub const ENV_RUSTFS_BUFFER_MIN_SIZE: &str = "RUSTFS_BUFFER_MIN_SIZE";
+
+/// Environment variable for maximum buffer size
+/// Default: 1MB (1048576 bytes)
+pub const ENV_RUSTFS_BUFFER_MAX_SIZE: &str = "RUSTFS_BUFFER_MAX_SIZE";
+
+/// Environment variable for default buffer size (used when file size is unknown)
+/// Default: 256KB (262144 bytes)
+pub const ENV_RUSTFS_BUFFER_DEFAULT_SIZE: &str = "RUSTFS_BUFFER_DEFAULT_SIZE";
+
+/// Default minimum buffer size: 64KB
+pub const DEFAULT_BUFFER_MIN_SIZE: usize = 64 * KI_B;
+
+/// Default maximum buffer size: 1MB
+pub const DEFAULT_BUFFER_MAX_SIZE: usize = MI_B;
+
+/// Default buffer size for unknown file size: 256KB
+pub const DEFAULT_BUFFER_UNKNOWN_SIZE: usize = 256 * KI_B;
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::MI_B;
+
+    #[test]
+    fn test_default_values() {
+        assert_eq!(DEFAULT_BUFFER_MIN_SIZE, 65536); // 64KB
+        assert_eq!(DEFAULT_BUFFER_MAX_SIZE, 1048576); // 1MB
+        assert_eq!(DEFAULT_BUFFER_UNKNOWN_SIZE, 262144); // 256KB
+    }
+
+    #[test]
+    fn test_constants() {
+        assert_eq!(KI_B, 1024);
+        assert_eq!(MI_B, 1024 * 1024);
+        assert_eq!(64 * KI_B, DEFAULT_BUFFER_MIN_SIZE);
+        assert_eq!(MI_B, DEFAULT_BUFFER_MAX_SIZE);
+        assert_eq!(256 * KI_B, DEFAULT_BUFFER_UNKNOWN_SIZE);
+    }
+
+    #[test]
+    fn test_env_var_names() {
+        assert_eq!(ENV_RUSTFS_BUFFER_MIN_SIZE, "RUSTFS_BUFFER_MIN_SIZE");
+        assert_eq!(ENV_RUSTFS_BUFFER_MAX_SIZE, "RUSTFS_BUFFER_MAX_SIZE");
+        assert_eq!(ENV_RUSTFS_BUFFER_DEFAULT_SIZE, "RUSTFS_BUFFER_DEFAULT_SIZE");
+    }
+}

crates/config/src/lib.rs

@@ -45,6 +45,8 @@ pub use constants::targets::*;
 #[cfg(feature = "constants")]
 pub use constants::tls::*;
 #[cfg(feature = "constants")]
+pub use constants::workload::*;
+#[cfg(feature = "constants")]
 pub mod oidc {
     pub use super::constants::oidc::*;
 }

crates/config/src/observability/mod.rs

@@ -22,6 +22,7 @@ pub const ENV_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 pub const ENV_OBS_TRACE_ENDPOINT: &str = "RUSTFS_OBS_TRACE_ENDPOINT";
 pub const ENV_OBS_METRIC_ENDPOINT: &str = "RUSTFS_OBS_METRIC_ENDPOINT";
 pub const ENV_OBS_LOG_ENDPOINT: &str = "RUSTFS_OBS_LOG_ENDPOINT";
+pub const ENV_OBS_PROFILING_ENDPOINT: &str = "RUSTFS_OBS_PROFILING_ENDPOINT";
 pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
 pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
 pub const ENV_OBS_METER_INTERVAL: &str = "RUSTFS_OBS_METER_INTERVAL";
@@ -33,12 +34,12 @@ pub const ENV_OBS_ENVIRONMENT: &str = "RUSTFS_OBS_ENVIRONMENT";
 pub const ENV_OBS_TRACES_EXPORT_ENABLED: &str = "RUSTFS_OBS_TRACES_EXPORT_ENABLED";
 pub const ENV_OBS_METRICS_EXPORT_ENABLED: &str = "RUSTFS_OBS_METRICS_EXPORT_ENABLED";
 pub const ENV_OBS_LOGS_EXPORT_ENABLED: &str = "RUSTFS_OBS_LOGS_EXPORT_ENABLED";
+pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_ENABLED";
 
 pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
 pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
 pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
 pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
-pub const ENV_OBS_LOG_ROTATION_SIZE_MB: &str = "RUSTFS_OBS_LOG_ROTATION_SIZE_MB";
 pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
 pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
 
@@ -47,6 +48,12 @@ pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZ
 pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
 pub const ENV_OBS_LOG_COMPRESS_OLD_FILES: &str = "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES";
 pub const ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL: &str = "RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL";
+pub const ENV_OBS_LOG_COMPRESSION_ALGORITHM: &str = "RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM";
+pub const ENV_OBS_LOG_PARALLEL_COMPRESS: &str = "RUSTFS_OBS_LOG_PARALLEL_COMPRESS";
+pub const ENV_OBS_LOG_PARALLEL_WORKERS: &str = "RUSTFS_OBS_LOG_PARALLEL_WORKERS";
+pub const ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL: &str = "RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL";
+pub const ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP: &str = "RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP";
+pub const ENV_OBS_LOG_ZSTD_WORKERS: &str = "RUSTFS_OBS_LOG_ZSTD_WORKERS";
 pub const ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: &str = "RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS";
 pub const ENV_OBS_LOG_EXCLUDE_PATTERNS: &str = "RUSTFS_OBS_LOG_EXCLUDE_PATTERNS";
 pub const ENV_OBS_LOG_DELETE_EMPTY_FILES: &str = "RUSTFS_OBS_LOG_DELETE_EMPTY_FILES";
@@ -60,13 +67,24 @@ pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; //
 pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
 pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP: &str = "gzip";
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD: &str = "zstd";
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM: &str = DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD;
+pub const DEFAULT_OBS_LOG_PARALLEL_COMPRESS: bool = true;
+pub const DEFAULT_OBS_LOG_PARALLEL_WORKERS: usize = 6;
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL: i32 = 8;
+pub const DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP: bool = true;
+pub const DEFAULT_OBS_LOG_ZSTD_WORKERS: usize = 1;
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION: &str = "gz";
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION);
-pub const DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: u64 = 30;
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_EXTENSION: &str = "zst";
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_ZSTD_COMPRESSION_EXTENSION);
+pub const DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: u64 = 30; // Retain compressed files for 30 days
 pub const DEFAULT_OBS_LOG_DELETE_EMPTY_FILES: bool = true;
 pub const DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS: u64 = 3600; // 1 hour
 pub const DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS: u64 = 1800; // 0.5 hours
 pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
+pub const DEFAULT_OBS_LOG_MATCH_MODE_PREFIX: &str = "prefix";
 pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";
 
 /// Default values for observability configuration
@@ -90,6 +108,7 @@ mod tests {
         assert_eq!(ENV_OBS_TRACE_ENDPOINT, "RUSTFS_OBS_TRACE_ENDPOINT");
         assert_eq!(ENV_OBS_METRIC_ENDPOINT, "RUSTFS_OBS_METRIC_ENDPOINT");
         assert_eq!(ENV_OBS_LOG_ENDPOINT, "RUSTFS_OBS_LOG_ENDPOINT");
+        assert_eq!(ENV_OBS_PROFILING_ENDPOINT, "RUSTFS_OBS_PROFILING_ENDPOINT");
         assert_eq!(ENV_OBS_USE_STDOUT, "RUSTFS_OBS_USE_STDOUT");
         assert_eq!(ENV_OBS_SAMPLE_RATIO, "RUSTFS_OBS_SAMPLE_RATIO");
         assert_eq!(ENV_OBS_METER_INTERVAL, "RUSTFS_OBS_METER_INTERVAL");
@@ -100,17 +119,23 @@ mod tests {
         assert_eq!(ENV_OBS_LOG_STDOUT_ENABLED, "RUSTFS_OBS_LOG_STDOUT_ENABLED");
         assert_eq!(ENV_OBS_LOG_DIRECTORY, "RUSTFS_OBS_LOG_DIRECTORY");
         assert_eq!(ENV_OBS_LOG_FILENAME, "RUSTFS_OBS_LOG_FILENAME");
-        assert_eq!(ENV_OBS_LOG_ROTATION_SIZE_MB, "RUSTFS_OBS_LOG_ROTATION_SIZE_MB");
         assert_eq!(ENV_OBS_LOG_ROTATION_TIME, "RUSTFS_OBS_LOG_ROTATION_TIME");
         assert_eq!(ENV_OBS_LOG_KEEP_FILES, "RUSTFS_OBS_LOG_KEEP_FILES");
         assert_eq!(ENV_OBS_TRACES_EXPORT_ENABLED, "RUSTFS_OBS_TRACES_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
+        assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
         // Test log cleanup related env keys
         assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_COMPRESS_OLD_FILES, "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES");
         assert_eq!(ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL, "RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL");
+        assert_eq!(ENV_OBS_LOG_COMPRESSION_ALGORITHM, "RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM");
+        assert_eq!(ENV_OBS_LOG_PARALLEL_COMPRESS, "RUSTFS_OBS_LOG_PARALLEL_COMPRESS");
+        assert_eq!(ENV_OBS_LOG_PARALLEL_WORKERS, "RUSTFS_OBS_LOG_PARALLEL_WORKERS");
+        assert_eq!(ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL, "RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL");
+        assert_eq!(ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, "RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP");
+        assert_eq!(ENV_OBS_LOG_ZSTD_WORKERS, "RUSTFS_OBS_LOG_ZSTD_WORKERS");
         assert_eq!(
             ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS,
             "RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS"
@@ -129,6 +154,10 @@ mod tests {
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_DEVELOPMENT, "development");
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_TEST, "test");
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_STAGING, "staging");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP, "gzip");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD, "zstd");
+        assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE_PREFIX, "prefix");
         assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE, "suffix");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, "zstd");
     }
 }

crates/credentials/Cargo.toml

@@ -15,7 +15,7 @@ base64-simd = { workspace = true }
 rand = { workspace = true }
 serde = { workspace = true }
 serde_json.workspace = true
-time = { workspace = true, features = ["serde-human-readable"] }
+time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }
 
 [lints]
 workspace = true

crates/credentials/src/credentials.rs

@@ -233,15 +233,18 @@ impl<'a> fmt::Debug for Masked<'a> {
         match self.0 {
             None => Ok(()),
             Some(s) => {
-                let len = s.len();
+                let len = s.chars().count();
                 if len == 0 {
                     Ok(())
                 } else if len == 1 {
                     write!(f, "***")
                 } else if len == 2 {
-                    write!(f, "{}***|{}", &s[0..1], len)
+                    let first = s.chars().next().ok_or(fmt::Error)?;
+                    write!(f, "{}***|{}", first, len)
                 } else {
-                    write!(f, "{}***{}|{}", &s[0..1], &s[len - 1..], len)
+                    let first = s.chars().next().ok_or(fmt::Error)?;
+                    let last = s.chars().last().ok_or(fmt::Error)?;
+                    write!(f, "{}***{}|{}", first, last, len)
                 }
             }
         }
@@ -270,15 +273,25 @@ impl<'a> fmt::Display for Masked<'a> {
 ///
 #[derive(Serialize, Deserialize, Clone, Default)]
 pub struct Credentials {
+    #[serde(rename = "accessKey", alias = "access_key", default)]
     pub access_key: String,
+    #[serde(rename = "secretKey", alias = "secret_key", default)]
     pub secret_key: String,
+    #[serde(rename = "sessionToken", alias = "session_token", default)]
     pub session_token: String,
+    #[serde(default, with = "crate::serde_datetime::option")]
     pub expiration: Option<OffsetDateTime>,
+    #[serde(default)]
     pub status: String,
+    #[serde(rename = "parentUser", alias = "parent_user", default)]
     pub parent_user: String,
+    #[serde(default)]
     pub groups: Option<Vec<String>>,
+    #[serde(default)]
     pub claims: Option<HashMap<String, Value>>,
+    #[serde(default)]
     pub name: Option<String>,
+    #[serde(default)]
     pub description: Option<String>,
 }
 
@@ -482,5 +495,37 @@ mod tests {
 
         // Test longer string
         assert_eq!(format!("{:?}", Masked(Some("secretpassword"))), "s***d|14");
+
+        // Test Unicode input should not panic and should keep character boundary
+        assert_eq!(format!("{:?}", Masked(Some("中"))), "***");
+        assert_eq!(format!("{:?}", Masked(Some("中文"))), "中***|2");
+        assert_eq!(format!("{:?}", Masked(Some("中文测试"))), "中***试|4");
+    }
+
+    #[test]
+    fn test_credentials_expiration_serialize_as_rfc3339() {
+        use time::OffsetDateTime;
+
+        let c = Credentials {
+            access_key: "ak".to_string(),
+            secret_key: "sk12345678".to_string(),
+            expiration: Some(OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+
+        let json = serde_json::to_string(&c).expect("serialize");
+        assert!(
+            json.contains('T') && (json.contains('Z') || json.contains("+00:00")),
+            "Credentials expiration should be RFC3339; got: {}",
+            json
+        );
+    }
+
+    #[test]
+    fn test_credentials_deserialize_minio_style_rfc3339_expiration() {
+        let minio_style = r#"{"accessKey":"ak","secretKey":"sk12345678","expiration":"2025-03-07T12:00:00Z"}"#;
+        let c: Credentials = serde_json::from_str(minio_style).expect("deserialize");
+        assert_eq!(c.access_key, "ak");
+        assert!(c.expiration.is_some());
     }
 }

crates/credentials/src/lib.rs

@@ -14,6 +14,7 @@
 
 mod constants;
 mod credentials;
+mod serde_datetime;
 
 pub use constants::*;
 pub use credentials::*;

crates/credentials/src/serde_datetime.rs

@@ -0,0 +1,68 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Serde helpers for expiration timestamp: serialize as RFC3339 (MinIO-compatible),
+//! deserialize from RFC3339 or legacy RustFS human-readable format.
+
+use time::OffsetDateTime;
+use time::format_description;
+use time::format_description::well_known::Rfc3339;
+
+static LEGACY_FORMAT: std::sync::OnceLock<time::format_description::OwnedFormatItem> = std::sync::OnceLock::new();
+
+fn legacy_format() -> &'static time::format_description::OwnedFormatItem {
+    LEGACY_FORMAT.get_or_init(|| {
+        format_description::parse_owned::<2>(
+            "[year]-[month]-[day] [hour]:[minute]:[second].[subsecond] [offset_hour sign:mandatory]:[offset_minute]:[offset_second]",
+        )
+        .expect("legacy format description is valid")
+    });
+    LEGACY_FORMAT.get().expect("initialized above")
+}
+
+fn parse_rfc3339_or_legacy(s: &str) -> Result<OffsetDateTime, time::Error> {
+    OffsetDateTime::parse(s, &Rfc3339).or_else(|_| OffsetDateTime::parse(s, legacy_format()).map_err(Into::into))
+}
+
+/// Option<OffsetDateTime>: serialize as RFC3339; deserialize from RFC3339 or legacy.
+pub mod option {
+    use serde::{Deserialize, Deserializer, Serializer};
+    use time::OffsetDateTime;
+
+    use super::{Rfc3339, parse_rfc3339_or_legacy};
+
+    pub fn serialize<S>(opt: &Option<OffsetDateTime>, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        match opt {
+            Some(dt) => {
+                let s = dt.format(&Rfc3339).map_err(serde::ser::Error::custom)?;
+                serializer.serialize_some(&s)
+            }
+            None => serializer.serialize_none(),
+        }
+    }
+
+    pub fn deserialize<'de, D>(deserializer: D) -> Result<Option<OffsetDateTime>, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let opt: Option<&str> = Option::deserialize(deserializer)?;
+        match opt {
+            None => Ok(None),
+            Some(s) => parse_rfc3339_or_legacy(s).map(Some).map_err(serde::de::Error::custom),
+        }
+    }
+}

crates/crypto/Cargo.toml

@@ -56,3 +56,6 @@ crypto = [
     "dep:rand",
     "dep:sha2",
 ]
+
+[lib]
+doctest = false

crates/crypto/src/encdec.rs

@@ -21,5 +21,8 @@ pub(crate) mod id;
 pub(crate) mod decrypt;
 pub(crate) mod encrypt;
 
+#[cfg(any(test, feature = "crypto"))]
+pub(crate) mod stream_io;
+
 #[cfg(test)]
 mod tests;

crates/crypto/src/encdec/stream_io.rs

@@ -0,0 +1,219 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! sio-go compatible stream encryption for IAM config.
+//! Header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes.
+//! Body: DARE-style fragmented AEAD (bufSize=16384, per-fragment nonce).
+
+#![allow(deprecated)] // AeadInPlace deprecated in favor of AeadInOut; keep for aead 0.6 compatibility
+
+use crate::encdec::id::ID;
+use crate::error::Error;
+use aes_gcm::{
+    Aes256Gcm,
+    aead::{AeadCore, AeadInPlace, KeyInit as _, array::Array},
+};
+use chacha20poly1305::ChaCha20Poly1305;
+
+const STREAM_IO_HEADER_LEN: usize = 41;
+const SIO_BUF_SIZE: usize = 16384;
+const SIO_NONCE_PREFIX_LEN: usize = 8;
+const AES_GCM_OVERHEAD: usize = 16;
+const CHACHA_OVERHEAD: usize = 16;
+
+/// Decrypt data in stream_io (sio-go) format.
+pub fn decrypt_stream_io(password: &[u8], data: &[u8]) -> Result<Vec<u8>, Error> {
+    if data.len() < STREAM_IO_HEADER_LEN {
+        return Err(Error::ErrUnexpectedHeader);
+    }
+    let salt = &data[0..32];
+    let id = ID::try_from(data[32])?;
+    let nonce_prefix = &data[33..41];
+    let body = &data[STREAM_IO_HEADER_LEN..];
+
+    let key = id.get_key(password, salt)?;
+
+    match id {
+        ID::Argon2idChaCHa20Poly1305 => decrypt_stream(
+            ChaCha20Poly1305::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            nonce_prefix,
+            body,
+            CHACHA_OVERHEAD,
+        ),
+        _ => decrypt_stream(
+            Aes256Gcm::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            nonce_prefix,
+            body,
+            AES_GCM_OVERHEAD,
+        ),
+    }
+}
+
+fn decrypt_stream<A>(aead: A, nonce_prefix: &[u8], body: &[u8], overhead: usize) -> Result<Vec<u8>, Error>
+where
+    A: AeadInPlace,
+{
+    let ciphertext_len = SIO_BUF_SIZE + overhead;
+    let ad = build_associated_data(&aead, nonce_prefix)?;
+    let mut plain = Vec::with_capacity(body.len());
+
+    let mut seq_num: u32 = 1;
+    let mut pos = 0;
+
+    while pos < body.len() {
+        let remaining = body.len() - pos;
+        let frag_len = remaining.min(ciphertext_len);
+        let is_last = (pos + frag_len) == body.len();
+
+        if frag_len < overhead {
+            return Err(Error::ErrDecryptFailed(aes_gcm::aead::Error));
+        }
+
+        let mut nonce = [0u8; 12];
+        nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+        nonce[8..12].copy_from_slice(&seq_num.to_le_bytes());
+
+        let mut ad_mut = ad.clone();
+        ad_mut[0] = if is_last { 0x80 } else { 0x00 };
+
+        let fragment = &body[pos..pos + frag_len];
+        let tag_len = overhead;
+        let (ct, tag) = fragment.split_at(frag_len - tag_len);
+
+        let mut buffer = ct.to_vec();
+        let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+            .map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        let tag_arr =
+            Array::<u8, <A as AeadCore>::TagSize>::try_from(tag).map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        aead.decrypt_in_place_detached(&nonce_arr, &ad_mut, &mut buffer, &tag_arr)
+            .map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        plain.extend_from_slice(&buffer);
+
+        pos += frag_len;
+        seq_num += 1;
+
+        if is_last {
+            break;
+        }
+    }
+
+    Ok(plain)
+}
+
+fn build_associated_data<A>(aead: &A, nonce_prefix: &[u8]) -> Result<Vec<u8>, Error>
+where
+    A: AeadInPlace,
+{
+    let mut nonce = [0u8; 12];
+    nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+    nonce[8..12].copy_from_slice(&0u32.to_le_bytes());
+
+    let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+        .map_err(|_| Error::ErrEncryptFailed(aes_gcm::aead::Error))?;
+    let mut empty: [u8; 0] = [];
+    let tag = aead
+        .encrypt_in_place_detached(&nonce_arr, &[] as &[u8], &mut empty)
+        .map_err(Error::ErrEncryptFailed)?;
+
+    let mut ad = vec![0u8; 1 + tag.len()];
+    ad[0] = 0x00;
+    ad[1..].copy_from_slice(tag.as_slice());
+    Ok(ad)
+}
+
+/// Encrypt data in stream_io (sio-go) format.
+pub fn encrypt_stream_io(password: &[u8], data: &[u8]) -> Result<Vec<u8>, Error> {
+    let salt: [u8; 32] = rand::random();
+
+    #[cfg(feature = "fips")]
+    let id = ID::Pbkdf2AESGCM;
+
+    #[cfg(not(feature = "fips"))]
+    let id = if crate::encdec::encrypt::native_aes() {
+        ID::Argon2idAESGCM
+    } else {
+        ID::Argon2idChaCHa20Poly1305
+    };
+
+    let key = id.get_key(password, &salt)?;
+    let nonce_prefix: [u8; SIO_NONCE_PREFIX_LEN] = rand::random();
+
+    let mut out = Vec::with_capacity(STREAM_IO_HEADER_LEN + data.len() + 32);
+    out.extend_from_slice(&salt);
+    out.push(id as u8);
+    out.extend_from_slice(&nonce_prefix);
+
+    match id {
+        ID::Argon2idChaCHa20Poly1305 => encrypt_stream(
+            ChaCha20Poly1305::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            &nonce_prefix,
+            data,
+            &mut out,
+            CHACHA_OVERHEAD,
+        )?,
+        _ => encrypt_stream(
+            Aes256Gcm::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            &nonce_prefix,
+            data,
+            &mut out,
+            AES_GCM_OVERHEAD,
+        )?,
+    }
+
+    Ok(out)
+}
+
+fn encrypt_stream<A>(
+    aead: A,
+    nonce_prefix: &[u8; SIO_NONCE_PREFIX_LEN],
+    data: &[u8],
+    out: &mut Vec<u8>,
+    _overhead: usize,
+) -> Result<(), Error>
+where
+    A: AeadInPlace,
+{
+    let ad = build_associated_data(&aead, nonce_prefix)?;
+    let mut seq_num: u32 = 1;
+    let mut pos = 0;
+
+    while pos < data.len() {
+        let remaining = data.len() - pos;
+        let is_last = remaining <= SIO_BUF_SIZE;
+
+        let chunk_len = if is_last { remaining } else { SIO_BUF_SIZE };
+        let chunk = &data[pos..pos + chunk_len];
+
+        let mut nonce = [0u8; 12];
+        nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+        nonce[8..12].copy_from_slice(&seq_num.to_le_bytes());
+
+        let mut ad_mut = ad.clone();
+        ad_mut[0] = if is_last { 0x80 } else { 0x00 };
+
+        let mut buffer = chunk.to_vec();
+        let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+            .map_err(|_| Error::ErrEncryptFailed(aes_gcm::aead::Error))?;
+        let tag = aead
+            .encrypt_in_place_detached(&nonce_arr, &ad_mut, &mut buffer)
+            .map_err(Error::ErrEncryptFailed)?;
+        out.extend_from_slice(&buffer);
+        out.extend_from_slice(tag.as_slice());
+
+        pos += chunk_len;
+        seq_num += 1;
+    }
+
+    Ok(())
+}

crates/crypto/src/encdec/tests.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::{decrypt_data, encrypt_data};
+use crate::{decrypt_data, decrypt_stream_io, encrypt_data, encrypt_stream_io};
 
 const PASSWORD: &[u8] = "test_password".as_bytes();
 const LONG_PASSWORD: &[u8] = "very_long_password_with_many_characters_for_testing_purposes_123456789".as_bytes();
@@ -317,3 +317,67 @@ fn test_concurrent_encryption_safety() -> Result<(), crate::Error> {
 
     Ok(())
 }
+
+#[test]
+fn test_stream_io_roundtrip() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = br#"{"Version":1,"policy":"readonly"}"#;
+    let encrypted = encrypt_stream_io(password, data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert_eq!(data, decrypted.as_slice());
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_large_data_roundtrip() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = vec![0xAB; 32 * 1024]; // > SIO_BUF_SIZE to test fragmentation
+    let encrypted = encrypt_stream_io(password, &data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert_eq!(data, decrypted);
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_wrong_password_fails() {
+    let password = b"access:secret";
+    let data = br#"{"Version":1}"#;
+    let encrypted = encrypt_stream_io(password, data).expect("encrypt should succeed");
+    let result = decrypt_stream_io(b"wrong:password", &encrypted);
+    assert!(result.is_err(), "decrypt with wrong password should fail");
+}
+
+#[test]
+fn test_stream_io_empty_data() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data: &[u8] = &[];
+    let encrypted = encrypt_stream_io(password, data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert!(decrypted.is_empty());
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_header_format() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = b"test";
+    let encrypted = encrypt_stream_io(password, data)?;
+    // stream_io header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes
+    const STREAM_IO_HEADER_LEN: usize = 41;
+    assert!(encrypted.len() >= STREAM_IO_HEADER_LEN, "encrypted should have at least 41-byte header");
+    assert!(
+        encrypted[32] == 0x00 || encrypted[32] == 0x01 || encrypted[32] == 0x02,
+        "alg_id should be 0x00, 0x01, or 0x02"
+    );
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_truncated_data_fails() {
+    let password = b"access:secret";
+    let data = b"test";
+    let encrypted = encrypt_stream_io(password, data).expect("encrypt should succeed");
+    let truncated = &encrypted[..40]; // less than 41-byte header
+    let result = decrypt_stream_io(password, truncated);
+    assert!(result.is_err(), "truncated data should fail decrypt");
+}

crates/crypto/src/lib.rs

@@ -19,6 +19,9 @@ mod jwt;
 
 pub use encdec::decrypt::decrypt_data;
 pub use encdec::encrypt::encrypt_data;
+
+#[cfg(feature = "crypto")]
+pub use encdec::stream_io::{decrypt_stream_io, encrypt_stream_io};
 pub use error::Error;
 pub use jwt::decode::decode as jwt_decode;
 pub use jwt::encode::encode as jwt_encode;

crates/e2e_test/AGENTS.md

@@ -0,0 +1,26 @@
+# E2E Test Crate Instructions
+
+Applies to `crates/e2e_test/`.
+
+## Test Reliability
+
+- Keep end-to-end tests deterministic and environment-aware.
+- Prefer readiness checks and explicit polling over fixed sleep-based timing assumptions.
+- Ensure tests isolate resources and clean up temporary state.
+
+## Environment Safety
+
+- For local KMS-related E2E runs, keep proxy bypass settings:
+  - `NO_PROXY=127.0.0.1,localhost`
+  - clear `HTTP_PROXY` and `HTTPS_PROXY`
+- Do not hardcode machine-specific paths or credentials.
+
+## Scope and Cost
+
+- Place exhaustive integration behavior here; keep unit behavior in source crates.
+- Keep new E2E scenarios focused and avoid redundant overlap with existing suites.
+
+## Suggested Validation
+
+- `cargo test --package e2e_test`
+- Full gate before commit: `make pre-commit`

crates/e2e_test/src/delete_objects_versioning_test.rs

@@ -16,15 +16,13 @@
 //! "In a versioned Bucket, DeleteMarkers are not appearing straight after
 //! a delete_objects is called."
 //!
-//! Root cause: `delete_versions_internal` wrote new xl.meta to disk via
-//! `write_all_private` without invalidating the `GlobalFileCache`. Subsequent
-//! calls to `read_metadata` returned the stale cached xl.meta (without the
-//! delete marker), making `list_object_versions` show the old version as
-//! `IsLatest=true` rather than the new delete marker.
+//! Root cause: metadata updates could become temporarily invisible to
+//! `list_object_versions`, so the old version was still reported as
+//! `IsLatest=true` instead of the newly-created delete marker.
 //!
-//! Fix: `write_all_private` now calls `get_global_file_cache().invalidate()`
-//! after every successful write, and `rename_data` also invalidates the cache
-//! for the destination path after the atomic rename.
+//! Fix: metadata write, delete, and rename paths now make the updated
+//! `xl.meta` immediately visible, and the old file-cache shortcut has been
+//! removed from the read path.
 
 #[cfg(test)]
 mod tests {

crates/e2e_test/src/lib.rs

@@ -66,6 +66,11 @@ mod compression_test;
 // Regression test for Issue #1878: DeleteMarkers not visible immediately after delete_objects
 #[cfg(test)]
 mod delete_objects_versioning_test;
+
+// Regression test for Issue #2252: ListObjectVersions misses newest version after put -> delete -> put
+#[cfg(test)]
+mod list_object_versions_regression_test;
+
 #[cfg(test)]
 mod protocols;
 

crates/e2e_test/src/list_object_versions_regression_test.rs

@@ -0,0 +1,182 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Regression test for Issue #2252:
+//! "ListObjectVersions misses the newest version after create -> delete -> create."
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::Client;
+    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
+    use serial_test::serial;
+    use tracing::info;
+
+    fn create_s3_client(env: &RustFSTestEnvironment) -> Client {
+        env.create_s3_client()
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_list_object_versions_immediately_returns_latest_put_after_delete_marker() {
+        init_logging();
+        info!("🧪 TEST: ListObjectVersions returns the newest version immediately after put -> delete -> put");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-list-object-versions-2252";
+        let key = "test-prefix/test-object.txt";
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("Failed to create bucket");
+
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await
+            .expect("Failed to enable versioning");
+
+        let first_put = client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(aws_sdk_s3::primitives::ByteStream::from_static(b"first version"))
+            .send()
+            .await
+            .expect("Failed to put first object version");
+        let first_version_id = first_put
+            .version_id()
+            .map(str::to_string)
+            .expect("First put should return a version_id");
+
+        let delete_resp = client
+            .delete_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("Failed to create delete marker");
+        let delete_marker_version_id = delete_resp
+            .version_id()
+            .map(str::to_string)
+            .expect("DeleteObject should return a delete marker version_id");
+
+        let second_put = client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(aws_sdk_s3::primitives::ByteStream::from_static(b"second version"))
+            .send()
+            .await
+            .expect("Failed to put second object version");
+        let second_version_id = second_put
+            .version_id()
+            .map(str::to_string)
+            .expect("Second put should return a version_id");
+
+        let first_listing = client
+            .list_object_versions()
+            .bucket(bucket)
+            .prefix(key)
+            .send()
+            .await
+            .expect("Failed to list object versions immediately after second put");
+        let second_listing = client
+            .list_object_versions()
+            .bucket(bucket)
+            .prefix(key)
+            .send()
+            .await
+            .expect("Failed to list object versions a second time");
+
+        let first_versions = first_listing.versions().to_vec();
+        let first_delete_markers = first_listing.delete_markers().to_vec();
+        let second_versions = second_listing.versions().to_vec();
+        let second_delete_markers = second_listing.delete_markers().to_vec();
+
+        info!(
+            "First listing: {} versions, {} delete markers",
+            first_versions.len(),
+            first_delete_markers.len()
+        );
+        info!(
+            "Second listing: {} versions, {} delete markers",
+            second_versions.len(),
+            second_delete_markers.len()
+        );
+
+        assert_eq!(
+            first_versions.len(),
+            2,
+            "First ListObjectVersions call should return both object versions immediately (regression #2252)"
+        );
+        assert_eq!(
+            first_delete_markers.len(),
+            1,
+            "First ListObjectVersions call should return the delete marker immediately (regression #2252)"
+        );
+        assert_eq!(
+            second_versions.len(),
+            2,
+            "Second ListObjectVersions call should still return both object versions"
+        );
+        assert_eq!(
+            second_delete_markers.len(),
+            1,
+            "Second ListObjectVersions call should still return the delete marker"
+        );
+
+        let first_latest_version = first_versions
+            .iter()
+            .find(|version| version.version_id() == Some(second_version_id.as_str()))
+            .expect("First listing should include the newest object version");
+        assert_eq!(
+            first_latest_version.is_latest(),
+            Some(true),
+            "Newest object version should be latest on the first listing"
+        );
+
+        let first_original_version = first_versions
+            .iter()
+            .find(|version| version.version_id() == Some(first_version_id.as_str()))
+            .expect("First listing should include the original object version");
+        assert_eq!(
+            first_original_version.is_latest(),
+            Some(false),
+            "Original object version should no longer be latest"
+        );
+
+        let first_delete_marker = first_delete_markers
+            .iter()
+            .find(|marker| marker.version_id() == Some(delete_marker_version_id.as_str()))
+            .expect("First listing should include the delete marker");
+        assert_eq!(
+            first_delete_marker.is_latest(),
+            Some(false),
+            "Delete marker should no longer be latest after the second put"
+        );
+    }
+}

crates/e2e_test/src/protocols/README.md

@@ -1,6 +1,6 @@
 # Protocol E2E Tests
 
-FTPS protocol end-to-end tests for RustFS.
+FTPS and WebDAV protocol end-to-end tests for RustFS.
 
 ## Prerequisites
 
@@ -19,11 +19,21 @@ brew install sshpass openssh
 
 ## Running Tests
 
-Run all protocol tests:
+Run all protocol tests (FTPS + WebDAV):
+```bash
+RUSTFS_BUILD_FEATURES=ftps,webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
+```
+
+Run FTPS tests only:
 ```bash
 RUSTFS_BUILD_FEATURES=ftps cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
 ```
 
+Run WebDAV tests only:
+```bash
+RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
+```
+
 ## Test Coverage
 
 ### FTPS Tests
@@ -38,3 +48,13 @@ RUSTFS_BUILD_FEATURES=ftps cargo test --package e2e_test test_protocol_core_suit
 - cdup
 - rmdir delete bucket
 
+### WebDAV Tests
+- PROPFIND at root (list buckets)
+- MKCOL (create bucket)
+- PUT (upload file)
+- GET (download file)
+- PROPFIND on bucket (list objects)
+- DELETE file
+- DELETE bucket
+- Authentication failure test
+

crates/e2e_test/src/protocols/mod.rs

@@ -12,8 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Protocol tests for FTPS
+//! Protocol tests for FTPS and WebDAV
 
 pub mod ftps_core;
 pub mod test_env;
 pub mod test_runner;
+pub mod webdav_core;

crates/e2e_test/src/protocols/test_runner.rs

@@ -16,6 +16,7 @@
 
 use crate::common::init_logging;
 use crate::protocols::ftps_core::test_ftps_core_operations;
+use crate::protocols::webdav_core::test_webdav_core_operations;
 use std::time::Instant;
 use tokio::time::{Duration, sleep};
 use tracing::{error, info};
@@ -59,9 +60,14 @@ struct TestDefinition {
 impl ProtocolTestSuite {
     /// Create default test suite
     pub fn new() -> Self {
-        let tests = vec![TestDefinition {
-            name: "test_ftps_core_operations".to_string(),
-        }];
+        let tests = vec![
+            TestDefinition {
+                name: "test_ftps_core_operations".to_string(),
+            },
+            TestDefinition {
+                name: "test_webdav_core_operations".to_string(),
+            },
+        ];
 
         Self { tests }
     }
@@ -83,6 +89,10 @@ impl ProtocolTestSuite {
                     info!("=== Starting FTPS Module Test ===");
                     "FTPS core operations (put, ls, mkdir, rmdir, delete)"
                 }
+                "test_webdav_core_operations" => {
+                    info!("=== Starting WebDAV Core Test ===");
+                    "WebDAV core operations (MKCOL, PUT, GET, DELETE, PROPFIND)"
+                }
                 _ => "",
             };
 
@@ -121,6 +131,7 @@ impl ProtocolTestSuite {
     async fn run_single_test(&self, test_def: &TestDefinition) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         match test_def.name.as_str() {
             "test_ftps_core_operations" => test_ftps_core_operations().await.map_err(|e| e.into()),
+            "test_webdav_core_operations" => test_webdav_core_operations().await.map_err(|e| e.into()),
             _ => Err(format!("Test {} not implemented", test_def.name).into()),
         }
     }

crates/e2e_test/src/protocols/webdav_core.rs

@@ -0,0 +1,207 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Core WebDAV tests
+
+use crate::common::rustfs_binary_path;
+use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
+use anyhow::Result;
+use base64::Engine;
+use reqwest::Client;
+use tokio::process::Command;
+use tracing::info;
+
+// Fixed WebDAV port for testing
+const WEBDAV_PORT: u16 = 9080;
+const WEBDAV_ADDRESS: &str = "127.0.0.1:9080";
+
+/// Create HTTP client with basic auth
+fn create_client() -> Client {
+    Client::builder()
+        .danger_accept_invalid_certs(true)
+        .build()
+        .expect("Failed to create HTTP client")
+}
+
+/// Get basic auth header value
+fn basic_auth_header() -> String {
+    let credentials = format!("{}:{}", DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY);
+    let encoded = base64::engine::general_purpose::STANDARD.encode(credentials);
+    format!("Basic {}", encoded)
+}
+
+/// Test WebDAV: MKCOL (create bucket), PUT, GET, DELETE, PROPFIND operations
+pub async fn test_webdav_core_operations() -> Result<()> {
+    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
+
+    // Start server manually
+    info!("Starting WebDAV server on {}", WEBDAV_ADDRESS);
+    let binary_path = rustfs_binary_path();
+    let mut server_process = Command::new(&binary_path)
+        .env("RUSTFS_WEBDAV_ENABLE", "true")
+        .env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
+        .env("RUSTFS_WEBDAV_TLS_ENABLED", "false") // No TLS for testing
+        .arg(&env.temp_dir)
+        .spawn()?;
+
+    // Ensure server is cleaned up even on failure
+    let result = async {
+        // Wait for server to be ready
+        ProtocolTestEnvironment::wait_for_port_ready(WEBDAV_PORT, 30)
+            .await
+            .map_err(|e| anyhow::anyhow!("{}", e))?;
+
+        let client = create_client();
+        let auth_header = basic_auth_header();
+        let base_url = format!("http://{}", WEBDAV_ADDRESS);
+
+        // Test PROPFIND at root (list buckets)
+        info!("Testing WebDAV: PROPFIND at root (list buckets)");
+        let resp = client
+            .request(reqwest::Method::from_bytes(b"PROPFIND").unwrap(), &base_url)
+            .header("Authorization", &auth_header)
+            .header("Depth", "1")
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 207,
+            "PROPFIND at root should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: PROPFIND at root successful");
+
+        // Test MKCOL (create bucket)
+        let bucket_name = "webdav-test-bucket";
+        info!("Testing WebDAV: MKCOL (create bucket '{}')", bucket_name);
+        let resp = client
+            .request(reqwest::Method::from_bytes(b"MKCOL").unwrap(), format!("{}/{}", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MKCOL bucket '{}' successful", bucket_name);
+
+        // Test PUT (upload file)
+        let filename = "test-file.txt";
+        let file_content = "Hello, WebDAV!";
+        info!("Testing WebDAV: PUT (upload file '{}')", filename);
+        let resp = client
+            .put(format!("{}/{}/{}", base_url, bucket_name, filename))
+            .header("Authorization", &auth_header)
+            .body(file_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: PUT file '{}' successful", filename);
+
+        // Test GET (download file)
+        info!("Testing WebDAV: GET (download file '{}')", filename);
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(resp.status().is_success(), "GET should succeed, got: {}", resp.status());
+        let downloaded_content = resp.text().await?;
+        assert_eq!(downloaded_content, file_content, "Downloaded content should match uploaded content");
+        info!("PASS: GET file '{}' successful, content matches", filename);
+
+        // Test PROPFIND on bucket (list objects)
+        info!("Testing WebDAV: PROPFIND on bucket (list objects)");
+        let resp = client
+            .request(reqwest::Method::from_bytes(b"PROPFIND").unwrap(), format!("{}/{}", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .header("Depth", "1")
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 207,
+            "PROPFIND on bucket should succeed, got: {}",
+            resp.status()
+        );
+        let body = resp.text().await?;
+        assert!(body.contains(filename), "File should appear in PROPFIND response");
+        info!("PASS: PROPFIND on bucket successful, file '{}' found", filename);
+
+        // Test DELETE file
+        info!("Testing WebDAV: DELETE file '{}'", filename);
+        let resp = client
+            .delete(format!("{}/{}/{}", base_url, bucket_name, filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204,
+            "DELETE file should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: DELETE file '{}' successful", filename);
+
+        // Verify file is deleted
+        info!("Testing WebDAV: Verify file is deleted");
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().as_u16() == 404,
+            "GET deleted file should return 404, got: {}",
+            resp.status()
+        );
+        info!("PASS: Verified file '{}' is deleted", filename);
+
+        // Test DELETE bucket
+        info!("Testing WebDAV: DELETE bucket '{}'", bucket_name);
+        let resp = client
+            .delete(format!("{}/{}", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204,
+            "DELETE bucket should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: DELETE bucket '{}' successful", bucket_name);
+
+        // Test authentication failure
+        info!("Testing WebDAV: Authentication failure");
+        let resp = client
+            .request(reqwest::Method::from_bytes(b"PROPFIND").unwrap(), &base_url)
+            .header("Authorization", "Basic aW52YWxpZDppbnZhbGlk") // invalid:invalid
+            .send()
+            .await?;
+        assert_eq!(resp.status().as_u16(), 401, "Invalid auth should return 401, got: {}", resp.status());
+        info!("PASS: Authentication failure test successful");
+
+        info!("WebDAV core tests passed");
+        Ok(())
+    }
+    .await;
+
+    // Always cleanup server process
+    let _ = server_process.kill().await;
+    let _ = server_process.wait().await;
+
+    result
+}

crates/e2e_test/src/reliant/grpc_lock_client.rs

@@ -61,6 +61,7 @@ impl GrpcLockClient {
             metadata: LockMetadata::default(),
             priority: LockPriority::Normal,
             deadlock_detection: false,
+            suppress_contention_logs: false,
         }
     }
 }

crates/e2e_test/src/reliant/lifecycle.rs

@@ -19,7 +19,6 @@ use aws_sdk_s3::config::{Credentials, Region};
 use bytes::Bytes;
 use serial_test::serial;
 use std::error::Error;
-use tokio::time::sleep;
 
 const ENDPOINT: &str = "http://localhost:9000";
 const ACCESS_KEY: &str = "rustfsadmin";
@@ -62,6 +61,7 @@ async fn setup_test_bucket(client: &Client) -> Result<(), Box<dyn Error>> {
 #[ignore = "requires running RustFS server at localhost:9000"]
 async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error::Error>> {
     use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
+    use chrono::{Duration as ChronoDuration, Utc};
     use tokio::time::Duration;
 
     let client = create_aws_s3_client().await?;
@@ -70,6 +70,7 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
     // Upload test object first
     let test_content = "Test object for lifecycle expiration";
     let lifecycle_object_key = "lifecycle-test-object.txt";
+    let untouched_object_key = "keep-object.txt";
     client
         .put_object()
         .bucket(BUCKET)
@@ -77,13 +78,29 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
         .body(Bytes::from(test_content.as_bytes()).into())
         .send()
         .await?;
+    client
+        .put_object()
+        .bucket(BUCKET)
+        .key(untouched_object_key)
+        .body(Bytes::from("should-stay".as_bytes()).into())
+        .send()
+        .await?;
 
     // Verify object exists initially
     let resp = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await?;
     assert!(resp.content_length().unwrap_or(0) > 0);
+    let untouched_resp = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
+    assert!(untouched_resp.content_length().unwrap_or(0) > 0);
 
-    // Configure lifecycle rule: expire after current time + 3 seconds
-    let expiration = LifecycleExpiration::builder().days(0).build();
+    // Use a past midnight UTC date to trigger immediate lifecycle expiry without requiring days=0.
+    let yesterday_midnight_utc = Utc::now()
+        .date_naive()
+        .and_hms_opt(0, 0, 0)
+        .expect("midnight should always be valid")
+        - ChronoDuration::days(1);
+    let expiration = LifecycleExpiration::builder()
+        .date(aws_sdk_s3::primitives::DateTime::from_secs(yesterday_midnight_utc.and_utc().timestamp()))
+        .build();
     let filter = LifecycleRuleFilter::builder().prefix(lifecycle_object_key).build();
     let rule = LifecycleRule::builder()
         .id("expire-test-object")
@@ -105,29 +122,73 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
     let rules = resp.rules();
     assert!(rules.iter().any(|r| r.id().unwrap_or("") == "expire-test-object"));
 
-    // Wait for lifecycle processing (scanner runs every 1 second)
-    sleep(Duration::from_secs(3)).await;
-
-    // After lifecycle processing, the object should be deleted by the lifecycle rule
-    let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
-
-    match get_result {
-        Ok(_) => {
-            panic!("Expected object to be deleted by lifecycle rule, but it still exists");
-        }
-        Err(e) => {
-            if let Some(service_error) = e.as_service_error() {
-                if service_error.is_no_such_key() {
-                    println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
-                } else {
-                    panic!("Expected NoSuchKey error, but got: {e:?}");
+    // Poll for deletion instead of using a fixed sleep to keep the test deterministic.
+    // Default scanner cycle interval is 60s with jitter, so allow enough time for one full cycle.
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(150);
+    loop {
+        let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
+        match get_result {
+            Ok(_) => {
+                if tokio::time::Instant::now() >= deadline {
+                    panic!("Expected object to be deleted by lifecycle rule within 150s, but it still exists");
+                }
+                tokio::time::sleep(Duration::from_secs(1)).await;
+            }
+            Err(e) => {
+                if let Some(service_error) = e.as_service_error() {
+                    if service_error.is_no_such_key() {
+                        println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
+                        break;
+                    }
+                    panic!("Expected NoSuchKey error, but got: {e:?}");
+                } else {
+                    panic!("Expected service error, but got: {e:?}");
                 }
-            } else {
-                panic!("Expected service error, but got: {e:?}");
             }
         }
     }
 
     println!("Lifecycle configuration test completed.");
+
+    // Non-matching prefix object should remain available.
+    let untouched_after = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
+    assert!(untouched_after.content_length().unwrap_or(0) > 0);
+
+    Ok(())
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+#[serial]
+#[ignore = "requires running RustFS server at localhost:9000"]
+async fn test_bucket_lifecycle_rejects_zero_days() -> Result<(), Box<dyn std::error::Error>> {
+    use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
+
+    let client = create_aws_s3_client().await?;
+    setup_test_bucket(&client).await?;
+
+    let expiration = LifecycleExpiration::builder().days(0).build();
+    let filter = LifecycleRuleFilter::builder().prefix("zero-days/").build();
+    let rule = LifecycleRule::builder()
+        .id("expire-zero-days")
+        .filter(filter)
+        .expiration(expiration)
+        .status(aws_sdk_s3::types::ExpirationStatus::Enabled)
+        .build()?;
+    let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
+
+    let err = client
+        .put_bucket_lifecycle_configuration()
+        .bucket(BUCKET)
+        .lifecycle_configuration(lifecycle)
+        .send()
+        .await
+        .expect_err("zero-day lifecycle expiration should be rejected");
+
+    let err_msg = format!("{err:?}");
+    assert!(
+        err_msg.contains("InvalidArgument") && err_msg.contains("greater than 0"),
+        "unexpected error: {err_msg}"
+    );
+
     Ok(())
 }

crates/ecstore/AGENTS.md

@@ -0,0 +1,28 @@
+# ECStore Crate Instructions
+
+Applies to `crates/ecstore/`.
+
+## Data Durability and Integrity
+
+- Do not weaken quorum checks, bitrot checks, or metadata validation paths.
+- Treat any change affecting read/write/repair correctness as high risk and test accordingly.
+- Prefer explicit failure over silent data corruption or implicit success.
+
+## Performance-Critical Paths
+
+- Be careful with allocations and locking in hot paths.
+- Keep network and disk operations async-friendly; avoid introducing unnecessary blocking.
+- Benchmark-sensitive changes should include measurable rationale.
+
+## Cross-Module Coordination
+
+- Validate behavior impacts on:
+  - `rustfs/src/storage/`
+  - `crates/filemeta/`
+  - `crates/heal/`
+  - `crates/checksums/`
+
+## Suggested Validation
+
+- `cargo test -p rustfs-ecstore`
+- Full gate before commit: `make pre-commit`

crates/ecstore/Cargo.toml

@@ -44,6 +44,7 @@ rustfs-credentials = { workspace = true }
 rustfs-common.workspace = true
 rustfs-policy.workspace = true
 rustfs-protos.workspace = true
+rustfs-s3-common = { workspace = true }
 async-trait.workspace = true
 bytes.workspace = true
 byteorder = { workspace = true }
@@ -65,6 +66,7 @@ http-body = { workspace = true }
 http-body-util.workspace = true
 url.workspace = true
 uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
+reed-solomon-erasure = { workspace = true }
 reed-solomon-simd = { workspace = true }
 lazy_static.workspace = true
 rustfs-lock.workspace = true
@@ -114,11 +116,15 @@ faster-hex = { workspace = true }
 ratelimit = { workspace = true }
 aws-smithy-http-client.workspace = true
 
+# Observability and Metrics
+metrics = { workspace = true }
+
 [dev-dependencies]
 tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true }
 tracing-subscriber = { workspace = true }
+serial_test = { workspace = true }
 
 [build-dependencies]
 shadow-rs = { workspace = true, features = ["build", "metadata"] }
@@ -130,3 +136,6 @@ harness = false
 [[bench]]
 name = "comparison_benchmark"
 harness = false
+
+[lib]
+doctest = false

crates/ecstore/benches/erasure_benchmark.rs

@@ -118,41 +118,38 @@ fn bench_encode_performance(c: &mut Criterion) {
         });
         group.finish();
 
-        // Test direct SIMD implementation for large shards (>= 512 bytes)
+        // Test direct reed-solomon-erasure implementation for large shards (>= 512 bytes)
         let shard_size = calc_shard_size(config.data_size, config.data_shards);
-        if shard_size >= 512 {
-            let mut simd_group = c.benchmark_group("encode_simd_direct");
-            simd_group.throughput(Throughput::Bytes(config.data_size as u64));
-            simd_group.sample_size(10);
-            simd_group.measurement_time(Duration::from_secs(5));
+        if shard_size >= 512 && config.parity_shards > 0 {
+            use reed_solomon_erasure::galois_8::ReedSolomon;
 
-            simd_group.bench_with_input(BenchmarkId::new("simd_direct", &config.name), &(&data, &config), |b, (data, config)| {
-                b.iter(|| {
-                    // Direct SIMD implementation
-                    let per_shard_size = calc_shard_size(data.len(), config.data_shards);
-                    match reed_solomon_simd::ReedSolomonEncoder::new(config.data_shards, config.parity_shards, per_shard_size) {
-                        Ok(mut encoder) => {
-                            // Create properly sized buffer and fill with data
-                            let mut buffer = vec![0u8; per_shard_size * config.data_shards];
+            let mut rse_group = c.benchmark_group("encode_rse_direct");
+            rse_group.throughput(Throughput::Bytes(config.data_size as u64));
+            rse_group.sample_size(10);
+            rse_group.measurement_time(Duration::from_secs(5));
+
+            if let Ok(rs) = ReedSolomon::new(config.data_shards, config.parity_shards) {
+                let total_shards = config.data_shards + config.parity_shards;
+                let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
+                let need_total = per_shard_size * total_shards;
+
+                rse_group.bench_with_input(
+                    BenchmarkId::new("rse_direct", &config.name),
+                    &(&data, need_total, per_shard_size),
+                    |b, (data, need_total, per_shard_size)| {
+                        b.iter(|| {
+                            let mut buffer = vec![0u8; *need_total];
                             let copy_len = data.len().min(buffer.len());
                             buffer[..copy_len].copy_from_slice(&data[..copy_len]);
 
-                            // Add data shards with correct shard size
-                            for chunk in buffer.chunks_exact(per_shard_size) {
-                                encoder.add_original_shard(black_box(chunk)).unwrap();
-                            }
-
-                            let result = encoder.encode().unwrap();
-                            black_box(result);
-                        }
-                        Err(_) => {
-                            // SIMD doesn't support this configuration, skip
-                            black_box(());
-                        }
-                    }
-                });
-            });
-            simd_group.finish();
+                            let mut slices: Vec<&mut [u8]> = buffer.chunks_exact_mut(*per_shard_size).collect();
+                            rs.encode(&mut slices).unwrap();
+                            black_box(buffer);
+                        });
+                    },
+                );
+            }
+            rse_group.finish();
         }
     }
 }
@@ -203,47 +200,33 @@ fn bench_decode_performance(c: &mut Criterion) {
         );
         group.finish();
 
-        // Test direct SIMD decoding for large shards
+        // Test direct reed-solomon-erasure decoding for large shards
         let shard_size = calc_shard_size(config.data_size, config.data_shards);
-        if shard_size >= 512 {
-            let mut simd_group = c.benchmark_group("decode_simd_direct");
-            simd_group.throughput(Throughput::Bytes(config.data_size as u64));
-            simd_group.sample_size(10);
-            simd_group.measurement_time(Duration::from_secs(5));
+        if shard_size >= 512 && config.parity_shards > 0 {
+            use reed_solomon_erasure::galois_8::ReedSolomon;
 
-            simd_group.bench_with_input(
-                BenchmarkId::new("simd_direct", &config.name),
-                &(&encoded_shards, &config),
-                |b, (shards, config)| {
-                    b.iter(|| {
-                        let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
-                        match reed_solomon_simd::ReedSolomonDecoder::new(config.data_shards, config.parity_shards, per_shard_size)
-                        {
-                            Ok(mut decoder) => {
-                                // Add available shards (except lost ones)
-                                for (i, shard) in shards.iter().enumerate() {
-                                    if i != config.data_shards - 1 && i != config.data_shards {
-                                        if i < config.data_shards {
-                                            decoder.add_original_shard(i, black_box(shard)).unwrap();
-                                        } else {
-                                            let recovery_idx = i - config.data_shards;
-                                            decoder.add_recovery_shard(recovery_idx, black_box(shard)).unwrap();
-                                        }
-                                    }
-                                }
+            if let Ok(rs) = ReedSolomon::new(config.data_shards, config.parity_shards) {
+                let mut rse_group = c.benchmark_group("decode_rse_direct");
+                rse_group.throughput(Throughput::Bytes(config.data_size as u64));
+                rse_group.sample_size(10);
+                rse_group.measurement_time(Duration::from_secs(5));
 
-                                let result = decoder.decode().unwrap();
-                                black_box(result);
-                            }
-                            Err(_) => {
-                                // SIMD doesn't support this configuration, skip
-                                black_box(());
-                            }
-                        }
-                    });
-                },
-            );
-            simd_group.finish();
+                rse_group.bench_with_input(
+                    BenchmarkId::new("rse_direct", &config.name),
+                    &(&encoded_shards, &config),
+                    |b, (shards, config)| {
+                        b.iter(|| {
+                            let mut shards_opt: Vec<Option<Vec<u8>>> = shards.iter().map(|s| Some(s.to_vec())).collect();
+                            shards_opt[config.data_shards - 1] = None;
+                            shards_opt[config.data_shards] = None;
+
+                            rs.reconstruct_data(&mut shards_opt).unwrap();
+                            black_box(shards_opt);
+                        });
+                    },
+                );
+                rse_group.finish();
+            }
         }
     }
 }

crates/ecstore/src/bitrot.rs

@@ -39,6 +39,7 @@ pub async fn create_bitrot_reader(
     length: usize,
     shard_size: usize,
     checksum_algo: HashAlgorithm,
+    skip_verify: bool,
 ) -> disk::error::Result<Option<BitrotReader<Box<dyn AsyncRead + Send + Sync + Unpin>>>> {
     // Calculate the total length to read, including the checksum overhead
     let length = length.div_ceil(shard_size) * checksum_algo.size() + length;
@@ -47,13 +48,18 @@ pub async fn create_bitrot_reader(
         // Use inline data
         let mut rd = Cursor::new(data.to_vec());
         rd.set_position(offset as u64);
-        let reader = BitrotReader::new(Box::new(rd) as Box<dyn AsyncRead + Send + Sync + Unpin>, shard_size, checksum_algo);
+        let reader = BitrotReader::new(
+            Box::new(rd) as Box<dyn AsyncRead + Send + Sync + Unpin>,
+            shard_size,
+            checksum_algo,
+            skip_verify,
+        );
         Ok(Some(reader))
     } else if let Some(disk) = disk {
         // Read from disk
         match disk.read_file_stream(bucket, path, offset, length - offset).await {
             Ok(rd) => {
-                let reader = BitrotReader::new(rd, shard_size, checksum_algo);
+                let reader = BitrotReader::new(rd, shard_size, checksum_algo, skip_verify);
                 Ok(Some(reader))
             }
             Err(e) => Err(e),
@@ -113,10 +119,10 @@ mod tests {
     async fn test_create_bitrot_reader_with_inline_data() {
         let test_data = b"hello world test data";
         let shard_size = 16;
-        let checksum_algo = HashAlgorithm::HighwayHash256;
+        let checksum_algo = HashAlgorithm::HighwayHash256S;
 
         let result =
-            create_bitrot_reader(Some(test_data), None, "test-bucket", "test-path", 0, 0, shard_size, checksum_algo).await;
+            create_bitrot_reader(Some(test_data), None, "test-bucket", "test-path", 0, 0, shard_size, checksum_algo, false).await;
 
         assert!(result.is_ok());
         assert!(result.unwrap().is_some());
@@ -125,9 +131,10 @@ mod tests {
     #[tokio::test]
     async fn test_create_bitrot_reader_without_data_or_disk() {
         let shard_size = 16;
-        let checksum_algo = HashAlgorithm::HighwayHash256;
+        let checksum_algo = HashAlgorithm::HighwayHash256S;
 
-        let result = create_bitrot_reader(None, None, "test-bucket", "test-path", 0, 1024, shard_size, checksum_algo).await;
+        let result =
+            create_bitrot_reader(None, None, "test-bucket", "test-path", 0, 1024, shard_size, checksum_algo, false).await;
 
         assert!(result.is_ok());
         assert!(result.unwrap().is_none());
@@ -144,7 +151,7 @@ mod tests {
             "test-path",
             1024, // length
             1024, // shard_size
-            HashAlgorithm::HighwayHash256,
+            HashAlgorithm::HighwayHash256S,
         )
         .await;
 
@@ -176,7 +183,7 @@ mod tests {
             "test-path",
             1024, // length
             1024, // shard_size
-            HashAlgorithm::HighwayHash256,
+            HashAlgorithm::HighwayHash256S,
         )
         .await;
 

crates/ecstore/src/bucket/bandwidth/monitor.rs

@@ -14,11 +14,16 @@
 
 use crate::bucket::bandwidth::reader::BucketOptions;
 use ratelimit::{Error as RatelimitError, Ratelimiter};
+use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
 use std::sync::{Arc, Mutex, RwLock};
-use std::time::Duration;
+use std::time::{Duration, Instant};
 use tracing::warn;
 
+/// BETA_BUCKET is the weight used to calculate exponential moving average
+const BETA_BUCKET: f64 = 0.1;
+
 #[derive(Clone)]
 pub struct BucketThrottle {
     limiter: Arc<Mutex<Ratelimiter>>,
@@ -71,25 +76,202 @@ impl BucketThrottle {
     }
 }
 
+#[derive(Debug)]
+pub struct BucketMeasurement {
+    bytes_since_last_window: AtomicU64,
+    start_time: Mutex<Option<Instant>>,
+    exp_moving_avg: Mutex<f64>,
+}
+
+impl BucketMeasurement {
+    pub fn new(init_time: Instant) -> Self {
+        Self {
+            bytes_since_last_window: AtomicU64::new(0),
+            start_time: Mutex::new(Some(init_time)),
+            exp_moving_avg: Mutex::new(0.0),
+        }
+    }
+
+    pub fn increment_bytes(&self, bytes: u64) {
+        self.bytes_since_last_window.fetch_add(bytes, Ordering::Relaxed);
+    }
+
+    pub fn update_exponential_moving_average(&self, end_time: Instant) {
+        let mut start_time = self.start_time.lock().unwrap_or_else(|e| {
+            warn!("bucket measurement start_time mutex poisoned, recovering");
+            e.into_inner()
+        });
+        let previous_start = *start_time;
+        *start_time = Some(end_time);
+
+        let Some(prev_start) = previous_start else {
+            return;
+        };
+
+        if prev_start > end_time {
+            return;
+        }
+
+        let duration = end_time.duration_since(prev_start);
+        if duration.is_zero() {
+            return;
+        }
+
+        let bytes_since_last_window = self.bytes_since_last_window.swap(0, Ordering::Relaxed);
+        let increment = bytes_since_last_window as f64 / duration.as_secs_f64();
+
+        let mut exp_moving_avg = self.exp_moving_avg.lock().unwrap_or_else(|e| {
+            warn!("bucket measurement exp_moving_avg mutex poisoned, recovering");
+            e.into_inner()
+        });
+
+        if *exp_moving_avg == 0.0 {
+            *exp_moving_avg = increment;
+            return;
+        }
+
+        *exp_moving_avg = exponential_moving_average(BETA_BUCKET, *exp_moving_avg, increment);
+    }
+
+    pub fn get_exp_moving_avg_bytes_per_second(&self) -> f64 {
+        *self.exp_moving_avg.lock().unwrap_or_else(|e| {
+            warn!("bucket measurement exp_moving_avg mutex poisoned, recovering");
+            e.into_inner()
+        })
+    }
+}
+
+fn exponential_moving_average(beta: f64, previous_avg: f64, increment_avg: f64) -> f64 {
+    (1f64 - beta) * increment_avg + beta * previous_avg
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BandwidthDetails {
+    pub limit_bytes_per_sec: i64,
+    pub current_bandwidth_bytes_per_sec: f64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BucketBandwidthReport {
+    pub bucket_stats: HashMap<BucketOptions, BandwidthDetails>,
+}
+
 pub struct Monitor {
     t_lock: RwLock<HashMap<BucketOptions, BucketThrottle>>,
+    m_lock: RwLock<HashMap<BucketOptions, BucketMeasurement>>,
     pub node_count: u64,
 }
 
 impl Monitor {
     pub fn new(num_nodes: u64) -> Arc<Self> {
         let node_cnt = num_nodes.max(1);
-        Arc::new(Monitor {
+        let m = Arc::new(Monitor {
             t_lock: RwLock::new(HashMap::new()),
+            m_lock: RwLock::new(HashMap::new()),
             node_count: node_cnt,
-        })
+        });
+        if let Ok(handle) = tokio::runtime::Handle::try_current() {
+            let weak = Arc::downgrade(&m);
+            handle.spawn(async move {
+                let mut interval = tokio::time::interval(Duration::from_secs(2));
+                loop {
+                    interval.tick().await;
+                    let Some(monitor) = weak.upgrade() else { break };
+                    monitor.update_moving_avg();
+                }
+            });
+        }
+        m
+    }
+
+    pub fn init_measurement(&self, opts: &BucketOptions) {
+        let mut guard = self.m_lock.write().unwrap_or_else(|e| {
+            warn!("bucket monitor measurement rwlock write poisoned, recovering");
+            e.into_inner()
+        });
+        guard
+            .entry(opts.clone())
+            .or_insert_with(|| BucketMeasurement::new(Instant::now()));
+    }
+
+    pub fn update_measurement(&self, opts: &BucketOptions, bytes: u64) {
+        {
+            let guard = self.m_lock.read().unwrap_or_else(|e| {
+                warn!("bucket monitor measurement rwlock read poisoned, recovering");
+                e.into_inner()
+            });
+            if let Some(measurement) = guard.get(opts) {
+                measurement.increment_bytes(bytes);
+                return;
+            }
+        }
+
+        // Miss path: write lock + insert once, then increment.
+        let mut guard = self.m_lock.write().unwrap_or_else(|e| {
+            warn!("bucket monitor measurement rwlock write poisoned, recovering");
+            e.into_inner()
+        });
+
+        // Double-check after lock upgrade in case another thread inserted it.
+        let measurement = guard
+            .entry(opts.clone())
+            .or_insert_with(|| BucketMeasurement::new(Instant::now()));
+
+        measurement.increment_bytes(bytes);
+    }
+
+    pub fn update_moving_avg(&self) {
+        let now = Instant::now();
+        let guard = self.m_lock.read().unwrap_or_else(|e| {
+            warn!("bucket monitor measurement rwlock read poisoned, recovering");
+            e.into_inner()
+        });
+        for measurement in guard.values() {
+            measurement.update_exponential_moving_average(now);
+        }
+    }
+
+    pub fn get_report(&self, select_bucket: impl Fn(&str) -> bool) -> BucketBandwidthReport {
+        let t_guard = self.t_lock.read().unwrap_or_else(|e| {
+            warn!("bucket monitor throttle rwlock read poisoned, recovering");
+            e.into_inner()
+        });
+        let m_guard = self.m_lock.read().unwrap_or_else(|e| {
+            warn!("bucket monitor measurement rwlock read poisoned, recovering");
+            e.into_inner()
+        });
+        let mut bucket_stats = HashMap::new();
+        for (opts, throttle) in t_guard.iter() {
+            if !select_bucket(&opts.name) {
+                continue;
+            }
+            let mut current_bandwidth_bytes_per_sec = 0.0;
+            if let Some(measurement) = m_guard.get(opts) {
+                current_bandwidth_bytes_per_sec = measurement.get_exp_moving_avg_bytes_per_second();
+            }
+            bucket_stats.insert(
+                opts.clone(),
+                BandwidthDetails {
+                    limit_bytes_per_sec: throttle.node_bandwidth_per_sec * self.node_count as i64,
+                    current_bandwidth_bytes_per_sec,
+                },
+            );
+        }
+        BucketBandwidthReport { bucket_stats }
     }
 
     pub fn delete_bucket(&self, bucket: &str) {
         self.t_lock
             .write()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock write poisoned, recovering");
+                warn!("bucket monitor throttle rwlock write poisoned, recovering");
+                e.into_inner()
+            })
+            .retain(|opts, _| opts.name != bucket);
+        self.m_lock
+            .write()
+            .unwrap_or_else(|e| {
+                warn!("bucket monitor measurement rwlock write poisoned, recovering");
                 e.into_inner()
             })
             .retain(|opts, _| opts.name != bucket);
@@ -103,7 +285,14 @@ impl Monitor {
         self.t_lock
             .write()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock write poisoned, recovering");
+                warn!("bucket monitor throttle rwlock write poisoned, recovering");
+                e.into_inner()
+            })
+            .remove(&opts);
+        self.m_lock
+            .write()
+            .unwrap_or_else(|e| {
+                warn!("bucket monitor measurement rwlock write poisoned, recovering");
                 e.into_inner()
             })
             .remove(&opts);
@@ -113,7 +302,7 @@ impl Monitor {
         self.t_lock
             .read()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock read poisoned, recovering");
+                warn!("bucket monitor throttle rwlock read poisoned, recovering");
                 e.into_inner()
             })
             .get(opts)
@@ -160,7 +349,7 @@ impl Monitor {
         self.t_lock
             .write()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock write poisoned, recovering");
+                warn!("bucket monitor throttle rwlock write poisoned, recovering");
                 e.into_inner()
             })
             .insert(opts, throttle);
@@ -174,7 +363,7 @@ impl Monitor {
         self.t_lock
             .read()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock read poisoned, recovering");
+                warn!("bucket monitor throttle rwlock read poisoned, recovering");
                 e.into_inner()
             })
             .contains_key(&opt)
@@ -184,6 +373,7 @@ impl Monitor {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use std::panic::{AssertUnwindSafe, catch_unwind};
 
     #[test]
     fn test_set_and_get_throttle_with_node_split() {
@@ -318,4 +508,82 @@ mod tests {
         assert_eq!(t2.burst(), 500);
         assert_eq!(t2.node_bandwidth_per_sec, 500);
     }
+
+    #[test]
+    fn test_bucket_measurement_recovers_from_poisoned_mutexes() {
+        let measurement = BucketMeasurement::new(Instant::now());
+
+        let _ = catch_unwind(AssertUnwindSafe(|| {
+            let _guard = measurement.start_time.lock().unwrap();
+            panic!("poison start_time mutex");
+        }));
+        measurement.increment_bytes(64);
+        measurement.update_exponential_moving_average(Instant::now() + Duration::from_secs(1));
+
+        let _ = catch_unwind(AssertUnwindSafe(|| {
+            let _guard = measurement.exp_moving_avg.lock().unwrap();
+            panic!("poison exp_moving_avg mutex");
+        }));
+        measurement.increment_bytes(32);
+        measurement.update_exponential_moving_average(Instant::now() + Duration::from_secs(2));
+
+        let value = measurement.get_exp_moving_avg_bytes_per_second();
+        assert!(value.is_finite());
+        assert!(value >= 0.0);
+    }
+
+    #[test]
+    fn test_get_report_limit_and_current_bandwidth_after_measurement() {
+        let monitor = Monitor::new(4);
+        monitor.set_bandwidth_limit("b1", "arn1", 400);
+        let opts = BucketOptions {
+            name: "b1".to_string(),
+            replication_arn: "arn1".to_string(),
+        };
+        monitor.init_measurement(&opts);
+        monitor.update_measurement(&opts, 500);
+        monitor.update_measurement(&opts, 500);
+        std::thread::sleep(Duration::from_millis(110));
+        monitor.update_moving_avg();
+
+        let report = monitor.get_report(|name| name == "b1");
+        let details = report.bucket_stats.get(&opts).expect("report should contain b1/arn1");
+        assert_eq!(details.limit_bytes_per_sec, 400);
+        assert!(
+            details.current_bandwidth_bytes_per_sec > 0.0,
+            "current_bandwidth should be positive after update_measurement and update_moving_avg"
+        );
+        assert!(
+            details.current_bandwidth_bytes_per_sec < 20000.0,
+            "current_bandwidth should be in reasonable range"
+        );
+    }
+
+    #[test]
+    fn test_get_report_select_bucket_filters() {
+        let monitor = Monitor::new(2);
+        monitor.set_bandwidth_limit("b1", "arn1", 100);
+        monitor.set_bandwidth_limit("b2", "arn2", 200);
+        let opts_b1 = BucketOptions {
+            name: "b1".to_string(),
+            replication_arn: "arn1".to_string(),
+        };
+        let opts_b2 = BucketOptions {
+            name: "b2".to_string(),
+            replication_arn: "arn2".to_string(),
+        };
+        monitor.init_measurement(&opts_b1);
+        monitor.init_measurement(&opts_b2);
+
+        let report_all = monitor.get_report(|_| true);
+        assert_eq!(report_all.bucket_stats.len(), 2);
+
+        let report_b1 = monitor.get_report(|name| name == "b1");
+        assert_eq!(report_b1.bucket_stats.len(), 1);
+        assert_eq!(report_b1.bucket_stats.get(&opts_b1).unwrap().limit_bytes_per_sec, 100);
+
+        let report_b2 = monitor.get_report(|name| name == "b2");
+        assert_eq!(report_b2.bucket_stats.len(), 1);
+        assert_eq!(report_b2.bucket_stats.get(&opts_b2).unwrap().limit_bytes_per_sec, 200);
+    }
 }

crates/ecstore/src/bucket/bandwidth/reader.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use crate::bucket::bandwidth::monitor::Monitor;
+use serde::{Deserialize, Serialize};
 use std::pin::Pin;
 use std::sync::Arc;
 use std::task::{Context, Poll};
@@ -20,7 +21,7 @@ use tokio::io::{AsyncRead, ReadBuf};
 use tokio::time::Sleep;
 use tracing::{debug, warn};
 
-#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
 pub struct BucketOptions {
     pub name: String,
     pub replication_arn: String,
@@ -54,6 +55,9 @@ impl<R> MonitoredReader<R> {
             header_size = opts.header_size,
             "MonitoredReader created"
         );
+        if throttle.is_some() {
+            m.init_measurement(&opts.bucket_options);
+        }
         MonitoredReader {
             r,
             m,
@@ -117,7 +121,15 @@ impl<R: AsyncRead + Unpin> AsyncRead for MonitoredReader<R> {
             }
         }
 
-        poll_limited_read(&mut this.r, cx, buf, need, &mut this.temp_buf)
+        let filled_before = buf.filled().len();
+        let result = poll_limited_read(&mut this.r, cx, buf, need, &mut this.temp_buf);
+        if let Poll::Ready(Ok(())) = result {
+            let read_bytes = buf.filled().len().saturating_sub(filled_before) as u64;
+            if read_bytes > 0 {
+                this.m.update_measurement(&this.opts.bucket_options, read_bytes);
+            }
+        }
+        result
     }
 }
 

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -43,11 +43,13 @@ use rustfs_config::{DEFAULT_TRUST_LEAF_CERT_AS_CA, ENV_TRUST_LEAF_CERT_AS_CA, RU
 use rustfs_filemeta::{ReplicationStatusType, ReplicationType};
 use rustfs_utils::http::{
     AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE,
-    AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, RUSTFS_BUCKET_REPLICATION_CHECK,
-    RUSTFS_BUCKET_REPLICATION_DELETE_MARKER, RUSTFS_BUCKET_REPLICATION_REQUEST, RUSTFS_BUCKET_SOURCE_ETAG,
-    RUSTFS_BUCKET_SOURCE_MTIME, RUSTFS_BUCKET_SOURCE_VERSION_ID, RUSTFS_FORCE_DELETE, is_amz_header, is_minio_header,
+    AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, is_amz_header, is_minio_header,
     is_rustfs_header, is_standard_header, is_storageclass_header,
 };
+use rustfs_utils::http::{
+    SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
+    SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, insert_header,
+};
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::error::Error;
@@ -68,8 +70,6 @@ use uuid::Uuid;
 const DEFAULT_HEALTH_CHECK_DURATION: Duration = Duration::from_secs(5);
 const DEFAULT_HEALTH_CHECK_RELOAD_DURATION: Duration = Duration::from_secs(30 * 60);
 
-const REPLICATION_REQUEST_TRUE: HeaderValue = HeaderValue::from_static("true");
-
 pub static GLOBAL_BUCKET_TARGET_SYS: OnceLock<BucketTargetSys> = OnceLock::new();
 
 #[derive(Debug, Clone)]
@@ -804,7 +804,7 @@ impl BucketTargetSys {
 }
 
 async fn build_aws_s3_http_client_from_tls_path() -> Option<aws_sdk_s3::config::SharedHttpClient> {
-    let tls_path = std::env::var("RUSTFS_TLS_PATH").ok()?;
+    let tls_path = rustfs_utils::get_env_str(rustfs_config::ENV_RUSTFS_TLS_PATH, rustfs_config::DEFAULT_RUSTFS_TLS_PATH);
     if tls_path.is_empty() {
         return None;
     }
@@ -862,7 +862,6 @@ fn should_force_path_style(target: &BucketTarget) -> bool {
         // Explicit path-style or legacy boolean-like values.
         "path" | "on" | "true" => true,
         // `auto` and empty are defaulted to path-style for custom S3-compatible endpoints.
-        // RustFS/MinIO-style deployments typically do not configure virtual-hosted-style routing.
         "auto" | "" => true,
         // Unknown values: prefer compatibility with S3-compatible services.
         _ => true,
@@ -1082,23 +1081,21 @@ impl PutObjectOptions {
         }
 
         if !self.internal.source_version_id.is_empty() {
-            header.insert(
-                RUSTFS_BUCKET_SOURCE_VERSION_ID,
-                HeaderValue::from_str(&self.internal.source_version_id).expect("err"),
-            );
+            insert_header(&mut header, SUFFIX_SOURCE_VERSION_ID, &self.internal.source_version_id);
         }
         if self.internal.source_etag.is_empty() {
-            header.insert(RUSTFS_BUCKET_SOURCE_ETAG, HeaderValue::from_str(&self.internal.source_etag).expect("err"));
+            insert_header(&mut header, SUFFIX_SOURCE_ETAG, &self.internal.source_etag);
         }
         if self.internal.source_mtime.unix_timestamp() != 0 {
-            header.insert(
-                RUSTFS_BUCKET_SOURCE_MTIME,
-                HeaderValue::from_str(&self.internal.source_mtime.format(&Rfc3339).unwrap_or_default()).expect("err"),
+            insert_header(
+                &mut header,
+                SUFFIX_SOURCE_MTIME,
+                self.internal.source_mtime.format(&Rfc3339).unwrap_or_default(),
             );
         }
 
         if self.internal.replication_request {
-            header.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, REPLICATION_REQUEST_TRUE);
+            insert_header(&mut header, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
 
         header
@@ -1267,10 +1264,8 @@ impl TargetClient {
         let builder = self.client.put_object();
 
         let version_id = opts.internal.source_version_id.clone();
-        if !version_id.is_empty()
-            && let Ok(header_value) = HeaderValue::from_str(&version_id)
-        {
-            headers.insert(RUSTFS_BUCKET_SOURCE_VERSION_ID, header_value);
+        if !version_id.is_empty() {
+            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
         }
 
         match builder
@@ -1304,13 +1299,11 @@ impl TargetClient {
     ) -> Result<String, S3ClientError> {
         let mut headers = HeaderMap::new();
         let version_id = opts.internal.source_version_id.clone();
-        if !version_id.is_empty()
-            && let Ok(header_value) = HeaderValue::from_str(&version_id)
-        {
-            headers.insert(RUSTFS_BUCKET_SOURCE_VERSION_ID, header_value);
+        if !version_id.is_empty() {
+            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
         }
         if opts.internal.replication_request {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, REPLICATION_REQUEST_TRUE);
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
 
         match self
@@ -1419,21 +1412,18 @@ impl TargetClient {
     ) -> Result<(), S3ClientError> {
         let mut headers = HeaderMap::new();
         if opts.force_delete {
-            headers.insert(RUSTFS_FORCE_DELETE, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_FORCE_DELETE, "true");
         }
         if opts.governance_bypass {
             headers.insert(AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, "true".parse().unwrap());
         }
 
         if opts.replication_delete_marker {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_DELETE_MARKER, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_DELETEMARKER, "true");
         }
 
         if let Some(t) = opts.replication_mtime {
-            headers.insert(
-                RUSTFS_BUCKET_SOURCE_MTIME,
-                t.format(&Rfc3339).unwrap_or_default().as_str().parse().unwrap(),
-            );
+            insert_header(&mut headers, SUFFIX_SOURCE_MTIME, t.format(&Rfc3339).unwrap_or_default());
         }
 
         if !opts.replication_status.is_empty() {
@@ -1441,10 +1431,10 @@ impl TargetClient {
         }
 
         if opts.replication_request {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
         if opts.replication_validity_check {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_CHECK, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
         }
 
         match self

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -28,13 +28,14 @@ use crate::client::object_api_utils::new_getobjectreader;
 use crate::error::Error;
 use crate::error::StorageError;
 use crate::error::{error_resp_to_object_err, is_err_object_not_found, is_err_version_not_found, is_network_or_host_down};
-use crate::event::name::EventName;
 use crate::event_notification::{EventArgs, send_event};
 use crate::global::GLOBAL_LocalNodeName;
 use crate::global::{GLOBAL_LifecycleSys, GLOBAL_TierConfigMgr, get_global_deployment_id};
 use crate::store::ECStore;
 use crate::store_api::StorageAPI;
-use crate::store_api::{GetObjectReader, HTTPRangeSpec, ObjectInfo, ObjectOperations, ObjectOptions, ObjectToDelete};
+use crate::store_api::{
+    GetObjectReader, HTTPRangeSpec, ListOperations, ObjectInfo, ObjectOperations, ObjectOptions, ObjectToDelete,
+};
 use crate::tier::warm_backend::WarmBackendGetOpts;
 use async_channel::{Receiver as A_Receiver, Sender as A_Sender, bounded};
 use bytes::BytesMut;
@@ -45,8 +46,8 @@ use rustfs_common::data_usage::TierStats;
 use rustfs_common::heal_channel::rep_has_active_rules;
 use rustfs_common::metrics::{IlmAction, Metrics};
 use rustfs_filemeta::{NULL_VERSION_ID, RestoreStatusOps, is_restored_object_on_disk};
-use rustfs_utils::path::encode_dir_object;
-use rustfs_utils::string::strings_has_prefix_fold;
+use rustfs_s3_common::EventName;
+use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::strings_has_prefix_fold};
 use s3s::Body;
 use s3s::dto::{
     BucketLifecycleConfiguration, DefaultRetention, ReplicationConfiguration, RestoreRequest, RestoreRequestType, RestoreStatus,
@@ -97,8 +98,14 @@ impl LifecycleSys {
     }
 
     pub async fn get(&self, bucket: &str) -> Option<BucketLifecycleConfiguration> {
-        let lc = get_lifecycle_config(bucket).await.expect("get_lifecycle_config err!").0;
-        Some(lc)
+        match get_lifecycle_config(bucket).await {
+            Ok((lc, _)) => Some(lc),
+            Err(err) if err == Error::ConfigNotFound => None,
+            Err(err) => {
+                warn!(bucket, error = ?err, "failed to load lifecycle config");
+                None
+            }
+        }
     }
 
     pub fn trace(_oi: &ObjectInfo) -> TraceFn {
@@ -471,10 +478,7 @@ impl TransitionState {
     }
 
     pub async fn init(api: Arc<ECStore>) {
-        let max_workers = std::env::var("RUSTFS_MAX_TRANSITION_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<i64>().ok())
-            .unwrap_or_else(|| std::cmp::min(num_cpus::get() as i64, 16));
+        let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
         let mut n = max_workers;
         let tw = 8; //globalILMConfig.getTransitionWorkers();
         if tw > 0 {
@@ -569,17 +573,11 @@ impl TransitionState {
     pub async fn update_workers_inner(api: Arc<ECStore>, n: i64) {
         let mut n = n;
         if n == 0 {
-            let max_workers = std::env::var("RUSTFS_MAX_TRANSITION_WORKERS")
-                .ok()
-                .and_then(|s| s.parse::<i64>().ok())
-                .unwrap_or_else(|| std::cmp::min(num_cpus::get() as i64, 16));
+            let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
             n = max_workers;
         }
         // Allow environment override of maximum workers
-        let absolute_max = std::env::var("RUSTFS_ABSOLUTE_MAX_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<i64>().ok())
-            .unwrap_or(32);
+        let absolute_max = get_env_i64("RUSTFS_ABSOLUTE_MAX_WORKERS", 32);
         n = std::cmp::min(n, absolute_max);
 
         let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
@@ -603,10 +601,7 @@ impl TransitionState {
 }
 
 pub async fn init_background_expiry(api: Arc<ECStore>) {
-    let mut workers = std::env::var("RUSTFS_MAX_EXPIRY_WORKERS")
-        .ok()
-        .and_then(|s| s.parse::<usize>().ok())
-        .unwrap_or_else(|| std::cmp::min(num_cpus::get(), 16));
+    let mut workers = get_env_usize("RUSTFS_MAX_EXPIRY_WORKERS", std::cmp::min(num_cpus::get(), 16));
     //globalILMConfig.getExpirationWorkers()
     if let Ok(env_expiration_workers) = env::var("_RUSTFS_ILM_EXPIRATION_WORKERS") {
         if let Ok(num_expirations) = env_expiration_workers.parse::<usize>() {
@@ -615,10 +610,7 @@ pub async fn init_background_expiry(api: Arc<ECStore>) {
     }
 
     if workers == 0 {
-        workers = std::env::var("RUSTFS_DEFAULT_EXPIRY_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<usize>().ok())
-            .unwrap_or(8);
+        workers = get_env_usize("RUSTFS_DEFAULT_EXPIRY_WORKERS", 8);
     }
 
     //let expiry_state = GLOBAL_ExpiryStSate.write().await;
@@ -656,18 +648,48 @@ pub async fn validate_transition_tier(lc: &BucketLifecycleConfiguration) -> Resu
 }
 
 pub async fn enqueue_transition_immediate(oi: &ObjectInfo, src: LcEventSrc) {
-    let lc = GLOBAL_LifecycleSys.get(&oi.bucket).await;
-    if !lc.is_none() {
-        let event = lc.expect("err").eval(&oi.to_lifecycle_opts()).await;
-        match event.action {
-            IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
-                if oi.delete_marker || oi.is_dir {
-                    return;
-                }
-                GLOBAL_TransitionState.queue_transition_task(oi, &event, &src).await;
-            }
-            _ => (),
+    if let Some(lc) = GLOBAL_LifecycleSys.get(&oi.bucket).await {
+        enqueue_transition_with_lifecycle(oi, &lc, &src).await;
+    }
+}
+
+pub async fn enqueue_transition_for_existing_objects(api: Arc<ECStore>, bucket: &str) -> Result<(), Error> {
+    let Some(lc) = GLOBAL_LifecycleSys.get(bucket).await else {
+        return Ok(());
+    };
+    let mut marker = None;
+    let mut version_marker = None;
+    let src = LcEventSrc::Scanner;
+
+    loop {
+        let page = api
+            .clone()
+            .list_object_versions(bucket, "", marker.clone(), version_marker.clone(), None, 1000)
+            .await?;
+
+        for object in &page.objects {
+            enqueue_transition_with_lifecycle(object, &lc, &src).await;
         }
+
+        if !page.is_truncated {
+            return Ok(());
+        }
+
+        marker = page.next_marker;
+        version_marker = page.next_version_idmarker;
+    }
+}
+
+async fn enqueue_transition_with_lifecycle(oi: &ObjectInfo, lc: &BucketLifecycleConfiguration, src: &LcEventSrc) {
+    let event = lc.eval(&oi.to_lifecycle_opts()).await;
+    match event.action {
+        IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
+            if oi.delete_marker || oi.is_dir {
+                return;
+            }
+            GLOBAL_TransitionState.queue_transition_task(oi, &event, src).await;
+        }
+        _ => (),
     }
 }
 
@@ -689,13 +711,13 @@ pub async fn expire_transitioned_object(
     //let tags = LcAuditEvent::new(src, lcEvent).Tags();
     if lc_event.action == IlmAction::DeleteRestoredAction {
         opts.transition.expire_restored = true;
-        match api.delete_object(&oi.bucket, &oi.name, opts).await {
+        return match api.delete_object(&oi.bucket, &oi.name, opts).await {
             Ok(dobj) => {
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
-                return Ok(dobj);
+                Ok(dobj)
             }
-            Err(err) => return Err(std::io::Error::other(err)),
-        }
+            Err(err) => Err(std::io::Error::other(err)),
+        };
     }
 
     let ret = delete_object_from_remote_tier(
@@ -732,7 +754,7 @@ pub async fn expire_transitioned_object(
         ..Default::default()
     };
     send_event(EventArgs {
-        event_name: event_name.as_ref().to_string(),
+        event_name: event_name.to_string(),
         bucket_name: obj_info.bucket.clone(),
         object: obj_info,
         user_agent: "Internal: [ILM-Expiry]".to_string(),
@@ -847,8 +869,8 @@ pub async fn post_restore_opts(version_id: &str, bucket: &str, object: &str) ->
         }
     }
     Ok(ObjectOptions {
-        versioned: versioned,
-        version_suspended: version_suspended,
+        versioned,
+        version_suspended,
         version_id: Some(vid.to_string()),
         ..Default::default()
     })
@@ -1033,12 +1055,12 @@ pub async fn eval_action_from_lifecycle(
     let lock_enabled = if let Some(lr) = lr { lr.mode.is_some() } else { false };
 
     match event.action {
-        lifecycle::IlmAction::DeleteAllVersionsAction | lifecycle::IlmAction::DelMarkerDeleteAllVersionsAction => {
+        IlmAction::DeleteAllVersionsAction | IlmAction::DelMarkerDeleteAllVersionsAction => {
             if lock_enabled {
                 return lifecycle::Event::default();
             }
         }
-        lifecycle::IlmAction::DeleteVersionAction | lifecycle::IlmAction::DeleteRestoredVersionAction => {
+        IlmAction::DeleteVersionAction | IlmAction::DeleteRestoredVersionAction => {
             if oi.version_id.is_none() {
                 return lifecycle::Event::default();
             }
@@ -1139,12 +1161,12 @@ pub async fn apply_expiry_on_non_transitioned_objects(
         event_name = EventName::ObjectRemovedDeleteMarkerCreated;
     }
     match lc_event.action {
-        lifecycle::IlmAction::DeleteAllVersionsAction => event_name = EventName::ObjectRemovedDeleteAllVersions,
-        lifecycle::IlmAction::DelMarkerDeleteAllVersionsAction => event_name = EventName::ILMDelMarkerExpirationDelete,
+        IlmAction::DeleteAllVersionsAction => event_name = EventName::ObjectRemovedDeleteAllVersions,
+        IlmAction::DelMarkerDeleteAllVersionsAction => event_name = EventName::LifecycleDelMarkerExpirationDelete,
         _ => (),
     }
     send_event(EventArgs {
-        event_name: event_name.as_ref().to_string(),
+        event_name: event_name.to_string(),
         bucket_name: dobj.bucket.clone(),
         object: dobj,
         user_agent: "Internal: [ILM-Expiry]".to_string(),
@@ -1152,7 +1174,7 @@ pub async fn apply_expiry_on_non_transitioned_objects(
         ..Default::default()
     });
 
-    if lc_event.action != lifecycle::IlmAction::NoneAction {
+    if lc_event.action != IlmAction::NoneAction {
         let mut num_versions = 1_u64;
         if lc_event.action.delete_all() {
             num_versions = oi.num_versions as u64;
@@ -1172,15 +1194,15 @@ pub async fn apply_expiry_rule(event: &lifecycle::Event, src: &LcEventSrc, oi: &
 pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc, oi: &ObjectInfo) -> bool {
     let mut success = false;
     match event.action {
-        lifecycle::IlmAction::DeleteVersionAction
-        | lifecycle::IlmAction::DeleteAction
-        | lifecycle::IlmAction::DeleteRestoredAction
-        | lifecycle::IlmAction::DeleteRestoredVersionAction
-        | lifecycle::IlmAction::DeleteAllVersionsAction
-        | lifecycle::IlmAction::DelMarkerDeleteAllVersionsAction => {
+        IlmAction::DeleteVersionAction
+        | IlmAction::DeleteAction
+        | IlmAction::DeleteRestoredAction
+        | IlmAction::DeleteRestoredVersionAction
+        | IlmAction::DeleteAllVersionsAction
+        | IlmAction::DelMarkerDeleteAllVersionsAction => {
             success = apply_expiry_rule(event, src, oi).await;
         }
-        lifecycle::IlmAction::TransitionAction | lifecycle::IlmAction::TransitionVersionAction => {
+        IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
             success = apply_transition_rule(event, src, oi).await;
         }
         _ => (),

crates/ecstore/src/bucket/lifecycle/lifecycle.rs

@@ -20,8 +20,8 @@
 
 use rustfs_filemeta::{ReplicationStatusType, VersionPurgeStatusType};
 use s3s::dto::{
-    BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, NoncurrentVersionTransition,
-    ObjectLockConfiguration, ObjectLockEnabled, RestoreRequest, Transition, TransitionStorageClass,
+    BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleAndOperator,
+    NoncurrentVersionTransition, ObjectLockConfiguration, ObjectLockEnabled, RestoreRequest, Transition, TransitionStorageClass,
 };
 use std::cmp::Ordering;
 use std::collections::HashMap;
@@ -49,6 +49,8 @@ const ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS: &str = "Lifecycle expiration days m
 const ERR_LIFECYCLE_INVALID_EXPIRATION_DATE_NOT_MIDNIGHT: &str = "Expiration.Date must be at midnight UTC";
 const ERR_LIFECYCLE_INVALID_RULE_ID_TOO_LONG: &str = "Rule ID must be at most 255 characters";
 const ERR_LIFECYCLE_INVALID_RULE_STATUS: &str = "Rule status must be either Enabled or Disabled";
+const ERR_LIFECYCLE_DEL_MARKER_WITH_TAGS: &str = "Rule with DelMarkerExpiration cannot have tags based filtering";
+const ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION: &str = "Rule must have at least one of Expiration, Transition, NoncurrentVersionExpiration, NoncurrentVersionTransition, or DelMarkerExpiration";
 
 pub use rustfs_common::metrics::IlmAction;
 
@@ -117,23 +119,66 @@ impl RuleValidate for LifecycleRule {
     }*/
 
     fn validate(&self) -> Result<(), std::io::Error> {
-        /*self.validate_id()?;
-        self.validate_status()?;
-        self.validate_expiration()?;
-        self.validate_noncurrent_expiration()?;
-        self.validate_prefix_and_filter()?;
-        self.validate_transition()?;
-        self.validate_noncurrent_transition()?;
-        if (!self.Filter.Tag.IsEmpty() || len(self.Filter.And.Tags) != 0) && !self.delmarker_expiration.Empty() {
-          return errInvalidRuleDelMarkerExpiration
+        // Rule with DelMarkerExpiration cannot have tags based filtering
+        let has_tag_filter = self
+            .filter
+            .as_ref()
+            .map_or(false, |f| f.tag.is_some() || f.and.as_ref().and_then(|a| a.tags.as_ref()).is_some());
+        if has_tag_filter && self.del_marker_expiration.is_some() {
+            return Err(std::io::Error::other(ERR_LIFECYCLE_DEL_MARKER_WITH_TAGS));
+        }
+        // Rule must have at least one action
+        let has_expiration = self.expiration.is_some();
+        let has_transition = self.transitions.as_ref().map_or(false, |t| !t.is_empty());
+        let has_noncurrent_expiration = self
+            .noncurrent_version_expiration
+            .as_ref()
+            .and_then(|e| e.noncurrent_days)
+            .map_or(false, |d| d != 0);
+        let has_noncurrent_transition = self
+            .noncurrent_version_transitions
+            .as_ref()
+            .and_then(|t| t.first())
+            .and_then(|t| t.storage_class.as_ref())
+            .is_some();
+        let has_abort_incomplete_multipart_upload = self.abort_incomplete_multipart_upload.is_some();
+        let has_del_marker_expiration = self
+            .del_marker_expiration
+            .as_ref()
+            .and_then(|d| d.days)
+            .map_or(false, |d| d > 0);
+        if !has_expiration
+            && !has_transition
+            && !has_noncurrent_expiration
+            && !has_noncurrent_transition
+            && !has_abort_incomplete_multipart_upload
+            && !has_del_marker_expiration
+        {
+            return Err(std::io::Error::other(ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION));
         }
-        if !self.expiration.set && !self.transition.set && !self.noncurrent_version_expiration.set && !self.noncurrent_version_transitions.unwrap()[0].set && self.delmarker_expiration.Empty() {
-          return errXMLNotWellFormed
-        }*/
         Ok(())
     }
 }
 
+fn lifecycle_rule_prefix(rule: &LifecycleRule) -> Option<&str> {
+    // Prefer a non-empty legacy prefix; treat an empty legacy prefix as if it were not set
+    if let Some(p) = rule.prefix.as_deref() {
+        if !p.is_empty() {
+            return Some(p);
+        }
+    }
+
+    let Some(filter) = rule.filter.as_ref() else {
+        return None;
+    };
+
+    if let Some(p) = filter.prefix.as_deref() {
+        return Some(p);
+    }
+
+    filter.and.as_ref().and_then(|and| and.prefix.as_deref())
+}
+
 #[async_trait::async_trait]
 pub trait Lifecycle {
     async fn has_transition(&self) -> bool;
@@ -177,8 +222,11 @@ impl Lifecycle for BucketLifecycleConfiguration {
                 continue;
             }
 
-            let rule_prefix = &rule.prefix.clone().unwrap_or_default();
-            if prefix.len() > 0 && rule_prefix.len() > 0 && !prefix.starts_with(rule_prefix) && !rule_prefix.starts_with(&prefix)
+            let rule_prefix = lifecycle_rule_prefix(rule).unwrap_or("");
+            if !prefix.is_empty()
+                && !rule_prefix.is_empty()
+                && !prefix.starts_with(rule_prefix)
+                && !rule_prefix.starts_with(prefix)
             {
                 continue;
             }
@@ -297,8 +345,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
             if rule.status.as_str() == ExpirationStatus::DISABLED {
                 continue;
             }
-            if let Some(prefix) = rule.prefix.clone() {
-                if !obj.name.starts_with(prefix.as_str()) {
+            if let Some(rule_prefix) = lifecycle_rule_prefix(rule) {
+                if !obj.name.starts_with(rule_prefix) {
                     continue;
                 }
             }
@@ -414,43 +462,32 @@ impl Lifecycle for BucketLifecycleConfiguration {
 
         if let Some(ref lc_rules) = self.filter_rules(obj).await {
             for rule in lc_rules.iter() {
-                if obj.expired_object_deletemarker() {
+                if obj.is_latest && obj.expired_object_deletemarker() {
                     if let Some(expiration) = rule.expiration.as_ref() {
-                        if let Some(expired_object_delete_marker) = expiration.expired_object_delete_marker {
-                            events.push(Event {
-                                action: IlmAction::DeleteVersionAction,
-                                rule_id: rule.id.clone().unwrap_or_default(),
-                                due: Some(now),
-                                noncurrent_days: 0,
-                                newer_noncurrent_versions: 0,
-                                storage_class: "".into(),
-                            });
-                            break;
-                        }
-
-                        if let Some(days) = expiration.days {
-                            let expected_expiry = expected_expiry_time(mod_time, days /*, date*/);
-                            if now.unix_timestamp() >= expected_expiry.unix_timestamp() {
+                        if expiration.expired_object_delete_marker.is_some_and(|v| v) {
+                            // Preserve explicit date/days scheduling when configured.
+                            // If only ExpiredObjectDeleteMarker=true is set, delete immediately.
+                            let due = expiration.next_due(obj).unwrap_or(now);
+                            if now.unix_timestamp() >= due.unix_timestamp() {
                                 events.push(Event {
                                     action: IlmAction::DeleteVersionAction,
                                     rule_id: rule.id.clone().unwrap_or_default(),
-                                    due: Some(expected_expiry),
+                                    due: Some(due),
                                     noncurrent_days: 0,
                                     newer_noncurrent_versions: 0,
                                     storage_class: "".into(),
                                 });
+                                // Stop after scheduling an expired delete-marker event.
                                 break;
                             }
                         }
                     }
-                }
-
-                if obj.is_latest {
-                    if let Some(ref expiration) = rule.expiration {
-                        if let Some(expired_object_delete_marker) = expiration.expired_object_delete_marker {
-                            if obj.delete_marker && expired_object_delete_marker {
-                                let due = expiration.next_due(obj);
-                                if let Some(due) = due {
+                    // DelMarkerExpiration: expire delete marker after N days from mod_time
+                    if obj.delete_marker {
+                        if let Some(ref dme) = rule.del_marker_expiration {
+                            if let Some(days) = dme.days {
+                                if days > 0 {
+                                    let due = expected_expiry_time(mod_time, days);
                                     if now.unix_timestamp() >= due.unix_timestamp() {
                                         events.push(Event {
                                             action: IlmAction::DelMarkerDeleteAllVersionsAction,
@@ -461,8 +498,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
                                             storage_class: "".into(),
                                         });
                                     }
+                                    continue;
                                 }
-                                continue;
                             }
                         }
                     }
@@ -694,8 +731,16 @@ impl LifecycleCalculate for LifecycleExpiration {
         if !obj.is_latest || !obj.delete_marker {
             return None;
         }
+        // Check date first (date-based expiration takes priority over days).
+        // A zero unix timestamp means "not set" (default value) and is skipped.
+        if let Some(ref date) = self.date {
+            let expiry_date = OffsetDateTime::from(date.clone());
+            if expiry_date.unix_timestamp() != 0 {
+                return Some(expiry_date);
+            }
+        }
         match self.days {
-            Some(days) => Some(expected_expiry_time(obj.mod_time.unwrap(), days)),
+            Some(days) => obj.mod_time.map(|mod_time| expected_expiry_time(mod_time, days)),
             None => None,
         }
     }
@@ -723,7 +768,7 @@ impl LifecycleCalculate for NoncurrentVersionTransition {
 #[async_trait::async_trait]
 impl LifecycleCalculate for Transition {
     fn next_due(&self, obj: &ObjectOpts) -> Option<OffsetDateTime> {
-        if !obj.is_latest || self.days.is_none() {
+        if !obj.is_latest {
             return None;
         }
 
@@ -860,10 +905,15 @@ impl Default for TransitionOptions {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use s3s::dto::LifecycleRuleFilter;
+    use serial_test::serial;
+    use std::sync::Arc;
 
     #[tokio::test]
+    #[serial]
     async fn validate_rejects_non_positive_expiration_days() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -871,6 +921,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -889,8 +940,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_accepts_positive_expiration_days() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -898,6 +951,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -913,8 +967,36 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
+    async fn validate_accepts_abort_incomplete_multipart_upload_only_rule() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: Some(s3s::dto::AbortIncompleteMultipartUpload {
+                    days_after_initiation: Some(2),
+                }),
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("abort-only".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: Some("test/".to_string()),
+                transitions: None,
+            }],
+        };
+
+        lc.validate(&ObjectLockConfiguration::default())
+            .await
+            .expect("expected validation to pass");
+    }
+
+    #[tokio::test]
+    #[serial]
     async fn validate_rejects_non_midnight_expiration_date() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -922,6 +1004,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -937,9 +1020,11 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn predict_expiration_selects_closest_expiry_for_put_object() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![
                 LifecycleRule {
                     status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
@@ -948,6 +1033,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: Some("rule-days".to_string()),
                     noncurrent_version_expiration: None,
@@ -962,6 +1048,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: Some("rule-date".to_string()),
                     noncurrent_version_expiration: None,
@@ -988,8 +1075,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_accepts_multiple_rules_without_ids() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![
                 LifecycleRule {
                     status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
@@ -998,6 +1087,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: None,
                     noncurrent_version_expiration: None,
@@ -1012,6 +1102,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: None,
                     noncurrent_version_expiration: None,
@@ -1028,8 +1119,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_rejects_rule_id_too_long() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -1037,6 +1130,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: Some("a".repeat(256)),
                 noncurrent_version_expiration: None,
@@ -1052,8 +1146,315 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
+    async fn validate_rejects_duplicate_rule_ids() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![
+                LifecycleRule {
+                    status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                    expiration: Some(LifecycleExpiration {
+                        days: Some(1),
+                        ..Default::default()
+                    }),
+                    abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
+                    filter: None,
+                    id: Some("dup-rule".to_string()),
+                    noncurrent_version_expiration: None,
+                    noncurrent_version_transitions: None,
+                    prefix: None,
+                    transitions: None,
+                },
+                LifecycleRule {
+                    status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                    expiration: Some(LifecycleExpiration {
+                        days: Some(2),
+                        ..Default::default()
+                    }),
+                    abort_incomplete_multipart_upload: None,
+                    filter: None,
+                    id: Some("dup-rule".to_string()),
+                    noncurrent_version_expiration: None,
+                    noncurrent_version_transitions: None,
+                    prefix: None,
+                    transitions: None,
+                    del_marker_expiration: None,
+                },
+            ],
+        };
+
+        let err = lc.validate(&ObjectLockConfiguration::default()).await.unwrap_err();
+        assert_eq!(err.to_string(), ERR_LIFECYCLE_DUPLICATE_ID);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_expires_latest_object_after_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("expire-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteAction);
+        assert_eq!(event.rule_id, "expire-days");
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_keeps_latest_object_before_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(2),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("expire-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::hours(12), 0).await;
+
+        assert_eq!(event.action, IlmAction::NoneAction);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_transitions_latest_object_after_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("transition-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: Some(vec![Transition {
+                    days: Some(1),
+                    date: None,
+                    storage_class: Some(TransitionStorageClass::from_static("COLDTIER44")),
+                }]),
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            transition_status: "".to_string(),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::TransitionAction);
+        assert_eq!(event.rule_id, "transition-days");
+        assert_eq!(event.storage_class, "COLDTIER44");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_transitions_latest_object_after_date_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let transition_date = base_time - Duration::days(1);
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("transition-date".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: Some(vec![Transition {
+                    days: None,
+                    date: Some(transition_date.into()),
+                    storage_class: Some(TransitionStorageClass::from_static("WARM")),
+                }]),
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            transition_status: "".to_string(),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(1), 0).await;
+
+        assert_eq!(event.action, IlmAction::TransitionAction);
+        assert_eq!(event.rule_id, "transition-date");
+        assert_eq!(event.storage_class, "WARM");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_expires_noncurrent_version_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-expire".to_string()),
+                noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
+                    noncurrent_days: Some(1),
+                    newer_noncurrent_versions: None,
+                }),
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            successor_mod_time: Some(base_time),
+            is_latest: false,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-expire");
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_transitions_noncurrent_version_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-transition".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
+                    noncurrent_days: Some(1),
+                    newer_noncurrent_versions: None,
+                    storage_class: Some(TransitionStorageClass::from_static("COLDTIER44")),
+                }]),
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            successor_mod_time: Some(base_time),
+            is_latest: false,
+            transition_status: "".to_string(),
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::TransitionVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-transition");
+        assert_eq!(event.storage_class, "COLDTIER44");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn noncurrent_versions_expiration_limit_returns_configured_limits() {
+        let lc = Arc::new(BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-limit".to_string()),
+                noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
+                    noncurrent_days: Some(7),
+                    newer_noncurrent_versions: Some(3),
+                }),
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        });
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: false,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.noncurrent_versions_expiration_limit(&opts).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-limit");
+        assert_eq!(event.noncurrent_days, 7);
+        assert_eq!(event.newer_noncurrent_versions, 3);
+    }
+
+    #[tokio::test]
+    #[serial]
     async fn validate_rejects_invalid_status_case_sensitive() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static("enabled"),
                 expiration: Some(LifecycleExpiration {
@@ -1061,6 +1462,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -1074,4 +1476,260 @@ mod tests {
 
         assert_eq!(err.to_string(), ERR_LIFECYCLE_INVALID_RULE_STATUS);
     }
+
+    #[tokio::test]
+    #[serial]
+    async fn filter_rules_respects_filter_prefix() {
+        let mut filter = LifecycleRuleFilter::default();
+        filter.prefix = Some("prefix".to_string());
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: Some(filter),
+                id: Some("rule".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let match_obj = ObjectOpts {
+            name: "prefix/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let matched = lc.filter_rules(&match_obj).await.unwrap();
+        assert_eq!(matched.len(), 1);
+
+        let non_match_obj = ObjectOpts {
+            name: "other/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let not_matched = lc.filter_rules(&non_match_obj).await.unwrap();
+        assert_eq!(not_matched.len(), 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn filter_rules_respects_filter_and_prefix() {
+        let mut filter = LifecycleRuleFilter::default();
+
+        let mut and = LifecycleRuleAndOperator::default();
+        and.prefix = Some("prefix".to_string());
+        filter.and = Some(and);
+
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: Some(filter),
+                id: Some("rule-and-prefix".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let match_obj = ObjectOpts {
+            name: "prefix/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let matched = lc.filter_rules(&match_obj).await.unwrap();
+        assert_eq!(matched.len(), 1);
+
+        let non_match_obj = ObjectOpts {
+            name: "other/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let not_matched = lc.filter_rules(&non_match_obj).await.unwrap();
+        assert_eq!(not_matched.len(), 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expired_object_delete_marker_requires_single_version() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 2,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+        assert_eq!(event.action, IlmAction::NoneAction);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expired_object_delete_marker_deletes_only_delete_marker_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    async fn expired_object_delete_marker_without_date_or_days_deletes_immediately() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker-immediate".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.due, Some(now));
+    }
+
+    #[tokio::test]
+    async fn expired_object_delete_marker_date_based_not_yet_due() {
+        // A date-based rule that has not yet reached its expiry date must not
+        // trigger immediate deletion (unwrap_or(now) must not override the date).
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let future_date = base_time + Duration::days(10);
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    date: Some(future_date.into()),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-date-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        // now is before the configured date — must not schedule deletion
+        let now_before = base_time + Duration::days(5);
+        let event_before = lc.eval_inner(&opts, now_before, 0).await;
+        assert_eq!(event_before.action, IlmAction::NoneAction);
+
+        // now is after the configured date — must schedule deletion
+        let now_after = base_time + Duration::days(11);
+        let event_after = lc.eval_inner(&opts, now_after, 0).await;
+        assert_eq!(event_after.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event_after.due, Some(future_date));
+    }
 }

crates/ecstore/src/bucket/lifecycle/tier_last_day_stats.rs

@@ -51,6 +51,10 @@ impl LastDayTierStats {
         self.bins[now_idx] = self.bins[now_idx].add(&ts);
     }
 
+    pub fn total(&self) -> TierStats {
+        self.bins.iter().fold(TierStats::default(), |acc, bin| acc.add(bin))
+    }
+
     fn forward_to(&mut self, t: &mut OffsetDateTime) {
         if t.unix_timestamp() == 0 {
             *t = OffsetDateTime::now_utc();
@@ -99,4 +103,30 @@ impl LastDayTierStats {
 }
 
 #[cfg(test)]
-mod test {}
+mod test {
+    use super::*;
+
+    #[test]
+    fn total_sums_all_recorded_stats() {
+        let mut stats = LastDayTierStats::default();
+        stats.add_stats(TierStats {
+            total_size: 10,
+            num_versions: 1,
+            num_objects: 1,
+        });
+        stats.add_stats(TierStats {
+            total_size: 20,
+            num_versions: 2,
+            num_objects: 0,
+        });
+
+        assert_eq!(
+            stats.total(),
+            TierStats {
+                total_size: 30,
+                num_versions: 3,
+                num_objects: 1,
+            }
+        );
+    }
+}

crates/ecstore/src/bucket/metadata.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
 use super::object_lock::ObjectLockApi;
 use super::versioning::VersioningApi;
 use super::{quota::BucketQuota, target::BucketTargets};
@@ -22,7 +23,6 @@ use crate::error::{Error, Result};
 use crate::new_object_layer_fn;
 use crate::store::ECStore;
 use byteorder::{BigEndian, ByteOrder, LittleEndian};
-use rmp_serde::Serializer as rmpSerializer;
 use rustfs_policy::policy::BucketPolicy;
 use s3s::dto::{
     BucketLifecycleConfiguration, CORSConfiguration, NotificationConfiguration, ObjectLockConfiguration,
@@ -30,12 +30,199 @@ use s3s::dto::{
     VersioningConfiguration,
 };
 use serde::Serializer;
-use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
+use std::io::{Read, Write};
 use std::sync::Arc;
-use time::OffsetDateTime;
+use time::{Date, OffsetDateTime, PrimitiveDateTime, Time as CivilTime, UtcOffset};
 use tracing::error;
 
+fn read_msgp_str<R: Read>(rd: &mut R) -> Result<String> {
+    let len = rmp::decode::read_str_len(rd)? as usize;
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    Ok(String::from_utf8(buf)?)
+}
+
+fn read_msgp_bool<R: Read>(rd: &mut R) -> Result<bool> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    match marker {
+        rmp::Marker::True => Ok(true),
+        rmp::Marker::False => Ok(false),
+        rmp::Marker::FixPos(v) => Ok(v != 0),
+        rmp::Marker::U8 => Ok(read_u8(rd)? != 0),
+        rmp::Marker::U16 => Ok(read_u16_raw(rd)? != 0),
+        rmp::Marker::U32 => Ok(read_u32_raw(rd)? != 0),
+        rmp::Marker::U64 => Ok(read_u64_raw(rd)? != 0),
+        rmp::Marker::I8 => Ok(read_i8_raw(rd)? != 0),
+        rmp::Marker::I16 => Ok(read_i16_raw(rd)? != 0),
+        rmp::Marker::I32 => Ok(read_i32_raw(rd)? != 0),
+        rmp::Marker::I64 => Ok(read_i64_raw(rd)? != 0),
+        rmp::Marker::FixNeg(v) => Ok(v != 0),
+        _ => Err(Error::other(format!("expected bool or int-like bool, got marker: {marker:?}"))),
+    }
+}
+
+fn read_msgp_time_value<R: Read>(rd: &mut R) -> Result<OffsetDateTime> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    match marker {
+        rmp::Marker::Null => Ok(OffsetDateTime::UNIX_EPOCH),
+        rmp::Marker::Ext8 => read_msgp_ext8_time(rd),
+        rmp::Marker::FixArray(len) => read_msgp_legacy_compact_time(rd, u32::from(len)),
+        rmp::Marker::Array16 => {
+            let len = read_u16_raw(rd)?;
+            read_msgp_legacy_compact_time(rd, u32::from(len))
+        }
+        rmp::Marker::Array32 => {
+            let len = read_u32_raw(rd)?;
+            read_msgp_legacy_compact_time(rd, len)
+        }
+        rmp::Marker::Bin8 => {
+            let len = usize::from(read_u8(rd)?);
+            read_msgp_time_value_from_embedded_bin(rd, len)
+        }
+        rmp::Marker::Bin16 => {
+            let len = usize::from(read_u16_raw(rd)?);
+            read_msgp_time_value_from_embedded_bin(rd, len)
+        }
+        rmp::Marker::Bin32 => {
+            let len = read_u32_raw(rd)? as usize;
+            read_msgp_time_value_from_embedded_bin(rd, len)
+        }
+        _ => Err(Error::other(format!("expected time ext or nil, got marker: {marker:?}"))),
+    }
+}
+
+fn read_u8<R: Read>(rd: &mut R) -> Result<u8> {
+    let mut buf = [0u8; 1];
+    rd.read_exact(&mut buf)?;
+    Ok(buf[0])
+}
+
+fn read_u16_raw<R: Read>(rd: &mut R) -> Result<u16> {
+    let mut buf = [0u8; 2];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_u16(&buf))
+}
+
+fn read_u32_raw<R: Read>(rd: &mut R) -> Result<u32> {
+    let mut buf = [0u8; 4];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_u32(&buf))
+}
+
+fn read_u64_raw<R: Read>(rd: &mut R) -> Result<u64> {
+    let mut buf = [0u8; 8];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_u64(&buf))
+}
+
+fn read_i8_raw<R: Read>(rd: &mut R) -> Result<i8> {
+    Ok(read_u8(rd)? as i8)
+}
+
+fn read_i16_raw<R: Read>(rd: &mut R) -> Result<i16> {
+    let mut buf = [0u8; 2];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_i16(&buf))
+}
+
+fn read_i32_raw<R: Read>(rd: &mut R) -> Result<i32> {
+    let mut buf = [0u8; 4];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_i32(&buf))
+}
+
+fn read_i64_raw<R: Read>(rd: &mut R) -> Result<i64> {
+    let mut buf = [0u8; 8];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_i64(&buf))
+}
+
+fn read_msgp_time_value_from_embedded_bin<R: Read>(rd: &mut R, len: usize) -> Result<OffsetDateTime> {
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    let mut cur = std::io::Cursor::new(buf);
+    read_msgp_time_value(&mut cur)
+}
+
+fn read_msgp_legacy_compact_time<R: Read>(rd: &mut R, len: u32) -> Result<OffsetDateTime> {
+    if len != 9 {
+        return Err(Error::other(format!("invalid legacy compact time len: {len}")));
+    }
+
+    let year: i32 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let ordinal: u16 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let hour: u8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let minute: u8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let second: u8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let nanosecond: u32 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let offset_hour: i8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let offset_minute: i8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let offset_second: i8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+
+    let date =
+        Date::from_ordinal_date(year, ordinal).map_err(|e| Error::other(format!("invalid legacy compact time date: {e}")))?;
+    let time = CivilTime::from_hms_nano(hour, minute, second, nanosecond)
+        .map_err(|e| Error::other(format!("invalid legacy compact time time: {e}")))?;
+    let offset = UtcOffset::from_hms(offset_hour, offset_minute, offset_second)
+        .map_err(|e| Error::other(format!("invalid legacy compact time offset: {e}")))?;
+
+    Ok(PrimitiveDateTime::new(date, time)
+        .assume_offset(offset)
+        .to_offset(UtcOffset::UTC))
+}
+
+fn read_msgp_bin<R: Read>(rd: &mut R) -> Result<Vec<u8>> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    match marker {
+        rmp::Marker::Null => Ok(Vec::new()),
+        rmp::Marker::Bin8 => {
+            let len = usize::from(read_u8(rd)?);
+            read_exact_bytes(rd, len)
+        }
+        rmp::Marker::Bin16 => {
+            let len = usize::from(read_u16_raw(rd)?);
+            read_exact_bytes(rd, len)
+        }
+        rmp::Marker::Bin32 => {
+            let len = read_u32_raw(rd)? as usize;
+            read_exact_bytes(rd, len)
+        }
+        rmp::Marker::FixArray(len) => read_msgp_legacy_byte_array(rd, u32::from(len)),
+        rmp::Marker::Array16 => {
+            let len = read_u16_raw(rd)?;
+            read_msgp_legacy_byte_array(rd, u32::from(len))
+        }
+        rmp::Marker::Array32 => {
+            let len = read_u32_raw(rd)?;
+            read_msgp_legacy_byte_array(rd, len)
+        }
+        _ => Err(Error::other(format!("expected bin or byte array, got marker: {marker:?}"))),
+    }
+}
+
+fn read_exact_bytes<R: Read>(rd: &mut R, len: usize) -> Result<Vec<u8>> {
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    Ok(buf)
+}
+
+fn read_msgp_legacy_byte_array<R: Read>(rd: &mut R, len: u32) -> Result<Vec<u8>> {
+    let mut buf = Vec::with_capacity(len as usize);
+    for _ in 0..len {
+        let value: i64 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+        let byte = u8::try_from(value).map_err(|_| Error::other(format!("byte value out of range: {value}")))?;
+        buf.push(byte);
+    }
+    Ok(buf)
+}
+
+fn write_bin_field<W: Write>(wr: &mut W, key: &str, val: &[u8]) -> Result<()> {
+    rmp::encode::write_str(wr, key)?;
+    rmp::encode::write_bin(wr, val)?;
+    Ok(())
+}
+
 pub const BUCKET_METADATA_FILE: &str = ".metadata.bin";
 pub const BUCKET_METADATA_FORMAT: u16 = 1;
 pub const BUCKET_METADATA_VERSION: u16 = 1;
@@ -54,8 +241,7 @@ pub const BUCKET_CORS_CONFIG: &str = "cors.xml";
 pub const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = "public-access-block.xml";
 pub const BUCKET_ACL_CONFIG: &str = "bucket-acl.json";
 
-#[derive(Debug, Deserialize, Serialize, Clone)]
-#[serde(rename_all = "PascalCase", default)]
+#[derive(Debug, Clone)]
 pub struct BucketMetadata {
     pub name: String,
     pub created: OffsetDateTime,
@@ -90,36 +276,21 @@ pub struct BucketMetadata {
     pub public_access_block_config_updated_at: OffsetDateTime,
     pub bucket_acl_config_updated_at: OffsetDateTime,
 
-    #[serde(skip)]
     pub new_field_updated_at: OffsetDateTime,
 
-    #[serde(skip)]
     pub policy_config: Option<BucketPolicy>,
-    #[serde(skip)]
     pub notification_config: Option<NotificationConfiguration>,
-    #[serde(skip)]
     pub lifecycle_config: Option<BucketLifecycleConfiguration>,
-    #[serde(skip)]
     pub object_lock_config: Option<ObjectLockConfiguration>,
-    #[serde(skip)]
     pub versioning_config: Option<VersioningConfiguration>,
-    #[serde(skip)]
     pub sse_config: Option<ServerSideEncryptionConfiguration>,
-    #[serde(skip)]
     pub tagging_config: Option<Tagging>,
-    #[serde(skip)]
     pub quota_config: Option<BucketQuota>,
-    #[serde(skip)]
     pub replication_config: Option<ReplicationConfiguration>,
-    #[serde(skip)]
     pub bucket_target_config: Option<BucketTargets>,
-    #[serde(skip)]
     pub bucket_target_config_meta: Option<HashMap<String, String>>,
-    #[serde(skip)]
     pub cors_config: Option<CORSConfiguration>,
-    #[serde(skip)]
     pub public_access_block_config: Option<PublicAccessBlockConfiguration>,
-    #[serde(skip)]
     pub bucket_acl_config: Option<String>,
 }
 
@@ -198,17 +369,141 @@ impl BucketMetadata {
         self.lock_enabled || (self.versioning_config.as_ref().is_some_and(|v| v.enabled()))
     }
 
+    /// Decode from msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
+    pub fn decode_from<R: Read>(&mut self, rd: &mut R) -> Result<()> {
+        let mut fields = rmp::decode::read_map_len(rd)?;
+        *self = Self::default();
+
+        while fields > 0 {
+            fields -= 1;
+
+            let key_len = rmp::decode::read_str_len(rd)?;
+            let mut key_buf = vec![0u8; key_len as usize];
+            rd.read_exact(&mut key_buf)?;
+            let key = String::from_utf8(key_buf)?;
+
+            match key.as_str() {
+                "Name" => self.name = read_msgp_str(rd)?,
+                "Created" => self.created = read_msgp_time_value(rd)?,
+                "LockEnabled" => self.lock_enabled = read_msgp_bool(rd)?,
+                "PolicyConfigJSON" | "PolicyConfigJson" => self.policy_config_json = read_msgp_bin(rd)?,
+                "NotificationConfigXML" | "NotificationConfigXml" => self.notification_config_xml = read_msgp_bin(rd)?,
+                "LifecycleConfigXML" | "LifecycleConfigXml" => self.lifecycle_config_xml = read_msgp_bin(rd)?,
+                "ObjectLockConfigXML" | "ObjectLockConfigXml" => self.object_lock_config_xml = read_msgp_bin(rd)?,
+                "VersioningConfigXML" | "VersioningConfigXml" => self.versioning_config_xml = read_msgp_bin(rd)?,
+                "EncryptionConfigXML" | "EncryptionConfigXml" => self.encryption_config_xml = read_msgp_bin(rd)?,
+                "TaggingConfigXML" | "TaggingConfigXml" => self.tagging_config_xml = read_msgp_bin(rd)?,
+                "QuotaConfigJSON" | "QuotaConfigJson" => self.quota_config_json = read_msgp_bin(rd)?,
+                "ReplicationConfigXML" | "ReplicationConfigXml" => self.replication_config_xml = read_msgp_bin(rd)?,
+                "BucketTargetsConfigJSON" | "BucketTargetsConfigJson" => self.bucket_targets_config_json = read_msgp_bin(rd)?,
+                "BucketTargetsConfigMetaJSON" | "BucketTargetsConfigMetaJson" => {
+                    self.bucket_targets_config_meta_json = read_msgp_bin(rd)?
+                }
+                "PolicyConfigUpdatedAt" => self.policy_config_updated_at = read_msgp_time_value(rd)?,
+                "ObjectLockConfigUpdatedAt" => self.object_lock_config_updated_at = read_msgp_time_value(rd)?,
+                "EncryptionConfigUpdatedAt" => self.encryption_config_updated_at = read_msgp_time_value(rd)?,
+                "TaggingConfigUpdatedAt" => self.tagging_config_updated_at = read_msgp_time_value(rd)?,
+                "QuotaConfigUpdatedAt" => self.quota_config_updated_at = read_msgp_time_value(rd)?,
+                "ReplicationConfigUpdatedAt" => self.replication_config_updated_at = read_msgp_time_value(rd)?,
+                "VersioningConfigUpdatedAt" => self.versioning_config_updated_at = read_msgp_time_value(rd)?,
+                "LifecycleConfigUpdatedAt" => self.lifecycle_config_updated_at = read_msgp_time_value(rd)?,
+                "NotificationConfigUpdatedAt" => self.notification_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketTargetsConfigUpdatedAt" => self.bucket_targets_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketTargetsConfigMetaUpdatedAt" => self.bucket_targets_config_meta_updated_at = read_msgp_time_value(rd)?,
+                "CorsConfigXML" | "CorsConfigXml" => self.cors_config_xml = read_msgp_bin(rd)?,
+                "PublicAccessBlockConfigXML" | "PublicAccessBlockConfigXml" => {
+                    self.public_access_block_config_xml = read_msgp_bin(rd)?
+                }
+                "BucketAclConfigJSON" | "BucketAclConfigJson" => self.bucket_acl_config_json = read_msgp_bin(rd)?,
+                "CorsConfigUpdatedAt" => self.cors_config_updated_at = read_msgp_time_value(rd)?,
+                "PublicAccessBlockConfigUpdatedAt" => self.public_access_block_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketAclConfigUpdatedAt" => self.bucket_acl_config_updated_at = read_msgp_time_value(rd)?,
+                other => {
+                    tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
+                    skip_msgp_value(rd)?;
+                }
+            }
+        }
+
+        Ok(())
+    }
+
+    /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
+    pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
+        // Map size: MinIO fields (25) + RustFS extensions (6)
+        let map_len: u32 = 31;
+        rmp::encode::write_map_len(wr, map_len)?;
+
+        // MinIO field order (same as Go struct)
+        rmp::encode::write_str(wr, "Name")?;
+        rmp::encode::write_str(wr, &self.name)?;
+
+        rmp::encode::write_str(wr, "Created")?;
+        write_msgp_time(wr, self.created)?;
+
+        rmp::encode::write_str(wr, "LockEnabled")?;
+        rmp::encode::write_bool(wr, self.lock_enabled)?;
+
+        write_bin_field(wr, "PolicyConfigJSON", &self.policy_config_json)?;
+        write_bin_field(wr, "NotificationConfigXML", &self.notification_config_xml)?;
+        write_bin_field(wr, "LifecycleConfigXML", &self.lifecycle_config_xml)?;
+        write_bin_field(wr, "ObjectLockConfigXML", &self.object_lock_config_xml)?;
+        write_bin_field(wr, "VersioningConfigXML", &self.versioning_config_xml)?;
+        write_bin_field(wr, "EncryptionConfigXML", &self.encryption_config_xml)?;
+        write_bin_field(wr, "TaggingConfigXML", &self.tagging_config_xml)?;
+        write_bin_field(wr, "QuotaConfigJSON", &self.quota_config_json)?;
+        write_bin_field(wr, "ReplicationConfigXML", &self.replication_config_xml)?;
+        write_bin_field(wr, "BucketTargetsConfigJSON", &self.bucket_targets_config_json)?;
+        write_bin_field(wr, "BucketTargetsConfigMetaJSON", &self.bucket_targets_config_meta_json)?;
+
+        rmp::encode::write_str(wr, "PolicyConfigUpdatedAt")?;
+        write_msgp_time(wr, self.policy_config_updated_at)?;
+        rmp::encode::write_str(wr, "ObjectLockConfigUpdatedAt")?;
+        write_msgp_time(wr, self.object_lock_config_updated_at)?;
+        rmp::encode::write_str(wr, "EncryptionConfigUpdatedAt")?;
+        write_msgp_time(wr, self.encryption_config_updated_at)?;
+        rmp::encode::write_str(wr, "TaggingConfigUpdatedAt")?;
+        write_msgp_time(wr, self.tagging_config_updated_at)?;
+        rmp::encode::write_str(wr, "QuotaConfigUpdatedAt")?;
+        write_msgp_time(wr, self.quota_config_updated_at)?;
+        rmp::encode::write_str(wr, "ReplicationConfigUpdatedAt")?;
+        write_msgp_time(wr, self.replication_config_updated_at)?;
+        rmp::encode::write_str(wr, "VersioningConfigUpdatedAt")?;
+        write_msgp_time(wr, self.versioning_config_updated_at)?;
+        rmp::encode::write_str(wr, "LifecycleConfigUpdatedAt")?;
+        write_msgp_time(wr, self.lifecycle_config_updated_at)?;
+        rmp::encode::write_str(wr, "NotificationConfigUpdatedAt")?;
+        write_msgp_time(wr, self.notification_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketTargetsConfigUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_targets_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketTargetsConfigMetaUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_targets_config_meta_updated_at)?;
+
+        // RustFS extensions
+        write_bin_field(wr, "CorsConfigXML", &self.cors_config_xml)?;
+        write_bin_field(wr, "PublicAccessBlockConfigXML", &self.public_access_block_config_xml)?;
+        write_bin_field(wr, "BucketAclConfigJSON", &self.bucket_acl_config_json)?;
+        rmp::encode::write_str(wr, "CorsConfigUpdatedAt")?;
+        write_msgp_time(wr, self.cors_config_updated_at)?;
+        rmp::encode::write_str(wr, "PublicAccessBlockConfigUpdatedAt")?;
+        write_msgp_time(wr, self.public_access_block_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketAclConfigUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_acl_config_updated_at)?;
+
+        Ok(())
+    }
+
     pub fn marshal_msg(&self) -> Result<Vec<u8>> {
         let mut buf = Vec::new();
-
-        self.serialize(&mut rmpSerializer::new(&mut buf).with_struct_map())?;
-
+        self.encode_to(&mut buf)?;
         Ok(buf)
     }
 
     pub fn unmarshal(buf: &[u8]) -> Result<Self> {
-        let t: BucketMetadata = rmp_serde::from_slice(buf)?;
-        Ok(t)
+        let mut bm = Self::default();
+        let mut cur = std::io::Cursor::new(buf);
+        bm.decode_from(&mut cur)?;
+        Ok(bm)
     }
 
     pub fn check_header(buf: &[u8]) -> Result<()> {
@@ -372,53 +667,90 @@ impl BucketMetadata {
         Ok(())
     }
 
-    fn parse_all_configs(&mut self, _api: Arc<ECStore>) -> Result<()> {
+    fn parse_policy_config(&mut self) -> Result<()> {
         if !self.policy_config_json.is_empty() {
             self.policy_config = Some(serde_json::from_slice(&self.policy_config_json)?);
-        }
-        if !self.notification_config_xml.is_empty() {
-            self.notification_config = Some(deserialize::<NotificationConfiguration>(&self.notification_config_xml)?);
-        }
-        if !self.lifecycle_config_xml.is_empty() {
-            self.lifecycle_config = Some(deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml)?);
-        }
-
-        if !self.object_lock_config_xml.is_empty() {
-            self.object_lock_config = Some(deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml)?);
-        }
-        if !self.versioning_config_xml.is_empty() {
-            self.versioning_config = Some(deserialize::<VersioningConfiguration>(&self.versioning_config_xml)?);
-        }
-        if !self.encryption_config_xml.is_empty() {
-            self.sse_config = Some(deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml)?);
-        }
-        if !self.tagging_config_xml.is_empty() {
-            self.tagging_config = Some(deserialize::<Tagging>(&self.tagging_config_xml)?);
-        }
-        if !self.quota_config_json.is_empty() {
-            self.quota_config = Some(serde_json::from_slice(&self.quota_config_json)?);
-        }
-        if !self.replication_config_xml.is_empty() {
-            self.replication_config = Some(deserialize::<ReplicationConfiguration>(&self.replication_config_xml)?);
-        }
-        //let temp = self.bucket_targets_config_json.clone();
-        if !self.bucket_targets_config_json.is_empty() {
-            let bucket_targets: BucketTargets = serde_json::from_slice(&self.bucket_targets_config_json)?;
-            self.bucket_target_config = Some(bucket_targets);
         } else {
-            self.bucket_target_config = Some(BucketTargets::default())
+            self.policy_config = None;
         }
-        if !self.cors_config_xml.is_empty() {
-            self.cors_config = Some(deserialize::<CORSConfiguration>(&self.cors_config_xml)?);
+        Ok(())
+    }
+
+    fn parse_all_configs(&mut self, _api: Arc<ECStore>) -> Result<()> {
+        if let Err(e) = self.parse_policy_config() {
+            tracing::warn!(bucket = %self.name, config = "policy", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.public_access_block_config_xml.is_empty() {
-            self.public_access_block_config =
-                Some(deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)?);
+        if !self.notification_config_xml.is_empty()
+            && let Err(e) = deserialize::<NotificationConfiguration>(&self.notification_config_xml)
+                .map(|c| self.notification_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "notification", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.bucket_acl_config_json.is_empty() {
-            let acl = String::from_utf8(self.bucket_acl_config_json.clone())
-                .map_err(|e| Error::other(format!("invalid UTF-8 in bucket ACL: {}", e)))?;
-            self.bucket_acl_config = Some(acl);
+        if !self.lifecycle_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "lifecycle", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.object_lock_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml).map(|c| self.object_lock_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "object_lock", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.versioning_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<VersioningConfiguration>(&self.versioning_config_xml).map(|c| self.versioning_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "versioning", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.encryption_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml).map(|c| self.sse_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "encryption", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.tagging_config_xml.is_empty()
+            && let Err(e) = deserialize::<Tagging>(&self.tagging_config_xml).map(|c| self.tagging_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "tagging", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.quota_config_json.is_empty()
+            && let Err(e) = serde_json::from_slice(&self.quota_config_json).map(|c| self.quota_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "quota", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.replication_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "replication", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.bucket_targets_config_json.is_empty() {
+            if let Err(e) = serde_json::from_slice::<BucketTargets>(&self.bucket_targets_config_json)
+                .map(|t| self.bucket_target_config = Some(t))
+            {
+                tracing::warn!(bucket = %self.name, config = "bucket_targets", error = %e, "parse_all_configs: failed to parse");
+                self.bucket_target_config = Some(BucketTargets::default());
+            }
+        } else {
+            self.bucket_target_config = Some(BucketTargets::default());
+        }
+        if !self.cors_config_xml.is_empty()
+            && let Err(e) = deserialize::<CORSConfiguration>(&self.cors_config_xml).map(|c| self.cors_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "cors", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.public_access_block_config_xml.is_empty()
+            && let Err(e) = deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)
+                .map(|c| self.public_access_block_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "public_access_block", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.bucket_acl_config_json.is_empty()
+            && let Err(e) = String::from_utf8(self.bucket_acl_config_json.clone()).map(|acl| self.bucket_acl_config = Some(acl))
+        {
+            tracing::warn!(bucket = %self.name, config = "bucket_acl", error = %e, "parse_all_configs: failed to parse");
         }
 
         Ok(())
@@ -471,7 +803,6 @@ async fn read_bucket_metadata(api: Arc<ECStore>, bucket: &str) -> Result<BucketM
 
     Ok(bm)
 }
-
 fn _write_time<S>(t: &OffsetDateTime, s: S) -> std::result::Result<S::Ok, S::Error>
 where
     S: Serializer,
@@ -666,4 +997,21 @@ mod test {
         println!("   - Lifecycle config size: {} bytes", deserialized_bm.lifecycle_config_xml.len());
         println!("   - Serialized buffer size: {} bytes", buf.len());
     }
+
+    /// After policy deletion (policy_config_json cleared), parse_policy_config sets policy_config to None.
+    #[test]
+    fn test_parse_policy_config_clears_cache_when_json_empty() {
+        let policy_json = r#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::b/*"}]}"#;
+        let mut bm = BucketMetadata::new("b");
+        bm.policy_config_json = policy_json.as_bytes().to_vec();
+        bm.parse_policy_config().unwrap();
+        assert!(bm.policy_config.is_some(), "policy_config should be set when JSON non-empty");
+
+        bm.policy_config_json.clear();
+        bm.parse_policy_config().unwrap();
+        assert!(
+            bm.policy_config.is_none(),
+            "policy_config should be None after JSON cleared (e.g. policy deleted)"
+        );
+    }
 }

crates/ecstore/src/bucket/metadata_sys.rs

@@ -360,12 +360,14 @@ impl BucketMetadataSys {
             };
 
             if !meta.lifecycle_config_xml.is_empty() {
-                let cfg = deserialize::<BucketLifecycleConfiguration>(&meta.lifecycle_config_xml)?;
-                // TODO: FIXME:
-                // for _v in cfg.rules.iter() {
-                //     break;
-                // }
-                if let Some(_v) = cfg.rules.first() {}
+                if let Ok(cfg) = deserialize::<BucketLifecycleConfiguration>(&meta.lifecycle_config_xml) {
+                    if let Some(_v) = cfg.rules.first() {}
+                } else {
+                    tracing::warn!(
+                        bucket = %bucket,
+                        "delete: failed to parse lifecycle config XML"
+                    );
+                }
             }
 
             // TODO: other lifecycle handle

crates/ecstore/src/bucket/metadata_test.rs

@@ -0,0 +1,398 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::metadata::BucketMetadata;
+use time::OffsetDateTime;
+
+/// Full BucketMetadata hex (all fields populated).
+const TEST_BUCKET_METADATA_HEX: &str = "de0019a44e616d65b27275737466732d636f6d7061742d74657374a743726561746564c70c050000000065920080075bcd15ab4c6f636b456e61626c6564c3b0506f6c696379436f6e6669674a534f4ec4907b2256657273696f6e223a22323031322d31302d3137222c2253746174656d656e74223a5b7b22456666656374223a22416c6c6f77222c225072696e636970616c223a222a222c22416374696f6e223a2273333a4765744f626a656374222c225265736f75726365223a2261726e3a6177733a73333a3a3a7275737466732d636f6d7061742d746573742f2a227d5d7db54e6f74696669636174696f6e436f6e666967584d4cc4963c4e6f74696669636174696f6e436f6e66696775726174696f6e3e3c436c6f75645761746368436f6e66696775726174696f6e3e3c49643e6e313c2f49643e3c4576656e743e73333a4f626a656374437265617465643a2a3c2f4576656e743e3c2f436c6f75645761746368436f6e66696775726174696f6e3e3c2f4e6f74696669636174696f6e436f6e66696775726174696f6e3eb24c6966656379636c65436f6e666967584d4cc48c3c4c6966656379636c65436f6e66696775726174696f6e3e3c52756c653e3c49443e72756c65313c2f49443e3c5374617475733e456e61626c65643c2f5374617475733e3c45787069726174696f6e3e3c446179733e33303c2f446179733e3c2f45787069726174696f6e3e3c2f52756c653e3c2f4c6966656379636c65436f6e66696775726174696f6e3eb34f626a6563744c6f636b436f6e666967584d4cc4b83c4f626a6563744c6f636b436f6e66696775726174696f6e3e3c4f626a6563744c6f636b456e61626c65643e456e61626c65643c2f4f626a6563744c6f636b456e61626c65643e3c52756c653e3c44656661756c74526574656e74696f6e3e3c4d6f64653e474f5645524e414e43453c2f4d6f64653e3c446179733e373c2f446179733e3c2f44656661756c74526574656e74696f6e3e3c2f52756c653e3c2f4f626a6563744c6f636b436f6e66696775726174696f6e3eb356657273696f6e696e67436f6e666967584d4cc44b3c56657273696f6e696e67436f6e66696775726174696f6e3e3c5374617475733e456e61626c65643c2f5374617475733e3c2f56657273696f6e696e67436f6e66696775726174696f6e3eb3456e6372797074696f6e436f6e666967584d4cc4c03c53657276657253696465456e6372797074696f6e436f6e66696775726174696f6e3e3c52756c653e3c4170706c7953657276657253696465456e6372797074696f6e427944656661756c743e3c535345416c676f726974686d3e4145533235363c2f535345416c676f726974686d3e3c2f4170706c7953657276657253696465456e6372797074696f6e427944656661756c743e3c2f52756c653e3c2f53657276657253696465456e6372797074696f6e436f6e66696775726174696f6e3eb054616767696e67436f6e666967584d4cc4503c54616767696e673e3c5461675365743e3c5461673e3c4b65793e456e763c2f4b65793e3c56616c75653e546573743c2f56616c75653e3c2f5461673e3c2f5461675365743e3c2f54616767696e673eaf51756f7461436f6e6669674a534f4ec4707b2271756f7461223a313037333734313832342c2271756f74615f74797065223a2248617264222c22637265617465645f6174223a22323032342d30312d30315430303a30303a30305a222c22757064617465645f6174223a22323032342d30312d30315430303a30303a30305a227db45265706c69636174696f6e436f6e666967584d4cc4e73c5265706c69636174696f6e436f6e66696775726174696f6e3e3c526f6c653e61726e3a6177733a69616d3a3a3132333435363738393031323a726f6c652f7265706c3c2f526f6c653e3c52756c653e3c49443e72313c2f49443e3c5374617475733e456e61626c65643c2f5374617475733e3c5072656669783e646f632f3c2f5072656669783e3c44657374696e6174696f6e3e3c4275636b65743e61726e3a6177733a73333a3a3a646573743c2f4275636b65743e3c2f44657374696e6174696f6e3e3c2f52756c653e3c2f5265706c69636174696f6e436f6e66696775726174696f6e3eb74275636b657454617267657473436f6e6669674a534f4ec4535b7b22656e64706f696e74223a22687474703a2f2f7461726765742e6578616d706c652e636f6d222c227461726765744275636b6574223a227462222c22726567696f6e223a2275732d656173742d31227d5dbb4275636b657454617267657473436f6e6669674d6574614a534f4ec42d7b227265706c69636174696f6e4964223a227265706c2d31222c2273796e634d6f6465223a226173796e63227db5506f6c696379436f6e666967557064617465644174c70c050000000065a5022000000000b94f626a6563744c6f636b436f6e666967557064617465644174c70c050000000065a5022000000000b9456e6372797074696f6e436f6e666967557064617465644174c70c050000000065a5022000000000b654616767696e67436f6e666967557064617465644174c70c050000000065a5022000000000b451756f7461436f6e666967557064617465644174c70c050000000065a5022000000000ba5265706c69636174696f6e436f6e666967557064617465644174c70c050000000065a5022000000000b956657273696f6e696e67436f6e666967557064617465644174c70c050000000065a5022000000000b84c6966656379636c65436f6e666967557064617465644174c70c050000000065a5022000000000bb4e6f74696669636174696f6e436f6e666967557064617465644174c70c050000000065a5022000000000bc4275636b657454617267657473436f6e666967557064617465644174c70c050000000065a5022000000000d9204275636b657454617267657473436f6e6669674d657461557064617465644174c70c050000000065a5022000000000";
+
+#[tokio::test]
+async fn marshal_msg() {
+    let bm = BucketMetadata::new("dada");
+
+    let buf = bm.marshal_msg().unwrap();
+
+    let new = BucketMetadata::unmarshal(&buf).unwrap();
+
+    assert_eq!(bm.name, new.name);
+}
+
+/// Verifies that serialized time uses msgp ext type 5.
+#[tokio::test]
+async fn marshal_msg_uses_time_format() {
+    let mut bm = BucketMetadata::new("test-bucket");
+    bm.created = OffsetDateTime::from_unix_timestamp(1704067200).unwrap(); // 2024-01-01 00:00:00 UTC
+
+    let buf = bm.marshal_msg().unwrap();
+
+    // msgp uses ext8 (0xc7), len 12, type 5 for time
+    assert!(
+        buf.windows(3).any(|w| w == [0xc7, 0x0c, 0x05]),
+        "serialized data should contain msgp time ext (0xc7 0x0c 0x05)"
+    );
+}
+
+#[tokio::test]
+async fn unmarshal_test_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let mut bytes = vec![0u8; TEST_BUCKET_METADATA_HEX.len() / 2];
+    hex_decode(TEST_BUCKET_METADATA_HEX.as_bytes(), &mut bytes).expect("valid hex");
+    let bm = BucketMetadata::unmarshal(&bytes).expect("RustFS must unmarshal MinIO format");
+
+    assert_eq!(bm.name, "rustfs-compat-test");
+    assert_eq!(bm.created.unix_timestamp(), 1704067200);
+    assert_eq!(bm.created.nanosecond(), 123456789);
+    assert!(bm.lock_enabled);
+
+    assert!(!bm.policy_config_json.is_empty());
+    assert!(bm.policy_config_json.starts_with(b"{\"Version\""));
+    assert!(!bm.notification_config_xml.is_empty());
+    assert!(bm.notification_config_xml.starts_with(b"<Notification"));
+    assert!(!bm.lifecycle_config_xml.is_empty());
+    assert!(bm.lifecycle_config_xml.starts_with(b"<Lifecycle"));
+    assert!(!bm.object_lock_config_xml.is_empty());
+    assert!(bm.object_lock_config_xml.starts_with(b"<ObjectLock"));
+    assert!(!bm.versioning_config_xml.is_empty());
+    assert!(bm.versioning_config_xml.starts_with(b"<Versioning"));
+    assert!(!bm.encryption_config_xml.is_empty());
+    assert!(bm.encryption_config_xml.starts_with(b"<ServerSide"));
+    assert!(!bm.tagging_config_xml.is_empty());
+    assert!(bm.tagging_config_xml.starts_with(b"<Tagging"));
+    assert!(!bm.quota_config_json.is_empty());
+    assert!(bm.quota_config_json.starts_with(b"{\"quota\""));
+    assert!(!bm.replication_config_xml.is_empty());
+    assert!(bm.replication_config_xml.starts_with(b"<Replication"));
+    assert!(!bm.bucket_targets_config_json.is_empty());
+    assert!(bm.bucket_targets_config_json.starts_with(b"[{"));
+    assert!(!bm.bucket_targets_config_meta_json.is_empty());
+    assert!(bm.bucket_targets_config_meta_json.starts_with(b"{\"replication"));
+
+    let updated_sec = 1705312800; // 2024-01-15 12:00:00 UTC
+    assert_eq!(bm.policy_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.object_lock_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.encryption_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.tagging_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.quota_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.replication_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.versioning_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.lifecycle_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.notification_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.bucket_targets_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.bucket_targets_config_meta_updated_at.unix_timestamp(), updated_sec);
+
+    assert!(bm.cors_config_xml.is_empty());
+    assert!(bm.public_access_block_config_xml.is_empty());
+    assert!(bm.bucket_acl_config_json.is_empty());
+}
+
+#[test]
+fn unmarshal_legacy_compact_time_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let legacy_hex = concat!(
+        "83",
+        "a44e616d65",
+        "a474657374",
+        "a743726561746564",
+        "99cd07e9cd01100c1021ce2026b1fa000000",
+        "ab4c6f636b456e61626c6564",
+        "c2"
+    );
+
+    let mut bytes = vec![0u8; legacy_hex.len() / 2];
+    hex_decode(legacy_hex.as_bytes(), &mut bytes).expect("valid hex");
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("legacy compact time should decode");
+
+    assert_eq!(bm.name, "test");
+    assert_eq!(bm.created.unix_timestamp(), 1759148193);
+    assert_eq!(bm.created.nanosecond(), 539406842);
+    assert!(!bm.lock_enabled);
+}
+
+#[test]
+fn unmarshal_bin_wrapped_ext_time_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let wrapped_hex = concat!(
+        "83",
+        "a44e616d65",
+        "a464616461",
+        "a743726561746564",
+        "c40fc70c05fffffff1886e090000000000",
+        "ab4c6f636b456e61626c6564",
+        "c2"
+    );
+
+    let mut bytes = vec![0u8; wrapped_hex.len() / 2];
+    hex_decode(wrapped_hex.as_bytes(), &mut bytes).expect("valid hex");
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("bin-wrapped ext time should decode");
+
+    assert_eq!(bm.name, "dada");
+    assert_eq!(bm.created.unix_timestamp(), -62135596800);
+    assert!(!bm.lock_enabled);
+}
+
+#[test]
+fn unmarshal_legacy_rmp_serde_field_aliases_and_byte_arrays() {
+    use faster_hex::hex_decode;
+
+    let legacy_hex = concat!(
+        "85",
+        "a44e616d65",
+        "a474657374",
+        "a743726561746564",
+        "99cd07e9cd01100c1021ce2026b1fa000000",
+        "ab4c6f636b456e61626c6564",
+        "c2",
+        "b0506f6c696379436f6e6669674a736f6e",
+        "93010203",
+        "bb4275636b657454617267657473436f6e6669674d6574614a736f6e",
+        "920405"
+    );
+
+    let mut bytes = vec![0u8; legacy_hex.len() / 2];
+    hex_decode(legacy_hex.as_bytes(), &mut bytes).expect("valid hex");
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("legacy field aliases and byte arrays should decode");
+
+    assert_eq!(bm.name, "test");
+    assert_eq!(bm.created.unix_timestamp(), 1759148193);
+    assert_eq!(bm.created.nanosecond(), 539406842);
+    assert_eq!(bm.policy_config_json, vec![1, 2, 3]);
+    assert_eq!(bm.bucket_targets_config_meta_json, vec![4, 5]);
+    assert!(!bm.lock_enabled);
+}
+
+#[test]
+fn unmarshal_legacy_bin16_and_array16_bucket_metadata() {
+    let policy = vec![b'x'; 257];
+    let targets_meta: Vec<u8> = (0u8..=16).collect();
+
+    let mut bytes = Vec::new();
+    rmp::encode::write_map_len(&mut bytes, 5).unwrap();
+    rmp::encode::write_str(&mut bytes, "Name").unwrap();
+    rmp::encode::write_str(&mut bytes, "test-bucket").unwrap();
+    rmp::encode::write_str(&mut bytes, "Created").unwrap();
+    bytes.extend_from_slice(&[
+        0xc7, 0x0c, 0x05, 0x00, 0x00, 0x00, 0x00, 0x65, 0x92, 0x00, 0x80, 0x00, 0x00, 0x00, 0x00,
+    ]);
+    rmp::encode::write_str(&mut bytes, "LockEnabled").unwrap();
+    bytes.push(0x01);
+    rmp::encode::write_str(&mut bytes, "PolicyConfigJson").unwrap();
+    rmp::encode::write_bin(&mut bytes, &policy).unwrap();
+    rmp::encode::write_str(&mut bytes, "BucketTargetsConfigMetaJson").unwrap();
+    rmp::encode::write_array_len(&mut bytes, targets_meta.len() as u32).unwrap();
+    for byte in &targets_meta {
+        rmp::encode::write_uint(&mut bytes, u64::from(*byte)).unwrap();
+    }
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("legacy bin16 and array16 should decode");
+
+    assert_eq!(bm.name, "test-bucket");
+    assert_eq!(bm.created.unix_timestamp(), 1704067200);
+    assert!(bm.lock_enabled);
+    assert_eq!(bm.policy_config_json, policy);
+    assert_eq!(bm.bucket_targets_config_meta_json, targets_meta);
+}
+
+#[test]
+fn unmarshal_legacy_numeric_bool_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let legacy_hex = concat!(
+        "83",
+        "a44e616d65",
+        "ab746573742d6275636b6574",
+        "a743726561746564",
+        "c70c05000000006592008000000000",
+        "ab4c6f636b456e61626c6564",
+        "01"
+    );
+
+    let mut bytes = vec![0u8; legacy_hex.len() / 2];
+    hex_decode(legacy_hex.as_bytes(), &mut bytes).expect("valid hex");
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("legacy numeric bool should decode");
+
+    assert_eq!(bm.name, "test-bucket");
+    assert_eq!(bm.created.unix_timestamp(), 1704067200);
+    assert_eq!(bm.created.nanosecond(), 0);
+    assert!(bm.lock_enabled);
+}
+
+#[tokio::test]
+async fn marshal_msg_complete_example() {
+    // Create a complete BucketMetadata with various configurations
+    let mut bm = BucketMetadata::new("test-bucket");
+
+    // Set creation time to current time
+    bm.created = OffsetDateTime::now_utc();
+    bm.lock_enabled = true;
+
+    // Add policy configuration
+    let policy_json = r#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::test-bucket/*"}]}"#;
+    bm.policy_config_json = policy_json.as_bytes().to_vec();
+    bm.policy_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add lifecycle configuration
+    let lifecycle_xml = r#"<LifecycleConfiguration><Rule><ID>rule1</ID><Status>Enabled</Status><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>"#;
+    bm.lifecycle_config_xml = lifecycle_xml.as_bytes().to_vec();
+    bm.lifecycle_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add versioning configuration
+    let versioning_xml = r#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#;
+    bm.versioning_config_xml = versioning_xml.as_bytes().to_vec();
+    bm.versioning_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add encryption configuration
+    let encryption_xml = r#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
+    bm.encryption_config_xml = encryption_xml.as_bytes().to_vec();
+    bm.encryption_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add tagging configuration
+    let tagging_xml = r#"<Tagging><TagSet><Tag><Key>Environment</Key><Value>Test</Value></Tag><Tag><Key>Owner</Key><Value>RustFS</Value></Tag></TagSet></Tagging>"#;
+    bm.tagging_config_xml = tagging_xml.as_bytes().to_vec();
+    bm.tagging_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add quota configuration
+    let quota_json =
+        r#"{"quota":1073741824,"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#; // 1GB quota
+    bm.quota_config_json = quota_json.as_bytes().to_vec();
+    bm.quota_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add object lock configuration
+    let object_lock_xml = r#"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled><Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>7</Days></DefaultRetention></Rule></ObjectLockConfiguration>"#;
+    bm.object_lock_config_xml = object_lock_xml.as_bytes().to_vec();
+    bm.object_lock_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add notification configuration
+    let notification_xml = r#"<NotificationConfiguration><CloudWatchConfiguration><Id>notification1</Id><Event>s3:ObjectCreated:*</Event><CloudWatchConfiguration><LogGroupName>test-log-group</LogGroupName></CloudWatchConfiguration></CloudWatchConfiguration></NotificationConfiguration>"#;
+    bm.notification_config_xml = notification_xml.as_bytes().to_vec();
+    bm.notification_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add replication configuration
+    let replication_xml = r#"<ReplicationConfiguration><Role>arn:aws:iam::123456789012:role/replication-role</Role><Rule><ID>rule1</ID><Status>Enabled</Status><Prefix>documents/</Prefix><Destination><Bucket>arn:aws:s3:::destination-bucket</Bucket></Destination></Rule></ReplicationConfiguration>"#;
+    bm.replication_config_xml = replication_xml.as_bytes().to_vec();
+    bm.replication_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add bucket targets configuration
+    let bucket_targets_json = r#"[{"endpoint":"http://target1.example.com","credentials":{"accessKey":"key1","secretKey":"secret1"},"targetBucket":"target-bucket-1","region":"us-east-1"},{"endpoint":"http://target2.example.com","credentials":{"accessKey":"key2","secretKey":"secret2"},"targetBucket":"target-bucket-2","region":"us-west-2"}]"#;
+    bm.bucket_targets_config_json = bucket_targets_json.as_bytes().to_vec();
+    bm.bucket_targets_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add bucket targets meta configuration
+    let bucket_targets_meta_json = r#"{"replicationId":"repl-123","syncMode":"async","bandwidth":"100MB"}"#;
+    bm.bucket_targets_config_meta_json = bucket_targets_meta_json.as_bytes().to_vec();
+    bm.bucket_targets_config_meta_updated_at = OffsetDateTime::now_utc();
+
+    // Add public access block configuration
+    let public_access_block_xml = r#"<PublicAccessBlockConfiguration><BlockPublicAcls>true</BlockPublicAcls><IgnorePublicAcls>true</IgnorePublicAcls><BlockPublicPolicy>true</BlockPublicPolicy><RestrictPublicBuckets>false</RestrictPublicBuckets></PublicAccessBlockConfiguration>"#;
+    bm.public_access_block_config_xml = public_access_block_xml.as_bytes().to_vec();
+    bm.public_access_block_config_updated_at = OffsetDateTime::now_utc();
+
+    let bucket_acl = r#"{"owner":{"id":"rustfsadmin","display_name":"RustFS Tester"},"grants":[{"grantee":{"grantee_type":"CanonicalUser","id":"rustfsadmin","display_name":"RustFS Tester","uri":null,"email_address":null},"permission":"FULL_CONTROL"}]}"#;
+    bm.bucket_acl_config_json = bucket_acl.as_bytes().to_vec();
+    bm.bucket_acl_config_updated_at = OffsetDateTime::now_utc();
+
+    // Test serialization
+    let buf = bm.marshal_msg().unwrap();
+    assert!(!buf.is_empty(), "Serialized buffer should not be empty");
+
+    // Test deserialization
+    let deserialized_bm = BucketMetadata::unmarshal(&buf).unwrap();
+
+    // Verify all fields are correctly serialized and deserialized
+    assert_eq!(bm.name, deserialized_bm.name);
+    assert_eq!(bm.created.unix_timestamp(), deserialized_bm.created.unix_timestamp());
+    assert_eq!(bm.lock_enabled, deserialized_bm.lock_enabled);
+
+    // Verify configuration data
+    assert_eq!(bm.policy_config_json, deserialized_bm.policy_config_json);
+    assert_eq!(bm.lifecycle_config_xml, deserialized_bm.lifecycle_config_xml);
+    assert_eq!(bm.versioning_config_xml, deserialized_bm.versioning_config_xml);
+    assert_eq!(bm.encryption_config_xml, deserialized_bm.encryption_config_xml);
+    assert_eq!(bm.tagging_config_xml, deserialized_bm.tagging_config_xml);
+    assert_eq!(bm.quota_config_json, deserialized_bm.quota_config_json);
+    assert_eq!(bm.public_access_block_config_xml, deserialized_bm.public_access_block_config_xml);
+    assert_eq!(bm.bucket_acl_config_json, deserialized_bm.bucket_acl_config_json);
+    assert_eq!(bm.object_lock_config_xml, deserialized_bm.object_lock_config_xml);
+    assert_eq!(bm.notification_config_xml, deserialized_bm.notification_config_xml);
+    assert_eq!(bm.replication_config_xml, deserialized_bm.replication_config_xml);
+    assert_eq!(bm.bucket_targets_config_json, deserialized_bm.bucket_targets_config_json);
+    assert_eq!(bm.bucket_targets_config_meta_json, deserialized_bm.bucket_targets_config_meta_json);
+
+    // Verify timestamps (comparing unix timestamps to avoid precision issues)
+    assert_eq!(
+        bm.policy_config_updated_at.unix_timestamp(),
+        deserialized_bm.policy_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.lifecycle_config_updated_at.unix_timestamp(),
+        deserialized_bm.lifecycle_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.versioning_config_updated_at.unix_timestamp(),
+        deserialized_bm.versioning_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.encryption_config_updated_at.unix_timestamp(),
+        deserialized_bm.encryption_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.tagging_config_updated_at.unix_timestamp(),
+        deserialized_bm.tagging_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.quota_config_updated_at.unix_timestamp(),
+        deserialized_bm.quota_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.object_lock_config_updated_at.unix_timestamp(),
+        deserialized_bm.object_lock_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.notification_config_updated_at.unix_timestamp(),
+        deserialized_bm.notification_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.replication_config_updated_at.unix_timestamp(),
+        deserialized_bm.replication_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.bucket_targets_config_updated_at.unix_timestamp(),
+        deserialized_bm.bucket_targets_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.bucket_targets_config_meta_updated_at.unix_timestamp(),
+        deserialized_bm.bucket_targets_config_meta_updated_at.unix_timestamp()
+    );
+
+    // Test that the serialized data contains expected content
+    let buf_str = String::from_utf8_lossy(&buf);
+    assert!(buf_str.contains("test-bucket"), "Serialized data should contain bucket name");
+
+    // Verify the buffer size is reasonable (should be larger due to all the config data)
+    assert!(buf.len() > 1000, "Buffer should be substantial in size due to all configurations");
+
+    println!("✅ Complete BucketMetadata serialization test passed");
+    println!("   - Bucket name: {}", deserialized_bm.name);
+    println!("   - Lock enabled: {}", deserialized_bm.lock_enabled);
+    println!("   - Policy config size: {} bytes", deserialized_bm.policy_config_json.len());
+    println!("   - Lifecycle config size: {} bytes", deserialized_bm.lifecycle_config_xml.len());
+    println!("   - Serialized buffer size: {} bytes", buf.len());
+}

crates/ecstore/src/bucket/migration.rs

@@ -0,0 +1,420 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Migration of bucket metadata and IAM config from legacy format to RustFS format.
+
+use crate::bucket::metadata::BUCKET_METADATA_FILE;
+use crate::bucket::replication::{decode_resync_file, encode_resync_file};
+use crate::disk::{BUCKET_META_PREFIX, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
+use crate::store_api::{BucketOptions, ObjectOptions, PutObjReader, StorageAPI};
+use http::HeaderMap;
+use rustfs_policy::auth::UserIdentity;
+use rustfs_policy::policy::PolicyDoc;
+use rustfs_utils::path::SLASH_SEPARATOR;
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use time::OffsetDateTime;
+use tracing::{debug, info, warn};
+
+/// IAM config prefix under meta bucket (e.g. config/iam/).
+const IAM_CONFIG_PREFIX: &str = "config/iam";
+const IAM_FORMAT_FILE_PATH: &str = "config/iam/format.json";
+const IAM_USERS_PREFIX: &str = "config/iam/users/";
+const IAM_SERVICE_ACCOUNTS_PREFIX: &str = "config/iam/service-accounts/";
+const IAM_STS_PREFIX: &str = "config/iam/sts/";
+const IAM_GROUPS_PREFIX: &str = "config/iam/groups/";
+const IAM_POLICIES_PREFIX: &str = "config/iam/policies/";
+const IAM_POLICY_DB_PREFIX: &str = "config/iam/policydb/";
+const REPLICATION_META_DIR: &str = ".replication";
+const RESYNC_META_FILE: &str = "resync.bin";
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatIamFormat {
+    #[serde(default)]
+    version: i64,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatGroupInfo {
+    #[serde(default)]
+    version: i64,
+    #[serde(default = "default_group_status")]
+    status: String,
+    #[serde(default)]
+    members: Vec<String>,
+    #[serde(
+        rename = "updatedAt",
+        alias = "update_at",
+        default,
+        with = "rustfs_policy::serde_datetime::option"
+    )]
+    update_at: Option<OffsetDateTime>,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatMappedPolicy {
+    #[serde(default)]
+    version: i64,
+    #[serde(rename = "policy", alias = "policies", default)]
+    policy: String,
+    #[serde(
+        rename = "updatedAt",
+        alias = "update_at",
+        default,
+        with = "rustfs_policy::serde_datetime::option"
+    )]
+    update_at: Option<OffsetDateTime>,
+}
+
+fn default_group_status() -> String {
+    "enabled".to_string()
+}
+
+fn normalize_iam_config_blob(path: &str, data: &[u8]) -> std::result::Result<Option<Vec<u8>>, String> {
+    if path == IAM_FORMAT_FILE_PATH {
+        let mut format: CompatIamFormat =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM format failed: {err}"))?;
+        if format.version <= 0 {
+            format.version = 1;
+        }
+        return serde_json::to_vec(&format)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM format failed: {err}"));
+    }
+
+    if is_identity_path(path) {
+        let mut identity: UserIdentity =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM identity failed: {err}"))?;
+        if identity.update_at.is_none() {
+            identity.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&identity)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM identity failed: {err}"));
+    }
+
+    if is_group_path(path) {
+        let mut group: CompatGroupInfo = serde_json::from_slice(data).map_err(|err| format!("parse IAM group failed: {err}"))?;
+        if group.update_at.is_none() {
+            group.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&group)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM group failed: {err}"));
+    }
+
+    if is_policy_doc_path(path) {
+        let mut doc = PolicyDoc::try_from(data.to_vec()).map_err(|err| format!("parse IAM policy doc failed: {err}"))?;
+        if doc.create_date.is_none() {
+            doc.create_date = doc.update_date;
+        }
+        if doc.update_date.is_none() {
+            doc.update_date = doc.create_date;
+        }
+        return serde_json::to_vec(&doc)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM policy doc failed: {err}"));
+    }
+
+    if is_policy_mapping_path(path) {
+        let mut mapped: CompatMappedPolicy =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM policy mapping failed: {err}"))?;
+        if mapped.update_at.is_none() {
+            mapped.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&mapped)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM policy mapping failed: {err}"));
+    }
+
+    Ok(None)
+}
+
+fn is_identity_path(path: &str) -> bool {
+    (path.starts_with(IAM_USERS_PREFIX) || path.starts_with(IAM_SERVICE_ACCOUNTS_PREFIX) || path.starts_with(IAM_STS_PREFIX))
+        && path.ends_with("/identity.json")
+}
+
+fn is_group_path(path: &str) -> bool {
+    path.starts_with(IAM_GROUPS_PREFIX) && path.ends_with("/members.json")
+}
+
+fn is_policy_doc_path(path: &str) -> bool {
+    path.starts_with(IAM_POLICIES_PREFIX) && path.ends_with("/policy.json")
+}
+
+fn is_policy_mapping_path(path: &str) -> bool {
+    path.starts_with(IAM_POLICY_DB_PREFIX) && path.ends_with(".json")
+}
+
+fn is_resync_meta_path(path: &str) -> bool {
+    path.ends_with(&format!("{REPLICATION_META_DIR}/{RESYNC_META_FILE}"))
+}
+
+fn normalize_bucket_meta_blob(path: &str, data: &[u8]) -> std::result::Result<Option<Vec<u8>>, String> {
+    if !is_resync_meta_path(path) {
+        return Ok(None);
+    }
+    let status = decode_resync_file(data).map_err(|err| format!("decode resync meta failed: {err}"))?;
+    encode_resync_file(&status)
+        .map(Some)
+        .map_err(|err| format!("encode resync meta failed: {err}"))
+}
+
+/// Migrates bucket metadata from legacy format to RustFS.
+/// Uses list_bucket (from disk volumes) to get bucket names, since list_objects_v2 on the legacy
+/// meta bucket may not work (legacy format differs from object layer expectations).
+/// Skips buckets that already exist in RustFS (idempotent).
+pub async fn try_migrate_bucket_metadata<S: StorageAPI>(store: Arc<S>) {
+    let buckets_list = match store
+        .list_bucket(&BucketOptions {
+            no_metadata: true,
+            ..Default::default()
+        })
+        .await
+    {
+        Ok(b) => b,
+        Err(e) => {
+            warn!("list buckets failed (skip migration): {e}");
+            return;
+        }
+    };
+
+    let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
+
+    if buckets.is_empty() {
+        debug!("No migrating bucket metadata found");
+        return;
+    }
+
+    debug!("Found {} migrating bucket metadata, migrating...", buckets.len());
+
+    let opts = ObjectOptions {
+        max_parity: true,
+        no_lock: true,
+        ..Default::default()
+    };
+    let h = HeaderMap::new();
+
+    for bucket in buckets {
+        let meta_path = format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{bucket}{SLASH_SEPARATOR}{BUCKET_METADATA_FILE}");
+        migrate_one_if_missing(store.clone(), &opts, &h, &meta_path, &format!("bucket metadata: {bucket}")).await;
+
+        let resync_path = format!(
+            "{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{bucket}{SLASH_SEPARATOR}{REPLICATION_META_DIR}{SLASH_SEPARATOR}{RESYNC_META_FILE}"
+        );
+        migrate_one_if_missing(store.clone(), &opts, &h, &resync_path, &format!("bucket replication resync: {bucket}")).await;
+    }
+}
+
+async fn migrate_one_if_missing<S: StorageAPI>(
+    store: Arc<S>,
+    opts: &ObjectOptions,
+    headers: &HeaderMap,
+    path: &str,
+    label: &str,
+) {
+    if store
+        .get_object_info(RUSTFS_META_BUCKET, path, &ObjectOptions::default())
+        .await
+        .is_ok()
+    {
+        debug!("{label} already exists in RustFS, skip");
+        return;
+    }
+
+    let mut rd = match store
+        .get_object_reader(MIGRATING_META_BUCKET, path, None, headers.clone(), opts)
+        .await
+    {
+        Ok(r) => r,
+        Err(e) => {
+            debug!("read migrating {label}: {e}");
+            return;
+        }
+    };
+
+    let data = match rd.read_all().await {
+        Ok(d) if !d.is_empty() => d,
+        Ok(_) => return,
+        Err(e) => {
+            debug!("read migrating {label} body: {e}");
+            return;
+        }
+    };
+
+    let data = match normalize_bucket_meta_blob(path, &data) {
+        Ok(Some(normalized)) => normalized,
+        Ok(None) => data,
+        Err(e) => {
+            warn!("skip {label} migration due to incompatible format: {e}");
+            return;
+        }
+    };
+
+    if let Err(e) = store
+        .put_object(RUSTFS_META_BUCKET, path, &mut PutObjReader::from_vec(data), opts)
+        .await
+    {
+        warn!("write {label}: {e}");
+    } else {
+        info!("Migrated {label}");
+    }
+}
+
+/// Migrates IAM config from legacy meta bucket `config/iam/` to RustFS meta bucket.
+/// Lists all objects under the IAM prefix in the source, copies each to the target if not present.
+/// Skips objects that already exist in RustFS (idempotent).
+/// If list_objects_v2 on the legacy bucket fails (e.g. format differs), migration is skipped.
+pub async fn try_migrate_iam_config<S: StorageAPI>(store: Arc<S>) {
+    let opts = ObjectOptions {
+        max_parity: true,
+        no_lock: true,
+        ..Default::default()
+    };
+    let h = HeaderMap::new();
+    let prefix = format!("{IAM_CONFIG_PREFIX}/");
+    let mut continuation: Option<String> = None;
+    let mut total_migrated = 0usize;
+
+    loop {
+        let list_result = match store
+            .clone()
+            .list_objects_v2(MIGRATING_META_BUCKET, &prefix, continuation, None, 500, false, None, false)
+            .await
+        {
+            Ok(r) => r,
+            Err(e) => {
+                debug!("list IAM config from legacy bucket failed (skip migration): {e}");
+                return;
+            }
+        };
+
+        for obj in list_result.objects {
+            let path = &obj.name;
+            if path.is_empty() || path.ends_with('/') {
+                continue;
+            }
+            if store
+                .get_object_info(RUSTFS_META_BUCKET, path, &ObjectOptions::default())
+                .await
+                .is_ok()
+            {
+                debug!("IAM config already exists in RustFS, skip: {path}");
+                continue;
+            }
+            let mut rd = match store
+                .get_object_reader(MIGRATING_META_BUCKET, path, None, h.clone(), &opts)
+                .await
+            {
+                Ok(r) => r,
+                Err(e) => {
+                    debug!("read migrating IAM config {path}: {e}");
+                    continue;
+                }
+            };
+            let data = match rd.read_all().await {
+                Ok(d) if !d.is_empty() => d,
+                Ok(_) => continue,
+                Err(e) => {
+                    debug!("read migrating IAM config {path} body: {e}");
+                    continue;
+                }
+            };
+            let data = match normalize_iam_config_blob(path, &data) {
+                Ok(Some(normalized)) => normalized,
+                Ok(None) => {
+                    debug!("skip unsupported IAM config path during migration: {path}");
+                    continue;
+                }
+                Err(e) => {
+                    warn!("skip IAM config migration due to incompatible format, path: {path}, err: {e}");
+                    continue;
+                }
+            };
+            if let Err(e) = store
+                .put_object(RUSTFS_META_BUCKET, path, &mut PutObjReader::from_vec(data), &opts)
+                .await
+            {
+                warn!("write IAM config {path}: {e}");
+            } else {
+                info!("Migrated IAM config: {path}");
+                total_migrated += 1;
+            }
+        }
+
+        continuation = list_result.next_continuation_token.or(list_result.continuation_token);
+        if !list_result.is_truncated || continuation.is_none() {
+            break;
+        }
+    }
+
+    if total_migrated > 0 {
+        info!("IAM migration complete: {} object(s) migrated", total_migrated);
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{normalize_bucket_meta_blob, normalize_iam_config_blob};
+    use crate::bucket::replication::{
+        BucketReplicationResyncStatus, ResyncStatusType, TargetReplicationResyncStatus, decode_resync_file, encode_resync_file,
+    };
+    use std::collections::HashMap;
+
+    #[test]
+    fn test_normalize_policy_mapping_legacy_timestamp_and_fields() {
+        let path = "config/iam/policydb/users/alice.json";
+        let input = r#"{"version":1,"policies":"readwrite","update_at":"2026-03-09 02:22:44.998954 +00:00:00"}"#;
+
+        let output = normalize_iam_config_blob(path, input.as_bytes())
+            .expect("normalize should succeed")
+            .expect("path should be supported");
+
+        let v: serde_json::Value = serde_json::from_slice(&output).expect("output should be valid JSON");
+        assert_eq!(v.get("policy").and_then(|x| x.as_str()), Some("readwrite"));
+        assert!(v.get("policies").is_none(), "legacy field should be normalized");
+
+        let updated_at = v
+            .get("updatedAt")
+            .and_then(|x| x.as_str())
+            .expect("updatedAt should exist as string");
+        assert!(updated_at.contains('T'), "updatedAt should be RFC3339-like");
+    }
+
+    #[test]
+    fn test_normalize_bucket_meta_blob_resync_reencode() {
+        let path = ".buckets/test/.replication/resync.bin";
+        let mut status = BucketReplicationResyncStatus::new();
+        status.id = 123;
+        status.targets_map = HashMap::from([(
+            "arn:replication::1:dest".to_string(),
+            TargetReplicationResyncStatus {
+                resync_id: "reset-1".to_string(),
+                resync_status: ResyncStatusType::ResyncStarted,
+                replicated_count: 1,
+                ..Default::default()
+            },
+        )]);
+
+        let input = encode_resync_file(&status).expect("encode should succeed");
+        let output = normalize_bucket_meta_blob(path, &input)
+            .expect("normalize should succeed")
+            .expect("resync path should be normalized");
+
+        let decoded = decode_resync_file(&output).expect("decode should succeed");
+        assert_eq!(decoded.id, 123);
+        assert_eq!(decoded.targets_map["arn:replication::1:dest"].resync_id, "reset-1");
+    }
+}

crates/ecstore/src/bucket/mod.rs

@@ -18,6 +18,10 @@ pub mod error;
 pub mod lifecycle;
 pub mod metadata;
 pub mod metadata_sys;
+#[cfg(test)]
+mod metadata_test;
+pub mod migration;
+mod msgp_decode;
 pub mod object_lock;
 pub mod policy_sys;
 pub mod quota;

crates/ecstore/src/bucket/msgp_decode.rs

@@ -0,0 +1,189 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! MessagePack decode helpers for bucket metadata, aligned with msgp format.
+
+use crate::error::{Error, Result};
+use byteorder::{BigEndian, ByteOrder};
+use rmp::Marker;
+use std::io::{Read, Write};
+use time::OffsetDateTime;
+
+/// Skip a single MessagePack value. Used for unknown map keys.
+pub(crate) fn skip_msgp_value<R: Read>(rd: &mut R) -> Result<()> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let skip_len: usize = match marker {
+        Marker::Null | Marker::False | Marker::True => 0,
+        Marker::FixPos(_) | Marker::FixNeg(_) => 0,
+        Marker::U8 => 1,
+        Marker::U16 => 2,
+        Marker::U32 => 4,
+        Marker::U64 => 8,
+        Marker::I8 => 1,
+        Marker::I16 => 2,
+        Marker::I32 => 4,
+        Marker::I64 => 8,
+        Marker::F32 => 4,
+        Marker::F64 => 8,
+        Marker::FixStr(n) => n as usize,
+        Marker::Str8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            b[0] as usize
+        }
+        Marker::Str16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u16::from_be_bytes(b) as usize
+        }
+        Marker::Str32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u32::from_be_bytes(b) as usize
+        }
+        Marker::Bin8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            b[0] as usize
+        }
+        Marker::Bin16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u16::from_be_bytes(b) as usize
+        }
+        Marker::Bin32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u32::from_be_bytes(b) as usize
+        }
+        Marker::FixArray(n) => {
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Array16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u16::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Array32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u32::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::FixMap(n) => {
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Map16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u16::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Map32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u32::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::FixExt1 => 1,
+        Marker::FixExt2 => 2,
+        Marker::FixExt4 => 4,
+        Marker::FixExt8 => 8,
+        Marker::FixExt16 => 16,
+        Marker::Ext8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = b[0] as usize;
+            1 + len // type byte + data
+        }
+        Marker::Ext16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = u16::from_be_bytes(b) as usize;
+            2 + len
+        }
+        Marker::Ext32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = u32::from_be_bytes(b) as usize;
+            4 + len
+        }
+        Marker::Reserved => 0,
+    };
+    if skip_len > 0 {
+        let mut buf = vec![0u8; skip_len];
+        rd.read_exact(&mut buf).map_err(Error::other)?;
+    }
+    Ok(())
+}
+
+/// msgp time format: ext8 (0xc7), len 12, type 5, 8 bytes sec (BE) + 4 bytes nsec (BE).
+pub(crate) const MSGP_TIME_EXT_TYPE: i8 = 5;
+pub(crate) const MSGP_TIME_LEN: u8 = 12;
+
+/// Read msgp ext8 time - caller must have already read the marker and verified it's ext8.
+/// Ext8 format: 1 byte len, 1 byte type, then data bytes.
+pub(crate) fn read_msgp_ext8_time<R: Read>(rd: &mut R) -> Result<OffsetDateTime> {
+    let mut len_buf = [0u8; 1];
+    rd.read_exact(&mut len_buf).map_err(Error::other)?;
+    let len = len_buf[0] as usize;
+    if len != MSGP_TIME_LEN as usize {
+        return Err(Error::other(format!("invalid msgp time len: {len}")));
+    }
+    let mut type_buf = [0u8; 1];
+    rd.read_exact(&mut type_buf).map_err(Error::other)?;
+    if type_buf[0] != MSGP_TIME_EXT_TYPE as u8 {
+        return Err(Error::other(format!("invalid msgp time type: {}", type_buf[0])));
+    }
+    let mut buf = [0u8; 12];
+    rd.read_exact(&mut buf).map_err(Error::other)?;
+    let sec = BigEndian::read_i64(&buf[0..8]);
+    let nsec = BigEndian::read_u32(&buf[8..12]);
+    OffsetDateTime::from_unix_timestamp(sec)
+        .map_err(|_| Error::other("invalid timestamp"))?
+        .replace_nanosecond(nsec)
+        .map_err(|_| Error::other("invalid nanosecond"))
+}
+
+/// Write msgp time as ext8 (0xc7), len 12, type 5. Always uses ext format (never nil).
+pub(crate) fn write_msgp_time<W: Write>(wr: &mut W, t: OffsetDateTime) -> Result<()> {
+    wr.write_all(&[0xc7, MSGP_TIME_LEN, MSGP_TIME_EXT_TYPE as u8])
+        .map_err(Error::other)?;
+    let mut buf = [0u8; 12];
+    BigEndian::write_i64(&mut buf[0..8], t.unix_timestamp());
+    BigEndian::write_u32(&mut buf[8..12], t.nanosecond());
+    wr.write_all(&buf).map_err(Error::other)
+}

crates/ecstore/src/bucket/quota/mod.rs

@@ -15,7 +15,6 @@
 pub mod checker;
 
 use crate::error::Result;
-use rmp_serde::Serializer as rmpSerializer;
 use rustfs_config::{
     QUOTA_API_PATH, QUOTA_EXCEEDED_ERROR_CODE, QUOTA_INTERNAL_ERROR_CODE, QUOTA_INVALID_CONFIG_ERROR_CODE,
     QUOTA_NOT_FOUND_ERROR_CODE,
@@ -28,27 +27,35 @@ use time::OffsetDateTime;
 pub enum QuotaType {
     /// Hard quota: reject immediately when exceeded
     #[default]
+    #[serde(alias = "HARD", alias = "hard")]
     Hard,
 }
 
+/// Bucket quota configuration. quota_type defaults to Hard when omitted.
 #[derive(Debug, Deserialize, Serialize, Default, Clone, PartialEq)]
 pub struct BucketQuota {
+    #[serde(default)]
     pub quota: Option<u64>,
+    /// Defaults to Hard when missing.
+    #[serde(default)]
     pub quota_type: QuotaType,
     /// Timestamp when this quota configuration was set (for audit purposes)
+    #[serde(default, with = "time::serde::rfc3339::option")]
     pub created_at: Option<OffsetDateTime>,
+    /// Accept updated_at for compatibility; not used.
+    #[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
+    pub updated_at: Option<OffsetDateTime>,
 }
 
 impl BucketQuota {
+    /// Serialize to JSON bytes. Same format as parse_all_configs.
     pub fn marshal_msg(&self) -> Result<Vec<u8>> {
-        let mut buf = Vec::new();
-        self.serialize(&mut rmpSerializer::new(&mut buf).with_struct_map())?;
-        Ok(buf)
+        serde_json::to_vec(self).map_err(Into::into)
     }
 
+    /// Deserialize from JSON bytes. Same format as parse_all_configs.
     pub fn unmarshal(buf: &[u8]) -> Result<Self> {
-        let t: BucketQuota = rmp_serde::from_slice(buf)?;
-        Ok(t)
+        serde_json::from_slice(buf).map_err(Into::into)
     }
 
     pub fn new(quota: Option<u64>) -> Self {
@@ -57,6 +64,7 @@ impl BucketQuota {
             quota,
             quota_type: QuotaType::Hard,
             created_at: Some(now),
+            updated_at: None,
         }
     }
 
@@ -156,3 +164,57 @@ impl QuotaErrorResponse {
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// Legacy format: quota, created_at, updated_at (no quota_type)
+    #[test]
+    fn deserialize_format_without_quota_type() {
+        let json = r#"{"quota":1073741824,"created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+        assert!(q.created_at.is_some());
+        assert!(q.updated_at.is_some());
+    }
+
+    /// RustFS format: quota, quota_type, created_at
+    #[test]
+    fn deserialize_rustfs_format() {
+        let json = r#"{"quota":1073741824,"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+        assert!(q.created_at.is_some());
+        assert!(q.created_at.is_some_and(|t| t.unix_timestamp() == 1704067200));
+    }
+
+    /// E2E format uses "HARD" (uppercase)
+    #[test]
+    fn deserialize_quota_type_hard_uppercase() {
+        let json = r#"{"quota":2048,"quota_type":"HARD"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota_type, QuotaType::Hard);
+    }
+
+    /// marshal_msg/unmarshal use JSON, same as parse_all_configs
+    #[test]
+    fn marshal_unmarshal_roundtrip() {
+        let q = BucketQuota::new(Some(1073741824));
+        let buf = q.marshal_msg().expect("marshal");
+        let restored = BucketQuota::unmarshal(&buf).expect("unmarshal");
+        assert_eq!(q.quota, restored.quota);
+        assert_eq!(q.quota_type, restored.quota_type);
+    }
+
+    /// unmarshal accepts format without quota_type
+    #[test]
+    fn unmarshal_format_without_quota_type() {
+        let json = r#"{"quota":1073741824,"created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#;
+        let q = BucketQuota::unmarshal(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+    }
+}