fix(ci): revert docker alpha latest tag (#2209)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

.agents/skills/code-change-verification/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: code-change-verification
+description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
+---
+
+# Code Change Verification
+
+Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
+
+## Quick Start
+
+1. Read the scope: commit, PR, patch, or file list.
+2. Map each changed area by risk and user impact.
+3. Inspect each risky change in context.
+4. Report findings first, ordered by severity.
+5. Close with residual risks and verification recommendations.
+
+## Core Workflow
+
+### 1) Scope and assumptions
+- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
+- If context is missing, state assumptions before deeper analysis.
+- Focus only on requested scope; avoid reviewing unrelated files.
+
+### 2) Risk map
+- Prioritize in this order:
+  - Data correctness and user-visible behavior
+  - API/contract compatibility
+  - Security and authz/authn boundaries
+  - Concurrency and lifecycle correctness
+  - Performance and resource usage
+- Give higher priority to stateful paths, migration logic, defaults, and error handling.
+
+### 3) Evidence-based inspection
+- Read each modified hunk with neighboring context.
+- Trace call paths and call-site expectations.
+- Check for:
+  - invariant breaks and missing guards
+  - unchecked assumptions and null/empty/error-path handling
+  - stale tests, fixtures, and configs
+  - hidden coupling to shared helpers/constants/features
+- If a point is uncertain, mark it as an open question instead of guessing.
+
+### 4) Findings-first output
+- Order findings by severity:
+  - P0: critical failure, security breach, or data loss risk
+  - P1: high-impact regression
+  - P2: medium risk correctness gap
+  - P3: low risk/quality debt
+- For each finding include:
+  - Severity
+  - `path:line` reference
+  - concise issue statement
+  - impact and likely failure mode
+  - specific fix or mitigation
+  - validation step to confirm
+- If no issues exist, explicitly state `No findings` and why.
+
+### 5) Close
+- Report assumptions and unknowns.
+- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
+
+## Output Template
+
+1. Findings
+2. No findings (if applicable)
+3. Assumptions / Unknowns
+4. Recommended verification steps
+
+## Finding Template
+
+- `[P1] Missing timeout for downstream call`
+  - Location: `path/to/file.rs:123`
+  - Issue: ...
+  - Impact: ...
+  - Fix suggestion: ...
+  - Validation: ...
+

.agents/skills/code-change-verification/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Code Change Verification"
+  short_description: "Prioritize risks and verify code changes before merge."
+  default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

.agents/skills/pr-creation-checker/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: pr-creation-checker
+description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
+---
+
+# PR Creation Checker
+
+Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
+
+## Read sources of truth first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Use `Makefile` and `.config/make/` for local quality commands.
+- Use `.github/workflows/ci.yml` for CI expectations.
+- Do not restate long command matrices or template sections from memory when the files exist.
+
+## Workflow
+
+1. Collect PR context
+- Confirm base branch, current branch, change goal, and scope.
+- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
+- Confirm whether the branch includes only intended changes.
+
+2. Inspect change scope
+- Review the diff and summarize what changed.
+- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
+- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+
+3. Verify readiness requirements
+- Require `make pre-commit` before marking the PR ready.
+- If `make` is unavailable, use the equivalent commands from `.config/make/`.
+- Add scope-specific verification commands when the changed area needs more than the baseline.
+- If required checks fail, stop and return `BLOCKED`.
+
+4. Draft PR metadata
+- Write the PR title in English using Conventional Commits and keep it within 72 characters.
+- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
+- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
+- Keep the PR body in English.
+- Use the exact section headings from `.github/pull_request_template.md`.
+- Fill non-applicable sections with `N/A`.
+- Include verification commands in the PR description.
+- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
+- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
+
+5. Prepare reviewer context
+- Summarize why the change exists.
+- Summarize what was verified.
+- Call out risks, rollout notes, config impact, and rollback notes when applicable.
+- Mention assumptions or missing context instead of guessing.
+
+6. Prepare CLI-safe output
+- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
+- Return a ready-to-save PR body plus a short title.
+- If not ready, return blockers first and list the minimum steps needed to unblock.
+
+## Output format
+
+### Status
+- `READY` or `BLOCKED`
+
+### Title
+- `<type>(<scope>): <summary>`
+
+### PR Body
+- Reproduce the repository template headings exactly.
+- Fill every section.
+- Omit local absolute paths unless explicitly required.
+
+### Verification
+- List each command run.
+- State pass/fail.
+
+### Risks
+- List breaking changes, config changes, migration impact, or `N/A`.
+
+## Blocker rules
+
+- Return `BLOCKED` if `make pre-commit` has not passed.
+- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
+- Return `BLOCKED` if required template sections are missing.
+- Return `BLOCKED` if the title/body is not in English.
+- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
+
+## Reference
+
+- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "PR Creation Checker"
+  short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
+  default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -0,0 +1,14 @@
+# PR Readiness Checklist
+
+- Confirm the branch is based on current `main`.
+- Confirm the diff matches the stated scope.
+- Confirm no secrets, logs, temp files, or unrelated refactors are included.
+- Confirm `make pre-commit` passed, or document why it could not run.
+- Confirm extra verification commands are listed for risky changes.
+- Confirm the PR title uses Conventional Commits and stays within 72 characters.
+- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+- Confirm the PR body is in English.
+- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
+- Confirm non-applicable sections are filled with `N/A`.
+- Confirm the PR body does not include local absolute paths unless explicitly required.
+- Confirm multiline GitHub CLI commands use `--body-file`.

.agents/skills/test-coverage-improver/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: test-coverage-improver
+description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
+---
+
+# Test Coverage Improver
+
+Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
+
+## Usage assumptions
+- Focus scope is either changed lines/files, a module, or the whole repository.
+- Coverage artifact must be generated or provided in a supported format.
+- If required context is missing, call out assumptions explicitly before proposing work.
+
+## Workflow
+
+1. Define scope and baseline
+   - Confirm target language, framework, and branch.
+   - Confirm whether the scope is changed files only or full-repo.
+
+2. Produce coverage snapshot
+   - Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
+   - JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
+   - Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
+   - Collect total, per-file, and changed-line coverage.
+
+3. Rank highest-risk gaps
+   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
+   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
+   - Keep shortlist to 5–8 gaps.
+   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
+
+4. Propose high-impact tests
+   - For each shortlisted gap, output:
+     - Intent and expected behavior.
+     - Normal, edge, and failure scenarios.
+     - Assertions and side effects to verify.
+     - Setup needs (fixtures, mocks, integration dependencies).
+     - Estimated effort (`S/M/L`).
+
+5. Close with validation plan
+   - State which gaps remain after proposals.
+   - Provide concrete verification command and acceptance threshold.
+   - List assumptions or blockers (environment, fixtures, flaky dependencies).
+
+## Output template
+
+### Coverage Snapshot
+- total / branch coverage
+- changed-file coverage
+- top missing regions by size
+
+### Top Gaps (ranked)
+- `path:line-range` | risk score | why critical
+
+### Test Proposals
+- `path:line-range`
+  - Test name
+  - scenarios
+  - assertions
+  - effort
+
+### Validation Plan
+- command
+- pass criteria
+- remaining risk

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Test Coverage Improver"
+  short_description: "Find top uncovered risk areas and propose high-impact tests."
+  default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -0,0 +1,25 @@
+# Coverage Gap Prioritization Guide
+
+Use this rubric for each uncovered area.
+
+Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
+
+- Criticality:
+  - 5: authz/authn, data-loss, payment/consistency path
+  - 4: state mutation, cache invalidation, scheduling
+  - 3: error handling + fallbacks in user-visible flows
+  - 2: parsing/format conversion paths
+  - 1: logging-only or low-impact utilities
+
+- CoverageDebt:
+  - 0: 0–5 uncovered lines
+  - 1: 6–20 uncovered lines
+  - 2: 21–40 uncovered lines
+  - 3: 41+ uncovered lines
+
+- Volatility:
+  - 1: stable legacy code with few recent edits
+  - 2: changed in last 2 releases
+  - 3: touched in last 30 days or currently in active PR
+
+Sort by score descending, then by business impact.

.docker/compose/docker-compose.observability.yaml

@@ -104,7 +104,7 @@ services:
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/loki-config.yaml:/etc/loki/local-config.yaml:ro
+      - ../../.docker/observability/loki.yaml:/etc/loki/local-config.yaml:ro
       - loki-data:/loki
     ports:
       - "3100:3100"

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -0,0 +1,270 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  rustfs:
+    security_opt:
+      - "no-new-privileges:true"
+    image: rustfs/rustfs:latest
+    container_name: rustfs-server
+    ports:
+      - "9000:9000" # S3 API port
+      - "9001:9001" # Console port
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+    volumes:
+      - rustfs-data:/data/rustfs
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+    depends_on:
+      otel-collector:
+        condition: service_started
+
+  rustfs-init:
+    image: alpine
+    container_name: rustfs-init
+    volumes:
+      - rustfs-data:/data
+    networks:
+      - otel-network
+    command: >
+      sh -c "
+        chown -R 10001:10001 /data &&
+        echo 'Volume Permissions fixed' &&
+        exit 0
+      "
+    restart: no
+
+  # --- Tracing ---
+
+  tempo:
+    image: grafana/tempo:latest
+    container_name: tempo
+    command: [ "-config.file=/etc/tempo.yaml" ]
+    volumes:
+      - ./tempo.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200" # tempo
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - redpanda
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  redpanda:
+    image: redpandadata/redpanda:latest # for tempo ingest
+    container_name: redpanda
+    ports:
+      - "9092:9092"
+    networks:
+      - otel-network
+    restart: unless-stopped
+    command: >
+      redpanda start --overprovisioned
+      --mode=dev-container
+      --kafka-addr=PLAINTEXT://0.0.0.0:9092
+      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
+
+  jaeger:
+    image: jaegertracing/jaeger:latest
+    container_name: jaeger
+    environment:
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
+      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - ./jaeger.yaml:/etc/jaeger/config.yml
+      - jaeger-data:/badger
+    ports:
+      - "16686:16686" # Web UI
+      - "14269:14269" # Admin/Metrics
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  # --- Metrics ---
+
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: prometheus
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-data:/prometheus
+    ports:
+      - "9090:9090"
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--web.enable-otlp-receiver" # Enable OTLP
+      - "--web.enable-remote-write-receiver" # Enable remote write
+      - "--enable-feature=promql-experimental-functions" # Enable info()
+      - "--storage.tsdb.retention.time=30d"
+    restart: unless-stopped
+    networks:
+      - otel-network
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Logging ---
+
+  loki:
+    image: grafana/loki:latest
+    container_name: loki
+    volumes:
+      - ./loki.yaml:/etc/loki/loki.yaml:ro
+      - loki-data:/loki
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/loki.yaml
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  # --- Collection ---
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    ports:
+      - "1888:1888" # pprof
+      - "8888:8888" # Prometheus metrics for Collector
+      - "8889:8889" # Prometheus metrics for application indicators
+      - "13133:13133" # health check
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
+      - "55679:55679" # zpages
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Profiles ---
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    container_name: pyroscope
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - otel-network
+    restart: unless-stopped
+
+  # --- Visualization ---
+
+  grafana:
+    image: grafana/grafana:latest
+    container_name: grafana
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+      - GF_SECURITY_ADMIN_USER=admin
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/etc/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+    healthcheck:
+      test:
+        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  rustfs-data:
+  tempo-data:
+  jaeger-data:
+  prometheus-data:
+  loki-data:
+  grafana-data:
+
+networks:
+  otel-network:
+    driver: bridge
+    name: "network_otel"
+    ipam:
+      config:
+        - subnet: 172.28.0.0/16
+    driver_opts:
+      com.docker.network.enable_ipv6: "true"

.docker/observability/docker-compose-tempo-ha-override.yml

@@ -0,0 +1,62 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Docker Compose override file for High Availability Tempo setup
+#
+# Usage:
+# docker-compose -f docker-compose-example-for-rustfs.yml \
+#                -f docker-compose-tempo-ha-override.yml up
+
+services:
+  # Override Tempo to use high-availability configuration
+  tempo:
+    volumes:
+      - ./tempo-ha.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200"   # Tempo HTTP
+      - "4317:4317"   # OTLP gRPC
+      - "4318:4318"   # OTLP HTTP
+      - "7946:7946"   # Memberlist
+      - "14250:14250" # Jaeger gRPC
+      - "14268:14268" # Jaeger Thrift HTTP
+      - "9411:9411"   # Zipkin
+    environment:
+      - TEMPO_MEMBERLIST_BIND_PORT=7946
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    depends_on:
+      - redpanda
+
+volumes:
+  tempo-data:
+    driver: local
+    driver_opts:
+      type: tmpfs
+      device: tmpfs
+      o: "size=4g"  # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
+
+# Network configuration remains the same
+# networks:
+#   otel-network:
+#     driver: bridge
+#     name: "network_otel"
+#     ipam:
+#       config:
+#         - subnet: 172.28.0.0/16
+

.docker/observability/docker-compose.yml

@@ -13,100 +13,31 @@
 # limitations under the License.
 
 services:
-  rustfs:
-    security_opt:
-      - "no-new-privileges:true"
-    image: rustfs/rustfs:latest
-    container_name: rustfs-server
-    ports:
-      - "9000:9000" # S3 API port
-      - "9001:9001" # Console port
-    environment:
-      - RUSTFS_VOLUMES=/data/rustfs
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
-      - RUSTFS_OBS_LOGGER_LEVEL=info
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-    volumes:
-      - rustfs-data:/data/rustfs
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test:
-        [
-          "CMD",
-          "sh",
-          "-c",
-          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
-        ]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 40s
-    depends_on:
-      otel-collector:
-        condition: service_started
-
-  rustfs-init:
-    image: alpine
-    container_name: rustfs-init
-    volumes:
-      - rustfs-data:/data
-    networks:
-      - otel-network
-    command: >
-      sh -c "
-        chown -R 10001:10001 /data &&
-        echo 'Volume Permissions fixed' &&
-        exit 0
-      "
-    restart: no
 
   # --- Tracing ---
 
   tempo:
     image: grafana/tempo:latest
     container_name: tempo
-    command: ["-config.file=/etc/tempo.yaml"]
+    command: [ "-config.file=/etc/tempo.yaml" ]
     volumes:
       - ./tempo.yaml:/etc/tempo.yaml:ro
       - tempo-data:/var/tempo
     ports:
       - "3200:3200" # tempo
-      - "4317"      # otlp grpc
-      - "4318"      # otlp http
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+      - "7946" # memberlist
     networks:
       - otel-network
     restart: unless-stopped
-    depends_on:
-      - redpanda
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3200/ready"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
       interval: 10s
       timeout: 5s
       retries: 3
       start_period: 15s
 
-  redpanda:
-    image: redpandadata/redpanda:latest # for tempo ingest
-    container_name: redpanda
-    ports:
-      - "9092:9092"
-    networks:
-      - otel-network
-    restart: unless-stopped
-    command: >
-      redpanda start --overprovisioned
-      --mode=dev-container
-      --kafka-addr=PLAINTEXT://0.0.0.0:9092
-      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
-
   jaeger:
     image: jaegertracing/jaeger:latest
     container_name: jaeger
@@ -124,12 +55,12 @@ services:
       - "14269:14269" # Admin/Metrics
       - "4317" # otlp grpc
       - "4318" # otlp http
-    command: ["--config", "/etc/jaeger/config.yml"]
+    command: [ "--config", "/etc/jaeger/config.yml" ]
     networks:
       - otel-network
     restart: unless-stopped
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:14269"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
       interval: 10s
       timeout: 5s
       retries: 3
@@ -155,7 +86,7 @@ services:
     networks:
       - otel-network
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
       interval: 10s
       timeout: 5s
       retries: 3
@@ -175,7 +106,7 @@ services:
       - otel-network
     restart: unless-stopped
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3100/ready"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
       interval: 15s
       timeout: 10s
       retries: 5
@@ -188,12 +119,12 @@ services:
     volumes:
       - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
     ports:
-      - "1888:1888"   # pprof
-      - "8888:8888"   # Prometheus metrics for Collector
-      - "8889:8889"   # Prometheus metrics for application indicators
+      - "1888:1888" # pprof
+      - "8888:8888" # Prometheus metrics for Collector
+      - "8889:8889" # Prometheus metrics for application indicators
       - "13133:13133" # health check
-      - "4317:4317"   # OTLP gRPC
-      - "4318:4318"   # OTLP HTTP
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
       - "55679:55679" # zpages
     networks:
       - otel-network
@@ -204,11 +135,24 @@ services:
       - prometheus
       - loki
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:13133"]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
       interval: 10s
       timeout: 5s
       retries: 3
 
+  # --- Profiles ---
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    container_name: pyroscope
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - otel-network
+    restart: unless-stopped
+
   # --- Visualization ---
 
   grafana:
@@ -231,13 +175,13 @@ services:
       - tempo
       - loki
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health"]
+      test:
+        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
       interval: 10s
       timeout: 5s
       retries: 3
 
 volumes:
-  rustfs-data:
   tempo-data:
   jaeger-data:
   prometheus-data:

.docker/observability/grafana/provisioning/dashboards/dashboard.yml

@@ -1,3 +1,17 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 apiVersion: 1
 
 providers:

.docker/observability/grafana/provisioning/datasources/datasources.yaml

@@ -1,3 +1,17 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 apiVersion: 1
 
 datasources:
@@ -76,3 +90,9 @@ datasources:
         spanEndTimeShift: '-1s'
         filterByTraceID: true
         filterBySpanID: false
+
+  - name: Pyroscope
+    type: grafana-pyroscope-datasource
+    url: http://pyroscope:4040
+    jsonData:
+      minStep: '15s'

.docker/observability/tempo-ha.yaml

@@ -0,0 +1,286 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
+# Features:
+# - Distributed architecture with multiple components
+# - Kafka-based ingestion for fault tolerance
+# - Replication factor of 3 for data resilience
+# - Query frontend for load balancing
+# - Metrics generation from traces
+# - WAL for durability
+
+partition_ring_live_store: true
+stream_over_http_enabled: true
+
+server:
+  http_listen_port: 3200
+  http_server_read_timeout: 30s
+  http_server_write_timeout: 30s
+  grpc_server_max_recv_msg_size: 4194304  # 4MB
+  grpc_server_max_send_msg_size: 4194304
+  log_level: info
+  log_format: json
+
+# Memberlist configuration for distributed mode
+memberlist:
+  node_name: tempo
+  bind_port: 7946
+  join_members:
+    - tempo:7946
+  retransmit_factor: 4
+  node_timeout: 15s
+  retransmit_interval: 300ms
+  dead_node_reclaim_time: 30s
+
+# Distributor configuration - receives traces and routes to ingesters
+distributor:
+  ingester_write_path_enabled: true
+  kafka_write_path_enabled: true
+  rate_limit_bytes: 10MB
+  rate_limit_enabled: true
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:4317"
+          max_concurrent_streams: 0
+          max_receive_message_size: 4194304
+        http:
+          endpoint: "0.0.0.0:4318"
+          cors:
+            allowed_origins:
+              - "*"
+            max_age: 86400
+    jaeger:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:14250"
+        thrift_http:
+          endpoint: "0.0.0.0:14268"
+    zipkin:
+      endpoint: "0.0.0.0:9411"
+  ring:
+    kvstore:
+      store: memberlist
+    heartbeat_timeout: 5s
+    replication_factor: 3
+    heartbeat_interval: 5s
+
+# Ingester configuration - stores traces and querying
+ingester:
+  lifecycler:
+    address: tempo
+    ring:
+      kvstore:
+        store: memberlist
+      replication_factor: 3
+      max_cache_freshness_per_sec: 10s
+      heartbeat_interval: 5s
+      heartbeat_timeout: 5s
+    num_tokens: 128
+    tokens_file_path: /var/tempo/tokens.json
+    claim_on_rollout: true
+  trace_idle_period: 20s
+  max_block_bytes: 10_000_000
+  max_block_duration: 10m
+  chunk_size_bytes: 1_000_000
+  chunk_encoding: snappy
+  wal:
+    checkpoint_duration: 5s
+    max_wal_blocks: 4
+  metrics:
+    enabled: true
+    level: block
+    target_info_duration: 15m
+
+# WAL configuration for data durability
+wal:
+  checkpoint_duration: 5s
+  flush_on_shutdown: true
+  path: /var/tempo/wal
+
+# Kafka ingestion configuration - for high throughput scenarios
+ingest:
+  enabled: true
+  kafka:
+    brokers: [ redpanda:9092 ]
+    topic: tempo-ingest
+    encoding: protobuf
+    consumer_group: tempo-ingest-consumer
+    session_timeout: 10s
+    rebalance_timeout: 1m
+    partition: auto
+    verbosity: 2
+
+# Query frontend configuration - distributed querying
+query_frontend:
+  compression: gzip
+  downstream_url: http://localhost:3200
+  log_queries_longer_than: 5s
+  cache_uncompressed_bytes: 100MB
+  max_outstanding_requests_per_tenant: 100
+  max_query_length: 48h
+  max_query_lookback: 30d
+  default_result_cache_ttl: 1m
+  result_cache:
+    cache:
+      enable_fifocache: true
+      default_validity: 1m
+  rf1_after: "1999-01-01T00:00:00Z"
+  mcp_server:
+    enabled: true
+
+# Querier configuration - queries traces
+querier:
+  frontend_worker:
+    frontend_address: localhost:3200
+    grpc_client_config:
+      max_recv_msg_size: 104857600
+  max_concurrent_queries: 20
+  max_metric_bytes_per_trace: 1MB
+
+# Query scheduler configuration - for distributed querying
+query_scheduler:
+  use_scheduler_ring: false
+
+# Metrics generator configuration - generates metrics from traces
+metrics_generator:
+  enabled: true
+  registry:
+    enabled: true
+    external_labels:
+      source: tempo
+      cluster: rustfs-docker-ha
+      environment: production
+  storage:
+    path: /var/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+        resource_to_telemetry_conversion:
+          enabled: true
+  processor:
+    batch:
+      timeout: 10s
+      send_batch_size: 1024
+    memory_limiter:
+      check_interval: 5s
+      limit_mib: 512
+      spike_limit_mib: 128
+  processors:
+    - span-metrics
+    - local-blocks
+    - service-graphs
+  generate_native_histograms: both
+
+# Backend worker configuration
+backend_worker:
+  backend_scheduler_addr: localhost:3200
+  compaction:
+    block_retention: 24h
+    compacted_block_retention: 1h
+  ring:
+    kvstore:
+      store: memberlist
+
+# Backend scheduler configuration
+backend_scheduler:
+  enabled: true
+  provider:
+    compaction:
+      compaction:
+        block_retention: 24h
+        compacted_block_retention: 1h
+        concurrency: 25
+        v2_out_path: /var/tempo/blocks/compaction
+
+# Storage configuration - local backend with proper retention
+storage:
+  trace:
+    backend: local
+    wal:
+      path: /var/tempo/wal
+      checkpoint_duration: 5s
+      flush_on_shutdown: true
+    local:
+      path: /var/tempo/blocks
+      bloom_filter_false_positive: 0.05
+      bloom_shift: 4
+      index:
+        downsample_bytes: 1000000
+        page_size_bytes: 0
+        cache_size_bytes: 0
+    pool:
+      max_workers: 400
+      queue_depth: 10000
+
+# Compactor configuration - manages block compaction
+compactor:
+  compaction:
+    block_retention: 168h  # 7 days
+    compacted_block_retention: 1h
+    concurrency: 25
+    v2_out_path: /var/tempo/blocks/compaction
+    shard_count: 32
+    max_block_bytes: 107374182400  # 100GB
+    max_compaction_objects: 6000000
+    max_time_per_tenant: 5m
+    block_size_bytes: 107374182400
+  ring:
+    kvstore:
+      store: memberlist
+    heartbeat_interval: 5s
+    heartbeat_timeout: 5s
+
+# Limits configuration - rate limiting and quotas
+limits:
+  max_traces_per_user: 10000
+  max_bytes_per_trace: 10485760  # 10MB
+  max_search_bytes_per_trace: 0
+  forgiving_oversize_traces: true
+  rate_limit_bytes: 10MB
+  rate_limit_enabled: true
+  ingestion_burst_size_bytes: 20MB
+  ingestion_rate_limit_bytes: 10MB
+  max_bytes_per_second: 10485760
+  metrics_generator_max_active_series: 10000
+  metrics_generator_max_churned_series: 10000
+  metrics_generator_forta_out_of_order_ttl: 5m
+
+# Override configuration
+overrides:
+  defaults:
+    metrics_generator:
+      processors:
+        - span-metrics
+        - local-blocks
+        - service-graphs
+      generate_native_histograms: both
+      max_active_series: 10000
+      max_churned_series: 10000
+
+# Usage reporting configuration
+usage_report:
+  reporting_enabled: false
+
+# Tracing configuration for debugging
+tracing:
+  enabled: true
+  jaeger:
+    sampler:
+      name: probabilistic
+      param: 0.1
+    reporter_log_spans: false
+

.docker/observability/tempo.yaml

@@ -19,9 +19,16 @@ server:
   http_listen_port: 3200
   log_level: info
 
+memberlist:
+  node_name: tempo
+  bind_port: 7946
+  join_members:
+    - tempo:7946
+
+# Distributor configuration - receives traces and writes directly to ingesters
 distributor:
-  ingester_write_path_enabled: false
-  kafka_write_path_enabled: true
+  ingester_write_path_enabled: true
+  kafka_write_path_enabled: false
   receivers:
     otlp:
       protocols:
@@ -29,10 +36,21 @@ distributor:
           endpoint: "tempo:4317"
         http:
           endpoint: "tempo:4318"
-  #log_received_spans:
-  #    enabled: true
-  #  log_discarded_spans:
-  #    enabled: true
+  ring:
+    kvstore:
+      store: memberlist
+
+# Ingester configuration - consumes from Kafka and stores traces
+ingester:
+  lifecycler:
+    ring:
+      kvstore:
+        store: memberlist
+      replication_factor: 1
+    tokens_file_path: /var/tempo/tokens.json
+  trace_idle_period: 10s
+  max_block_bytes: 1_000_000
+  max_block_duration: 5m
 
 backend_scheduler:
   provider:
@@ -49,7 +67,8 @@ backend_worker:
       store: memberlist
 
 querier:
-  query_live_store: true
+  frontend_worker:
+    frontend_address: tempo:3200
 
 metrics_generator:
   registry:
@@ -78,17 +97,28 @@ storage:
 overrides:
   defaults:
     metrics_generator:
-      processors: ["span-metrics", "service-graphs", "local-blocks"]
+      processors: [ "span-metrics", "service-graphs", "local-blocks" ]
       generate_native_histograms: both
 
 ingest:
-  enabled: true
-  kafka:
-    address: redpanda:9092
-    topic: tempo-ingest
+  enabled: false
+  # Disabled because using direct ingester write path
+  # If you want Kafka path, enable this and set:
+  # kafka:
+  #   brokers: [redpanda:9092]
+  #   topic: tempo-ingest
+  #   encoding: protobuf
+  #   consumer_group: tempo-ingest-consumer
 
 block_builder:
   consume_cycle_duration: 30s
 
+compactor:
+  compaction:
+    block_retention: 168h  # 7 days
+  ring:
+    kvstore:
+      store: memberlist
+
 usage_report:
   reporting_enabled: false

.github/AGENTS.md

@@ -0,0 +1,30 @@
+# GitHub Workflow Instructions
+
+Applies to `.github/` and repository pull-request operations.
+
+## Pull Requests
+
+- PR titles and descriptions must be in English.
+- Use `.github/pull_request_template.md` for every PR body.
+- Keep all template section headings.
+- Use `N/A` for non-applicable sections.
+- Include verification commands in the PR details.
+- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
+- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
+- Recommended pattern:
+  - `cat > /tmp/pr_body.md <<'EOF'`
+  - `...markdown...`
+  - `EOF`
+  - `gh pr create ... --body-file /tmp/pr_body.md`
+
+## CI Alignment
+
+When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`.
+
+Current `test-and-lint` gate includes:
+
+- `cargo nextest run --all --exclude e2e_test`
+- `cargo test --all --doc`
+- `cargo fmt --all --check`
+- `cargo clippy --all-targets --all-features -- -D warnings`
+- `./scripts/check_layer_dependencies.sh`

.github/workflows/build.yml

@@ -23,6 +23,7 @@
 #
 # Manual Parameters:
 # - build_docker: Build and push Docker images (default: true)
+# - platforms: Comma-separated platform IDs or 'all' (default: all)
 
 name: Build and Release
 
@@ -44,22 +45,6 @@ on:
       - "**/*.svg"
       - ".gitignore"
       - ".dockerignore"
-  pull_request:
-    branches: [ main ]
-    paths-ignore:
-      - "**.md"
-      - "**.txt"
-      - ".github/**"
-      - "docs/**"
-      - "deploy/**"
-      - "scripts/dev_*.sh"
-      - "LICENSE*"
-      - "README*"
-      - "**/*.png"
-      - "**/*.jpg"
-      - "**/*.svg"
-      - ".gitignore"
-      - ".dockerignore"
   schedule:
     - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
   workflow_dispatch:
@@ -69,6 +54,11 @@ on:
         required: false
         default: true
         type: boolean
+      platforms:
+        description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
+        required: false
+        default: "all"
+        type: string
 
 permissions:
   contents: read
@@ -154,63 +144,70 @@ jobs:
           echo "  - Is prerelease: $is_prerelease"
 
   # Build RustFS binaries
+  prepare-platform-matrix:
+    name: Prepare Platform Matrix
+    runs-on: ubicloud-standard-2
+    outputs:
+      matrix: ${{ steps.select.outputs.matrix }}
+      selected: ${{ steps.select.outputs.selected }}
+    steps:
+      - name: Select target platforms
+        id: select
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
+          selected="$(echo "${selected}" | tr -d '[:space:]')"
+          if [[ -z "${selected}" ]]; then
+            selected="all"
+          fi
+
+          all='{"include":[
+            {"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
+          ]}'
+
+          if [[ "${selected}" == "all" ]]; then
+            matrix="$(jq -c . <<<"${all}")"
+          else
+            unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | ($all.include | map(.target_id)) as $known
+              | [$req[] | select(( $known | index(.) ) == null)]
+            ')"
+            if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
+              echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
+              echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
+              exit 1
+            fi
+
+            matrix="$(jq -c --arg selected "${selected}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | .include |= map(select(.target_id as $id | ($req | index($id))))
+            ' <<<"${all}")"
+          fi
+
+          echo "selected=${selected}" >> "$GITHUB_OUTPUT"
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
+          echo "Selected platforms: ${selected}"
+
   build-rustfs:
     name: Build RustFS
-    needs: [ build-check ]
-    if: needs.build-check.outputs.should_build == 'true'
+    needs: [ build-check, prepare-platform-matrix ]
+    if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
     timeout-minutes: 60
     env:
       RUSTFLAGS: ${{ matrix.rustflags }}
     strategy:
       fail-fast: false
-      matrix:
-        include:
-          # Linux builds
-          # Use x86-64-v2 (SSE4.2 baseline) instead of native to ensure distributed
-          # binaries run on older x86_64 CPUs (e.g. Intel Celeron/Atom, Synology NAS).
-          # See: https://github.com/rustfs/rustfs/issues/1838
-          - os: ubicloud-standard-2
-            target: x86_64-unknown-linux-musl
-            cross: false
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: aarch64-unknown-linux-musl
-            cross: true
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: x86_64-unknown-linux-gnu
-            cross: false
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: aarch64-unknown-linux-gnu
-            cross: true
-            platform: linux
-            rustflags: ''
-          # macOS builds
-          - os: macos-latest
-            target: aarch64-apple-darwin
-            cross: false
-            platform: macos
-            rustflags: ''
-          - os: macos-latest
-            target: x86_64-apple-darwin
-            cross: false
-            platform: macos
-            rustflags: ''
-          # Windows builds (temporarily disabled)
-          - os: windows-latest
-            target: x86_64-pc-windows-msvc
-            cross: false
-            platform: windows
-            rustflags: ''
-          #- os: windows-latest
-          #  target: aarch64-pc-windows-msvc
-          #  cross: true
-          #  platform: windows
+      matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6

.github/workflows/ci.yml

@@ -91,6 +91,8 @@ jobs:
 
   typos:
     name: Typos
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-2
     steps:
       - uses: actions/checkout@v6
@@ -116,9 +118,6 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Install cargo-nextest
-        uses: taiki-e/install-action@nextest
-
       - name: Run tests
         run: |
           cargo nextest run --all --exclude e2e_test
@@ -133,9 +132,40 @@ jobs:
       - name: Check layered dependencies
         run: ./scripts/check_layer_dependencies.sh
 
+  build-rustfs-debug-binary:
+    name: Build RustFS Debug Binary
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-rustfs-debug-binary-${{ hashFiles('**/Cargo.lock') }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build debug binary
+        run: |
+          touch rustfs/build.rs
+          cargo build -p rustfs --bins --jobs 2
+
+      - name: Upload debug binary
+        uses: actions/upload-artifact@v6
+        with:
+          name: rustfs-debug-binary
+          path: target/debug/rustfs
+          if-no-files-found: error
+          retention-days: 1
+
   e2e-tests:
     name: End-to-End Tests
-    needs: skip-check
+    needs: [ skip-check, build-rustfs-debug-binary ]
     if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-2
     timeout-minutes: 30
@@ -148,13 +178,17 @@ jobs:
           rm -rf /tmp/rustfs
           rm -f /tmp/rustfs.log
 
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
         with:
-          rust-version: stable
-          cache-shared-key: ci-e2e-${{ hashFiles('**/Cargo.lock') }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Setup Rust toolchain for s3s-e2e installation
+        uses: dtolnay/rust-toolchain@stable
 
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@v2
@@ -163,12 +197,6 @@ jobs:
           git: https://github.com/s3s-project/s3s.git
           rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
 
-      - name: Build debug binary
-        run: |
-          touch rustfs/build.rs
-          # Limit concurrency to prevent OOM
-          cargo build -p rustfs --bins --jobs 2
-
       - name: Run end-to-end tests
         run: |
           s3s-e2e --version
@@ -184,7 +212,7 @@ jobs:
 
   s3-implemented-tests:
     name: S3 Implemented Tests
-    needs: skip-check
+    needs: [ skip-check, build-rustfs-debug-binary ]
     if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-4
     timeout-minutes: 60
@@ -192,18 +220,14 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v6
 
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
         with:
-          rust-version: stable
-          cache-shared-key: ci-s3tests-${{ hashFiles('**/Cargo.lock') }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          name: rustfs-debug-binary
+          path: target/debug
 
-      - name: Build debug binary
-        run: |
-          touch rustfs/build.rs
-          cargo build -p rustfs --bins --jobs 2
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
 
       - name: Run implemented s3-tests
         run: |

.github/workflows/docker.yml

@@ -66,6 +66,7 @@ env:
   CARGO_TERM_COLOR: always
   REGISTRY_DOCKERHUB: rustfs/rustfs
   REGISTRY_GHCR: ghcr.io/${{ github.repository }}
+  REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
   DOCKER_PLATFORMS: linux/amd64,linux/arm64
 
 jobs:
@@ -287,12 +288,19 @@ jobs:
           username: ${{ env.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
-      # - name: Login to GitHub Container Registry
-      #   uses: docker/login-action@v3
-      #   with:
-      #     registry: ghcr.io
-      #     username: ${{ github.actor }}
-      #     password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ secrets.GHCR_USERNAME }}
+          password: ${{ secrets.GHCR_PASSWORD }}
+
+      - name: Login to Quay.io
+        uses: docker/login-action@v3
+        with:
+          registry: quay.io
+          username: ${{ secrets.QUAY_USERNAME }}
+          password: ${{ secrets.QUAY_PASSWORD }}
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -338,14 +346,15 @@ jobs:
 
           # Generate tags based on build type
           # Only support release and prerelease builds (no development builds)
-          TAGS="${{ env.REGISTRY_DOCKERHUB }}:${VERSION}${VARIANT_SUFFIX}"
+          TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
+          TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
 
           # Add channel tags for prereleases and latest for stable
           if [[ "$CREATE_LATEST" == "true" ]]; then
             # TODO: Temporary change - the current alpha version will also create the latest tag
             # After the version is stabilized, the logic here remains unchanged, but the upstream CREATE_LATEST setting needs to be restored.
             # Stable release (and temporary alpha versions)
-            TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX}"
+            TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
           elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
             # Prerelease channel tags (alpha, beta, rc)
             if [[ "$VERSION" == *"alpha"* ]]; then
@@ -357,7 +366,7 @@ jobs:
             fi
 
             if [[ -n "$CHANNEL" ]]; then
-              TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX}"
+              TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
             fi
           fi
 

.github/workflows/e2e-s3tests.yml

@@ -50,6 +50,11 @@ env:
 
   RUST_LOG: info
   PLATFORM: linux/amd64
+  BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
+  cancel-in-progress: true
 
 defaults:
   run:
@@ -57,12 +62,25 @@ defaults:
 
 jobs:
   s3tests-single:
-    if: github.event.inputs.test-mode == 'single'
+    if: github.event.inputs['test-mode'] == 'single'
     runs-on: ubicloud-standard-2
     timeout-minutes: 120
     steps:
       - uses: actions/checkout@v6
 
+      - name: Cache pip downloads
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-e2e-s3tests-
+
+      - name: Install Python tools
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl tox
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
       - name: Enable buildx
         uses: docker/setup-buildx-action@v3
 
@@ -70,8 +88,8 @@ jobs:
         run: |
           DOCKER_BUILDKIT=1 docker buildx build --load \
             --platform ${PLATFORM} \
-            --cache-from type=gha \
-            --cache-to type=gha,mode=max \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
             -t rustfs-ci \
             -f Dockerfile.source .
 
@@ -121,9 +139,6 @@ jobs:
 
       - name: Provision s3-tests alt user (required by suite)
         run: |
-          python3 -m pip install --user --upgrade pip awscurl
-          export PATH="$HOME/.local/bin:$PATH"
-
           # Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well.
           awscurl \
             --service s3 \
@@ -156,8 +171,6 @@ jobs:
 
       - name: Prepare s3-tests
         run: |
-          python3 -m pip install --user --upgrade pip tox
-          export PATH="$HOME/.local/bin:$PATH"
           git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
 
       - name: Run ceph s3-tests (debug friendly)
@@ -211,12 +224,25 @@ jobs:
           path: artifacts/**
 
   s3tests-multi:
-    if: github.event_name == 'workflow_dispatch' && github.event.inputs.test-mode == 'multi'
+    if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi'
     runs-on: ubicloud-standard-2
     timeout-minutes: 150
     steps:
       - uses: actions/checkout@v6
 
+      - name: Cache pip downloads
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-e2e-s3tests-
+
+      - name: Install Python tools
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl tox
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
       - name: Enable buildx
         uses: docker/setup-buildx-action@v3
 
@@ -224,8 +250,8 @@ jobs:
         run: |
           DOCKER_BUILDKIT=1 docker buildx build --load \
             --platform ${PLATFORM} \
-            --cache-from type=gha \
-            --cache-to type=gha,mode=max \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
             -t rustfs-ci \
             -f Dockerfile.source .
 
@@ -337,9 +363,6 @@ jobs:
 
       - name: Provision s3-tests alt user (required by suite)
         run: |
-          python3 -m pip install --user --upgrade pip awscurl
-          export PATH="$HOME/.local/bin:$PATH"
-
           awscurl \
             --service s3 \
             --region "${S3_REGION}" \
@@ -368,8 +391,6 @@ jobs:
 
       - name: Prepare s3-tests
         run: |
-          python3 -m pip install --user --upgrade pip tox
-          export PATH="$HOME/.local/bin:$PATH"
           git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
 
       - name: Run ceph s3-tests (multi, debug friendly)

.github/workflows/nix-flake-update.yml

@@ -31,11 +31,11 @@ jobs:
   update-flake:
     name: Update flake.lock
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 45
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
-
+        
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@v3
 
@@ -46,14 +46,18 @@ jobs:
         id: update
         uses: DeterminateSystems/update-flake-lock@main
         with:
+          git-author-name: heihutu
+          git-author-email: heihutu@gmail.com
+          git-committer-name: heihutu
+          git-committer-email: heihutu@gmail.com
           pr-title: "chore(deps): update flake.lock"
           pr-labels: |
             dependencies
             nix
             automated
           commit-msg: "chore(deps): update flake.lock"
-          pr-reviewers: houseme, heihutu
-          token: ${{ secrets.GITHUB_TOKEN }}
+          pr-reviewers: houseme, overtrue, majinghe
+          token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
 
       - name: Log PR details
         if: steps.update.outputs.pull-request-number

.github/workflows/nix.yml

@@ -43,7 +43,7 @@ jobs:
   nix-validation:
     name: Nix Build & Check
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 45
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -57,7 +57,7 @@ jobs:
 
       - name: Setup Magic Nix Cache
         uses: DeterminateSystems/magic-nix-cache-action@v13
-        
+
       - name: Setup Flake Checker
         uses: DeterminateSystems/flake-checker-action@v12
 

.gitignore

@@ -16,6 +16,7 @@ vendor
 cli/rustfs-gui/embedded-rustfs/rustfs
 *.log
 deploy/certs/*
+deploy/data/*
 *jsonl
 .env
 .rustfs.sys
@@ -45,3 +46,6 @@ docs
 # nix stuff
 result*
 *.gz
+rustfs-webdav.code-workspace
+
+.aiexclude

AGENTS.md

@@ -1,103 +1,67 @@
-# Repository Guidelines
+# RustFS Agent Instructions (Global)
 
-This file provides guidance for AI agents and developers working with code in this repository.
+This root file keeps repository-wide rules only.
+Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
 
-## Project Overview
+## Rule Precedence
 
-RustFS is a high-performance distributed object storage software built with Rust, providing S3-compatible APIs and advanced features like data lakes, AI, and big data support. It's designed as an alternative to MinIO with better performance and a more business-friendly Apache 2.0 license.
+1. System/developer instructions.
+2. This file (global defaults).
+3. The nearest `AGENTS.md` in the current path (more specific scope wins).
 
-## ⚠️ Pre-Commit Checklist (MANDATORY)
+If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
+
+## Communication and Language
+
+- Respond in the same language used by the requester.
+- Keep source code, comments, commit messages, and PR title/body in English.
+
+## Sources of Truth
+
+- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
+- Local quality commands: `Makefile` and `.config/make/`
+- CI quality gates: `.github/workflows/ci.yml`
+- PR template: `.github/pull_request_template.md`
+
+Avoid duplicating long crate lists or command matrices in instruction files.
+Reference the source files above instead.
+
+## Mandatory Before Commit
+
+Run and pass:
 
-**Before EVERY commit, you MUST run and pass ALL of the following:**
 ```bash
-cargo fmt --all --check    # Code formatting
-cargo clippy --all-targets --all-features -- -D warnings  # Lints
-cargo test --workspace --exclude e2e_test  # Unit tests
+make pre-commit
 ```
-Or simply run `make pre-commit` which covers all checks. **DO NOT commit if any check fails.**
 
-## Communication Rules
+If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
+Do not commit when required checks fail.
 
-- Respond to the user in the same language used by the user.
-- Code and documentation must be written in English only.
-- **Pull Request titles and descriptions must be written in English** to ensure consistency and accessibility for all contributors.
+## Git and PR Baseline
 
-## Project Structure
+- Use feature branches based on the latest `main`.
+- Follow Conventional Commits, with subject length <= 72 characters.
+- Keep PR title and description in English.
+- Use `.github/pull_request_template.md` and keep all section headings.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands in the PR description.
+- When using `gh pr create`/`gh pr edit`, use `--body-file` instead of inline `--body` for multiline markdown.
 
-The workspace root hosts shared dependencies in `Cargo.toml`. The service binary lives under `rustfs/src/main.rs`, while reusable crates sit in `crates/` (including `ecstore`, `iam`, `kms`, `madmin`, `s3select-api`, `s3select-query`, `config`, `crypto`, `lock`, `filemeta`, `rio`, `common`, `protos`, `audit-logger`, `notify`, `obs`, `workers`, `appauth`, `ahm`, `mcp`, `signer`, `checksums`, `utils`, `zip`, `targets`, and `e2e_test`). Deployment manifests are under `deploy/`, Docker assets sit at the root, and automation lives in `scripts/`.
+## Security Baseline
 
-### Core Architecture
+- Never commit secrets, credentials, or key material.
+- Use environment variables or vault tooling for sensitive configuration.
+- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
-- **Main Binary (`rustfs/`):** Entry point at `rustfs/src/main.rs`, includes admin, auth, config, server, storage, license management, and profiling modules.
-- **Key Crates:** Cargo workspace with 25+ crates supporting S3-compatible APIs, erasure coding storage, IAM, KMS, S3 Select, and observability.
-- **Build System:** Custom `build-rustfs.sh` script, multi-architecture Docker builds, Make/Just task runners, cross-compilation support.
+## Scoped Guidance in This Repository
 
-## Build, Test, and Development Commands
-
-### Quick Commands
-- `cargo check --all-targets` - Fast validation
-- `cargo build --release` or `make build` - Release build
-- `./build-rustfs.sh --dev` - Development build with debug symbols
-- `make pre-commit` - Run all quality checks (fmt, clippy, check, test)
-
-### Testing
-- `cargo test --workspace --exclude e2e_test` - Unit tests
-- `cargo test --package e2e_test` - E2E tests
-- For KMS tests: `NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1`
-
-### Cross-Platform Builds
-- `./build-rustfs.sh --platform x86_64-unknown-linux-musl` - Build for musl
-- `./build-rustfs.sh --platform aarch64-unknown-linux-gnu` - Build for ARM64
-- `make build-cross-all` - Build all supported architectures
-
-## Coding Style & Safety Requirements
-
-- **Formatting:** Follow `rustfmt.toml` (130-column width). Use `snake_case` for items, `PascalCase` for types, `SCREAMING_SNAKE_CASE` for constants.
-- **Safety:** `unsafe_code = "deny"` enforced at workspace level. Never use `unwrap()`, `expect()`, or panic-inducing code except in tests.
-- **Error Handling:** Prefer `anyhow` for applications, `thiserror` for libraries. Use proper error handling with `Result<T, E>` and `Option<T>`.
-- **Async:** Keep async code non-blocking. Offload CPU-heavy work with `tokio::task::spawn_blocking` when necessary.
-- **Language:** Code comments, function names, variable names, and all text in source files must be in English only.
-
-## Testing Guidelines
-
-Co-locate unit tests with their modules and give behavior-led names. Integration suites belong in each crate's `tests/` directory, while exhaustive end-to-end scenarios live in `crates/e2e_test/`. When fixing bugs or adding features, include regression tests that capture the new behavior.
-
-## KMS (Key Management Service)
-
-- **Implementation:** Complete with Local and Vault backends, auto-configures on startup with `--kms-enable` flag.
-- **Encryption:** Full S3-compatible server-side encryption (SSE-S3, SSE-KMS, SSE-C).
-- **Testing:** Comprehensive E2E tests in `crates/e2e_test/src/kms/`. Requires proxy bypass for local testing.
-- **Key Files:** `crates/kms/`, `rustfs/src/storage/ecfs.rs`, `rustfs/src/admin/handlers/kms*.rs`
-
-## Environment Variables
-
-- Global configuration environment variables must use flat `RUSTFS_*` names (no module segments), such as `RUSTFS_REGION`, `RUSTFS_ADDRESS`, `RUSTFS_VOLUMES`, and `RUSTFS_LICENSE`.
-- `RUSTFS_SCANNER_ENABLED` - Enable/disable background data scanner (default: true)
-- `RUSTFS_HEAL_ENABLED` - Enable/disable auto-heal functionality (default: true)
-- Deprecated aliases (for pre-beta compatibility) are documented in [the config module README](crates/config/README.md#environment-variable-naming-conventions) and must log warnings when used.
-- For KMS tests: `NO_PROXY=127.0.0.1,localhost` and clear proxy environment variables
-
-## Commit & Pull Request Guidelines
-
-Work on feature branches (e.g., `feat/...`) after syncing `main`. Follow Conventional Commits under 72 characters. Each commit must compile, format cleanly, and pass `make pre-commit`.
-
-**Pull Request Requirements:**
-- PR titles and descriptions **MUST be written in English**
-- Open PRs with a concise summary, note verification commands, link relevant issues
-- Follow the PR template format and fill in all required sections
-- Wait for reviewer approval before merging
-
-## Security & Configuration Tips
-
-Do not commit secrets or cloud credentials; prefer environment variables or vault tooling. Review IAM- and KMS-related changes with a second maintainer. Confirm proxy settings before running sensitive tests to avoid leaking traffic outside localhost.
-
-## Important Reminders
-
-- Always compile after code changes: Use `cargo build` to catch errors early
-- Don't bypass tests: All functionality must be properly tested, not worked around
-- Use proper error handling: Never use `unwrap()` or `expect()` in production code (except tests)
-- Do what has been asked; nothing more, nothing less
-- NEVER create files unless they're absolutely necessary for achieving your goal
-- ALWAYS prefer editing an existing file to creating a new one
-- NEVER proactively create documentation files (*.md) or README files unless explicitly requested
-- NEVER commit PR description files (e.g., PR_DESCRIPTION.md): These are temporary reference files for creating pull requests and should remain local only
+- `.github/AGENTS.md`
+- `crates/AGENTS.md`
+- `crates/config/AGENTS.md`
+- `crates/ecstore/AGENTS.md`
+- `crates/e2e_test/AGENTS.md`
+- `crates/iam/AGENTS.md`
+- `crates/kms/AGENTS.md`
+- `crates/policy/AGENTS.md`
+- `rustfs/src/admin/AGENTS.md`
+- `rustfs/src/storage/AGENTS.md`

CHANGELOG.md

@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
+
 ### Added
 - **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
   - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate

Cargo.toml

@@ -39,6 +39,7 @@ members = [
     "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
+    "crates/s3-common", # Common utilities and data structures for S3 compatibility
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
     "crates/scanner", # Scanner for data integrity checks and health monitoring
@@ -49,7 +50,7 @@ members = [
     "crates/workers", # Worker thread pools and task scheduling
     "crates/zip", # ZIP file handling and compression
 ]
-resolver = "2"
+resolver = "3"
 
 [workspace.package]
 edition = "2024"
@@ -94,6 +95,7 @@ rustfs-obs = { path = "crates/obs", version = "0.0.5" }
 rustfs-policy = { path = "crates/policy", version = "0.0.5" }
 rustfs-protos = { path = "crates/protos", version = "0.0.5" }
 rustfs-rio = { path = "crates/rio", version = "0.0.5" }
+rustfs-s3-common = { path = "crates/s3-common", version = "0.0.5" }
 rustfs-s3select-api = { path = "crates/s3select-api", version = "0.0.5" }
 rustfs-s3select-query = { path = "crates/s3select-query", version = "0.0.5" }
 rustfs-scanner = { path = "crates/scanner", version = "0.0.5" }
@@ -122,8 +124,8 @@ http = "1.4.0"
 http-body = "1.0.1"
 http-body-util = "0.1.3"
 reqwest = { version = "0.13.2", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
-socket2 = { version = "0.6.2", features = ["all"] }
-tokio = { version = "1.49.0", features = ["fs", "rt-multi-thread"] }
+socket2 = { version = "0.6.3", features = ["all"] }
+tokio = { version = "1.50.0", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
@@ -142,10 +144,11 @@ flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
 prost = "0.14.3"
 quick-xml = "0.39.2"
-rmcp = { version = "0.17.0" }
+rmcp = { version = "1.2.0" }
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
+serde_bytes = "0.11"
 serde_json = { version = "1.0.149", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
 schemars = "1.2.1"
@@ -153,6 +156,7 @@ schemars = "1.2.1"
 # Cryptography and Security
 aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
 argon2 = { version = "0.6.0-rc.7" }
+blake2 = "0.11.0-rc.5"
 blake3 = { version = "1.8.3", features = ["rayon", "mmap"] }
 chacha20poly1305 = { version = "0.11.0-rc.3" }
 crc-fast = "1.9.0"
@@ -160,7 +164,7 @@ hmac = { version = "0.13.0-rc.5" }
 jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
 openidconnect = { version = "4.0", default-features = false }
 pbkdf2 = "0.13.0-rc.9"
-rsa = { version = "0.10.0-rc.16" }
+rsa = { version = "0.10.0-rc.17" }
 rustls = { version = "0.23.37", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
 rustls-pki-types = "1.14.0"
 sha1 = "0.11.0-rc.5"
@@ -171,44 +175,48 @@ zeroize = { version = "1.8.2", features = ["derive"] }
 # Time and Date
 chrono = { version = "0.4.44", features = ["serde"] }
 humantime = "2.3.0"
-jiff = { version = "0.2.22", features = ["serde"] }
+jiff = { version = "0.2.23", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Utilities and Tools
 anyhow = "1.0.102"
 arc-swap = "1.8.2"
-astral-tokio-tar = "0.5.6"
+astral-tokio-tar = "0.6.0"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.14" }
-aws-credential-types = { version = "1.2.13" }
-aws-sdk-s3 = { version = "1.124.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-http-client = { version = "1.1.11", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.5" }
+aws-config = { version = "1.8.15" }
+aws-credential-types = { version = "1.2.14" }
+aws-sdk-s3 = { version = "1.126.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-types = { version = "1.4.6" }
 backtrace = "0.3.76"
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
 cfg-if = "1.0.4"
-clap = { version = "4.5.60", features = ["derive", "env"] }
+clap = { version = "4.6.0", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
 criterion = { version = "0.8", features = ["html_reports"] }
 crossbeam-queue = "0.3.12"
-datafusion = "52.2.0"
+crossbeam-channel = "0.5.15"
+crossbeam-deque = "0.8.6"
+crossbeam-utils = "0.8.21"
+datafusion = "52.3.0"
 derive_builder = "0.20.2"
 enumset = "1.1.10"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.8.0"
-google-cloud-auth = "1.6.0"
+google-cloud-storage = "1.9.0"
+google-cloud-auth = "1.7.0"
 hashbrown = { version = "0.16.1", features = ["serde", "rayon"] }
+hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
-libc = "0.2.182"
+libc = "0.2.183"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.10"
 lz4 = "1.28.1"
@@ -224,47 +232,49 @@ object_store = "0.12.5"
 parking_lot = "0.12.5"
 path-absolutize = "3.1.1"
 path-clean = "1.0.1"
+percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.0", features = ["serde"] }
 ratelimit = "0.10.0"
 rayon = "1.11.0"
-reed-solomon-simd = { version = "3.1.0" }
+reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
+reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
 rumqttc = { version = "0.25.1" }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.1" }
-s3s = { version = "0.13.0", features = ["minio"] }
+s3s = { git = "https://github.com/rustfs/s3s", rev = "d9556e3c0036bd3f2b330966009cbaa5aebf19a3", features = ["minio"] }
 serial_test = "3.4.0"
-shadow-rs = { version = "1.7.0", default-features = false }
+shadow-rs = { version = "1.7.1", default-features = false }
 siphasher = "1.0.2"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
-snafu = "0.8.9"
+snafu = "0.9.0"
 snap = "1.1.1"
 starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.38.3"
+sysinfo = "0.38.4"
 temp-env = "0.3.6"
-tempfile = "3.26.0"
+tempfile = "3.27.0"
 test-case = "3.3.1"
 thiserror = "2.0.18"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.4"
 tracing-error = "0.2.1"
 tracing-opentelemetry = "0.32.1"
-tracing-subscriber = { version = "0.3.22", features = ["env-filter", "time"] }
+tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.21.0", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.22.0", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.7.4" }
 walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
 windows = { version = "0.62.2" }
 xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
-zip = "8.1.0"
+zip = "8.2.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
@@ -275,6 +285,7 @@ opentelemetry-otlp = { version = "0.31.0", features = ["gzip-http", "reqwest-rus
 opentelemetry_sdk = { version = "0.31.0" }
 opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.31.0" }
+pyroscope = { version = "2.0.0", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
@@ -282,6 +293,9 @@ unftp-core = "0.1.0"
 suppaftp = { version = "8.0.2", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.7"
 
+# WebDAV
+dav-server = "0.11.0"
+
 # Performance Analysis and Memory Profiling
 mimalloc = "0.1"
 # Use tikv-jemallocator as memory allocator and enable performance analysis
@@ -291,7 +305,9 @@ tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profilin
 # Used to generate pprof-compatible memory profiling data and support symbolization and flame graphs
 jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
-pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
+# pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
+# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
+pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs", "rustfs-mcp"]

Dockerfile

@@ -86,12 +86,7 @@ RUN addgroup -g 10001 -S rustfs && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_CONSOLE_ADDRESS=":9001" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_CORS_ALLOWED_ORIGINS="*" \
+ENV RUSTFS_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \

Dockerfile.decommission-local

@@ -0,0 +1,65 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM rust:1.91-trixie
+
+RUN set -eux; \
+    export DEBIAN_FRONTEND=noninteractive; \
+    apt-get update; \
+    apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    curl \
+    git \
+    pkg-config \
+    libssl-dev \
+    lld \
+    protobuf-compiler \
+    flatbuffers-compiler; \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /usr/src/rustfs
+
+COPY . .
+
+RUN ./scripts/static.sh
+RUN cargo run --bin gproto
+RUN cargo build --release --locked --bin rustfs
+
+RUN set -eux; \
+    groupadd -g 10001 rustfs; \
+    useradd -u 10001 -g rustfs -M -s /usr/sbin/nologin rustfs
+
+WORKDIR /app
+
+RUN set -eux; \
+    mkdir -p /data /logs; \
+    chown -R rustfs:rustfs /data /logs /app; \
+    chmod 0750 /data /logs
+
+COPY entrypoint.sh /entrypoint.sh
+RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && chmod +x /entrypoint.sh
+
+ENV RUSTFS_VOLUMES="/data" \
+    RUST_LOG="warn" \
+    RUSTFS_OBS_LOG_DIRECTORY="/logs" \
+    RUSTFS_USERNAME="rustfs" \
+    RUSTFS_GROUPNAME="rustfs" \
+    RUSTFS_UID="10001" \
+    RUSTFS_GID="10001"
+
+EXPOSE 9000 9001
+
+ENTRYPOINT ["/entrypoint.sh"]
+CMD ["/usr/bin/rustfs"]

Dockerfile.source

@@ -157,11 +157,7 @@ WORKDIR /app
 ENV CARGO_INCREMENTAL=1
 
 # Ensure we have the same default env vars available
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_VOLUMES="/data" \
+ENV RUSTFS_VOLUMES="/data" \
     RUST_LOG="warn" \
     RUSTFS_OBS_LOG_DIRECTORY="/logs" \
     RUSTFS_USERNAME="rustfs" \
@@ -222,11 +218,7 @@ COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
 # Default environment (override in docker run/compose as needed)
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_VOLUMES="/data" \
+ENV RUSTFS_VOLUMES="/data" \
     RUST_LOG="warn" \
     RUSTFS_USERNAME="rustfs" \
     RUSTFS_GROUPNAME="rustfs" \

README.md

@@ -1,4 +1,4 @@
-[![RustFS](https://github.com/user-attachments/assets/3ba82e75-2f2d-4415-a4aa-1e4ffe9f22fd)](https://rustfs.com)
+[![RustFS](https://repository-images.githubusercontent.com/722597620/0fa936a2-8164-4f53-867f-def4beb64b21)](https://rustfs.com)
 
 <p align="center">RustFS is a high-performance, distributed object storage system built in Rust.</p>
 
@@ -42,6 +42,7 @@ Unlike other storage systems, RustFS is released under the permissible Apache 2.
 - **High Performance**: Built with Rust to ensure maximum speed and resource efficiency.
 - **Distributed Architecture**: Scalable and fault-tolerant design suitable for large-scale deployments.
 - **S3 Compatibility**: Seamless integration with existing S3-compatible applications and tools.
+- **OpenStack Swift API**: Native support for Swift protocol with Keystone authentication.
 - **OpenStack Keystone Integration**: Native support for OpenStack Keystone authentication with X-Auth-Token headers.
 - **Data Lake Support**: Optimized for high-throughput big data and AI workloads.
 - **Open Source**: Licensed under Apache 2.0, encouraging unrestricted community contributions and commercial usage.
@@ -56,6 +57,7 @@ Unlike other storage systems, RustFS is released under the permissible Apache 2.
 | **Event Notifications** | ✅ Available | **Distributed Mode**     | 🚧 Under Testing |
 | **K8s Helm Charts**     | ✅ Available | **RustFS KMS**           | 🚧 Under Testing |
 | **Keystone Auth**       | ✅ Available | **Multi-Tenancy**        | ✅ Available     |
+| **Swift API**           | ✅ Available | **Swift Metadata Ops**   | 🚧 Partial       |
 
 ## RustFS vs MinIO Performance
 

README_ZH.md

@@ -100,7 +100,7 @@ curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
 
 ### 2\. Docker 快速启动 (选项 2)
 
-RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用 Docker 的 `-v` 参数挂载宿主机目录，请务必确保宿主机目录的所有者已更改为 `1000`，否则会遇到权限拒绝错误。
+RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用 Docker 的 `-v` 参数挂载宿主机目录，请务必确保宿主机目录的所有者已更改为 `10001`，否则会遇到权限拒绝错误。
 
 ```bash
  # 创建数据和日志目录

_typos.toml

@@ -42,6 +42,8 @@ GAE = "GAE"
 # s3-tests original test names (cannot be changed)
 nonexisted = "nonexisted"
 consts = "consts"
+# Swift API - company/product names
+Hashi = "Hashi"  # HashiCorp
 
 [files]
 extend-exclude = []

crates/AGENTS.md

@@ -0,0 +1,20 @@
+# Crates Instructions
+
+Applies to all paths under `crates/`.
+
+## Library Design
+
+- Treat crate code as reusable library code by default.
+- Prefer `thiserror` for library-facing error types.
+- Do not use `unwrap()`, `expect()`, or panic-driven control flow outside tests.
+
+## Testing
+
+- Keep unit tests close to the module they test.
+- Keep integration tests under each crate's `tests/` directory.
+- Add regression tests for bug fixes and behavior changes.
+
+## Async and Performance
+
+- Keep async paths non-blocking.
+- Move CPU-heavy operations out of async hot paths with `tokio::task::spawn_blocking` when appropriate.

crates/audit/Cargo.toml

@@ -29,6 +29,7 @@ categories = ["web-programming", "development-tools", "asynchronous", "api-bindi
 rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
+rustfs-s3-common = { workspace = true }
 async-trait = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }

crates/audit/src/entity.rs

@@ -14,7 +14,7 @@
 
 use chrono::{DateTime, Utc};
 use hashbrown::HashMap;
-use rustfs_targets::EventName;
+use rustfs_s3_common::EventName;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
 

crates/common/src/data_usage.rs

@@ -458,10 +458,7 @@ impl DataUsageEntry {
         self.size += other.size;
 
         if let Some(o_rep) = &other.replication_stats {
-            if self.replication_stats.is_none() {
-                self.replication_stats = Some(ReplicationAllStats::default());
-            }
-            let s_rep = self.replication_stats.as_mut().unwrap();
+            let s_rep = self.replication_stats.get_or_insert_with(ReplicationAllStats::default);
             s_rep.targets.clear();
             s_rep.replica_size += o_rep.replica_size;
             s_rep.replica_count += o_rep.replica_count;
@@ -586,7 +583,7 @@ impl DataUsageCache {
                     return Some(root);
                 }
                 let mut flat = self.flatten(&root);
-                if flat.replication_stats.is_some() && flat.replication_stats.as_ref().unwrap().empty() {
+                if flat.replication_stats.as_ref().is_some_and(|stats| stats.empty()) {
                     flat.replication_stats = None;
                 }
                 Some(flat)
@@ -679,7 +676,9 @@ impl DataUsageCache {
         leaves.sort_by(|a, b| a.objects.cmp(&b.objects));
 
         while remove > 0 && !leaves.is_empty() {
-            let e = leaves.first().unwrap();
+            let Some(e) = leaves.first() else {
+                break;
+            };
             let candidate = e.path.clone();
             if candidate == *path && !compact_self {
                 break;
@@ -703,12 +702,9 @@ impl DataUsageCache {
     }
 
     pub fn total_children_rec(&self, path: &str) -> usize {
-        let root = self.find(path);
-
-        if root.is_none() {
+        let Some(root) = self.find(path) else {
             return 0;
-        }
-        let root = root.unwrap();
+        };
         if root.children.is_empty() {
             return 0;
         }
@@ -721,31 +717,36 @@ impl DataUsageCache {
     }
 
     pub fn merge(&mut self, o: &DataUsageCache) {
-        let mut existing_root = self.root();
-        let other_root = o.root();
-        if existing_root.is_none() && other_root.is_none() {
-            return;
-        }
-        if other_root.is_none() {
-            return;
-        }
-        if existing_root.is_none() {
+        let Some(mut existing_root) = self.root() else {
+            if o.root().is_none() {
+                return;
+            }
             *self = o.clone();
             return;
-        }
-        if o.info.last_update.gt(&self.info.last_update) {
+        };
+
+        let Some(other_root) = o.root() else {
+            return;
+        };
+
+        if o.info.last_update > self.info.last_update {
             self.info.last_update = o.info.last_update;
         }
 
-        existing_root.as_mut().unwrap().merge(other_root.as_ref().unwrap());
-        self.cache.insert(hash_path(&self.info.name).key(), existing_root.unwrap());
-        let e_hash = self.root_hash();
-        for key in other_root.as_ref().unwrap().children.iter() {
-            let entry = &o.cache[key];
+        existing_root.merge(&other_root);
+        self.cache.insert(hash_path(&self.info.name).key(), existing_root);
+
+        let root_hash = self.root_hash();
+        for key in other_root.children.iter() {
+            let Some(entry) = o.cache.get(key) else {
+                continue;
+            };
             let flat = o.flatten(entry);
-            let mut existing = self.cache[key].clone();
-            existing.merge(&flat);
-            self.replace_hashed(&DataUsageHash(key.clone()), &Some(e_hash.clone()), &existing);
+            if let Some(existing) = self.cache.get_mut(key) {
+                existing.merge(&flat);
+            } else {
+                self.replace_hashed(&DataUsageHash(key.clone()), &Some(root_hash.clone()), &flat);
+            }
         }
     }
 
@@ -1141,10 +1142,12 @@ impl DataUsageInfo {
         self.buckets_count = self.buckets_usage.len() as u64;
 
         // Update last update time
-        if let Some(other_update) = other.last_update
-            && (self.last_update.is_none() || other_update > self.last_update.unwrap())
-        {
-            self.last_update = Some(other_update);
+        if let Some(other_update) = other.last_update {
+            match self.last_update {
+                None => self.last_update = Some(other_update),
+                Some(self_update) if other_update > self_update => self.last_update = Some(other_update),
+                _ => {}
+            }
         }
     }
 }
@@ -1285,4 +1288,59 @@ mod tests {
         assert_eq!(summary1.total_size, 300);
         assert_eq!(summary1.versions, 15);
     }
+
+    #[test]
+    fn test_data_usage_cache_merge_adds_missing_child() {
+        let mut base = DataUsageCache::default();
+        base.info.name = "bucket".to_string();
+        base.replace("bucket", "", DataUsageEntry::default());
+
+        let mut other = DataUsageCache::default();
+        other.info.name = "bucket".to_string();
+        let child = DataUsageEntry {
+            size: 42,
+            ..Default::default()
+        };
+        other.replace("bucket/child", "bucket", child);
+
+        base.merge(&other);
+
+        let root = base.find("bucket").expect("root bucket should exist");
+        assert_eq!(root.size, 0);
+        let child_entry = base.find("bucket/child").expect("merged child should be added");
+        assert_eq!(child_entry.size, 42);
+    }
+
+    #[test]
+    fn test_data_usage_cache_merge_accumulates_existing_child() {
+        let mut base = DataUsageCache::default();
+        base.info.name = "bucket".to_string();
+        base.replace(
+            "bucket/child",
+            "bucket",
+            DataUsageEntry {
+                size: 10,
+                objects: 1,
+                ..Default::default()
+            },
+        );
+
+        let mut other = DataUsageCache::default();
+        other.info.name = "bucket".to_string();
+        other.replace(
+            "bucket/child",
+            "bucket",
+            DataUsageEntry {
+                size: 20,
+                objects: 2,
+                ..Default::default()
+            },
+        );
+
+        base.merge(&other);
+
+        let child_entry = base.find("bucket/child").expect("child should remain after merge");
+        assert_eq!(child_entry.size, 30);
+        assert_eq!(child_entry.objects, 3);
+    }
 }

crates/common/src/heal_channel.rs

@@ -296,12 +296,13 @@ type HealResponseSender = broadcast::Sender<HealChannelResponse>;
 static GLOBAL_HEAL_RESPONSE_SENDER: OnceLock<HealResponseSender> = OnceLock::new();
 
 /// Initialize global heal channel
-pub fn init_heal_channel() -> HealChannelReceiver {
+pub fn init_heal_channel() -> Result<HealChannelReceiver, &'static str> {
     let (tx, rx) = mpsc::unbounded_channel();
-    GLOBAL_HEAL_CHANNEL_SENDER
-        .set(tx)
-        .expect("Heal channel sender already initialized");
-    rx
+    if GLOBAL_HEAL_CHANNEL_SENDER.set(tx).is_ok() {
+        Ok(rx)
+    } else {
+        Err("Heal channel sender already initialized")
+    }
 }
 
 /// Get global heal channel sender

crates/common/src/last_minute.rs

@@ -155,10 +155,7 @@ impl LastMinuteLatency {
     }
 
     pub fn add(&mut self, t: &Duration) {
-        let sec = SystemTime::now()
-            .duration_since(UNIX_EPOCH)
-            .expect("Time went backwards")
-            .as_secs();
+        let sec = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs();
         self.forward_to(sec);
         let win_idx = sec % 60;
         self.totals[win_idx as usize].add(t);
@@ -174,10 +171,7 @@ impl LastMinuteLatency {
 
     pub fn get_total(&mut self) -> AccElem {
         let mut res = AccElem::default();
-        let sec = SystemTime::now()
-            .duration_since(UNIX_EPOCH)
-            .expect("Time went backwards")
-            .as_secs();
+        let sec = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs();
         self.forward_to(sec);
         for elem in self.totals.iter() {
             res.merge(elem);

crates/config/AGENTS.md

@@ -0,0 +1,28 @@
+# Config Crate Instructions
+
+Applies to `crates/config/`.
+
+## Environment Variable Naming
+
+- Global configuration variables must use flat `RUSTFS_*` names.
+- Do not introduce module-segmented names such as `RUSTFS_CONFIG_*`.
+
+Canonical examples:
+
+- `RUSTFS_REGION`
+- `RUSTFS_ADDRESS`
+- `RUSTFS_VOLUMES`
+- `RUSTFS_LICENSE`
+- `RUSTFS_SCANNER_ENABLED`
+- `RUSTFS_HEAL_ENABLED`
+
+## Compatibility
+
+- Deprecated aliases must keep warning behavior.
+- Document aliases in `crates/config/README.md`.
+- Any alias change should include tests for both canonical and deprecated forms.
+
+## Source of Truth
+
+- Constants: `crates/config/src/constants/app.rs`
+- Naming conventions: `crates/config/README.md#environment-variable-naming-conventions`

crates/config/README.md

@@ -49,6 +49,12 @@ Current guidance:
 - Deprecated example:
   - `RUSTFS_ENABLE_SCANNER` -> `RUSTFS_SCANNER_ENABLED`
   - `RUSTFS_ENABLE_HEAL` -> `RUSTFS_HEAL_ENABLED`
+  - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` -> `RUSTFS_SCANNER_START_DELAY_SECS`
+
+## Scanner environment aliases
+
+- `RUSTFS_SCANNER_START_DELAY_SECS` (canonical)
+- `RUSTFS_DATA_SCANNER_START_DELAY_SECS` (deprecated alias for compatibility)
 
 ## 📄 License
 

crates/config/src/constants/app.rs

@@ -131,6 +131,12 @@ pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
 /// Environment variable for server volumes.
 pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
 
+/// Environment variable for server tls path.
+pub const ENV_RUSTFS_TLS_PATH: &str = "RUSTFS_TLS_PATH";
+
+/// Default value for the server TLS path if `ENV_RUSTFS_TLS_PATH` is not set.
+pub const DEFAULT_RUSTFS_TLS_PATH: &str = "";
+
 /// Default port for rustfs
 /// This is the default port for rustfs.
 /// This is used to bind the server to a specific port.
@@ -221,6 +227,12 @@ pub const DEFAULT_OBS_METRICS_EXPORT_ENABLED: bool = true;
 /// Environment variable: RUSTFS_OBS_LOGS_EXPORT_ENABLED
 pub const DEFAULT_OBS_LOGS_EXPORT_ENABLED: bool = true;
 
+/// Default profiling export enabled
+/// It is used to enable or disable exporting profiles
+/// Default value: true
+/// Environment variable: RUSTFS_OBS_PROFILING_EXPORT_ENABLED
+pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = true;
+
 /// Default log local logging enabled for rustfs
 /// This is the default log local logging enabled for rustfs.
 /// It is used to enable or disable local logging of the application.

crates/config/src/constants/object.rs

@@ -167,3 +167,16 @@ pub const DEFAULT_OBJECT_CACHE_TTI_SECS: u64 = 120;
 ///
 /// Default is set to 5 hits.
 pub const DEFAULT_OBJECT_HOT_MIN_HITS_TO_EXTEND: usize = 5;
+
+/// Skip bitrot hash verification on GetObject reads.
+///
+/// When enabled, GetObject reads skip the per-shard hash
+/// computation and comparison, reducing CPU usage on the read path.
+/// The background scanner still performs full integrity verification.
+/// Does not affect writes, heals, or scanner operations.
+///
+/// Default is false (verify on every read, matching pre-existing behavior).
+pub const ENV_OBJECT_GET_SKIP_BITROT_VERIFY: &str = "RUSTFS_OBJECT_GET_SKIP_BITROT_VERIFY";
+
+/// Default: bitrot verification is enabled on GetObject reads (do not skip).
+pub const DEFAULT_OBJECT_GET_SKIP_BITROT_VERIFY: bool = false;

crates/config/src/constants/protocols.rs

@@ -45,3 +45,15 @@ pub const ENV_FTPS_CERTS_DIR: &str = "RUSTFS_FTPS_CERTS_DIR";
 pub const ENV_FTPS_CA_FILE: &str = "RUSTFS_FTPS_CA_FILE";
 pub const ENV_FTPS_PASSIVE_PORTS: &str = "RUSTFS_FTPS_PASSIVE_PORTS";
 pub const ENV_FTPS_EXTERNAL_IP: &str = "RUSTFS_FTPS_EXTERNAL_IP";
+
+/// Default WebDAV server bind address
+pub const DEFAULT_WEBDAV_ADDRESS: &str = "0.0.0.0:8080";
+
+/// WebDAV environment variable names
+pub const ENV_WEBDAV_ENABLE: &str = "RUSTFS_WEBDAV_ENABLE";
+pub const ENV_WEBDAV_ADDRESS: &str = "RUSTFS_WEBDAV_ADDRESS";
+pub const ENV_WEBDAV_TLS_ENABLED: &str = "RUSTFS_WEBDAV_TLS_ENABLED";
+pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
+pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
+pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
+pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";

crates/config/src/constants/scanner.rs

@@ -12,17 +12,107 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-/// Environment variable name that specifies the data scanner start delay in seconds.
-/// - Purpose: Define the delay between data scanner operations.
+use std::time::Duration;
+
+/// Canonical environment variable name that specifies the scanner start delay in seconds.
+/// If set, this overrides the cycle interval derived from `RUSTFS_SCANNER_SPEED`.
 /// - Unit: seconds (u64).
-/// - Valid values: any positive integer.
-/// - Semantics: This delay controls how frequently the data scanner checks for and processes data; shorter delays lead to more responsive scanning but may increase system load.
-/// - Example: `export RUSTFS_DATA_SCANNER_START_DELAY_SECS=10`
-/// - Note: Choose an appropriate delay that balances scanning responsiveness with overall system performance.
+/// - Example: `export RUSTFS_SCANNER_START_DELAY_SECS=10`
+pub const ENV_SCANNER_START_DELAY_SECS: &str = "RUSTFS_SCANNER_START_DELAY_SECS";
+
+/// Deprecated compatibility alias for scanner start delay.
+/// Prefer `RUSTFS_SCANNER_START_DELAY_SECS`.
+#[deprecated(note = "Use RUSTFS_SCANNER_START_DELAY_SECS instead")]
 pub const ENV_DATA_SCANNER_START_DELAY_SECS: &str = "RUSTFS_DATA_SCANNER_START_DELAY_SECS";
 
-/// Default data scanner start delay in seconds if not specified in the environment variable.
-/// - Value: 10 seconds.
-/// - Rationale: This default interval provides a reasonable balance between scanning responsiveness and system load for most deployments.
-/// - Adjustments: Users may modify this value via the `RUSTFS_DATA_SCANNER_START_DELAY_SECS` environment variable based on their specific scanning requirements and system performance.
-pub const DEFAULT_DATA_SCANNER_START_DELAY_SECS: u64 = 60;
+/// Environment variable that selects the scanner speed preset.
+/// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
+/// Controls the sleep factor, maximum sleep duration, and cycle interval.
+/// - Example: `export RUSTFS_SCANNER_SPEED=slow`
+pub const ENV_SCANNER_SPEED: &str = "RUSTFS_SCANNER_SPEED";
+
+/// Default scanner speed preset.
+pub const DEFAULT_SCANNER_SPEED: &str = "default";
+
+/// Environment variable that controls whether the scanner sleeps between operations.
+/// When `true` (default), the scanner throttles itself. When `false`, it runs at full speed.
+/// - Example: `export RUSTFS_SCANNER_IDLE_MODE=false`
+pub const ENV_SCANNER_IDLE_MODE: &str = "RUSTFS_SCANNER_IDLE_MODE";
+
+/// Default scanner idle mode.
+pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
+
+/// Scanner speed preset controlling throttling behavior.
+///
+/// Each preset defines three parameters:
+/// - **sleep_factor**: Multiplier applied to elapsed work time to compute inter-object sleep.
+/// - **max_sleep**: Upper bound on any single throttle sleep.
+/// - **cycle_interval**: Base delay between scan cycles.
+///
+/// | Preset    | Factor | Max Sleep | Cycle Interval |
+/// |-----------|--------|-----------|----------------|
+/// | `fastest` | 0      | 0         | 1 second       |
+/// | `fast`    | 1x     | 100ms     | 1 minute       |
+/// | `default` | 2x     | 1 second  | 1 minute       |
+/// | `slow`    | 10x    | 15 seconds| 1 minute       |
+/// | `slowest` | 100x   | 15 seconds| 30 minutes     |
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum ScannerSpeed {
+    Fastest,
+    Fast,
+    #[default]
+    Default,
+    Slow,
+    Slowest,
+}
+
+impl ScannerSpeed {
+    pub fn sleep_factor(self) -> f64 {
+        match self {
+            Self::Fastest => 0.0,
+            Self::Fast => 1.0,
+            Self::Default => 2.0,
+            Self::Slow => 10.0,
+            Self::Slowest => 100.0,
+        }
+    }
+
+    pub fn max_sleep(self) -> Duration {
+        match self {
+            Self::Fastest => Duration::ZERO,
+            Self::Fast => Duration::from_millis(100),
+            Self::Default => Duration::from_secs(1),
+            Self::Slow | Self::Slowest => Duration::from_secs(15),
+        }
+    }
+
+    pub fn cycle_interval(self) -> Duration {
+        match self {
+            Self::Fastest => Duration::from_secs(1),
+            Self::Fast | Self::Default | Self::Slow => Duration::from_secs(60),
+            Self::Slowest => Duration::from_secs(30 * 60),
+        }
+    }
+
+    pub fn from_env_str(s: &str) -> Self {
+        match s.trim().to_ascii_lowercase().as_str() {
+            "fastest" => Self::Fastest,
+            "fast" => Self::Fast,
+            "slow" => Self::Slow,
+            "slowest" => Self::Slowest,
+            _ => Self::Default,
+        }
+    }
+}
+
+impl std::fmt::Display for ScannerSpeed {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::Fastest => write!(f, "fastest"),
+            Self::Fast => write!(f, "fast"),
+            Self::Default => write!(f, "default"),
+            Self::Slow => write!(f, "slow"),
+            Self::Slowest => write!(f, "slowest"),
+        }
+    }
+}

crates/config/src/observability/mod.rs

@@ -22,6 +22,7 @@ pub const ENV_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 pub const ENV_OBS_TRACE_ENDPOINT: &str = "RUSTFS_OBS_TRACE_ENDPOINT";
 pub const ENV_OBS_METRIC_ENDPOINT: &str = "RUSTFS_OBS_METRIC_ENDPOINT";
 pub const ENV_OBS_LOG_ENDPOINT: &str = "RUSTFS_OBS_LOG_ENDPOINT";
+pub const ENV_OBS_PROFILING_ENDPOINT: &str = "RUSTFS_OBS_PROFILING_ENDPOINT";
 pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
 pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
 pub const ENV_OBS_METER_INTERVAL: &str = "RUSTFS_OBS_METER_INTERVAL";
@@ -33,12 +34,12 @@ pub const ENV_OBS_ENVIRONMENT: &str = "RUSTFS_OBS_ENVIRONMENT";
 pub const ENV_OBS_TRACES_EXPORT_ENABLED: &str = "RUSTFS_OBS_TRACES_EXPORT_ENABLED";
 pub const ENV_OBS_METRICS_EXPORT_ENABLED: &str = "RUSTFS_OBS_METRICS_EXPORT_ENABLED";
 pub const ENV_OBS_LOGS_EXPORT_ENABLED: &str = "RUSTFS_OBS_LOGS_EXPORT_ENABLED";
+pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_ENABLED";
 
 pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
 pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
 pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
 pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
-pub const ENV_OBS_LOG_ROTATION_SIZE_MB: &str = "RUSTFS_OBS_LOG_ROTATION_SIZE_MB";
 pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
 pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
 
@@ -47,6 +48,12 @@ pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZ
 pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
 pub const ENV_OBS_LOG_COMPRESS_OLD_FILES: &str = "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES";
 pub const ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL: &str = "RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL";
+pub const ENV_OBS_LOG_COMPRESSION_ALGORITHM: &str = "RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM";
+pub const ENV_OBS_LOG_PARALLEL_COMPRESS: &str = "RUSTFS_OBS_LOG_PARALLEL_COMPRESS";
+pub const ENV_OBS_LOG_PARALLEL_WORKERS: &str = "RUSTFS_OBS_LOG_PARALLEL_WORKERS";
+pub const ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL: &str = "RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL";
+pub const ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP: &str = "RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP";
+pub const ENV_OBS_LOG_ZSTD_WORKERS: &str = "RUSTFS_OBS_LOG_ZSTD_WORKERS";
 pub const ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: &str = "RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS";
 pub const ENV_OBS_LOG_EXCLUDE_PATTERNS: &str = "RUSTFS_OBS_LOG_EXCLUDE_PATTERNS";
 pub const ENV_OBS_LOG_DELETE_EMPTY_FILES: &str = "RUSTFS_OBS_LOG_DELETE_EMPTY_FILES";
@@ -60,13 +67,24 @@ pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; //
 pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
 pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP: &str = "gzip";
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD: &str = "zstd";
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM: &str = DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD;
+pub const DEFAULT_OBS_LOG_PARALLEL_COMPRESS: bool = true;
+pub const DEFAULT_OBS_LOG_PARALLEL_WORKERS: usize = 6;
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL: i32 = 8;
+pub const DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP: bool = true;
+pub const DEFAULT_OBS_LOG_ZSTD_WORKERS: usize = 1;
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION: &str = "gz";
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION);
-pub const DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: u64 = 30;
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_EXTENSION: &str = "zst";
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_ZSTD_COMPRESSION_EXTENSION);
+pub const DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: u64 = 30; // Retain compressed files for 30 days
 pub const DEFAULT_OBS_LOG_DELETE_EMPTY_FILES: bool = true;
 pub const DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS: u64 = 3600; // 1 hour
 pub const DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS: u64 = 1800; // 0.5 hours
 pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
+pub const DEFAULT_OBS_LOG_MATCH_MODE_PREFIX: &str = "prefix";
 pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";
 
 /// Default values for observability configuration
@@ -90,6 +108,7 @@ mod tests {
         assert_eq!(ENV_OBS_TRACE_ENDPOINT, "RUSTFS_OBS_TRACE_ENDPOINT");
         assert_eq!(ENV_OBS_METRIC_ENDPOINT, "RUSTFS_OBS_METRIC_ENDPOINT");
         assert_eq!(ENV_OBS_LOG_ENDPOINT, "RUSTFS_OBS_LOG_ENDPOINT");
+        assert_eq!(ENV_OBS_PROFILING_ENDPOINT, "RUSTFS_OBS_PROFILING_ENDPOINT");
         assert_eq!(ENV_OBS_USE_STDOUT, "RUSTFS_OBS_USE_STDOUT");
         assert_eq!(ENV_OBS_SAMPLE_RATIO, "RUSTFS_OBS_SAMPLE_RATIO");
         assert_eq!(ENV_OBS_METER_INTERVAL, "RUSTFS_OBS_METER_INTERVAL");
@@ -100,17 +119,23 @@ mod tests {
         assert_eq!(ENV_OBS_LOG_STDOUT_ENABLED, "RUSTFS_OBS_LOG_STDOUT_ENABLED");
         assert_eq!(ENV_OBS_LOG_DIRECTORY, "RUSTFS_OBS_LOG_DIRECTORY");
         assert_eq!(ENV_OBS_LOG_FILENAME, "RUSTFS_OBS_LOG_FILENAME");
-        assert_eq!(ENV_OBS_LOG_ROTATION_SIZE_MB, "RUSTFS_OBS_LOG_ROTATION_SIZE_MB");
         assert_eq!(ENV_OBS_LOG_ROTATION_TIME, "RUSTFS_OBS_LOG_ROTATION_TIME");
         assert_eq!(ENV_OBS_LOG_KEEP_FILES, "RUSTFS_OBS_LOG_KEEP_FILES");
         assert_eq!(ENV_OBS_TRACES_EXPORT_ENABLED, "RUSTFS_OBS_TRACES_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
+        assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
         // Test log cleanup related env keys
         assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_COMPRESS_OLD_FILES, "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES");
         assert_eq!(ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL, "RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL");
+        assert_eq!(ENV_OBS_LOG_COMPRESSION_ALGORITHM, "RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM");
+        assert_eq!(ENV_OBS_LOG_PARALLEL_COMPRESS, "RUSTFS_OBS_LOG_PARALLEL_COMPRESS");
+        assert_eq!(ENV_OBS_LOG_PARALLEL_WORKERS, "RUSTFS_OBS_LOG_PARALLEL_WORKERS");
+        assert_eq!(ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL, "RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL");
+        assert_eq!(ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, "RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP");
+        assert_eq!(ENV_OBS_LOG_ZSTD_WORKERS, "RUSTFS_OBS_LOG_ZSTD_WORKERS");
         assert_eq!(
             ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS,
             "RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS"
@@ -129,6 +154,10 @@ mod tests {
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_DEVELOPMENT, "development");
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_TEST, "test");
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_STAGING, "staging");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP, "gzip");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD, "zstd");
+        assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE_PREFIX, "prefix");
         assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE, "suffix");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, "zstd");
     }
 }

crates/credentials/Cargo.toml

@@ -15,7 +15,7 @@ base64-simd = { workspace = true }
 rand = { workspace = true }
 serde = { workspace = true }
 serde_json.workspace = true
-time = { workspace = true, features = ["serde-human-readable"] }
+time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }
 
 [lints]
 workspace = true

crates/credentials/src/credentials.rs

@@ -233,15 +233,18 @@ impl<'a> fmt::Debug for Masked<'a> {
         match self.0 {
             None => Ok(()),
             Some(s) => {
-                let len = s.len();
+                let len = s.chars().count();
                 if len == 0 {
                     Ok(())
                 } else if len == 1 {
                     write!(f, "***")
                 } else if len == 2 {
-                    write!(f, "{}***|{}", &s[0..1], len)
+                    let first = s.chars().next().ok_or(fmt::Error)?;
+                    write!(f, "{}***|{}", first, len)
                 } else {
-                    write!(f, "{}***{}|{}", &s[0..1], &s[len - 1..], len)
+                    let first = s.chars().next().ok_or(fmt::Error)?;
+                    let last = s.chars().last().ok_or(fmt::Error)?;
+                    write!(f, "{}***{}|{}", first, last, len)
                 }
             }
         }
@@ -270,15 +273,25 @@ impl<'a> fmt::Display for Masked<'a> {
 ///
 #[derive(Serialize, Deserialize, Clone, Default)]
 pub struct Credentials {
+    #[serde(rename = "accessKey", alias = "access_key", default)]
     pub access_key: String,
+    #[serde(rename = "secretKey", alias = "secret_key", default)]
     pub secret_key: String,
+    #[serde(rename = "sessionToken", alias = "session_token", default)]
     pub session_token: String,
+    #[serde(default, with = "crate::serde_datetime::option")]
     pub expiration: Option<OffsetDateTime>,
+    #[serde(default)]
     pub status: String,
+    #[serde(rename = "parentUser", alias = "parent_user", default)]
     pub parent_user: String,
+    #[serde(default)]
     pub groups: Option<Vec<String>>,
+    #[serde(default)]
     pub claims: Option<HashMap<String, Value>>,
+    #[serde(default)]
     pub name: Option<String>,
+    #[serde(default)]
     pub description: Option<String>,
 }
 
@@ -482,5 +495,37 @@ mod tests {
 
         // Test longer string
         assert_eq!(format!("{:?}", Masked(Some("secretpassword"))), "s***d|14");
+
+        // Test Unicode input should not panic and should keep character boundary
+        assert_eq!(format!("{:?}", Masked(Some("中"))), "***");
+        assert_eq!(format!("{:?}", Masked(Some("中文"))), "中***|2");
+        assert_eq!(format!("{:?}", Masked(Some("中文测试"))), "中***试|4");
+    }
+
+    #[test]
+    fn test_credentials_expiration_serialize_as_rfc3339() {
+        use time::OffsetDateTime;
+
+        let c = Credentials {
+            access_key: "ak".to_string(),
+            secret_key: "sk12345678".to_string(),
+            expiration: Some(OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+
+        let json = serde_json::to_string(&c).expect("serialize");
+        assert!(
+            json.contains('T') && (json.contains('Z') || json.contains("+00:00")),
+            "Credentials expiration should be RFC3339; got: {}",
+            json
+        );
+    }
+
+    #[test]
+    fn test_credentials_deserialize_minio_style_rfc3339_expiration() {
+        let minio_style = r#"{"accessKey":"ak","secretKey":"sk12345678","expiration":"2025-03-07T12:00:00Z"}"#;
+        let c: Credentials = serde_json::from_str(minio_style).expect("deserialize");
+        assert_eq!(c.access_key, "ak");
+        assert!(c.expiration.is_some());
     }
 }

crates/credentials/src/lib.rs

@@ -14,6 +14,7 @@
 
 mod constants;
 mod credentials;
+mod serde_datetime;
 
 pub use constants::*;
 pub use credentials::*;

crates/credentials/src/serde_datetime.rs

@@ -0,0 +1,68 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Serde helpers for expiration timestamp: serialize as RFC3339 (MinIO-compatible),
+//! deserialize from RFC3339 or legacy RustFS human-readable format.
+
+use time::OffsetDateTime;
+use time::format_description;
+use time::format_description::well_known::Rfc3339;
+
+static LEGACY_FORMAT: std::sync::OnceLock<time::format_description::OwnedFormatItem> = std::sync::OnceLock::new();
+
+fn legacy_format() -> &'static time::format_description::OwnedFormatItem {
+    LEGACY_FORMAT.get_or_init(|| {
+        format_description::parse_owned::<2>(
+            "[year]-[month]-[day] [hour]:[minute]:[second].[subsecond] [offset_hour sign:mandatory]:[offset_minute]:[offset_second]",
+        )
+        .expect("legacy format description is valid")
+    });
+    LEGACY_FORMAT.get().expect("initialized above")
+}
+
+fn parse_rfc3339_or_legacy(s: &str) -> Result<OffsetDateTime, time::Error> {
+    OffsetDateTime::parse(s, &Rfc3339).or_else(|_| OffsetDateTime::parse(s, legacy_format()).map_err(Into::into))
+}
+
+/// Option<OffsetDateTime>: serialize as RFC3339; deserialize from RFC3339 or legacy.
+pub mod option {
+    use serde::{Deserialize, Deserializer, Serializer};
+    use time::OffsetDateTime;
+
+    use super::{Rfc3339, parse_rfc3339_or_legacy};
+
+    pub fn serialize<S>(opt: &Option<OffsetDateTime>, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        match opt {
+            Some(dt) => {
+                let s = dt.format(&Rfc3339).map_err(serde::ser::Error::custom)?;
+                serializer.serialize_some(&s)
+            }
+            None => serializer.serialize_none(),
+        }
+    }
+
+    pub fn deserialize<'de, D>(deserializer: D) -> Result<Option<OffsetDateTime>, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let opt: Option<&str> = Option::deserialize(deserializer)?;
+        match opt {
+            None => Ok(None),
+            Some(s) => parse_rfc3339_or_legacy(s).map(Some).map_err(serde::de::Error::custom),
+        }
+    }
+}

crates/crypto/src/encdec.rs

@@ -21,5 +21,8 @@ pub(crate) mod id;
 pub(crate) mod decrypt;
 pub(crate) mod encrypt;
 
+#[cfg(any(test, feature = "crypto"))]
+pub(crate) mod stream_io;
+
 #[cfg(test)]
 mod tests;

crates/crypto/src/encdec/stream_io.rs

@@ -0,0 +1,219 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! sio-go compatible stream encryption for IAM config.
+//! Header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes.
+//! Body: DARE-style fragmented AEAD (bufSize=16384, per-fragment nonce).
+
+#![allow(deprecated)] // AeadInPlace deprecated in favor of AeadInOut; keep for aead 0.6 compatibility
+
+use crate::encdec::id::ID;
+use crate::error::Error;
+use aes_gcm::{
+    Aes256Gcm,
+    aead::{AeadCore, AeadInPlace, KeyInit as _, array::Array},
+};
+use chacha20poly1305::ChaCha20Poly1305;
+
+const STREAM_IO_HEADER_LEN: usize = 41;
+const SIO_BUF_SIZE: usize = 16384;
+const SIO_NONCE_PREFIX_LEN: usize = 8;
+const AES_GCM_OVERHEAD: usize = 16;
+const CHACHA_OVERHEAD: usize = 16;
+
+/// Decrypt data in stream_io (sio-go) format.
+pub fn decrypt_stream_io(password: &[u8], data: &[u8]) -> Result<Vec<u8>, Error> {
+    if data.len() < STREAM_IO_HEADER_LEN {
+        return Err(Error::ErrUnexpectedHeader);
+    }
+    let salt = &data[0..32];
+    let id = ID::try_from(data[32])?;
+    let nonce_prefix = &data[33..41];
+    let body = &data[STREAM_IO_HEADER_LEN..];
+
+    let key = id.get_key(password, salt)?;
+
+    match id {
+        ID::Argon2idChaCHa20Poly1305 => decrypt_stream(
+            ChaCha20Poly1305::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            nonce_prefix,
+            body,
+            CHACHA_OVERHEAD,
+        ),
+        _ => decrypt_stream(
+            Aes256Gcm::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            nonce_prefix,
+            body,
+            AES_GCM_OVERHEAD,
+        ),
+    }
+}
+
+fn decrypt_stream<A>(aead: A, nonce_prefix: &[u8], body: &[u8], overhead: usize) -> Result<Vec<u8>, Error>
+where
+    A: AeadInPlace,
+{
+    let ciphertext_len = SIO_BUF_SIZE + overhead;
+    let ad = build_associated_data(&aead, nonce_prefix)?;
+    let mut plain = Vec::with_capacity(body.len());
+
+    let mut seq_num: u32 = 1;
+    let mut pos = 0;
+
+    while pos < body.len() {
+        let remaining = body.len() - pos;
+        let frag_len = remaining.min(ciphertext_len);
+        let is_last = (pos + frag_len) == body.len();
+
+        if frag_len < overhead {
+            return Err(Error::ErrDecryptFailed(aes_gcm::aead::Error));
+        }
+
+        let mut nonce = [0u8; 12];
+        nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+        nonce[8..12].copy_from_slice(&seq_num.to_le_bytes());
+
+        let mut ad_mut = ad.clone();
+        ad_mut[0] = if is_last { 0x80 } else { 0x00 };
+
+        let fragment = &body[pos..pos + frag_len];
+        let tag_len = overhead;
+        let (ct, tag) = fragment.split_at(frag_len - tag_len);
+
+        let mut buffer = ct.to_vec();
+        let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+            .map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        let tag_arr =
+            Array::<u8, <A as AeadCore>::TagSize>::try_from(tag).map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        aead.decrypt_in_place_detached(&nonce_arr, &ad_mut, &mut buffer, &tag_arr)
+            .map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        plain.extend_from_slice(&buffer);
+
+        pos += frag_len;
+        seq_num += 1;
+
+        if is_last {
+            break;
+        }
+    }
+
+    Ok(plain)
+}
+
+fn build_associated_data<A>(aead: &A, nonce_prefix: &[u8]) -> Result<Vec<u8>, Error>
+where
+    A: AeadInPlace,
+{
+    let mut nonce = [0u8; 12];
+    nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+    nonce[8..12].copy_from_slice(&0u32.to_le_bytes());
+
+    let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+        .map_err(|_| Error::ErrEncryptFailed(aes_gcm::aead::Error))?;
+    let mut empty: [u8; 0] = [];
+    let tag = aead
+        .encrypt_in_place_detached(&nonce_arr, &[] as &[u8], &mut empty)
+        .map_err(Error::ErrEncryptFailed)?;
+
+    let mut ad = vec![0u8; 1 + tag.len()];
+    ad[0] = 0x00;
+    ad[1..].copy_from_slice(tag.as_slice());
+    Ok(ad)
+}
+
+/// Encrypt data in stream_io (sio-go) format.
+pub fn encrypt_stream_io(password: &[u8], data: &[u8]) -> Result<Vec<u8>, Error> {
+    let salt: [u8; 32] = rand::random();
+
+    #[cfg(feature = "fips")]
+    let id = ID::Pbkdf2AESGCM;
+
+    #[cfg(not(feature = "fips"))]
+    let id = if crate::encdec::encrypt::native_aes() {
+        ID::Argon2idAESGCM
+    } else {
+        ID::Argon2idChaCHa20Poly1305
+    };
+
+    let key = id.get_key(password, &salt)?;
+    let nonce_prefix: [u8; SIO_NONCE_PREFIX_LEN] = rand::random();
+
+    let mut out = Vec::with_capacity(STREAM_IO_HEADER_LEN + data.len() + 32);
+    out.extend_from_slice(&salt);
+    out.push(id as u8);
+    out.extend_from_slice(&nonce_prefix);
+
+    match id {
+        ID::Argon2idChaCHa20Poly1305 => encrypt_stream(
+            ChaCha20Poly1305::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            &nonce_prefix,
+            data,
+            &mut out,
+            CHACHA_OVERHEAD,
+        )?,
+        _ => encrypt_stream(
+            Aes256Gcm::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            &nonce_prefix,
+            data,
+            &mut out,
+            AES_GCM_OVERHEAD,
+        )?,
+    }
+
+    Ok(out)
+}
+
+fn encrypt_stream<A>(
+    aead: A,
+    nonce_prefix: &[u8; SIO_NONCE_PREFIX_LEN],
+    data: &[u8],
+    out: &mut Vec<u8>,
+    _overhead: usize,
+) -> Result<(), Error>
+where
+    A: AeadInPlace,
+{
+    let ad = build_associated_data(&aead, nonce_prefix)?;
+    let mut seq_num: u32 = 1;
+    let mut pos = 0;
+
+    while pos < data.len() {
+        let remaining = data.len() - pos;
+        let is_last = remaining <= SIO_BUF_SIZE;
+
+        let chunk_len = if is_last { remaining } else { SIO_BUF_SIZE };
+        let chunk = &data[pos..pos + chunk_len];
+
+        let mut nonce = [0u8; 12];
+        nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+        nonce[8..12].copy_from_slice(&seq_num.to_le_bytes());
+
+        let mut ad_mut = ad.clone();
+        ad_mut[0] = if is_last { 0x80 } else { 0x00 };
+
+        let mut buffer = chunk.to_vec();
+        let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+            .map_err(|_| Error::ErrEncryptFailed(aes_gcm::aead::Error))?;
+        let tag = aead
+            .encrypt_in_place_detached(&nonce_arr, &ad_mut, &mut buffer)
+            .map_err(Error::ErrEncryptFailed)?;
+        out.extend_from_slice(&buffer);
+        out.extend_from_slice(tag.as_slice());
+
+        pos += chunk_len;
+        seq_num += 1;
+    }
+
+    Ok(())
+}

crates/crypto/src/encdec/tests.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::{decrypt_data, encrypt_data};
+use crate::{decrypt_data, decrypt_stream_io, encrypt_data, encrypt_stream_io};
 
 const PASSWORD: &[u8] = "test_password".as_bytes();
 const LONG_PASSWORD: &[u8] = "very_long_password_with_many_characters_for_testing_purposes_123456789".as_bytes();
@@ -317,3 +317,67 @@ fn test_concurrent_encryption_safety() -> Result<(), crate::Error> {
 
     Ok(())
 }
+
+#[test]
+fn test_stream_io_roundtrip() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = br#"{"Version":1,"policy":"readonly"}"#;
+    let encrypted = encrypt_stream_io(password, data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert_eq!(data, decrypted.as_slice());
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_large_data_roundtrip() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = vec![0xAB; 32 * 1024]; // > SIO_BUF_SIZE to test fragmentation
+    let encrypted = encrypt_stream_io(password, &data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert_eq!(data, decrypted);
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_wrong_password_fails() {
+    let password = b"access:secret";
+    let data = br#"{"Version":1}"#;
+    let encrypted = encrypt_stream_io(password, data).expect("encrypt should succeed");
+    let result = decrypt_stream_io(b"wrong:password", &encrypted);
+    assert!(result.is_err(), "decrypt with wrong password should fail");
+}
+
+#[test]
+fn test_stream_io_empty_data() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data: &[u8] = &[];
+    let encrypted = encrypt_stream_io(password, data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert!(decrypted.is_empty());
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_header_format() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = b"test";
+    let encrypted = encrypt_stream_io(password, data)?;
+    // stream_io header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes
+    const STREAM_IO_HEADER_LEN: usize = 41;
+    assert!(encrypted.len() >= STREAM_IO_HEADER_LEN, "encrypted should have at least 41-byte header");
+    assert!(
+        encrypted[32] == 0x00 || encrypted[32] == 0x01 || encrypted[32] == 0x02,
+        "alg_id should be 0x00, 0x01, or 0x02"
+    );
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_truncated_data_fails() {
+    let password = b"access:secret";
+    let data = b"test";
+    let encrypted = encrypt_stream_io(password, data).expect("encrypt should succeed");
+    let truncated = &encrypted[..40]; // less than 41-byte header
+    let result = decrypt_stream_io(password, truncated);
+    assert!(result.is_err(), "truncated data should fail decrypt");
+}

crates/crypto/src/lib.rs

@@ -19,6 +19,9 @@ mod jwt;
 
 pub use encdec::decrypt::decrypt_data;
 pub use encdec::encrypt::encrypt_data;
+
+#[cfg(feature = "crypto")]
+pub use encdec::stream_io::{decrypt_stream_io, encrypt_stream_io};
 pub use error::Error;
 pub use jwt::decode::decode as jwt_decode;
 pub use jwt::encode::encode as jwt_encode;

crates/e2e_test/AGENTS.md

@@ -0,0 +1,26 @@
+# E2E Test Crate Instructions
+
+Applies to `crates/e2e_test/`.
+
+## Test Reliability
+
+- Keep end-to-end tests deterministic and environment-aware.
+- Prefer readiness checks and explicit polling over fixed sleep-based timing assumptions.
+- Ensure tests isolate resources and clean up temporary state.
+
+## Environment Safety
+
+- For local KMS-related E2E runs, keep proxy bypass settings:
+  - `NO_PROXY=127.0.0.1,localhost`
+  - clear `HTTP_PROXY` and `HTTPS_PROXY`
+- Do not hardcode machine-specific paths or credentials.
+
+## Scope and Cost
+
+- Place exhaustive integration behavior here; keep unit behavior in source crates.
+- Keep new E2E scenarios focused and avoid redundant overlap with existing suites.
+
+## Suggested Validation
+
+- `cargo test --package e2e_test`
+- Full gate before commit: `make pre-commit`

crates/e2e_test/src/protocols/README.md

@@ -1,6 +1,6 @@
 # Protocol E2E Tests
 
-FTPS protocol end-to-end tests for RustFS.
+FTPS and WebDAV protocol end-to-end tests for RustFS.
 
 ## Prerequisites
 
@@ -19,11 +19,21 @@ brew install sshpass openssh
 
 ## Running Tests
 
-Run all protocol tests:
+Run all protocol tests (FTPS + WebDAV):
+```bash
+RUSTFS_BUILD_FEATURES=ftps,webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
+```
+
+Run FTPS tests only:
 ```bash
 RUSTFS_BUILD_FEATURES=ftps cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
 ```
 
+Run WebDAV tests only:
+```bash
+RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
+```
+
 ## Test Coverage
 
 ### FTPS Tests
@@ -38,3 +48,13 @@ RUSTFS_BUILD_FEATURES=ftps cargo test --package e2e_test test_protocol_core_suit
 - cdup
 - rmdir delete bucket
 
+### WebDAV Tests
+- PROPFIND at root (list buckets)
+- MKCOL (create bucket)
+- PUT (upload file)
+- GET (download file)
+- PROPFIND on bucket (list objects)
+- DELETE file
+- DELETE bucket
+- Authentication failure test
+

crates/e2e_test/src/protocols/mod.rs

@@ -12,8 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Protocol tests for FTPS
+//! Protocol tests for FTPS and WebDAV
 
 pub mod ftps_core;
 pub mod test_env;
 pub mod test_runner;
+pub mod webdav_core;

crates/e2e_test/src/protocols/test_runner.rs

@@ -16,6 +16,7 @@
 
 use crate::common::init_logging;
 use crate::protocols::ftps_core::test_ftps_core_operations;
+use crate::protocols::webdav_core::test_webdav_core_operations;
 use std::time::Instant;
 use tokio::time::{Duration, sleep};
 use tracing::{error, info};
@@ -59,9 +60,14 @@ struct TestDefinition {
 impl ProtocolTestSuite {
     /// Create default test suite
     pub fn new() -> Self {
-        let tests = vec![TestDefinition {
-            name: "test_ftps_core_operations".to_string(),
-        }];
+        let tests = vec![
+            TestDefinition {
+                name: "test_ftps_core_operations".to_string(),
+            },
+            TestDefinition {
+                name: "test_webdav_core_operations".to_string(),
+            },
+        ];
 
         Self { tests }
     }
@@ -83,6 +89,10 @@ impl ProtocolTestSuite {
                     info!("=== Starting FTPS Module Test ===");
                     "FTPS core operations (put, ls, mkdir, rmdir, delete)"
                 }
+                "test_webdav_core_operations" => {
+                    info!("=== Starting WebDAV Core Test ===");
+                    "WebDAV core operations (MKCOL, PUT, GET, DELETE, PROPFIND)"
+                }
                 _ => "",
             };
 
@@ -121,6 +131,7 @@ impl ProtocolTestSuite {
     async fn run_single_test(&self, test_def: &TestDefinition) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         match test_def.name.as_str() {
             "test_ftps_core_operations" => test_ftps_core_operations().await.map_err(|e| e.into()),
+            "test_webdav_core_operations" => test_webdav_core_operations().await.map_err(|e| e.into()),
             _ => Err(format!("Test {} not implemented", test_def.name).into()),
         }
     }

crates/e2e_test/src/protocols/webdav_core.rs

@@ -0,0 +1,207 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Core WebDAV tests
+
+use crate::common::rustfs_binary_path;
+use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
+use anyhow::Result;
+use base64::Engine;
+use reqwest::Client;
+use tokio::process::Command;
+use tracing::info;
+
+// Fixed WebDAV port for testing
+const WEBDAV_PORT: u16 = 9080;
+const WEBDAV_ADDRESS: &str = "127.0.0.1:9080";
+
+/// Create HTTP client with basic auth
+fn create_client() -> Client {
+    Client::builder()
+        .danger_accept_invalid_certs(true)
+        .build()
+        .expect("Failed to create HTTP client")
+}
+
+/// Get basic auth header value
+fn basic_auth_header() -> String {
+    let credentials = format!("{}:{}", DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY);
+    let encoded = base64::engine::general_purpose::STANDARD.encode(credentials);
+    format!("Basic {}", encoded)
+}
+
+/// Test WebDAV: MKCOL (create bucket), PUT, GET, DELETE, PROPFIND operations
+pub async fn test_webdav_core_operations() -> Result<()> {
+    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
+
+    // Start server manually
+    info!("Starting WebDAV server on {}", WEBDAV_ADDRESS);
+    let binary_path = rustfs_binary_path();
+    let mut server_process = Command::new(&binary_path)
+        .env("RUSTFS_WEBDAV_ENABLE", "true")
+        .env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
+        .env("RUSTFS_WEBDAV_TLS_ENABLED", "false") // No TLS for testing
+        .arg(&env.temp_dir)
+        .spawn()?;
+
+    // Ensure server is cleaned up even on failure
+    let result = async {
+        // Wait for server to be ready
+        ProtocolTestEnvironment::wait_for_port_ready(WEBDAV_PORT, 30)
+            .await
+            .map_err(|e| anyhow::anyhow!("{}", e))?;
+
+        let client = create_client();
+        let auth_header = basic_auth_header();
+        let base_url = format!("http://{}", WEBDAV_ADDRESS);
+
+        // Test PROPFIND at root (list buckets)
+        info!("Testing WebDAV: PROPFIND at root (list buckets)");
+        let resp = client
+            .request(reqwest::Method::from_bytes(b"PROPFIND").unwrap(), &base_url)
+            .header("Authorization", &auth_header)
+            .header("Depth", "1")
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 207,
+            "PROPFIND at root should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: PROPFIND at root successful");
+
+        // Test MKCOL (create bucket)
+        let bucket_name = "webdav-test-bucket";
+        info!("Testing WebDAV: MKCOL (create bucket '{}')", bucket_name);
+        let resp = client
+            .request(reqwest::Method::from_bytes(b"MKCOL").unwrap(), format!("{}/{}", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MKCOL bucket '{}' successful", bucket_name);
+
+        // Test PUT (upload file)
+        let filename = "test-file.txt";
+        let file_content = "Hello, WebDAV!";
+        info!("Testing WebDAV: PUT (upload file '{}')", filename);
+        let resp = client
+            .put(format!("{}/{}/{}", base_url, bucket_name, filename))
+            .header("Authorization", &auth_header)
+            .body(file_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: PUT file '{}' successful", filename);
+
+        // Test GET (download file)
+        info!("Testing WebDAV: GET (download file '{}')", filename);
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(resp.status().is_success(), "GET should succeed, got: {}", resp.status());
+        let downloaded_content = resp.text().await?;
+        assert_eq!(downloaded_content, file_content, "Downloaded content should match uploaded content");
+        info!("PASS: GET file '{}' successful, content matches", filename);
+
+        // Test PROPFIND on bucket (list objects)
+        info!("Testing WebDAV: PROPFIND on bucket (list objects)");
+        let resp = client
+            .request(reqwest::Method::from_bytes(b"PROPFIND").unwrap(), format!("{}/{}", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .header("Depth", "1")
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 207,
+            "PROPFIND on bucket should succeed, got: {}",
+            resp.status()
+        );
+        let body = resp.text().await?;
+        assert!(body.contains(filename), "File should appear in PROPFIND response");
+        info!("PASS: PROPFIND on bucket successful, file '{}' found", filename);
+
+        // Test DELETE file
+        info!("Testing WebDAV: DELETE file '{}'", filename);
+        let resp = client
+            .delete(format!("{}/{}/{}", base_url, bucket_name, filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204,
+            "DELETE file should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: DELETE file '{}' successful", filename);
+
+        // Verify file is deleted
+        info!("Testing WebDAV: Verify file is deleted");
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().as_u16() == 404,
+            "GET deleted file should return 404, got: {}",
+            resp.status()
+        );
+        info!("PASS: Verified file '{}' is deleted", filename);
+
+        // Test DELETE bucket
+        info!("Testing WebDAV: DELETE bucket '{}'", bucket_name);
+        let resp = client
+            .delete(format!("{}/{}", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204,
+            "DELETE bucket should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: DELETE bucket '{}' successful", bucket_name);
+
+        // Test authentication failure
+        info!("Testing WebDAV: Authentication failure");
+        let resp = client
+            .request(reqwest::Method::from_bytes(b"PROPFIND").unwrap(), &base_url)
+            .header("Authorization", "Basic aW52YWxpZDppbnZhbGlk") // invalid:invalid
+            .send()
+            .await?;
+        assert_eq!(resp.status().as_u16(), 401, "Invalid auth should return 401, got: {}", resp.status());
+        info!("PASS: Authentication failure test successful");
+
+        info!("WebDAV core tests passed");
+        Ok(())
+    }
+    .await;
+
+    // Always cleanup server process
+    let _ = server_process.kill().await;
+    let _ = server_process.wait().await;
+
+    result
+}

crates/e2e_test/src/reliant/lifecycle.rs

@@ -19,7 +19,6 @@ use aws_sdk_s3::config::{Credentials, Region};
 use bytes::Bytes;
 use serial_test::serial;
 use std::error::Error;
-use tokio::time::sleep;
 
 const ENDPOINT: &str = "http://localhost:9000";
 const ACCESS_KEY: &str = "rustfsadmin";
@@ -62,6 +61,7 @@ async fn setup_test_bucket(client: &Client) -> Result<(), Box<dyn Error>> {
 #[ignore = "requires running RustFS server at localhost:9000"]
 async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error::Error>> {
     use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
+    use chrono::{Duration as ChronoDuration, Utc};
     use tokio::time::Duration;
 
     let client = create_aws_s3_client().await?;
@@ -70,6 +70,7 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
     // Upload test object first
     let test_content = "Test object for lifecycle expiration";
     let lifecycle_object_key = "lifecycle-test-object.txt";
+    let untouched_object_key = "keep-object.txt";
     client
         .put_object()
         .bucket(BUCKET)
@@ -77,13 +78,29 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
         .body(Bytes::from(test_content.as_bytes()).into())
         .send()
         .await?;
+    client
+        .put_object()
+        .bucket(BUCKET)
+        .key(untouched_object_key)
+        .body(Bytes::from("should-stay".as_bytes()).into())
+        .send()
+        .await?;
 
     // Verify object exists initially
     let resp = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await?;
     assert!(resp.content_length().unwrap_or(0) > 0);
+    let untouched_resp = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
+    assert!(untouched_resp.content_length().unwrap_or(0) > 0);
 
-    // Configure lifecycle rule: expire after current time + 3 seconds
-    let expiration = LifecycleExpiration::builder().days(0).build();
+    // Use a past midnight UTC date to trigger immediate lifecycle expiry without requiring days=0.
+    let yesterday_midnight_utc = Utc::now()
+        .date_naive()
+        .and_hms_opt(0, 0, 0)
+        .expect("midnight should always be valid")
+        - ChronoDuration::days(1);
+    let expiration = LifecycleExpiration::builder()
+        .date(aws_sdk_s3::primitives::DateTime::from_secs(yesterday_midnight_utc.and_utc().timestamp()))
+        .build();
     let filter = LifecycleRuleFilter::builder().prefix(lifecycle_object_key).build();
     let rule = LifecycleRule::builder()
         .id("expire-test-object")
@@ -105,29 +122,73 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
     let rules = resp.rules();
     assert!(rules.iter().any(|r| r.id().unwrap_or("") == "expire-test-object"));
 
-    // Wait for lifecycle processing (scanner runs every 1 second)
-    sleep(Duration::from_secs(3)).await;
-
-    // After lifecycle processing, the object should be deleted by the lifecycle rule
-    let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
-
-    match get_result {
-        Ok(_) => {
-            panic!("Expected object to be deleted by lifecycle rule, but it still exists");
-        }
-        Err(e) => {
-            if let Some(service_error) = e.as_service_error() {
-                if service_error.is_no_such_key() {
-                    println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
-                } else {
-                    panic!("Expected NoSuchKey error, but got: {e:?}");
+    // Poll for deletion instead of using a fixed sleep to keep the test deterministic.
+    // Default scanner cycle interval is 60s with jitter, so allow enough time for one full cycle.
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(150);
+    loop {
+        let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
+        match get_result {
+            Ok(_) => {
+                if tokio::time::Instant::now() >= deadline {
+                    panic!("Expected object to be deleted by lifecycle rule within 150s, but it still exists");
+                }
+                tokio::time::sleep(Duration::from_secs(1)).await;
+            }
+            Err(e) => {
+                if let Some(service_error) = e.as_service_error() {
+                    if service_error.is_no_such_key() {
+                        println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
+                        break;
+                    }
+                    panic!("Expected NoSuchKey error, but got: {e:?}");
+                } else {
+                    panic!("Expected service error, but got: {e:?}");
                 }
-            } else {
-                panic!("Expected service error, but got: {e:?}");
             }
         }
     }
 
     println!("Lifecycle configuration test completed.");
+
+    // Non-matching prefix object should remain available.
+    let untouched_after = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
+    assert!(untouched_after.content_length().unwrap_or(0) > 0);
+
+    Ok(())
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+#[serial]
+#[ignore = "requires running RustFS server at localhost:9000"]
+async fn test_bucket_lifecycle_rejects_zero_days() -> Result<(), Box<dyn std::error::Error>> {
+    use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
+
+    let client = create_aws_s3_client().await?;
+    setup_test_bucket(&client).await?;
+
+    let expiration = LifecycleExpiration::builder().days(0).build();
+    let filter = LifecycleRuleFilter::builder().prefix("zero-days/").build();
+    let rule = LifecycleRule::builder()
+        .id("expire-zero-days")
+        .filter(filter)
+        .expiration(expiration)
+        .status(aws_sdk_s3::types::ExpirationStatus::Enabled)
+        .build()?;
+    let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
+
+    let err = client
+        .put_bucket_lifecycle_configuration()
+        .bucket(BUCKET)
+        .lifecycle_configuration(lifecycle)
+        .send()
+        .await
+        .expect_err("zero-day lifecycle expiration should be rejected");
+
+    let err_msg = format!("{err:?}");
+    assert!(
+        err_msg.contains("InvalidArgument") && err_msg.contains("greater than 0"),
+        "unexpected error: {err_msg}"
+    );
+
     Ok(())
 }

crates/ecstore/AGENTS.md

@@ -0,0 +1,28 @@
+# ECStore Crate Instructions
+
+Applies to `crates/ecstore/`.
+
+## Data Durability and Integrity
+
+- Do not weaken quorum checks, bitrot checks, or metadata validation paths.
+- Treat any change affecting read/write/repair correctness as high risk and test accordingly.
+- Prefer explicit failure over silent data corruption or implicit success.
+
+## Performance-Critical Paths
+
+- Be careful with allocations and locking in hot paths.
+- Keep network and disk operations async-friendly; avoid introducing unnecessary blocking.
+- Benchmark-sensitive changes should include measurable rationale.
+
+## Cross-Module Coordination
+
+- Validate behavior impacts on:
+  - `rustfs/src/storage/`
+  - `crates/filemeta/`
+  - `crates/heal/`
+  - `crates/checksums/`
+
+## Suggested Validation
+
+- `cargo test -p rustfs-ecstore`
+- Full gate before commit: `make pre-commit`

crates/ecstore/Cargo.toml

@@ -44,6 +44,7 @@ rustfs-credentials = { workspace = true }
 rustfs-common.workspace = true
 rustfs-policy.workspace = true
 rustfs-protos.workspace = true
+rustfs-s3-common = { workspace = true }
 async-trait.workspace = true
 bytes.workspace = true
 byteorder = { workspace = true }
@@ -65,6 +66,7 @@ http-body = { workspace = true }
 http-body-util.workspace = true
 url.workspace = true
 uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
+reed-solomon-erasure = { workspace = true }
 reed-solomon-simd = { workspace = true }
 lazy_static.workspace = true
 rustfs-lock.workspace = true
@@ -72,6 +74,7 @@ regex = { workspace = true }
 path-absolutize = { workspace = true }
 rmp.workspace = true
 rmp-serde.workspace = true
+serde_bytes.workspace = true
 tokio-util = { workspace = true, features = ["io", "compat"] }
 base64 = { workspace = true }
 hmac = { workspace = true }
@@ -119,6 +122,7 @@ tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true }
 tracing-subscriber = { workspace = true }
+serial_test = { workspace = true }
 
 [build-dependencies]
 shadow-rs = { workspace = true, features = ["build", "metadata"] }

crates/ecstore/benches/erasure_benchmark.rs

@@ -118,41 +118,38 @@ fn bench_encode_performance(c: &mut Criterion) {
         });
         group.finish();
 
-        // Test direct SIMD implementation for large shards (>= 512 bytes)
+        // Test direct reed-solomon-erasure implementation for large shards (>= 512 bytes)
         let shard_size = calc_shard_size(config.data_size, config.data_shards);
-        if shard_size >= 512 {
-            let mut simd_group = c.benchmark_group("encode_simd_direct");
-            simd_group.throughput(Throughput::Bytes(config.data_size as u64));
-            simd_group.sample_size(10);
-            simd_group.measurement_time(Duration::from_secs(5));
+        if shard_size >= 512 && config.parity_shards > 0 {
+            use reed_solomon_erasure::galois_8::ReedSolomon;
 
-            simd_group.bench_with_input(BenchmarkId::new("simd_direct", &config.name), &(&data, &config), |b, (data, config)| {
-                b.iter(|| {
-                    // Direct SIMD implementation
-                    let per_shard_size = calc_shard_size(data.len(), config.data_shards);
-                    match reed_solomon_simd::ReedSolomonEncoder::new(config.data_shards, config.parity_shards, per_shard_size) {
-                        Ok(mut encoder) => {
-                            // Create properly sized buffer and fill with data
-                            let mut buffer = vec![0u8; per_shard_size * config.data_shards];
+            let mut rse_group = c.benchmark_group("encode_rse_direct");
+            rse_group.throughput(Throughput::Bytes(config.data_size as u64));
+            rse_group.sample_size(10);
+            rse_group.measurement_time(Duration::from_secs(5));
+
+            if let Ok(rs) = ReedSolomon::new(config.data_shards, config.parity_shards) {
+                let total_shards = config.data_shards + config.parity_shards;
+                let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
+                let need_total = per_shard_size * total_shards;
+
+                rse_group.bench_with_input(
+                    BenchmarkId::new("rse_direct", &config.name),
+                    &(&data, need_total, per_shard_size),
+                    |b, (data, need_total, per_shard_size)| {
+                        b.iter(|| {
+                            let mut buffer = vec![0u8; *need_total];
                             let copy_len = data.len().min(buffer.len());
                             buffer[..copy_len].copy_from_slice(&data[..copy_len]);
 
-                            // Add data shards with correct shard size
-                            for chunk in buffer.chunks_exact(per_shard_size) {
-                                encoder.add_original_shard(black_box(chunk)).unwrap();
-                            }
-
-                            let result = encoder.encode().unwrap();
-                            black_box(result);
-                        }
-                        Err(_) => {
-                            // SIMD doesn't support this configuration, skip
-                            black_box(());
-                        }
-                    }
-                });
-            });
-            simd_group.finish();
+                            let mut slices: Vec<&mut [u8]> = buffer.chunks_exact_mut(*per_shard_size).collect();
+                            rs.encode(&mut slices).unwrap();
+                            black_box(buffer);
+                        });
+                    },
+                );
+            }
+            rse_group.finish();
         }
     }
 }
@@ -203,47 +200,33 @@ fn bench_decode_performance(c: &mut Criterion) {
         );
         group.finish();
 
-        // Test direct SIMD decoding for large shards
+        // Test direct reed-solomon-erasure decoding for large shards
         let shard_size = calc_shard_size(config.data_size, config.data_shards);
-        if shard_size >= 512 {
-            let mut simd_group = c.benchmark_group("decode_simd_direct");
-            simd_group.throughput(Throughput::Bytes(config.data_size as u64));
-            simd_group.sample_size(10);
-            simd_group.measurement_time(Duration::from_secs(5));
+        if shard_size >= 512 && config.parity_shards > 0 {
+            use reed_solomon_erasure::galois_8::ReedSolomon;
 
-            simd_group.bench_with_input(
-                BenchmarkId::new("simd_direct", &config.name),
-                &(&encoded_shards, &config),
-                |b, (shards, config)| {
-                    b.iter(|| {
-                        let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
-                        match reed_solomon_simd::ReedSolomonDecoder::new(config.data_shards, config.parity_shards, per_shard_size)
-                        {
-                            Ok(mut decoder) => {
-                                // Add available shards (except lost ones)
-                                for (i, shard) in shards.iter().enumerate() {
-                                    if i != config.data_shards - 1 && i != config.data_shards {
-                                        if i < config.data_shards {
-                                            decoder.add_original_shard(i, black_box(shard)).unwrap();
-                                        } else {
-                                            let recovery_idx = i - config.data_shards;
-                                            decoder.add_recovery_shard(recovery_idx, black_box(shard)).unwrap();
-                                        }
-                                    }
-                                }
+            if let Ok(rs) = ReedSolomon::new(config.data_shards, config.parity_shards) {
+                let mut rse_group = c.benchmark_group("decode_rse_direct");
+                rse_group.throughput(Throughput::Bytes(config.data_size as u64));
+                rse_group.sample_size(10);
+                rse_group.measurement_time(Duration::from_secs(5));
 
-                                let result = decoder.decode().unwrap();
-                                black_box(result);
-                            }
-                            Err(_) => {
-                                // SIMD doesn't support this configuration, skip
-                                black_box(());
-                            }
-                        }
-                    });
-                },
-            );
-            simd_group.finish();
+                rse_group.bench_with_input(
+                    BenchmarkId::new("rse_direct", &config.name),
+                    &(&encoded_shards, &config),
+                    |b, (shards, config)| {
+                        b.iter(|| {
+                            let mut shards_opt: Vec<Option<Vec<u8>>> = shards.iter().map(|s| Some(s.to_vec())).collect();
+                            shards_opt[config.data_shards - 1] = None;
+                            shards_opt[config.data_shards] = None;
+
+                            rs.reconstruct_data(&mut shards_opt).unwrap();
+                            black_box(shards_opt);
+                        });
+                    },
+                );
+                rse_group.finish();
+            }
         }
     }
 }

crates/ecstore/src/bitrot.rs

@@ -39,6 +39,7 @@ pub async fn create_bitrot_reader(
     length: usize,
     shard_size: usize,
     checksum_algo: HashAlgorithm,
+    skip_verify: bool,
 ) -> disk::error::Result<Option<BitrotReader<Box<dyn AsyncRead + Send + Sync + Unpin>>>> {
     // Calculate the total length to read, including the checksum overhead
     let length = length.div_ceil(shard_size) * checksum_algo.size() + length;
@@ -47,13 +48,18 @@ pub async fn create_bitrot_reader(
         // Use inline data
         let mut rd = Cursor::new(data.to_vec());
         rd.set_position(offset as u64);
-        let reader = BitrotReader::new(Box::new(rd) as Box<dyn AsyncRead + Send + Sync + Unpin>, shard_size, checksum_algo);
+        let reader = BitrotReader::new(
+            Box::new(rd) as Box<dyn AsyncRead + Send + Sync + Unpin>,
+            shard_size,
+            checksum_algo,
+            skip_verify,
+        );
         Ok(Some(reader))
     } else if let Some(disk) = disk {
         // Read from disk
         match disk.read_file_stream(bucket, path, offset, length - offset).await {
             Ok(rd) => {
-                let reader = BitrotReader::new(rd, shard_size, checksum_algo);
+                let reader = BitrotReader::new(rd, shard_size, checksum_algo, skip_verify);
                 Ok(Some(reader))
             }
             Err(e) => Err(e),
@@ -113,10 +119,10 @@ mod tests {
     async fn test_create_bitrot_reader_with_inline_data() {
         let test_data = b"hello world test data";
         let shard_size = 16;
-        let checksum_algo = HashAlgorithm::HighwayHash256;
+        let checksum_algo = HashAlgorithm::HighwayHash256S;
 
         let result =
-            create_bitrot_reader(Some(test_data), None, "test-bucket", "test-path", 0, 0, shard_size, checksum_algo).await;
+            create_bitrot_reader(Some(test_data), None, "test-bucket", "test-path", 0, 0, shard_size, checksum_algo, false).await;
 
         assert!(result.is_ok());
         assert!(result.unwrap().is_some());
@@ -125,9 +131,10 @@ mod tests {
     #[tokio::test]
     async fn test_create_bitrot_reader_without_data_or_disk() {
         let shard_size = 16;
-        let checksum_algo = HashAlgorithm::HighwayHash256;
+        let checksum_algo = HashAlgorithm::HighwayHash256S;
 
-        let result = create_bitrot_reader(None, None, "test-bucket", "test-path", 0, 1024, shard_size, checksum_algo).await;
+        let result =
+            create_bitrot_reader(None, None, "test-bucket", "test-path", 0, 1024, shard_size, checksum_algo, false).await;
 
         assert!(result.is_ok());
         assert!(result.unwrap().is_none());
@@ -144,7 +151,7 @@ mod tests {
             "test-path",
             1024, // length
             1024, // shard_size
-            HashAlgorithm::HighwayHash256,
+            HashAlgorithm::HighwayHash256S,
         )
         .await;
 
@@ -176,7 +183,7 @@ mod tests {
             "test-path",
             1024, // length
             1024, // shard_size
-            HashAlgorithm::HighwayHash256,
+            HashAlgorithm::HighwayHash256S,
         )
         .await;
 

crates/ecstore/src/bucket/bandwidth/monitor.rs

@@ -14,11 +14,16 @@
 
 use crate::bucket::bandwidth::reader::BucketOptions;
 use ratelimit::{Error as RatelimitError, Ratelimiter};
+use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
 use std::sync::{Arc, Mutex, RwLock};
-use std::time::Duration;
+use std::time::{Duration, Instant};
 use tracing::warn;
 
+/// BETA_BUCKET is the weight used to calculate exponential moving average
+const BETA_BUCKET: f64 = 0.1;
+
 #[derive(Clone)]
 pub struct BucketThrottle {
     limiter: Arc<Mutex<Ratelimiter>>,
@@ -71,25 +76,202 @@ impl BucketThrottle {
     }
 }
 
+#[derive(Debug)]
+pub struct BucketMeasurement {
+    bytes_since_last_window: AtomicU64,
+    start_time: Mutex<Option<Instant>>,
+    exp_moving_avg: Mutex<f64>,
+}
+
+impl BucketMeasurement {
+    pub fn new(init_time: Instant) -> Self {
+        Self {
+            bytes_since_last_window: AtomicU64::new(0),
+            start_time: Mutex::new(Some(init_time)),
+            exp_moving_avg: Mutex::new(0.0),
+        }
+    }
+
+    pub fn increment_bytes(&self, bytes: u64) {
+        self.bytes_since_last_window.fetch_add(bytes, Ordering::Relaxed);
+    }
+
+    pub fn update_exponential_moving_average(&self, end_time: Instant) {
+        let mut start_time = self.start_time.lock().unwrap_or_else(|e| {
+            warn!("bucket measurement start_time mutex poisoned, recovering");
+            e.into_inner()
+        });
+        let previous_start = *start_time;
+        *start_time = Some(end_time);
+
+        let Some(prev_start) = previous_start else {
+            return;
+        };
+
+        if prev_start > end_time {
+            return;
+        }
+
+        let duration = end_time.duration_since(prev_start);
+        if duration.is_zero() {
+            return;
+        }
+
+        let bytes_since_last_window = self.bytes_since_last_window.swap(0, Ordering::Relaxed);
+        let increment = bytes_since_last_window as f64 / duration.as_secs_f64();
+
+        let mut exp_moving_avg = self.exp_moving_avg.lock().unwrap_or_else(|e| {
+            warn!("bucket measurement exp_moving_avg mutex poisoned, recovering");
+            e.into_inner()
+        });
+
+        if *exp_moving_avg == 0.0 {
+            *exp_moving_avg = increment;
+            return;
+        }
+
+        *exp_moving_avg = exponential_moving_average(BETA_BUCKET, *exp_moving_avg, increment);
+    }
+
+    pub fn get_exp_moving_avg_bytes_per_second(&self) -> f64 {
+        *self.exp_moving_avg.lock().unwrap_or_else(|e| {
+            warn!("bucket measurement exp_moving_avg mutex poisoned, recovering");
+            e.into_inner()
+        })
+    }
+}
+
+fn exponential_moving_average(beta: f64, previous_avg: f64, increment_avg: f64) -> f64 {
+    (1f64 - beta) * increment_avg + beta * previous_avg
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BandwidthDetails {
+    pub limit_bytes_per_sec: i64,
+    pub current_bandwidth_bytes_per_sec: f64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BucketBandwidthReport {
+    pub bucket_stats: HashMap<BucketOptions, BandwidthDetails>,
+}
+
 pub struct Monitor {
     t_lock: RwLock<HashMap<BucketOptions, BucketThrottle>>,
+    m_lock: RwLock<HashMap<BucketOptions, BucketMeasurement>>,
     pub node_count: u64,
 }
 
 impl Monitor {
     pub fn new(num_nodes: u64) -> Arc<Self> {
         let node_cnt = num_nodes.max(1);
-        Arc::new(Monitor {
+        let m = Arc::new(Monitor {
             t_lock: RwLock::new(HashMap::new()),
+            m_lock: RwLock::new(HashMap::new()),
             node_count: node_cnt,
-        })
+        });
+        if let Ok(handle) = tokio::runtime::Handle::try_current() {
+            let weak = Arc::downgrade(&m);
+            handle.spawn(async move {
+                let mut interval = tokio::time::interval(Duration::from_secs(2));
+                loop {
+                    interval.tick().await;
+                    let Some(monitor) = weak.upgrade() else { break };
+                    monitor.update_moving_avg();
+                }
+            });
+        }
+        m
+    }
+
+    pub fn init_measurement(&self, opts: &BucketOptions) {
+        let mut guard = self.m_lock.write().unwrap_or_else(|e| {
+            warn!("bucket monitor measurement rwlock write poisoned, recovering");
+            e.into_inner()
+        });
+        guard
+            .entry(opts.clone())
+            .or_insert_with(|| BucketMeasurement::new(Instant::now()));
+    }
+
+    pub fn update_measurement(&self, opts: &BucketOptions, bytes: u64) {
+        {
+            let guard = self.m_lock.read().unwrap_or_else(|e| {
+                warn!("bucket monitor measurement rwlock read poisoned, recovering");
+                e.into_inner()
+            });
+            if let Some(measurement) = guard.get(opts) {
+                measurement.increment_bytes(bytes);
+                return;
+            }
+        }
+
+        // Miss path: write lock + insert once, then increment.
+        let mut guard = self.m_lock.write().unwrap_or_else(|e| {
+            warn!("bucket monitor measurement rwlock write poisoned, recovering");
+            e.into_inner()
+        });
+
+        // Double-check after lock upgrade in case another thread inserted it.
+        let measurement = guard
+            .entry(opts.clone())
+            .or_insert_with(|| BucketMeasurement::new(Instant::now()));
+
+        measurement.increment_bytes(bytes);
+    }
+
+    pub fn update_moving_avg(&self) {
+        let now = Instant::now();
+        let guard = self.m_lock.read().unwrap_or_else(|e| {
+            warn!("bucket monitor measurement rwlock read poisoned, recovering");
+            e.into_inner()
+        });
+        for measurement in guard.values() {
+            measurement.update_exponential_moving_average(now);
+        }
+    }
+
+    pub fn get_report(&self, select_bucket: impl Fn(&str) -> bool) -> BucketBandwidthReport {
+        let t_guard = self.t_lock.read().unwrap_or_else(|e| {
+            warn!("bucket monitor throttle rwlock read poisoned, recovering");
+            e.into_inner()
+        });
+        let m_guard = self.m_lock.read().unwrap_or_else(|e| {
+            warn!("bucket monitor measurement rwlock read poisoned, recovering");
+            e.into_inner()
+        });
+        let mut bucket_stats = HashMap::new();
+        for (opts, throttle) in t_guard.iter() {
+            if !select_bucket(&opts.name) {
+                continue;
+            }
+            let mut current_bandwidth_bytes_per_sec = 0.0;
+            if let Some(measurement) = m_guard.get(opts) {
+                current_bandwidth_bytes_per_sec = measurement.get_exp_moving_avg_bytes_per_second();
+            }
+            bucket_stats.insert(
+                opts.clone(),
+                BandwidthDetails {
+                    limit_bytes_per_sec: throttle.node_bandwidth_per_sec * self.node_count as i64,
+                    current_bandwidth_bytes_per_sec,
+                },
+            );
+        }
+        BucketBandwidthReport { bucket_stats }
     }
 
     pub fn delete_bucket(&self, bucket: &str) {
         self.t_lock
             .write()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock write poisoned, recovering");
+                warn!("bucket monitor throttle rwlock write poisoned, recovering");
+                e.into_inner()
+            })
+            .retain(|opts, _| opts.name != bucket);
+        self.m_lock
+            .write()
+            .unwrap_or_else(|e| {
+                warn!("bucket monitor measurement rwlock write poisoned, recovering");
                 e.into_inner()
             })
             .retain(|opts, _| opts.name != bucket);
@@ -103,7 +285,14 @@ impl Monitor {
         self.t_lock
             .write()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock write poisoned, recovering");
+                warn!("bucket monitor throttle rwlock write poisoned, recovering");
+                e.into_inner()
+            })
+            .remove(&opts);
+        self.m_lock
+            .write()
+            .unwrap_or_else(|e| {
+                warn!("bucket monitor measurement rwlock write poisoned, recovering");
                 e.into_inner()
             })
             .remove(&opts);
@@ -113,7 +302,7 @@ impl Monitor {
         self.t_lock
             .read()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock read poisoned, recovering");
+                warn!("bucket monitor throttle rwlock read poisoned, recovering");
                 e.into_inner()
             })
             .get(opts)
@@ -160,7 +349,7 @@ impl Monitor {
         self.t_lock
             .write()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock write poisoned, recovering");
+                warn!("bucket monitor throttle rwlock write poisoned, recovering");
                 e.into_inner()
             })
             .insert(opts, throttle);
@@ -174,7 +363,7 @@ impl Monitor {
         self.t_lock
             .read()
             .unwrap_or_else(|e| {
-                warn!("bucket monitor rwlock read poisoned, recovering");
+                warn!("bucket monitor throttle rwlock read poisoned, recovering");
                 e.into_inner()
             })
             .contains_key(&opt)
@@ -184,6 +373,7 @@ impl Monitor {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use std::panic::{AssertUnwindSafe, catch_unwind};
 
     #[test]
     fn test_set_and_get_throttle_with_node_split() {
@@ -318,4 +508,82 @@ mod tests {
         assert_eq!(t2.burst(), 500);
         assert_eq!(t2.node_bandwidth_per_sec, 500);
     }
+
+    #[test]
+    fn test_bucket_measurement_recovers_from_poisoned_mutexes() {
+        let measurement = BucketMeasurement::new(Instant::now());
+
+        let _ = catch_unwind(AssertUnwindSafe(|| {
+            let _guard = measurement.start_time.lock().unwrap();
+            panic!("poison start_time mutex");
+        }));
+        measurement.increment_bytes(64);
+        measurement.update_exponential_moving_average(Instant::now() + Duration::from_secs(1));
+
+        let _ = catch_unwind(AssertUnwindSafe(|| {
+            let _guard = measurement.exp_moving_avg.lock().unwrap();
+            panic!("poison exp_moving_avg mutex");
+        }));
+        measurement.increment_bytes(32);
+        measurement.update_exponential_moving_average(Instant::now() + Duration::from_secs(2));
+
+        let value = measurement.get_exp_moving_avg_bytes_per_second();
+        assert!(value.is_finite());
+        assert!(value >= 0.0);
+    }
+
+    #[test]
+    fn test_get_report_limit_and_current_bandwidth_after_measurement() {
+        let monitor = Monitor::new(4);
+        monitor.set_bandwidth_limit("b1", "arn1", 400);
+        let opts = BucketOptions {
+            name: "b1".to_string(),
+            replication_arn: "arn1".to_string(),
+        };
+        monitor.init_measurement(&opts);
+        monitor.update_measurement(&opts, 500);
+        monitor.update_measurement(&opts, 500);
+        std::thread::sleep(Duration::from_millis(110));
+        monitor.update_moving_avg();
+
+        let report = monitor.get_report(|name| name == "b1");
+        let details = report.bucket_stats.get(&opts).expect("report should contain b1/arn1");
+        assert_eq!(details.limit_bytes_per_sec, 400);
+        assert!(
+            details.current_bandwidth_bytes_per_sec > 0.0,
+            "current_bandwidth should be positive after update_measurement and update_moving_avg"
+        );
+        assert!(
+            details.current_bandwidth_bytes_per_sec < 20000.0,
+            "current_bandwidth should be in reasonable range"
+        );
+    }
+
+    #[test]
+    fn test_get_report_select_bucket_filters() {
+        let monitor = Monitor::new(2);
+        monitor.set_bandwidth_limit("b1", "arn1", 100);
+        monitor.set_bandwidth_limit("b2", "arn2", 200);
+        let opts_b1 = BucketOptions {
+            name: "b1".to_string(),
+            replication_arn: "arn1".to_string(),
+        };
+        let opts_b2 = BucketOptions {
+            name: "b2".to_string(),
+            replication_arn: "arn2".to_string(),
+        };
+        monitor.init_measurement(&opts_b1);
+        monitor.init_measurement(&opts_b2);
+
+        let report_all = monitor.get_report(|_| true);
+        assert_eq!(report_all.bucket_stats.len(), 2);
+
+        let report_b1 = monitor.get_report(|name| name == "b1");
+        assert_eq!(report_b1.bucket_stats.len(), 1);
+        assert_eq!(report_b1.bucket_stats.get(&opts_b1).unwrap().limit_bytes_per_sec, 100);
+
+        let report_b2 = monitor.get_report(|name| name == "b2");
+        assert_eq!(report_b2.bucket_stats.len(), 1);
+        assert_eq!(report_b2.bucket_stats.get(&opts_b2).unwrap().limit_bytes_per_sec, 200);
+    }
 }

crates/ecstore/src/bucket/bandwidth/reader.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use crate::bucket::bandwidth::monitor::Monitor;
+use serde::{Deserialize, Serialize};
 use std::pin::Pin;
 use std::sync::Arc;
 use std::task::{Context, Poll};
@@ -20,7 +21,7 @@ use tokio::io::{AsyncRead, ReadBuf};
 use tokio::time::Sleep;
 use tracing::{debug, warn};
 
-#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
 pub struct BucketOptions {
     pub name: String,
     pub replication_arn: String,
@@ -54,6 +55,9 @@ impl<R> MonitoredReader<R> {
             header_size = opts.header_size,
             "MonitoredReader created"
         );
+        if throttle.is_some() {
+            m.init_measurement(&opts.bucket_options);
+        }
         MonitoredReader {
             r,
             m,
@@ -117,7 +121,15 @@ impl<R: AsyncRead + Unpin> AsyncRead for MonitoredReader<R> {
             }
         }
 
-        poll_limited_read(&mut this.r, cx, buf, need, &mut this.temp_buf)
+        let filled_before = buf.filled().len();
+        let result = poll_limited_read(&mut this.r, cx, buf, need, &mut this.temp_buf);
+        if let Poll::Ready(Ok(())) = result {
+            let read_bytes = buf.filled().len().saturating_sub(filled_before) as u64;
+            if read_bytes > 0 {
+                this.m.update_measurement(&this.opts.bucket_options, read_bytes);
+            }
+        }
+        result
     }
 }
 

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -43,11 +43,13 @@ use rustfs_config::{DEFAULT_TRUST_LEAF_CERT_AS_CA, ENV_TRUST_LEAF_CERT_AS_CA, RU
 use rustfs_filemeta::{ReplicationStatusType, ReplicationType};
 use rustfs_utils::http::{
     AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE,
-    AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, RUSTFS_BUCKET_REPLICATION_CHECK,
-    RUSTFS_BUCKET_REPLICATION_DELETE_MARKER, RUSTFS_BUCKET_REPLICATION_REQUEST, RUSTFS_BUCKET_SOURCE_ETAG,
-    RUSTFS_BUCKET_SOURCE_MTIME, RUSTFS_BUCKET_SOURCE_VERSION_ID, RUSTFS_FORCE_DELETE, is_amz_header, is_minio_header,
+    AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, is_amz_header, is_minio_header,
     is_rustfs_header, is_standard_header, is_storageclass_header,
 };
+use rustfs_utils::http::{
+    SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
+    SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, insert_header,
+};
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::error::Error;
@@ -68,8 +70,6 @@ use uuid::Uuid;
 const DEFAULT_HEALTH_CHECK_DURATION: Duration = Duration::from_secs(5);
 const DEFAULT_HEALTH_CHECK_RELOAD_DURATION: Duration = Duration::from_secs(30 * 60);
 
-const REPLICATION_REQUEST_TRUE: HeaderValue = HeaderValue::from_static("true");
-
 pub static GLOBAL_BUCKET_TARGET_SYS: OnceLock<BucketTargetSys> = OnceLock::new();
 
 #[derive(Debug, Clone)]
@@ -804,7 +804,7 @@ impl BucketTargetSys {
 }
 
 async fn build_aws_s3_http_client_from_tls_path() -> Option<aws_sdk_s3::config::SharedHttpClient> {
-    let tls_path = std::env::var("RUSTFS_TLS_PATH").ok()?;
+    let tls_path = rustfs_utils::get_env_str(rustfs_config::ENV_RUSTFS_TLS_PATH, rustfs_config::DEFAULT_RUSTFS_TLS_PATH);
     if tls_path.is_empty() {
         return None;
     }
@@ -862,7 +862,6 @@ fn should_force_path_style(target: &BucketTarget) -> bool {
         // Explicit path-style or legacy boolean-like values.
         "path" | "on" | "true" => true,
         // `auto` and empty are defaulted to path-style for custom S3-compatible endpoints.
-        // RustFS/MinIO-style deployments typically do not configure virtual-hosted-style routing.
         "auto" | "" => true,
         // Unknown values: prefer compatibility with S3-compatible services.
         _ => true,
@@ -1082,23 +1081,21 @@ impl PutObjectOptions {
         }
 
         if !self.internal.source_version_id.is_empty() {
-            header.insert(
-                RUSTFS_BUCKET_SOURCE_VERSION_ID,
-                HeaderValue::from_str(&self.internal.source_version_id).expect("err"),
-            );
+            insert_header(&mut header, SUFFIX_SOURCE_VERSION_ID, &self.internal.source_version_id);
         }
         if self.internal.source_etag.is_empty() {
-            header.insert(RUSTFS_BUCKET_SOURCE_ETAG, HeaderValue::from_str(&self.internal.source_etag).expect("err"));
+            insert_header(&mut header, SUFFIX_SOURCE_ETAG, &self.internal.source_etag);
         }
         if self.internal.source_mtime.unix_timestamp() != 0 {
-            header.insert(
-                RUSTFS_BUCKET_SOURCE_MTIME,
-                HeaderValue::from_str(&self.internal.source_mtime.format(&Rfc3339).unwrap_or_default()).expect("err"),
+            insert_header(
+                &mut header,
+                SUFFIX_SOURCE_MTIME,
+                self.internal.source_mtime.format(&Rfc3339).unwrap_or_default(),
             );
         }
 
         if self.internal.replication_request {
-            header.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, REPLICATION_REQUEST_TRUE);
+            insert_header(&mut header, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
 
         header
@@ -1267,10 +1264,8 @@ impl TargetClient {
         let builder = self.client.put_object();
 
         let version_id = opts.internal.source_version_id.clone();
-        if !version_id.is_empty()
-            && let Ok(header_value) = HeaderValue::from_str(&version_id)
-        {
-            headers.insert(RUSTFS_BUCKET_SOURCE_VERSION_ID, header_value);
+        if !version_id.is_empty() {
+            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
         }
 
         match builder
@@ -1304,13 +1299,11 @@ impl TargetClient {
     ) -> Result<String, S3ClientError> {
         let mut headers = HeaderMap::new();
         let version_id = opts.internal.source_version_id.clone();
-        if !version_id.is_empty()
-            && let Ok(header_value) = HeaderValue::from_str(&version_id)
-        {
-            headers.insert(RUSTFS_BUCKET_SOURCE_VERSION_ID, header_value);
+        if !version_id.is_empty() {
+            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
         }
         if opts.internal.replication_request {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, REPLICATION_REQUEST_TRUE);
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
 
         match self
@@ -1419,21 +1412,18 @@ impl TargetClient {
     ) -> Result<(), S3ClientError> {
         let mut headers = HeaderMap::new();
         if opts.force_delete {
-            headers.insert(RUSTFS_FORCE_DELETE, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_FORCE_DELETE, "true");
         }
         if opts.governance_bypass {
             headers.insert(AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, "true".parse().unwrap());
         }
 
         if opts.replication_delete_marker {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_DELETE_MARKER, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_DELETEMARKER, "true");
         }
 
         if let Some(t) = opts.replication_mtime {
-            headers.insert(
-                RUSTFS_BUCKET_SOURCE_MTIME,
-                t.format(&Rfc3339).unwrap_or_default().as_str().parse().unwrap(),
-            );
+            insert_header(&mut headers, SUFFIX_SOURCE_MTIME, t.format(&Rfc3339).unwrap_or_default());
         }
 
         if !opts.replication_status.is_empty() {
@@ -1441,10 +1431,10 @@ impl TargetClient {
         }
 
         if opts.replication_request {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
         if opts.replication_validity_check {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_CHECK, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
         }
 
         match self

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -28,7 +28,6 @@ use crate::client::object_api_utils::new_getobjectreader;
 use crate::error::Error;
 use crate::error::StorageError;
 use crate::error::{error_resp_to_object_err, is_err_object_not_found, is_err_version_not_found, is_network_or_host_down};
-use crate::event::name::EventName;
 use crate::event_notification::{EventArgs, send_event};
 use crate::global::GLOBAL_LocalNodeName;
 use crate::global::{GLOBAL_LifecycleSys, GLOBAL_TierConfigMgr, get_global_deployment_id};
@@ -45,8 +44,8 @@ use rustfs_common::data_usage::TierStats;
 use rustfs_common::heal_channel::rep_has_active_rules;
 use rustfs_common::metrics::{IlmAction, Metrics};
 use rustfs_filemeta::{NULL_VERSION_ID, RestoreStatusOps, is_restored_object_on_disk};
-use rustfs_utils::path::encode_dir_object;
-use rustfs_utils::string::strings_has_prefix_fold;
+use rustfs_s3_common::EventName;
+use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::strings_has_prefix_fold};
 use s3s::Body;
 use s3s::dto::{
     BucketLifecycleConfiguration, DefaultRetention, ReplicationConfiguration, RestoreRequest, RestoreRequestType, RestoreStatus,
@@ -97,8 +96,14 @@ impl LifecycleSys {
     }
 
     pub async fn get(&self, bucket: &str) -> Option<BucketLifecycleConfiguration> {
-        let lc = get_lifecycle_config(bucket).await.expect("get_lifecycle_config err!").0;
-        Some(lc)
+        match get_lifecycle_config(bucket).await {
+            Ok((lc, _)) => Some(lc),
+            Err(err) if err == Error::ConfigNotFound => None,
+            Err(err) => {
+                warn!(bucket, error = ?err, "failed to load lifecycle config");
+                None
+            }
+        }
     }
 
     pub fn trace(_oi: &ObjectInfo) -> TraceFn {
@@ -471,10 +476,7 @@ impl TransitionState {
     }
 
     pub async fn init(api: Arc<ECStore>) {
-        let max_workers = std::env::var("RUSTFS_MAX_TRANSITION_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<i64>().ok())
-            .unwrap_or_else(|| std::cmp::min(num_cpus::get() as i64, 16));
+        let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
         let mut n = max_workers;
         let tw = 8; //globalILMConfig.getTransitionWorkers();
         if tw > 0 {
@@ -569,17 +571,11 @@ impl TransitionState {
     pub async fn update_workers_inner(api: Arc<ECStore>, n: i64) {
         let mut n = n;
         if n == 0 {
-            let max_workers = std::env::var("RUSTFS_MAX_TRANSITION_WORKERS")
-                .ok()
-                .and_then(|s| s.parse::<i64>().ok())
-                .unwrap_or_else(|| std::cmp::min(num_cpus::get() as i64, 16));
+            let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
             n = max_workers;
         }
         // Allow environment override of maximum workers
-        let absolute_max = std::env::var("RUSTFS_ABSOLUTE_MAX_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<i64>().ok())
-            .unwrap_or(32);
+        let absolute_max = get_env_i64("RUSTFS_ABSOLUTE_MAX_WORKERS", 32);
         n = std::cmp::min(n, absolute_max);
 
         let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
@@ -603,10 +599,7 @@ impl TransitionState {
 }
 
 pub async fn init_background_expiry(api: Arc<ECStore>) {
-    let mut workers = std::env::var("RUSTFS_MAX_EXPIRY_WORKERS")
-        .ok()
-        .and_then(|s| s.parse::<usize>().ok())
-        .unwrap_or_else(|| std::cmp::min(num_cpus::get(), 16));
+    let mut workers = get_env_usize("RUSTFS_MAX_EXPIRY_WORKERS", std::cmp::min(num_cpus::get(), 16));
     //globalILMConfig.getExpirationWorkers()
     if let Ok(env_expiration_workers) = env::var("_RUSTFS_ILM_EXPIRATION_WORKERS") {
         if let Ok(num_expirations) = env_expiration_workers.parse::<usize>() {
@@ -615,10 +608,7 @@ pub async fn init_background_expiry(api: Arc<ECStore>) {
     }
 
     if workers == 0 {
-        workers = std::env::var("RUSTFS_DEFAULT_EXPIRY_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<usize>().ok())
-            .unwrap_or(8);
+        workers = get_env_usize("RUSTFS_DEFAULT_EXPIRY_WORKERS", 8);
     }
 
     //let expiry_state = GLOBAL_ExpiryStSate.write().await;
@@ -689,13 +679,13 @@ pub async fn expire_transitioned_object(
     //let tags = LcAuditEvent::new(src, lcEvent).Tags();
     if lc_event.action == IlmAction::DeleteRestoredAction {
         opts.transition.expire_restored = true;
-        match api.delete_object(&oi.bucket, &oi.name, opts).await {
+        return match api.delete_object(&oi.bucket, &oi.name, opts).await {
             Ok(dobj) => {
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
-                return Ok(dobj);
+                Ok(dobj)
             }
-            Err(err) => return Err(std::io::Error::other(err)),
-        }
+            Err(err) => Err(std::io::Error::other(err)),
+        };
     }
 
     let ret = delete_object_from_remote_tier(
@@ -732,7 +722,7 @@ pub async fn expire_transitioned_object(
         ..Default::default()
     };
     send_event(EventArgs {
-        event_name: event_name.as_ref().to_string(),
+        event_name: event_name.to_string(),
         bucket_name: obj_info.bucket.clone(),
         object: obj_info,
         user_agent: "Internal: [ILM-Expiry]".to_string(),
@@ -847,8 +837,8 @@ pub async fn post_restore_opts(version_id: &str, bucket: &str, object: &str) ->
         }
     }
     Ok(ObjectOptions {
-        versioned: versioned,
-        version_suspended: version_suspended,
+        versioned,
+        version_suspended,
         version_id: Some(vid.to_string()),
         ..Default::default()
     })
@@ -1033,12 +1023,12 @@ pub async fn eval_action_from_lifecycle(
     let lock_enabled = if let Some(lr) = lr { lr.mode.is_some() } else { false };
 
     match event.action {
-        lifecycle::IlmAction::DeleteAllVersionsAction | lifecycle::IlmAction::DelMarkerDeleteAllVersionsAction => {
+        IlmAction::DeleteAllVersionsAction | IlmAction::DelMarkerDeleteAllVersionsAction => {
             if lock_enabled {
                 return lifecycle::Event::default();
             }
         }
-        lifecycle::IlmAction::DeleteVersionAction | lifecycle::IlmAction::DeleteRestoredVersionAction => {
+        IlmAction::DeleteVersionAction | IlmAction::DeleteRestoredVersionAction => {
             if oi.version_id.is_none() {
                 return lifecycle::Event::default();
             }
@@ -1139,12 +1129,12 @@ pub async fn apply_expiry_on_non_transitioned_objects(
         event_name = EventName::ObjectRemovedDeleteMarkerCreated;
     }
     match lc_event.action {
-        lifecycle::IlmAction::DeleteAllVersionsAction => event_name = EventName::ObjectRemovedDeleteAllVersions,
-        lifecycle::IlmAction::DelMarkerDeleteAllVersionsAction => event_name = EventName::ILMDelMarkerExpirationDelete,
+        IlmAction::DeleteAllVersionsAction => event_name = EventName::ObjectRemovedDeleteAllVersions,
+        IlmAction::DelMarkerDeleteAllVersionsAction => event_name = EventName::LifecycleDelMarkerExpirationDelete,
         _ => (),
     }
     send_event(EventArgs {
-        event_name: event_name.as_ref().to_string(),
+        event_name: event_name.to_string(),
         bucket_name: dobj.bucket.clone(),
         object: dobj,
         user_agent: "Internal: [ILM-Expiry]".to_string(),
@@ -1152,7 +1142,7 @@ pub async fn apply_expiry_on_non_transitioned_objects(
         ..Default::default()
     });
 
-    if lc_event.action != lifecycle::IlmAction::NoneAction {
+    if lc_event.action != IlmAction::NoneAction {
         let mut num_versions = 1_u64;
         if lc_event.action.delete_all() {
             num_versions = oi.num_versions as u64;
@@ -1172,15 +1162,15 @@ pub async fn apply_expiry_rule(event: &lifecycle::Event, src: &LcEventSrc, oi: &
 pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc, oi: &ObjectInfo) -> bool {
     let mut success = false;
     match event.action {
-        lifecycle::IlmAction::DeleteVersionAction
-        | lifecycle::IlmAction::DeleteAction
-        | lifecycle::IlmAction::DeleteRestoredAction
-        | lifecycle::IlmAction::DeleteRestoredVersionAction
-        | lifecycle::IlmAction::DeleteAllVersionsAction
-        | lifecycle::IlmAction::DelMarkerDeleteAllVersionsAction => {
+        IlmAction::DeleteVersionAction
+        | IlmAction::DeleteAction
+        | IlmAction::DeleteRestoredAction
+        | IlmAction::DeleteRestoredVersionAction
+        | IlmAction::DeleteAllVersionsAction
+        | IlmAction::DelMarkerDeleteAllVersionsAction => {
             success = apply_expiry_rule(event, src, oi).await;
         }
-        lifecycle::IlmAction::TransitionAction | lifecycle::IlmAction::TransitionVersionAction => {
+        IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
             success = apply_transition_rule(event, src, oi).await;
         }
         _ => (),

crates/ecstore/src/bucket/lifecycle/lifecycle.rs

@@ -20,8 +20,8 @@
 
 use rustfs_filemeta::{ReplicationStatusType, VersionPurgeStatusType};
 use s3s::dto::{
-    BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, NoncurrentVersionTransition,
-    ObjectLockConfiguration, ObjectLockEnabled, RestoreRequest, Transition, TransitionStorageClass,
+    BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleAndOperator,
+    NoncurrentVersionTransition, ObjectLockConfiguration, ObjectLockEnabled, RestoreRequest, Transition, TransitionStorageClass,
 };
 use std::cmp::Ordering;
 use std::collections::HashMap;
@@ -49,6 +49,8 @@ const ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS: &str = "Lifecycle expiration days m
 const ERR_LIFECYCLE_INVALID_EXPIRATION_DATE_NOT_MIDNIGHT: &str = "Expiration.Date must be at midnight UTC";
 const ERR_LIFECYCLE_INVALID_RULE_ID_TOO_LONG: &str = "Rule ID must be at most 255 characters";
 const ERR_LIFECYCLE_INVALID_RULE_STATUS: &str = "Rule status must be either Enabled or Disabled";
+const ERR_LIFECYCLE_DEL_MARKER_WITH_TAGS: &str = "Rule with DelMarkerExpiration cannot have tags based filtering";
+const ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION: &str = "Rule must have at least one of Expiration, Transition, NoncurrentVersionExpiration, NoncurrentVersionTransition, or DelMarkerExpiration";
 
 pub use rustfs_common::metrics::IlmAction;
 
@@ -117,23 +119,66 @@ impl RuleValidate for LifecycleRule {
     }*/
 
     fn validate(&self) -> Result<(), std::io::Error> {
-        /*self.validate_id()?;
-        self.validate_status()?;
-        self.validate_expiration()?;
-        self.validate_noncurrent_expiration()?;
-        self.validate_prefix_and_filter()?;
-        self.validate_transition()?;
-        self.validate_noncurrent_transition()?;
-        if (!self.Filter.Tag.IsEmpty() || len(self.Filter.And.Tags) != 0) && !self.delmarker_expiration.Empty() {
-          return errInvalidRuleDelMarkerExpiration
+        // Rule with DelMarkerExpiration cannot have tags based filtering
+        let has_tag_filter = self
+            .filter
+            .as_ref()
+            .map_or(false, |f| f.tag.is_some() || f.and.as_ref().and_then(|a| a.tags.as_ref()).is_some());
+        if has_tag_filter && self.del_marker_expiration.is_some() {
+            return Err(std::io::Error::other(ERR_LIFECYCLE_DEL_MARKER_WITH_TAGS));
+        }
+        // Rule must have at least one action
+        let has_expiration = self.expiration.is_some();
+        let has_transition = self.transitions.as_ref().map_or(false, |t| !t.is_empty());
+        let has_noncurrent_expiration = self
+            .noncurrent_version_expiration
+            .as_ref()
+            .and_then(|e| e.noncurrent_days)
+            .map_or(false, |d| d != 0);
+        let has_noncurrent_transition = self
+            .noncurrent_version_transitions
+            .as_ref()
+            .and_then(|t| t.first())
+            .and_then(|t| t.storage_class.as_ref())
+            .is_some();
+        let has_abort_incomplete_multipart_upload = self.abort_incomplete_multipart_upload.is_some();
+        let has_del_marker_expiration = self
+            .del_marker_expiration
+            .as_ref()
+            .and_then(|d| d.days)
+            .map_or(false, |d| d > 0);
+        if !has_expiration
+            && !has_transition
+            && !has_noncurrent_expiration
+            && !has_noncurrent_transition
+            && !has_abort_incomplete_multipart_upload
+            && !has_del_marker_expiration
+        {
+            return Err(std::io::Error::other(ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION));
         }
-        if !self.expiration.set && !self.transition.set && !self.noncurrent_version_expiration.set && !self.noncurrent_version_transitions.unwrap()[0].set && self.delmarker_expiration.Empty() {
-          return errXMLNotWellFormed
-        }*/
         Ok(())
     }
 }
 
+fn lifecycle_rule_prefix(rule: &LifecycleRule) -> Option<&str> {
+    // Prefer a non-empty legacy prefix; treat an empty legacy prefix as if it were not set
+    if let Some(p) = rule.prefix.as_deref() {
+        if !p.is_empty() {
+            return Some(p);
+        }
+    }
+
+    let Some(filter) = rule.filter.as_ref() else {
+        return None;
+    };
+
+    if let Some(p) = filter.prefix.as_deref() {
+        return Some(p);
+    }
+
+    filter.and.as_ref().and_then(|and| and.prefix.as_deref())
+}
+
 #[async_trait::async_trait]
 pub trait Lifecycle {
     async fn has_transition(&self) -> bool;
@@ -177,8 +222,11 @@ impl Lifecycle for BucketLifecycleConfiguration {
                 continue;
             }
 
-            let rule_prefix = &rule.prefix.clone().unwrap_or_default();
-            if prefix.len() > 0 && rule_prefix.len() > 0 && !prefix.starts_with(rule_prefix) && !rule_prefix.starts_with(&prefix)
+            let rule_prefix = lifecycle_rule_prefix(rule).unwrap_or("");
+            if !prefix.is_empty()
+                && !rule_prefix.is_empty()
+                && !prefix.starts_with(rule_prefix)
+                && !rule_prefix.starts_with(prefix)
             {
                 continue;
             }
@@ -297,8 +345,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
             if rule.status.as_str() == ExpirationStatus::DISABLED {
                 continue;
             }
-            if let Some(prefix) = rule.prefix.clone() {
-                if !obj.name.starts_with(prefix.as_str()) {
+            if let Some(rule_prefix) = lifecycle_rule_prefix(rule) {
+                if !obj.name.starts_with(rule_prefix) {
                     continue;
                 }
             }
@@ -414,43 +462,32 @@ impl Lifecycle for BucketLifecycleConfiguration {
 
         if let Some(ref lc_rules) = self.filter_rules(obj).await {
             for rule in lc_rules.iter() {
-                if obj.expired_object_deletemarker() {
+                if obj.is_latest && obj.expired_object_deletemarker() {
                     if let Some(expiration) = rule.expiration.as_ref() {
-                        if let Some(expired_object_delete_marker) = expiration.expired_object_delete_marker {
-                            events.push(Event {
-                                action: IlmAction::DeleteVersionAction,
-                                rule_id: rule.id.clone().unwrap_or_default(),
-                                due: Some(now),
-                                noncurrent_days: 0,
-                                newer_noncurrent_versions: 0,
-                                storage_class: "".into(),
-                            });
-                            break;
-                        }
-
-                        if let Some(days) = expiration.days {
-                            let expected_expiry = expected_expiry_time(mod_time, days /*, date*/);
-                            if now.unix_timestamp() >= expected_expiry.unix_timestamp() {
+                        if expiration.expired_object_delete_marker.is_some_and(|v| v) {
+                            // Preserve explicit date/days scheduling when configured.
+                            // If only ExpiredObjectDeleteMarker=true is set, delete immediately.
+                            let due = expiration.next_due(obj).unwrap_or(now);
+                            if now.unix_timestamp() >= due.unix_timestamp() {
                                 events.push(Event {
                                     action: IlmAction::DeleteVersionAction,
                                     rule_id: rule.id.clone().unwrap_or_default(),
-                                    due: Some(expected_expiry),
+                                    due: Some(due),
                                     noncurrent_days: 0,
                                     newer_noncurrent_versions: 0,
                                     storage_class: "".into(),
                                 });
+                                // Stop after scheduling an expired delete-marker event.
                                 break;
                             }
                         }
                     }
-                }
-
-                if obj.is_latest {
-                    if let Some(ref expiration) = rule.expiration {
-                        if let Some(expired_object_delete_marker) = expiration.expired_object_delete_marker {
-                            if obj.delete_marker && expired_object_delete_marker {
-                                let due = expiration.next_due(obj);
-                                if let Some(due) = due {
+                    // DelMarkerExpiration: expire delete marker after N days from mod_time
+                    if obj.delete_marker {
+                        if let Some(ref dme) = rule.del_marker_expiration {
+                            if let Some(days) = dme.days {
+                                if days > 0 {
+                                    let due = expected_expiry_time(mod_time, days);
                                     if now.unix_timestamp() >= due.unix_timestamp() {
                                         events.push(Event {
                                             action: IlmAction::DelMarkerDeleteAllVersionsAction,
@@ -461,8 +498,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
                                             storage_class: "".into(),
                                         });
                                     }
+                                    continue;
                                 }
-                                continue;
                             }
                         }
                     }
@@ -694,8 +731,16 @@ impl LifecycleCalculate for LifecycleExpiration {
         if !obj.is_latest || !obj.delete_marker {
             return None;
         }
+        // Check date first (date-based expiration takes priority over days).
+        // A zero unix timestamp means "not set" (default value) and is skipped.
+        if let Some(ref date) = self.date {
+            let expiry_date = OffsetDateTime::from(date.clone());
+            if expiry_date.unix_timestamp() != 0 {
+                return Some(expiry_date);
+            }
+        }
         match self.days {
-            Some(days) => Some(expected_expiry_time(obj.mod_time.unwrap(), days)),
+            Some(days) => obj.mod_time.map(|mod_time| expected_expiry_time(mod_time, days)),
             None => None,
         }
     }
@@ -860,10 +905,15 @@ impl Default for TransitionOptions {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use s3s::dto::LifecycleRuleFilter;
+    use serial_test::serial;
+    use std::sync::Arc;
 
     #[tokio::test]
+    #[serial]
     async fn validate_rejects_non_positive_expiration_days() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -871,6 +921,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -889,8 +940,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_accepts_positive_expiration_days() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -898,6 +951,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -913,8 +967,36 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
+    async fn validate_accepts_abort_incomplete_multipart_upload_only_rule() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: Some(s3s::dto::AbortIncompleteMultipartUpload {
+                    days_after_initiation: Some(2),
+                }),
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("abort-only".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: Some("test/".to_string()),
+                transitions: None,
+            }],
+        };
+
+        lc.validate(&ObjectLockConfiguration::default())
+            .await
+            .expect("expected validation to pass");
+    }
+
+    #[tokio::test]
+    #[serial]
     async fn validate_rejects_non_midnight_expiration_date() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -922,6 +1004,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -937,9 +1020,11 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn predict_expiration_selects_closest_expiry_for_put_object() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![
                 LifecycleRule {
                     status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
@@ -948,6 +1033,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: Some("rule-days".to_string()),
                     noncurrent_version_expiration: None,
@@ -962,6 +1048,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: Some("rule-date".to_string()),
                     noncurrent_version_expiration: None,
@@ -988,8 +1075,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_accepts_multiple_rules_without_ids() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![
                 LifecycleRule {
                     status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
@@ -998,6 +1087,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: None,
                     noncurrent_version_expiration: None,
@@ -1012,6 +1102,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: None,
                     noncurrent_version_expiration: None,
@@ -1028,8 +1119,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_rejects_rule_id_too_long() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -1037,6 +1130,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: Some("a".repeat(256)),
                 noncurrent_version_expiration: None,
@@ -1052,8 +1146,276 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
+    async fn validate_rejects_duplicate_rule_ids() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![
+                LifecycleRule {
+                    status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                    expiration: Some(LifecycleExpiration {
+                        days: Some(1),
+                        ..Default::default()
+                    }),
+                    abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
+                    filter: None,
+                    id: Some("dup-rule".to_string()),
+                    noncurrent_version_expiration: None,
+                    noncurrent_version_transitions: None,
+                    prefix: None,
+                    transitions: None,
+                },
+                LifecycleRule {
+                    status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                    expiration: Some(LifecycleExpiration {
+                        days: Some(2),
+                        ..Default::default()
+                    }),
+                    abort_incomplete_multipart_upload: None,
+                    filter: None,
+                    id: Some("dup-rule".to_string()),
+                    noncurrent_version_expiration: None,
+                    noncurrent_version_transitions: None,
+                    prefix: None,
+                    transitions: None,
+                    del_marker_expiration: None,
+                },
+            ],
+        };
+
+        let err = lc.validate(&ObjectLockConfiguration::default()).await.unwrap_err();
+        assert_eq!(err.to_string(), ERR_LIFECYCLE_DUPLICATE_ID);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_expires_latest_object_after_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("expire-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteAction);
+        assert_eq!(event.rule_id, "expire-days");
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_keeps_latest_object_before_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(2),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("expire-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::hours(12), 0).await;
+
+        assert_eq!(event.action, IlmAction::NoneAction);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_transitions_latest_object_after_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("transition-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: Some(vec![Transition {
+                    days: Some(1),
+                    date: None,
+                    storage_class: Some(TransitionStorageClass::from_static("COLDTIER44")),
+                }]),
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            transition_status: "".to_string(),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::TransitionAction);
+        assert_eq!(event.rule_id, "transition-days");
+        assert_eq!(event.storage_class, "COLDTIER44");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_expires_noncurrent_version_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-expire".to_string()),
+                noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
+                    noncurrent_days: Some(1),
+                    newer_noncurrent_versions: None,
+                }),
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            successor_mod_time: Some(base_time),
+            is_latest: false,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-expire");
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_transitions_noncurrent_version_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-transition".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
+                    noncurrent_days: Some(1),
+                    newer_noncurrent_versions: None,
+                    storage_class: Some(TransitionStorageClass::from_static("COLDTIER44")),
+                }]),
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            successor_mod_time: Some(base_time),
+            is_latest: false,
+            transition_status: "".to_string(),
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::TransitionVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-transition");
+        assert_eq!(event.storage_class, "COLDTIER44");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn noncurrent_versions_expiration_limit_returns_configured_limits() {
+        let lc = Arc::new(BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-limit".to_string()),
+                noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
+                    noncurrent_days: Some(7),
+                    newer_noncurrent_versions: Some(3),
+                }),
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        });
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: false,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.noncurrent_versions_expiration_limit(&opts).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-limit");
+        assert_eq!(event.noncurrent_days, 7);
+        assert_eq!(event.newer_noncurrent_versions, 3);
+    }
+
+    #[tokio::test]
+    #[serial]
     async fn validate_rejects_invalid_status_case_sensitive() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static("enabled"),
                 expiration: Some(LifecycleExpiration {
@@ -1061,6 +1423,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -1074,4 +1437,260 @@ mod tests {
 
         assert_eq!(err.to_string(), ERR_LIFECYCLE_INVALID_RULE_STATUS);
     }
+
+    #[tokio::test]
+    #[serial]
+    async fn filter_rules_respects_filter_prefix() {
+        let mut filter = LifecycleRuleFilter::default();
+        filter.prefix = Some("prefix".to_string());
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: Some(filter),
+                id: Some("rule".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let match_obj = ObjectOpts {
+            name: "prefix/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let matched = lc.filter_rules(&match_obj).await.unwrap();
+        assert_eq!(matched.len(), 1);
+
+        let non_match_obj = ObjectOpts {
+            name: "other/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let not_matched = lc.filter_rules(&non_match_obj).await.unwrap();
+        assert_eq!(not_matched.len(), 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn filter_rules_respects_filter_and_prefix() {
+        let mut filter = LifecycleRuleFilter::default();
+
+        let mut and = LifecycleRuleAndOperator::default();
+        and.prefix = Some("prefix".to_string());
+        filter.and = Some(and);
+
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: Some(filter),
+                id: Some("rule-and-prefix".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let match_obj = ObjectOpts {
+            name: "prefix/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let matched = lc.filter_rules(&match_obj).await.unwrap();
+        assert_eq!(matched.len(), 1);
+
+        let non_match_obj = ObjectOpts {
+            name: "other/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let not_matched = lc.filter_rules(&non_match_obj).await.unwrap();
+        assert_eq!(not_matched.len(), 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expired_object_delete_marker_requires_single_version() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 2,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+        assert_eq!(event.action, IlmAction::NoneAction);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expired_object_delete_marker_deletes_only_delete_marker_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    async fn expired_object_delete_marker_without_date_or_days_deletes_immediately() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker-immediate".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.due, Some(now));
+    }
+
+    #[tokio::test]
+    async fn expired_object_delete_marker_date_based_not_yet_due() {
+        // A date-based rule that has not yet reached its expiry date must not
+        // trigger immediate deletion (unwrap_or(now) must not override the date).
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let future_date = base_time + Duration::days(10);
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    date: Some(future_date.into()),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-date-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        // now is before the configured date — must not schedule deletion
+        let now_before = base_time + Duration::days(5);
+        let event_before = lc.eval_inner(&opts, now_before, 0).await;
+        assert_eq!(event_before.action, IlmAction::NoneAction);
+
+        // now is after the configured date — must schedule deletion
+        let now_after = base_time + Duration::days(11);
+        let event_after = lc.eval_inner(&opts, now_after, 0).await;
+        assert_eq!(event_after.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event_after.due, Some(future_date));
+    }
 }

crates/ecstore/src/bucket/metadata.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
 use super::object_lock::ObjectLockApi;
 use super::versioning::VersioningApi;
 use super::{quota::BucketQuota, target::BucketTargets};
@@ -22,7 +23,6 @@ use crate::error::{Error, Result};
 use crate::new_object_layer_fn;
 use crate::store::ECStore;
 use byteorder::{BigEndian, ByteOrder, LittleEndian};
-use rmp_serde::Serializer as rmpSerializer;
 use rustfs_policy::policy::BucketPolicy;
 use s3s::dto::{
     BucketLifecycleConfiguration, CORSConfiguration, NotificationConfiguration, ObjectLockConfiguration,
@@ -30,12 +30,41 @@ use s3s::dto::{
     VersioningConfiguration,
 };
 use serde::Serializer;
-use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
+use std::io::{Read, Write};
 use std::sync::Arc;
 use time::OffsetDateTime;
 use tracing::error;
 
+fn read_msgp_str<R: Read>(rd: &mut R) -> Result<String> {
+    let len = rmp::decode::read_str_len(rd)? as usize;
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    Ok(String::from_utf8(buf)?)
+}
+
+fn read_msgp_time_value<R: Read>(rd: &mut R) -> Result<OffsetDateTime> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    match marker {
+        rmp::Marker::Null => Ok(OffsetDateTime::UNIX_EPOCH),
+        rmp::Marker::Ext8 => read_msgp_ext8_time(rd),
+        _ => Err(Error::other(format!("expected time ext or nil, got marker: {marker:?}"))),
+    }
+}
+
+fn read_msgp_bin<R: Read>(rd: &mut R) -> Result<Vec<u8>> {
+    let len = rmp::decode::read_bin_len(rd)? as usize;
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    Ok(buf)
+}
+
+fn write_bin_field<W: Write>(wr: &mut W, key: &str, val: &[u8]) -> Result<()> {
+    rmp::encode::write_str(wr, key)?;
+    rmp::encode::write_bin(wr, val)?;
+    Ok(())
+}
+
 pub const BUCKET_METADATA_FILE: &str = ".metadata.bin";
 pub const BUCKET_METADATA_FORMAT: u16 = 1;
 pub const BUCKET_METADATA_VERSION: u16 = 1;
@@ -54,8 +83,7 @@ pub const BUCKET_CORS_CONFIG: &str = "cors.xml";
 pub const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = "public-access-block.xml";
 pub const BUCKET_ACL_CONFIG: &str = "bucket-acl.json";
 
-#[derive(Debug, Deserialize, Serialize, Clone)]
-#[serde(rename_all = "PascalCase", default)]
+#[derive(Debug, Clone)]
 pub struct BucketMetadata {
     pub name: String,
     pub created: OffsetDateTime,
@@ -90,36 +118,21 @@ pub struct BucketMetadata {
     pub public_access_block_config_updated_at: OffsetDateTime,
     pub bucket_acl_config_updated_at: OffsetDateTime,
 
-    #[serde(skip)]
     pub new_field_updated_at: OffsetDateTime,
 
-    #[serde(skip)]
     pub policy_config: Option<BucketPolicy>,
-    #[serde(skip)]
     pub notification_config: Option<NotificationConfiguration>,
-    #[serde(skip)]
     pub lifecycle_config: Option<BucketLifecycleConfiguration>,
-    #[serde(skip)]
     pub object_lock_config: Option<ObjectLockConfiguration>,
-    #[serde(skip)]
     pub versioning_config: Option<VersioningConfiguration>,
-    #[serde(skip)]
     pub sse_config: Option<ServerSideEncryptionConfiguration>,
-    #[serde(skip)]
     pub tagging_config: Option<Tagging>,
-    #[serde(skip)]
     pub quota_config: Option<BucketQuota>,
-    #[serde(skip)]
     pub replication_config: Option<ReplicationConfiguration>,
-    #[serde(skip)]
     pub bucket_target_config: Option<BucketTargets>,
-    #[serde(skip)]
     pub bucket_target_config_meta: Option<HashMap<String, String>>,
-    #[serde(skip)]
     pub cors_config: Option<CORSConfiguration>,
-    #[serde(skip)]
     pub public_access_block_config: Option<PublicAccessBlockConfiguration>,
-    #[serde(skip)]
     pub bucket_acl_config: Option<String>,
 }
 
@@ -198,17 +211,137 @@ impl BucketMetadata {
         self.lock_enabled || (self.versioning_config.as_ref().is_some_and(|v| v.enabled()))
     }
 
+    /// Decode from msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
+    pub fn decode_from<R: Read>(&mut self, rd: &mut R) -> Result<()> {
+        let mut fields = rmp::decode::read_map_len(rd)?;
+        *self = Self::default();
+
+        while fields > 0 {
+            fields -= 1;
+
+            let key_len = rmp::decode::read_str_len(rd)?;
+            let mut key_buf = vec![0u8; key_len as usize];
+            rd.read_exact(&mut key_buf)?;
+            let key = String::from_utf8(key_buf)?;
+
+            match key.as_str() {
+                "Name" => self.name = read_msgp_str(rd)?,
+                "Created" => self.created = read_msgp_time_value(rd)?,
+                "LockEnabled" => self.lock_enabled = rmp::decode::read_bool(rd)?,
+                "PolicyConfigJSON" => self.policy_config_json = read_msgp_bin(rd)?,
+                "NotificationConfigXML" => self.notification_config_xml = read_msgp_bin(rd)?,
+                "LifecycleConfigXML" => self.lifecycle_config_xml = read_msgp_bin(rd)?,
+                "ObjectLockConfigXML" => self.object_lock_config_xml = read_msgp_bin(rd)?,
+                "VersioningConfigXML" => self.versioning_config_xml = read_msgp_bin(rd)?,
+                "EncryptionConfigXML" => self.encryption_config_xml = read_msgp_bin(rd)?,
+                "TaggingConfigXML" => self.tagging_config_xml = read_msgp_bin(rd)?,
+                "QuotaConfigJSON" => self.quota_config_json = read_msgp_bin(rd)?,
+                "ReplicationConfigXML" => self.replication_config_xml = read_msgp_bin(rd)?,
+                "BucketTargetsConfigJSON" => self.bucket_targets_config_json = read_msgp_bin(rd)?,
+                "BucketTargetsConfigMetaJSON" => self.bucket_targets_config_meta_json = read_msgp_bin(rd)?,
+                "PolicyConfigUpdatedAt" => self.policy_config_updated_at = read_msgp_time_value(rd)?,
+                "ObjectLockConfigUpdatedAt" => self.object_lock_config_updated_at = read_msgp_time_value(rd)?,
+                "EncryptionConfigUpdatedAt" => self.encryption_config_updated_at = read_msgp_time_value(rd)?,
+                "TaggingConfigUpdatedAt" => self.tagging_config_updated_at = read_msgp_time_value(rd)?,
+                "QuotaConfigUpdatedAt" => self.quota_config_updated_at = read_msgp_time_value(rd)?,
+                "ReplicationConfigUpdatedAt" => self.replication_config_updated_at = read_msgp_time_value(rd)?,
+                "VersioningConfigUpdatedAt" => self.versioning_config_updated_at = read_msgp_time_value(rd)?,
+                "LifecycleConfigUpdatedAt" => self.lifecycle_config_updated_at = read_msgp_time_value(rd)?,
+                "NotificationConfigUpdatedAt" => self.notification_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketTargetsConfigUpdatedAt" => self.bucket_targets_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketTargetsConfigMetaUpdatedAt" => self.bucket_targets_config_meta_updated_at = read_msgp_time_value(rd)?,
+                "CorsConfigXML" => self.cors_config_xml = read_msgp_bin(rd)?,
+                "PublicAccessBlockConfigXML" => self.public_access_block_config_xml = read_msgp_bin(rd)?,
+                "BucketAclConfigJSON" => self.bucket_acl_config_json = read_msgp_bin(rd)?,
+                "CorsConfigUpdatedAt" => self.cors_config_updated_at = read_msgp_time_value(rd)?,
+                "PublicAccessBlockConfigUpdatedAt" => self.public_access_block_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketAclConfigUpdatedAt" => self.bucket_acl_config_updated_at = read_msgp_time_value(rd)?,
+                other => {
+                    tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
+                    skip_msgp_value(rd)?;
+                }
+            }
+        }
+
+        Ok(())
+    }
+
+    /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
+    pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
+        // Map size: MinIO fields (25) + RustFS extensions (6)
+        let map_len: u32 = 31;
+        rmp::encode::write_map_len(wr, map_len)?;
+
+        // MinIO field order (same as Go struct)
+        rmp::encode::write_str(wr, "Name")?;
+        rmp::encode::write_str(wr, &self.name)?;
+
+        rmp::encode::write_str(wr, "Created")?;
+        write_msgp_time(wr, self.created)?;
+
+        rmp::encode::write_str(wr, "LockEnabled")?;
+        rmp::encode::write_bool(wr, self.lock_enabled)?;
+
+        write_bin_field(wr, "PolicyConfigJSON", &self.policy_config_json)?;
+        write_bin_field(wr, "NotificationConfigXML", &self.notification_config_xml)?;
+        write_bin_field(wr, "LifecycleConfigXML", &self.lifecycle_config_xml)?;
+        write_bin_field(wr, "ObjectLockConfigXML", &self.object_lock_config_xml)?;
+        write_bin_field(wr, "VersioningConfigXML", &self.versioning_config_xml)?;
+        write_bin_field(wr, "EncryptionConfigXML", &self.encryption_config_xml)?;
+        write_bin_field(wr, "TaggingConfigXML", &self.tagging_config_xml)?;
+        write_bin_field(wr, "QuotaConfigJSON", &self.quota_config_json)?;
+        write_bin_field(wr, "ReplicationConfigXML", &self.replication_config_xml)?;
+        write_bin_field(wr, "BucketTargetsConfigJSON", &self.bucket_targets_config_json)?;
+        write_bin_field(wr, "BucketTargetsConfigMetaJSON", &self.bucket_targets_config_meta_json)?;
+
+        rmp::encode::write_str(wr, "PolicyConfigUpdatedAt")?;
+        write_msgp_time(wr, self.policy_config_updated_at)?;
+        rmp::encode::write_str(wr, "ObjectLockConfigUpdatedAt")?;
+        write_msgp_time(wr, self.object_lock_config_updated_at)?;
+        rmp::encode::write_str(wr, "EncryptionConfigUpdatedAt")?;
+        write_msgp_time(wr, self.encryption_config_updated_at)?;
+        rmp::encode::write_str(wr, "TaggingConfigUpdatedAt")?;
+        write_msgp_time(wr, self.tagging_config_updated_at)?;
+        rmp::encode::write_str(wr, "QuotaConfigUpdatedAt")?;
+        write_msgp_time(wr, self.quota_config_updated_at)?;
+        rmp::encode::write_str(wr, "ReplicationConfigUpdatedAt")?;
+        write_msgp_time(wr, self.replication_config_updated_at)?;
+        rmp::encode::write_str(wr, "VersioningConfigUpdatedAt")?;
+        write_msgp_time(wr, self.versioning_config_updated_at)?;
+        rmp::encode::write_str(wr, "LifecycleConfigUpdatedAt")?;
+        write_msgp_time(wr, self.lifecycle_config_updated_at)?;
+        rmp::encode::write_str(wr, "NotificationConfigUpdatedAt")?;
+        write_msgp_time(wr, self.notification_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketTargetsConfigUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_targets_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketTargetsConfigMetaUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_targets_config_meta_updated_at)?;
+
+        // RustFS extensions
+        write_bin_field(wr, "CorsConfigXML", &self.cors_config_xml)?;
+        write_bin_field(wr, "PublicAccessBlockConfigXML", &self.public_access_block_config_xml)?;
+        write_bin_field(wr, "BucketAclConfigJSON", &self.bucket_acl_config_json)?;
+        rmp::encode::write_str(wr, "CorsConfigUpdatedAt")?;
+        write_msgp_time(wr, self.cors_config_updated_at)?;
+        rmp::encode::write_str(wr, "PublicAccessBlockConfigUpdatedAt")?;
+        write_msgp_time(wr, self.public_access_block_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketAclConfigUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_acl_config_updated_at)?;
+
+        Ok(())
+    }
+
     pub fn marshal_msg(&self) -> Result<Vec<u8>> {
         let mut buf = Vec::new();
-
-        self.serialize(&mut rmpSerializer::new(&mut buf).with_struct_map())?;
-
+        self.encode_to(&mut buf)?;
         Ok(buf)
     }
 
     pub fn unmarshal(buf: &[u8]) -> Result<Self> {
-        let t: BucketMetadata = rmp_serde::from_slice(buf)?;
-        Ok(t)
+        let mut bm = Self::default();
+        let mut cur = std::io::Cursor::new(buf);
+        bm.decode_from(&mut cur)?;
+        Ok(bm)
     }
 
     pub fn check_header(buf: &[u8]) -> Result<()> {
@@ -372,53 +505,90 @@ impl BucketMetadata {
         Ok(())
     }
 
-    fn parse_all_configs(&mut self, _api: Arc<ECStore>) -> Result<()> {
+    fn parse_policy_config(&mut self) -> Result<()> {
         if !self.policy_config_json.is_empty() {
             self.policy_config = Some(serde_json::from_slice(&self.policy_config_json)?);
-        }
-        if !self.notification_config_xml.is_empty() {
-            self.notification_config = Some(deserialize::<NotificationConfiguration>(&self.notification_config_xml)?);
-        }
-        if !self.lifecycle_config_xml.is_empty() {
-            self.lifecycle_config = Some(deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml)?);
-        }
-
-        if !self.object_lock_config_xml.is_empty() {
-            self.object_lock_config = Some(deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml)?);
-        }
-        if !self.versioning_config_xml.is_empty() {
-            self.versioning_config = Some(deserialize::<VersioningConfiguration>(&self.versioning_config_xml)?);
-        }
-        if !self.encryption_config_xml.is_empty() {
-            self.sse_config = Some(deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml)?);
-        }
-        if !self.tagging_config_xml.is_empty() {
-            self.tagging_config = Some(deserialize::<Tagging>(&self.tagging_config_xml)?);
-        }
-        if !self.quota_config_json.is_empty() {
-            self.quota_config = Some(serde_json::from_slice(&self.quota_config_json)?);
-        }
-        if !self.replication_config_xml.is_empty() {
-            self.replication_config = Some(deserialize::<ReplicationConfiguration>(&self.replication_config_xml)?);
-        }
-        //let temp = self.bucket_targets_config_json.clone();
-        if !self.bucket_targets_config_json.is_empty() {
-            let bucket_targets: BucketTargets = serde_json::from_slice(&self.bucket_targets_config_json)?;
-            self.bucket_target_config = Some(bucket_targets);
         } else {
-            self.bucket_target_config = Some(BucketTargets::default())
+            self.policy_config = None;
         }
-        if !self.cors_config_xml.is_empty() {
-            self.cors_config = Some(deserialize::<CORSConfiguration>(&self.cors_config_xml)?);
+        Ok(())
+    }
+
+    fn parse_all_configs(&mut self, _api: Arc<ECStore>) -> Result<()> {
+        if let Err(e) = self.parse_policy_config() {
+            tracing::warn!(bucket = %self.name, config = "policy", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.public_access_block_config_xml.is_empty() {
-            self.public_access_block_config =
-                Some(deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)?);
+        if !self.notification_config_xml.is_empty()
+            && let Err(e) = deserialize::<NotificationConfiguration>(&self.notification_config_xml)
+                .map(|c| self.notification_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "notification", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.bucket_acl_config_json.is_empty() {
-            let acl = String::from_utf8(self.bucket_acl_config_json.clone())
-                .map_err(|e| Error::other(format!("invalid UTF-8 in bucket ACL: {}", e)))?;
-            self.bucket_acl_config = Some(acl);
+        if !self.lifecycle_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "lifecycle", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.object_lock_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml).map(|c| self.object_lock_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "object_lock", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.versioning_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<VersioningConfiguration>(&self.versioning_config_xml).map(|c| self.versioning_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "versioning", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.encryption_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml).map(|c| self.sse_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "encryption", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.tagging_config_xml.is_empty()
+            && let Err(e) = deserialize::<Tagging>(&self.tagging_config_xml).map(|c| self.tagging_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "tagging", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.quota_config_json.is_empty()
+            && let Err(e) = serde_json::from_slice(&self.quota_config_json).map(|c| self.quota_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "quota", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.replication_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "replication", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.bucket_targets_config_json.is_empty() {
+            if let Err(e) = serde_json::from_slice::<BucketTargets>(&self.bucket_targets_config_json)
+                .map(|t| self.bucket_target_config = Some(t))
+            {
+                tracing::warn!(bucket = %self.name, config = "bucket_targets", error = %e, "parse_all_configs: failed to parse");
+                self.bucket_target_config = Some(BucketTargets::default());
+            }
+        } else {
+            self.bucket_target_config = Some(BucketTargets::default());
+        }
+        if !self.cors_config_xml.is_empty()
+            && let Err(e) = deserialize::<CORSConfiguration>(&self.cors_config_xml).map(|c| self.cors_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "cors", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.public_access_block_config_xml.is_empty()
+            && let Err(e) = deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)
+                .map(|c| self.public_access_block_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "public_access_block", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.bucket_acl_config_json.is_empty()
+            && let Err(e) = String::from_utf8(self.bucket_acl_config_json.clone()).map(|acl| self.bucket_acl_config = Some(acl))
+        {
+            tracing::warn!(bucket = %self.name, config = "bucket_acl", error = %e, "parse_all_configs: failed to parse");
         }
 
         Ok(())
@@ -471,7 +641,6 @@ async fn read_bucket_metadata(api: Arc<ECStore>, bucket: &str) -> Result<BucketM
 
     Ok(bm)
 }
-
 fn _write_time<S>(t: &OffsetDateTime, s: S) -> std::result::Result<S::Ok, S::Error>
 where
     S: Serializer,
@@ -666,4 +835,21 @@ mod test {
         println!("   - Lifecycle config size: {} bytes", deserialized_bm.lifecycle_config_xml.len());
         println!("   - Serialized buffer size: {} bytes", buf.len());
     }
+
+    /// After policy deletion (policy_config_json cleared), parse_policy_config sets policy_config to None.
+    #[test]
+    fn test_parse_policy_config_clears_cache_when_json_empty() {
+        let policy_json = r#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::b/*"}]}"#;
+        let mut bm = BucketMetadata::new("b");
+        bm.policy_config_json = policy_json.as_bytes().to_vec();
+        bm.parse_policy_config().unwrap();
+        assert!(bm.policy_config.is_some(), "policy_config should be set when JSON non-empty");
+
+        bm.policy_config_json.clear();
+        bm.parse_policy_config().unwrap();
+        assert!(
+            bm.policy_config.is_none(),
+            "policy_config should be None after JSON cleared (e.g. policy deleted)"
+        );
+    }
 }

crates/ecstore/src/bucket/metadata_sys.rs

@@ -360,12 +360,14 @@ impl BucketMetadataSys {
             };
 
             if !meta.lifecycle_config_xml.is_empty() {
-                let cfg = deserialize::<BucketLifecycleConfiguration>(&meta.lifecycle_config_xml)?;
-                // TODO: FIXME:
-                // for _v in cfg.rules.iter() {
-                //     break;
-                // }
-                if let Some(_v) = cfg.rules.first() {}
+                if let Ok(cfg) = deserialize::<BucketLifecycleConfiguration>(&meta.lifecycle_config_xml) {
+                    if let Some(_v) = cfg.rules.first() {}
+                } else {
+                    tracing::warn!(
+                        bucket = %bucket,
+                        "delete: failed to parse lifecycle config XML"
+                    );
+                }
             }
 
             // TODO: other lifecycle handle

crates/ecstore/src/bucket/metadata_test.rs

@@ -0,0 +1,261 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::metadata::BucketMetadata;
+use time::OffsetDateTime;
+
+/// Full BucketMetadata hex (all fields populated).
+const TEST_BUCKET_METADATA_HEX: &str = "de0019a44e616d65b27275737466732d636f6d7061742d74657374a743726561746564c70c050000000065920080075bcd15ab4c6f636b456e61626c6564c3b0506f6c696379436f6e6669674a534f4ec4907b2256657273696f6e223a22323031322d31302d3137222c2253746174656d656e74223a5b7b22456666656374223a22416c6c6f77222c225072696e636970616c223a222a222c22416374696f6e223a2273333a4765744f626a656374222c225265736f75726365223a2261726e3a6177733a73333a3a3a7275737466732d636f6d7061742d746573742f2a227d5d7db54e6f74696669636174696f6e436f6e666967584d4cc4963c4e6f74696669636174696f6e436f6e66696775726174696f6e3e3c436c6f75645761746368436f6e66696775726174696f6e3e3c49643e6e313c2f49643e3c4576656e743e73333a4f626a656374437265617465643a2a3c2f4576656e743e3c2f436c6f75645761746368436f6e66696775726174696f6e3e3c2f4e6f74696669636174696f6e436f6e66696775726174696f6e3eb24c6966656379636c65436f6e666967584d4cc48c3c4c6966656379636c65436f6e66696775726174696f6e3e3c52756c653e3c49443e72756c65313c2f49443e3c5374617475733e456e61626c65643c2f5374617475733e3c45787069726174696f6e3e3c446179733e33303c2f446179733e3c2f45787069726174696f6e3e3c2f52756c653e3c2f4c6966656379636c65436f6e66696775726174696f6e3eb34f626a6563744c6f636b436f6e666967584d4cc4b83c4f626a6563744c6f636b436f6e66696775726174696f6e3e3c4f626a6563744c6f636b456e61626c65643e456e61626c65643c2f4f626a6563744c6f636b456e61626c65643e3c52756c653e3c44656661756c74526574656e74696f6e3e3c4d6f64653e474f5645524e414e43453c2f4d6f64653e3c446179733e373c2f446179733e3c2f44656661756c74526574656e74696f6e3e3c2f52756c653e3c2f4f626a6563744c6f636b436f6e66696775726174696f6e3eb356657273696f6e696e67436f6e666967584d4cc44b3c56657273696f6e696e67436f6e66696775726174696f6e3e3c5374617475733e456e61626c65643c2f5374617475733e3c2f56657273696f6e696e67436f6e66696775726174696f6e3eb3456e6372797074696f6e436f6e666967584d4cc4c03c53657276657253696465456e6372797074696f6e436f6e66696775726174696f6e3e3c52756c653e3c4170706c7953657276657253696465456e6372797074696f6e427944656661756c743e3c535345416c676f726974686d3e4145533235363c2f535345416c676f726974686d3e3c2f4170706c7953657276657253696465456e6372797074696f6e427944656661756c743e3c2f52756c653e3c2f53657276657253696465456e6372797074696f6e436f6e66696775726174696f6e3eb054616767696e67436f6e666967584d4cc4503c54616767696e673e3c5461675365743e3c5461673e3c4b65793e456e763c2f4b65793e3c56616c75653e546573743c2f56616c75653e3c2f5461673e3c2f5461675365743e3c2f54616767696e673eaf51756f7461436f6e6669674a534f4ec4707b2271756f7461223a313037333734313832342c2271756f74615f74797065223a2248617264222c22637265617465645f6174223a22323032342d30312d30315430303a30303a30305a222c22757064617465645f6174223a22323032342d30312d30315430303a30303a30305a227db45265706c69636174696f6e436f6e666967584d4cc4e73c5265706c69636174696f6e436f6e66696775726174696f6e3e3c526f6c653e61726e3a6177733a69616d3a3a3132333435363738393031323a726f6c652f7265706c3c2f526f6c653e3c52756c653e3c49443e72313c2f49443e3c5374617475733e456e61626c65643c2f5374617475733e3c5072656669783e646f632f3c2f5072656669783e3c44657374696e6174696f6e3e3c4275636b65743e61726e3a6177733a73333a3a3a646573743c2f4275636b65743e3c2f44657374696e6174696f6e3e3c2f52756c653e3c2f5265706c69636174696f6e436f6e66696775726174696f6e3eb74275636b657454617267657473436f6e6669674a534f4ec4535b7b22656e64706f696e74223a22687474703a2f2f7461726765742e6578616d706c652e636f6d222c227461726765744275636b6574223a227462222c22726567696f6e223a2275732d656173742d31227d5dbb4275636b657454617267657473436f6e6669674d6574614a534f4ec42d7b227265706c69636174696f6e4964223a227265706c2d31222c2273796e634d6f6465223a226173796e63227db5506f6c696379436f6e666967557064617465644174c70c050000000065a5022000000000b94f626a6563744c6f636b436f6e666967557064617465644174c70c050000000065a5022000000000b9456e6372797074696f6e436f6e666967557064617465644174c70c050000000065a5022000000000b654616767696e67436f6e666967557064617465644174c70c050000000065a5022000000000b451756f7461436f6e666967557064617465644174c70c050000000065a5022000000000ba5265706c69636174696f6e436f6e666967557064617465644174c70c050000000065a5022000000000b956657273696f6e696e67436f6e666967557064617465644174c70c050000000065a5022000000000b84c6966656379636c65436f6e666967557064617465644174c70c050000000065a5022000000000bb4e6f74696669636174696f6e436f6e666967557064617465644174c70c050000000065a5022000000000bc4275636b657454617267657473436f6e666967557064617465644174c70c050000000065a5022000000000d9204275636b657454617267657473436f6e6669674d657461557064617465644174c70c050000000065a5022000000000";
+
+#[tokio::test]
+async fn marshal_msg() {
+    let bm = BucketMetadata::new("dada");
+
+    let buf = bm.marshal_msg().unwrap();
+
+    let new = BucketMetadata::unmarshal(&buf).unwrap();
+
+    assert_eq!(bm.name, new.name);
+}
+
+/// Verifies that serialized time uses msgp ext type 5.
+#[tokio::test]
+async fn marshal_msg_uses_time_format() {
+    let mut bm = BucketMetadata::new("test-bucket");
+    bm.created = OffsetDateTime::from_unix_timestamp(1704067200).unwrap(); // 2024-01-01 00:00:00 UTC
+
+    let buf = bm.marshal_msg().unwrap();
+
+    // msgp uses ext8 (0xc7), len 12, type 5 for time
+    assert!(
+        buf.windows(3).any(|w| w == [0xc7, 0x0c, 0x05]),
+        "serialized data should contain msgp time ext (0xc7 0x0c 0x05)"
+    );
+}
+
+#[tokio::test]
+async fn unmarshal_test_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let mut bytes = vec![0u8; TEST_BUCKET_METADATA_HEX.len() / 2];
+    hex_decode(TEST_BUCKET_METADATA_HEX.as_bytes(), &mut bytes).expect("valid hex");
+    let bm = BucketMetadata::unmarshal(&bytes).expect("RustFS must unmarshal MinIO format");
+
+    assert_eq!(bm.name, "rustfs-compat-test");
+    assert_eq!(bm.created.unix_timestamp(), 1704067200);
+    assert_eq!(bm.created.nanosecond(), 123456789);
+    assert!(bm.lock_enabled);
+
+    assert!(!bm.policy_config_json.is_empty());
+    assert!(bm.policy_config_json.starts_with(b"{\"Version\""));
+    assert!(!bm.notification_config_xml.is_empty());
+    assert!(bm.notification_config_xml.starts_with(b"<Notification"));
+    assert!(!bm.lifecycle_config_xml.is_empty());
+    assert!(bm.lifecycle_config_xml.starts_with(b"<Lifecycle"));
+    assert!(!bm.object_lock_config_xml.is_empty());
+    assert!(bm.object_lock_config_xml.starts_with(b"<ObjectLock"));
+    assert!(!bm.versioning_config_xml.is_empty());
+    assert!(bm.versioning_config_xml.starts_with(b"<Versioning"));
+    assert!(!bm.encryption_config_xml.is_empty());
+    assert!(bm.encryption_config_xml.starts_with(b"<ServerSide"));
+    assert!(!bm.tagging_config_xml.is_empty());
+    assert!(bm.tagging_config_xml.starts_with(b"<Tagging"));
+    assert!(!bm.quota_config_json.is_empty());
+    assert!(bm.quota_config_json.starts_with(b"{\"quota\""));
+    assert!(!bm.replication_config_xml.is_empty());
+    assert!(bm.replication_config_xml.starts_with(b"<Replication"));
+    assert!(!bm.bucket_targets_config_json.is_empty());
+    assert!(bm.bucket_targets_config_json.starts_with(b"[{"));
+    assert!(!bm.bucket_targets_config_meta_json.is_empty());
+    assert!(bm.bucket_targets_config_meta_json.starts_with(b"{\"replication"));
+
+    let updated_sec = 1705312800; // 2024-01-15 12:00:00 UTC
+    assert_eq!(bm.policy_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.object_lock_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.encryption_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.tagging_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.quota_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.replication_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.versioning_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.lifecycle_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.notification_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.bucket_targets_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.bucket_targets_config_meta_updated_at.unix_timestamp(), updated_sec);
+
+    assert!(bm.cors_config_xml.is_empty());
+    assert!(bm.public_access_block_config_xml.is_empty());
+    assert!(bm.bucket_acl_config_json.is_empty());
+}
+
+#[tokio::test]
+async fn marshal_msg_complete_example() {
+    // Create a complete BucketMetadata with various configurations
+    let mut bm = BucketMetadata::new("test-bucket");
+
+    // Set creation time to current time
+    bm.created = OffsetDateTime::now_utc();
+    bm.lock_enabled = true;
+
+    // Add policy configuration
+    let policy_json = r#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::test-bucket/*"}]}"#;
+    bm.policy_config_json = policy_json.as_bytes().to_vec();
+    bm.policy_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add lifecycle configuration
+    let lifecycle_xml = r#"<LifecycleConfiguration><Rule><ID>rule1</ID><Status>Enabled</Status><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>"#;
+    bm.lifecycle_config_xml = lifecycle_xml.as_bytes().to_vec();
+    bm.lifecycle_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add versioning configuration
+    let versioning_xml = r#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#;
+    bm.versioning_config_xml = versioning_xml.as_bytes().to_vec();
+    bm.versioning_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add encryption configuration
+    let encryption_xml = r#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
+    bm.encryption_config_xml = encryption_xml.as_bytes().to_vec();
+    bm.encryption_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add tagging configuration
+    let tagging_xml = r#"<Tagging><TagSet><Tag><Key>Environment</Key><Value>Test</Value></Tag><Tag><Key>Owner</Key><Value>RustFS</Value></Tag></TagSet></Tagging>"#;
+    bm.tagging_config_xml = tagging_xml.as_bytes().to_vec();
+    bm.tagging_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add quota configuration
+    let quota_json =
+        r#"{"quota":1073741824,"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#; // 1GB quota
+    bm.quota_config_json = quota_json.as_bytes().to_vec();
+    bm.quota_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add object lock configuration
+    let object_lock_xml = r#"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled><Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>7</Days></DefaultRetention></Rule></ObjectLockConfiguration>"#;
+    bm.object_lock_config_xml = object_lock_xml.as_bytes().to_vec();
+    bm.object_lock_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add notification configuration
+    let notification_xml = r#"<NotificationConfiguration><CloudWatchConfiguration><Id>notification1</Id><Event>s3:ObjectCreated:*</Event><CloudWatchConfiguration><LogGroupName>test-log-group</LogGroupName></CloudWatchConfiguration></CloudWatchConfiguration></NotificationConfiguration>"#;
+    bm.notification_config_xml = notification_xml.as_bytes().to_vec();
+    bm.notification_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add replication configuration
+    let replication_xml = r#"<ReplicationConfiguration><Role>arn:aws:iam::123456789012:role/replication-role</Role><Rule><ID>rule1</ID><Status>Enabled</Status><Prefix>documents/</Prefix><Destination><Bucket>arn:aws:s3:::destination-bucket</Bucket></Destination></Rule></ReplicationConfiguration>"#;
+    bm.replication_config_xml = replication_xml.as_bytes().to_vec();
+    bm.replication_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add bucket targets configuration
+    let bucket_targets_json = r#"[{"endpoint":"http://target1.example.com","credentials":{"accessKey":"key1","secretKey":"secret1"},"targetBucket":"target-bucket-1","region":"us-east-1"},{"endpoint":"http://target2.example.com","credentials":{"accessKey":"key2","secretKey":"secret2"},"targetBucket":"target-bucket-2","region":"us-west-2"}]"#;
+    bm.bucket_targets_config_json = bucket_targets_json.as_bytes().to_vec();
+    bm.bucket_targets_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add bucket targets meta configuration
+    let bucket_targets_meta_json = r#"{"replicationId":"repl-123","syncMode":"async","bandwidth":"100MB"}"#;
+    bm.bucket_targets_config_meta_json = bucket_targets_meta_json.as_bytes().to_vec();
+    bm.bucket_targets_config_meta_updated_at = OffsetDateTime::now_utc();
+
+    // Add public access block configuration
+    let public_access_block_xml = r#"<PublicAccessBlockConfiguration><BlockPublicAcls>true</BlockPublicAcls><IgnorePublicAcls>true</IgnorePublicAcls><BlockPublicPolicy>true</BlockPublicPolicy><RestrictPublicBuckets>false</RestrictPublicBuckets></PublicAccessBlockConfiguration>"#;
+    bm.public_access_block_config_xml = public_access_block_xml.as_bytes().to_vec();
+    bm.public_access_block_config_updated_at = OffsetDateTime::now_utc();
+
+    let bucket_acl = r#"{"owner":{"id":"rustfsadmin","display_name":"RustFS Tester"},"grants":[{"grantee":{"grantee_type":"CanonicalUser","id":"rustfsadmin","display_name":"RustFS Tester","uri":null,"email_address":null},"permission":"FULL_CONTROL"}]}"#;
+    bm.bucket_acl_config_json = bucket_acl.as_bytes().to_vec();
+    bm.bucket_acl_config_updated_at = OffsetDateTime::now_utc();
+
+    // Test serialization
+    let buf = bm.marshal_msg().unwrap();
+    assert!(!buf.is_empty(), "Serialized buffer should not be empty");
+
+    // Test deserialization
+    let deserialized_bm = BucketMetadata::unmarshal(&buf).unwrap();
+
+    // Verify all fields are correctly serialized and deserialized
+    assert_eq!(bm.name, deserialized_bm.name);
+    assert_eq!(bm.created.unix_timestamp(), deserialized_bm.created.unix_timestamp());
+    assert_eq!(bm.lock_enabled, deserialized_bm.lock_enabled);
+
+    // Verify configuration data
+    assert_eq!(bm.policy_config_json, deserialized_bm.policy_config_json);
+    assert_eq!(bm.lifecycle_config_xml, deserialized_bm.lifecycle_config_xml);
+    assert_eq!(bm.versioning_config_xml, deserialized_bm.versioning_config_xml);
+    assert_eq!(bm.encryption_config_xml, deserialized_bm.encryption_config_xml);
+    assert_eq!(bm.tagging_config_xml, deserialized_bm.tagging_config_xml);
+    assert_eq!(bm.quota_config_json, deserialized_bm.quota_config_json);
+    assert_eq!(bm.public_access_block_config_xml, deserialized_bm.public_access_block_config_xml);
+    assert_eq!(bm.bucket_acl_config_json, deserialized_bm.bucket_acl_config_json);
+    assert_eq!(bm.object_lock_config_xml, deserialized_bm.object_lock_config_xml);
+    assert_eq!(bm.notification_config_xml, deserialized_bm.notification_config_xml);
+    assert_eq!(bm.replication_config_xml, deserialized_bm.replication_config_xml);
+    assert_eq!(bm.bucket_targets_config_json, deserialized_bm.bucket_targets_config_json);
+    assert_eq!(bm.bucket_targets_config_meta_json, deserialized_bm.bucket_targets_config_meta_json);
+
+    // Verify timestamps (comparing unix timestamps to avoid precision issues)
+    assert_eq!(
+        bm.policy_config_updated_at.unix_timestamp(),
+        deserialized_bm.policy_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.lifecycle_config_updated_at.unix_timestamp(),
+        deserialized_bm.lifecycle_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.versioning_config_updated_at.unix_timestamp(),
+        deserialized_bm.versioning_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.encryption_config_updated_at.unix_timestamp(),
+        deserialized_bm.encryption_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.tagging_config_updated_at.unix_timestamp(),
+        deserialized_bm.tagging_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.quota_config_updated_at.unix_timestamp(),
+        deserialized_bm.quota_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.object_lock_config_updated_at.unix_timestamp(),
+        deserialized_bm.object_lock_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.notification_config_updated_at.unix_timestamp(),
+        deserialized_bm.notification_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.replication_config_updated_at.unix_timestamp(),
+        deserialized_bm.replication_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.bucket_targets_config_updated_at.unix_timestamp(),
+        deserialized_bm.bucket_targets_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.bucket_targets_config_meta_updated_at.unix_timestamp(),
+        deserialized_bm.bucket_targets_config_meta_updated_at.unix_timestamp()
+    );
+
+    // Test that the serialized data contains expected content
+    let buf_str = String::from_utf8_lossy(&buf);
+    assert!(buf_str.contains("test-bucket"), "Serialized data should contain bucket name");
+
+    // Verify the buffer size is reasonable (should be larger due to all the config data)
+    assert!(buf.len() > 1000, "Buffer should be substantial in size due to all configurations");
+
+    println!("✅ Complete BucketMetadata serialization test passed");
+    println!("   - Bucket name: {}", deserialized_bm.name);
+    println!("   - Lock enabled: {}", deserialized_bm.lock_enabled);
+    println!("   - Policy config size: {} bytes", deserialized_bm.policy_config_json.len());
+    println!("   - Lifecycle config size: {} bytes", deserialized_bm.lifecycle_config_xml.len());
+    println!("   - Serialized buffer size: {} bytes", buf.len());
+}

crates/ecstore/src/bucket/migration.rs

@@ -0,0 +1,420 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Migration of bucket metadata and IAM config from legacy format to RustFS format.
+
+use crate::bucket::metadata::BUCKET_METADATA_FILE;
+use crate::bucket::replication::{decode_resync_file, encode_resync_file};
+use crate::disk::{BUCKET_META_PREFIX, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
+use crate::store_api::{BucketOptions, ObjectOptions, PutObjReader, StorageAPI};
+use http::HeaderMap;
+use rustfs_policy::auth::UserIdentity;
+use rustfs_policy::policy::PolicyDoc;
+use rustfs_utils::path::SLASH_SEPARATOR;
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use time::OffsetDateTime;
+use tracing::{debug, info, warn};
+
+/// IAM config prefix under meta bucket (e.g. config/iam/).
+const IAM_CONFIG_PREFIX: &str = "config/iam";
+const IAM_FORMAT_FILE_PATH: &str = "config/iam/format.json";
+const IAM_USERS_PREFIX: &str = "config/iam/users/";
+const IAM_SERVICE_ACCOUNTS_PREFIX: &str = "config/iam/service-accounts/";
+const IAM_STS_PREFIX: &str = "config/iam/sts/";
+const IAM_GROUPS_PREFIX: &str = "config/iam/groups/";
+const IAM_POLICIES_PREFIX: &str = "config/iam/policies/";
+const IAM_POLICY_DB_PREFIX: &str = "config/iam/policydb/";
+const REPLICATION_META_DIR: &str = ".replication";
+const RESYNC_META_FILE: &str = "resync.bin";
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatIamFormat {
+    #[serde(default)]
+    version: i64,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatGroupInfo {
+    #[serde(default)]
+    version: i64,
+    #[serde(default = "default_group_status")]
+    status: String,
+    #[serde(default)]
+    members: Vec<String>,
+    #[serde(
+        rename = "updatedAt",
+        alias = "update_at",
+        default,
+        with = "rustfs_policy::serde_datetime::option"
+    )]
+    update_at: Option<OffsetDateTime>,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatMappedPolicy {
+    #[serde(default)]
+    version: i64,
+    #[serde(rename = "policy", alias = "policies", default)]
+    policy: String,
+    #[serde(
+        rename = "updatedAt",
+        alias = "update_at",
+        default,
+        with = "rustfs_policy::serde_datetime::option"
+    )]
+    update_at: Option<OffsetDateTime>,
+}
+
+fn default_group_status() -> String {
+    "enabled".to_string()
+}
+
+fn normalize_iam_config_blob(path: &str, data: &[u8]) -> std::result::Result<Option<Vec<u8>>, String> {
+    if path == IAM_FORMAT_FILE_PATH {
+        let mut format: CompatIamFormat =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM format failed: {err}"))?;
+        if format.version <= 0 {
+            format.version = 1;
+        }
+        return serde_json::to_vec(&format)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM format failed: {err}"));
+    }
+
+    if is_identity_path(path) {
+        let mut identity: UserIdentity =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM identity failed: {err}"))?;
+        if identity.update_at.is_none() {
+            identity.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&identity)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM identity failed: {err}"));
+    }
+
+    if is_group_path(path) {
+        let mut group: CompatGroupInfo = serde_json::from_slice(data).map_err(|err| format!("parse IAM group failed: {err}"))?;
+        if group.update_at.is_none() {
+            group.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&group)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM group failed: {err}"));
+    }
+
+    if is_policy_doc_path(path) {
+        let mut doc = PolicyDoc::try_from(data.to_vec()).map_err(|err| format!("parse IAM policy doc failed: {err}"))?;
+        if doc.create_date.is_none() {
+            doc.create_date = doc.update_date;
+        }
+        if doc.update_date.is_none() {
+            doc.update_date = doc.create_date;
+        }
+        return serde_json::to_vec(&doc)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM policy doc failed: {err}"));
+    }
+
+    if is_policy_mapping_path(path) {
+        let mut mapped: CompatMappedPolicy =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM policy mapping failed: {err}"))?;
+        if mapped.update_at.is_none() {
+            mapped.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&mapped)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM policy mapping failed: {err}"));
+    }
+
+    Ok(None)
+}
+
+fn is_identity_path(path: &str) -> bool {
+    (path.starts_with(IAM_USERS_PREFIX) || path.starts_with(IAM_SERVICE_ACCOUNTS_PREFIX) || path.starts_with(IAM_STS_PREFIX))
+        && path.ends_with("/identity.json")
+}
+
+fn is_group_path(path: &str) -> bool {
+    path.starts_with(IAM_GROUPS_PREFIX) && path.ends_with("/members.json")
+}
+
+fn is_policy_doc_path(path: &str) -> bool {
+    path.starts_with(IAM_POLICIES_PREFIX) && path.ends_with("/policy.json")
+}
+
+fn is_policy_mapping_path(path: &str) -> bool {
+    path.starts_with(IAM_POLICY_DB_PREFIX) && path.ends_with(".json")
+}
+
+fn is_resync_meta_path(path: &str) -> bool {
+    path.ends_with(&format!("{REPLICATION_META_DIR}/{RESYNC_META_FILE}"))
+}
+
+fn normalize_bucket_meta_blob(path: &str, data: &[u8]) -> std::result::Result<Option<Vec<u8>>, String> {
+    if !is_resync_meta_path(path) {
+        return Ok(None);
+    }
+    let status = decode_resync_file(data).map_err(|err| format!("decode resync meta failed: {err}"))?;
+    encode_resync_file(&status)
+        .map(Some)
+        .map_err(|err| format!("encode resync meta failed: {err}"))
+}
+
+/// Migrates bucket metadata from legacy format to RustFS.
+/// Uses list_bucket (from disk volumes) to get bucket names, since list_objects_v2 on the legacy
+/// meta bucket may not work (legacy format differs from object layer expectations).
+/// Skips buckets that already exist in RustFS (idempotent).
+pub async fn try_migrate_bucket_metadata<S: StorageAPI>(store: Arc<S>) {
+    let buckets_list = match store
+        .list_bucket(&BucketOptions {
+            no_metadata: true,
+            ..Default::default()
+        })
+        .await
+    {
+        Ok(b) => b,
+        Err(e) => {
+            warn!("list buckets failed (skip migration): {e}");
+            return;
+        }
+    };
+
+    let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
+
+    if buckets.is_empty() {
+        debug!("No migrating bucket metadata found");
+        return;
+    }
+
+    debug!("Found {} migrating bucket metadata, migrating...", buckets.len());
+
+    let opts = ObjectOptions {
+        max_parity: true,
+        no_lock: true,
+        ..Default::default()
+    };
+    let h = HeaderMap::new();
+
+    for bucket in buckets {
+        let meta_path = format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{bucket}{SLASH_SEPARATOR}{BUCKET_METADATA_FILE}");
+        migrate_one_if_missing(store.clone(), &opts, &h, &meta_path, &format!("bucket metadata: {bucket}")).await;
+
+        let resync_path = format!(
+            "{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{bucket}{SLASH_SEPARATOR}{REPLICATION_META_DIR}{SLASH_SEPARATOR}{RESYNC_META_FILE}"
+        );
+        migrate_one_if_missing(store.clone(), &opts, &h, &resync_path, &format!("bucket replication resync: {bucket}")).await;
+    }
+}
+
+async fn migrate_one_if_missing<S: StorageAPI>(
+    store: Arc<S>,
+    opts: &ObjectOptions,
+    headers: &HeaderMap,
+    path: &str,
+    label: &str,
+) {
+    if store
+        .get_object_info(RUSTFS_META_BUCKET, path, &ObjectOptions::default())
+        .await
+        .is_ok()
+    {
+        debug!("{label} already exists in RustFS, skip");
+        return;
+    }
+
+    let mut rd = match store
+        .get_object_reader(MIGRATING_META_BUCKET, path, None, headers.clone(), opts)
+        .await
+    {
+        Ok(r) => r,
+        Err(e) => {
+            debug!("read migrating {label}: {e}");
+            return;
+        }
+    };
+
+    let data = match rd.read_all().await {
+        Ok(d) if !d.is_empty() => d,
+        Ok(_) => return,
+        Err(e) => {
+            debug!("read migrating {label} body: {e}");
+            return;
+        }
+    };
+
+    let data = match normalize_bucket_meta_blob(path, &data) {
+        Ok(Some(normalized)) => normalized,
+        Ok(None) => data,
+        Err(e) => {
+            warn!("skip {label} migration due to incompatible format: {e}");
+            return;
+        }
+    };
+
+    if let Err(e) = store
+        .put_object(RUSTFS_META_BUCKET, path, &mut PutObjReader::from_vec(data), opts)
+        .await
+    {
+        warn!("write {label}: {e}");
+    } else {
+        info!("Migrated {label}");
+    }
+}
+
+/// Migrates IAM config from legacy meta bucket `config/iam/` to RustFS meta bucket.
+/// Lists all objects under the IAM prefix in the source, copies each to the target if not present.
+/// Skips objects that already exist in RustFS (idempotent).
+/// If list_objects_v2 on the legacy bucket fails (e.g. format differs), migration is skipped.
+pub async fn try_migrate_iam_config<S: StorageAPI>(store: Arc<S>) {
+    let opts = ObjectOptions {
+        max_parity: true,
+        no_lock: true,
+        ..Default::default()
+    };
+    let h = HeaderMap::new();
+    let prefix = format!("{IAM_CONFIG_PREFIX}/");
+    let mut continuation: Option<String> = None;
+    let mut total_migrated = 0usize;
+
+    loop {
+        let list_result = match store
+            .clone()
+            .list_objects_v2(MIGRATING_META_BUCKET, &prefix, continuation, None, 500, false, None, false)
+            .await
+        {
+            Ok(r) => r,
+            Err(e) => {
+                debug!("list IAM config from legacy bucket failed (skip migration): {e}");
+                return;
+            }
+        };
+
+        for obj in list_result.objects {
+            let path = &obj.name;
+            if path.is_empty() || path.ends_with('/') {
+                continue;
+            }
+            if store
+                .get_object_info(RUSTFS_META_BUCKET, path, &ObjectOptions::default())
+                .await
+                .is_ok()
+            {
+                debug!("IAM config already exists in RustFS, skip: {path}");
+                continue;
+            }
+            let mut rd = match store
+                .get_object_reader(MIGRATING_META_BUCKET, path, None, h.clone(), &opts)
+                .await
+            {
+                Ok(r) => r,
+                Err(e) => {
+                    debug!("read migrating IAM config {path}: {e}");
+                    continue;
+                }
+            };
+            let data = match rd.read_all().await {
+                Ok(d) if !d.is_empty() => d,
+                Ok(_) => continue,
+                Err(e) => {
+                    debug!("read migrating IAM config {path} body: {e}");
+                    continue;
+                }
+            };
+            let data = match normalize_iam_config_blob(path, &data) {
+                Ok(Some(normalized)) => normalized,
+                Ok(None) => {
+                    debug!("skip unsupported IAM config path during migration: {path}");
+                    continue;
+                }
+                Err(e) => {
+                    warn!("skip IAM config migration due to incompatible format, path: {path}, err: {e}");
+                    continue;
+                }
+            };
+            if let Err(e) = store
+                .put_object(RUSTFS_META_BUCKET, path, &mut PutObjReader::from_vec(data), &opts)
+                .await
+            {
+                warn!("write IAM config {path}: {e}");
+            } else {
+                info!("Migrated IAM config: {path}");
+                total_migrated += 1;
+            }
+        }
+
+        continuation = list_result.next_continuation_token.or(list_result.continuation_token);
+        if !list_result.is_truncated || continuation.is_none() {
+            break;
+        }
+    }
+
+    if total_migrated > 0 {
+        info!("IAM migration complete: {} object(s) migrated", total_migrated);
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{normalize_bucket_meta_blob, normalize_iam_config_blob};
+    use crate::bucket::replication::{
+        BucketReplicationResyncStatus, ResyncStatusType, TargetReplicationResyncStatus, decode_resync_file, encode_resync_file,
+    };
+    use std::collections::HashMap;
+
+    #[test]
+    fn test_normalize_policy_mapping_legacy_timestamp_and_fields() {
+        let path = "config/iam/policydb/users/alice.json";
+        let input = r#"{"version":1,"policies":"readwrite","update_at":"2026-03-09 02:22:44.998954 +00:00:00"}"#;
+
+        let output = normalize_iam_config_blob(path, input.as_bytes())
+            .expect("normalize should succeed")
+            .expect("path should be supported");
+
+        let v: serde_json::Value = serde_json::from_slice(&output).expect("output should be valid JSON");
+        assert_eq!(v.get("policy").and_then(|x| x.as_str()), Some("readwrite"));
+        assert!(v.get("policies").is_none(), "legacy field should be normalized");
+
+        let updated_at = v
+            .get("updatedAt")
+            .and_then(|x| x.as_str())
+            .expect("updatedAt should exist as string");
+        assert!(updated_at.contains('T'), "updatedAt should be RFC3339-like");
+    }
+
+    #[test]
+    fn test_normalize_bucket_meta_blob_resync_reencode() {
+        let path = ".buckets/test/.replication/resync.bin";
+        let mut status = BucketReplicationResyncStatus::new();
+        status.id = 123;
+        status.targets_map = HashMap::from([(
+            "arn:replication::1:dest".to_string(),
+            TargetReplicationResyncStatus {
+                resync_id: "reset-1".to_string(),
+                resync_status: ResyncStatusType::ResyncStarted,
+                replicated_count: 1,
+                ..Default::default()
+            },
+        )]);
+
+        let input = encode_resync_file(&status).expect("encode should succeed");
+        let output = normalize_bucket_meta_blob(path, &input)
+            .expect("normalize should succeed")
+            .expect("resync path should be normalized");
+
+        let decoded = decode_resync_file(&output).expect("decode should succeed");
+        assert_eq!(decoded.id, 123);
+        assert_eq!(decoded.targets_map["arn:replication::1:dest"].resync_id, "reset-1");
+    }
+}

crates/ecstore/src/bucket/mod.rs

@@ -18,6 +18,10 @@ pub mod error;
 pub mod lifecycle;
 pub mod metadata;
 pub mod metadata_sys;
+#[cfg(test)]
+mod metadata_test;
+pub mod migration;
+mod msgp_decode;
 pub mod object_lock;
 pub mod policy_sys;
 pub mod quota;

crates/ecstore/src/bucket/msgp_decode.rs

@@ -0,0 +1,189 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! MessagePack decode helpers for bucket metadata, aligned with msgp format.
+
+use crate::error::{Error, Result};
+use byteorder::{BigEndian, ByteOrder};
+use rmp::Marker;
+use std::io::{Read, Write};
+use time::OffsetDateTime;
+
+/// Skip a single MessagePack value. Used for unknown map keys.
+pub(crate) fn skip_msgp_value<R: Read>(rd: &mut R) -> Result<()> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let skip_len: usize = match marker {
+        Marker::Null | Marker::False | Marker::True => 0,
+        Marker::FixPos(_) | Marker::FixNeg(_) => 0,
+        Marker::U8 => 1,
+        Marker::U16 => 2,
+        Marker::U32 => 4,
+        Marker::U64 => 8,
+        Marker::I8 => 1,
+        Marker::I16 => 2,
+        Marker::I32 => 4,
+        Marker::I64 => 8,
+        Marker::F32 => 4,
+        Marker::F64 => 8,
+        Marker::FixStr(n) => n as usize,
+        Marker::Str8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            b[0] as usize
+        }
+        Marker::Str16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u16::from_be_bytes(b) as usize
+        }
+        Marker::Str32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u32::from_be_bytes(b) as usize
+        }
+        Marker::Bin8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            b[0] as usize
+        }
+        Marker::Bin16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u16::from_be_bytes(b) as usize
+        }
+        Marker::Bin32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u32::from_be_bytes(b) as usize
+        }
+        Marker::FixArray(n) => {
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Array16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u16::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Array32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u32::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::FixMap(n) => {
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Map16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u16::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Map32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u32::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::FixExt1 => 1,
+        Marker::FixExt2 => 2,
+        Marker::FixExt4 => 4,
+        Marker::FixExt8 => 8,
+        Marker::FixExt16 => 16,
+        Marker::Ext8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = b[0] as usize;
+            1 + len // type byte + data
+        }
+        Marker::Ext16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = u16::from_be_bytes(b) as usize;
+            2 + len
+        }
+        Marker::Ext32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = u32::from_be_bytes(b) as usize;
+            4 + len
+        }
+        Marker::Reserved => 0,
+    };
+    if skip_len > 0 {
+        let mut buf = vec![0u8; skip_len];
+        rd.read_exact(&mut buf).map_err(Error::other)?;
+    }
+    Ok(())
+}
+
+/// msgp time format: ext8 (0xc7), len 12, type 5, 8 bytes sec (BE) + 4 bytes nsec (BE).
+pub(crate) const MSGP_TIME_EXT_TYPE: i8 = 5;
+pub(crate) const MSGP_TIME_LEN: u8 = 12;
+
+/// Read msgp ext8 time - caller must have already read the marker and verified it's ext8.
+/// Ext8 format: 1 byte len, 1 byte type, then data bytes.
+pub(crate) fn read_msgp_ext8_time<R: Read>(rd: &mut R) -> Result<OffsetDateTime> {
+    let mut len_buf = [0u8; 1];
+    rd.read_exact(&mut len_buf).map_err(Error::other)?;
+    let len = len_buf[0] as usize;
+    if len != MSGP_TIME_LEN as usize {
+        return Err(Error::other(format!("invalid msgp time len: {len}")));
+    }
+    let mut type_buf = [0u8; 1];
+    rd.read_exact(&mut type_buf).map_err(Error::other)?;
+    if type_buf[0] != MSGP_TIME_EXT_TYPE as u8 {
+        return Err(Error::other(format!("invalid msgp time type: {}", type_buf[0])));
+    }
+    let mut buf = [0u8; 12];
+    rd.read_exact(&mut buf).map_err(Error::other)?;
+    let sec = BigEndian::read_i64(&buf[0..8]);
+    let nsec = BigEndian::read_u32(&buf[8..12]);
+    OffsetDateTime::from_unix_timestamp(sec)
+        .map_err(|_| Error::other("invalid timestamp"))?
+        .replace_nanosecond(nsec)
+        .map_err(|_| Error::other("invalid nanosecond"))
+}
+
+/// Write msgp time as ext8 (0xc7), len 12, type 5. Always uses ext format (never nil).
+pub(crate) fn write_msgp_time<W: Write>(wr: &mut W, t: OffsetDateTime) -> Result<()> {
+    wr.write_all(&[0xc7, MSGP_TIME_LEN, MSGP_TIME_EXT_TYPE as u8])
+        .map_err(Error::other)?;
+    let mut buf = [0u8; 12];
+    BigEndian::write_i64(&mut buf[0..8], t.unix_timestamp());
+    BigEndian::write_u32(&mut buf[8..12], t.nanosecond());
+    wr.write_all(&buf).map_err(Error::other)
+}

crates/ecstore/src/bucket/quota/mod.rs

@@ -15,7 +15,6 @@
 pub mod checker;
 
 use crate::error::Result;
-use rmp_serde::Serializer as rmpSerializer;
 use rustfs_config::{
     QUOTA_API_PATH, QUOTA_EXCEEDED_ERROR_CODE, QUOTA_INTERNAL_ERROR_CODE, QUOTA_INVALID_CONFIG_ERROR_CODE,
     QUOTA_NOT_FOUND_ERROR_CODE,
@@ -28,27 +27,35 @@ use time::OffsetDateTime;
 pub enum QuotaType {
     /// Hard quota: reject immediately when exceeded
     #[default]
+    #[serde(alias = "HARD", alias = "hard")]
     Hard,
 }
 
+/// Bucket quota configuration. quota_type defaults to Hard when omitted.
 #[derive(Debug, Deserialize, Serialize, Default, Clone, PartialEq)]
 pub struct BucketQuota {
+    #[serde(default)]
     pub quota: Option<u64>,
+    /// Defaults to Hard when missing.
+    #[serde(default)]
     pub quota_type: QuotaType,
     /// Timestamp when this quota configuration was set (for audit purposes)
+    #[serde(default, with = "time::serde::rfc3339::option")]
     pub created_at: Option<OffsetDateTime>,
+    /// Accept updated_at for compatibility; not used.
+    #[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
+    pub updated_at: Option<OffsetDateTime>,
 }
 
 impl BucketQuota {
+    /// Serialize to JSON bytes. Same format as parse_all_configs.
     pub fn marshal_msg(&self) -> Result<Vec<u8>> {
-        let mut buf = Vec::new();
-        self.serialize(&mut rmpSerializer::new(&mut buf).with_struct_map())?;
-        Ok(buf)
+        serde_json::to_vec(self).map_err(Into::into)
     }
 
+    /// Deserialize from JSON bytes. Same format as parse_all_configs.
     pub fn unmarshal(buf: &[u8]) -> Result<Self> {
-        let t: BucketQuota = rmp_serde::from_slice(buf)?;
-        Ok(t)
+        serde_json::from_slice(buf).map_err(Into::into)
     }
 
     pub fn new(quota: Option<u64>) -> Self {
@@ -57,6 +64,7 @@ impl BucketQuota {
             quota,
             quota_type: QuotaType::Hard,
             created_at: Some(now),
+            updated_at: None,
         }
     }
 
@@ -156,3 +164,57 @@ impl QuotaErrorResponse {
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// Legacy format: quota, created_at, updated_at (no quota_type)
+    #[test]
+    fn deserialize_format_without_quota_type() {
+        let json = r#"{"quota":1073741824,"created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+        assert!(q.created_at.is_some());
+        assert!(q.updated_at.is_some());
+    }
+
+    /// RustFS format: quota, quota_type, created_at
+    #[test]
+    fn deserialize_rustfs_format() {
+        let json = r#"{"quota":1073741824,"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+        assert!(q.created_at.is_some());
+        assert!(q.created_at.is_some_and(|t| t.unix_timestamp() == 1704067200));
+    }
+
+    /// E2E format uses "HARD" (uppercase)
+    #[test]
+    fn deserialize_quota_type_hard_uppercase() {
+        let json = r#"{"quota":2048,"quota_type":"HARD"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota_type, QuotaType::Hard);
+    }
+
+    /// marshal_msg/unmarshal use JSON, same as parse_all_configs
+    #[test]
+    fn marshal_unmarshal_roundtrip() {
+        let q = BucketQuota::new(Some(1073741824));
+        let buf = q.marshal_msg().expect("marshal");
+        let restored = BucketQuota::unmarshal(&buf).expect("unmarshal");
+        assert_eq!(q.quota, restored.quota);
+        assert_eq!(q.quota_type, restored.quota_type);
+    }
+
+    /// unmarshal accepts format without quota_type
+    #[test]
+    fn unmarshal_format_without_quota_type() {
+        let json = r#"{"quota":1073741824,"created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#;
+        let q = BucketQuota::unmarshal(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+    }
+}

crates/ecstore/src/bucket/replication/mod.rs

@@ -23,4 +23,5 @@ pub use config::*;
 pub use datatypes::*;
 pub use replication_pool::*;
 pub use replication_resyncer::*;
+pub use replication_state::BucketStats;
 pub use rule::*;

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -20,8 +20,8 @@ use crate::bucket::replication::ResyncStatusType;
 use crate::bucket::replication::replicate_delete;
 use crate::bucket::replication::replicate_object;
 use crate::bucket::replication::replication_resyncer::{
-    BucketReplicationResyncStatus, DeletedObjectReplicationInfo, ReplicationConfig, ReplicationResyncer,
-    get_heal_replicate_object_info,
+    BucketReplicationResyncStatus, DeletedObjectReplicationInfo, REPLICATION_DIR, RESYNC_FILE_NAME, ReplicationConfig,
+    ReplicationResyncer, decode_resync_file, get_heal_replicate_object_info,
 };
 use crate::bucket::replication::replication_state::ReplicationStats;
 use crate::config::com::read_config;
@@ -41,7 +41,7 @@ use rustfs_filemeta::VersionPurgeStatusType;
 use rustfs_filemeta::replication_statuses_map;
 use rustfs_filemeta::version_purge_statuses_map;
 use rustfs_filemeta::{REPLICATE_EXISTING, REPLICATE_HEAL, REPLICATE_HEAL_DELETE};
-use rustfs_utils::http::RESERVED_METADATA_PREFIX_LOWER;
+use rustfs_utils::http::{SUFFIX_REPLICATION_TIMESTAMP, get_str};
 use std::any::Any;
 use std::sync::Arc;
 use std::sync::atomic::AtomicI32;
@@ -56,8 +56,7 @@ use tokio::sync::mpsc::Sender;
 use tokio::task::JoinHandle;
 use tokio::time::Duration;
 use tokio_util::sync::CancellationToken;
-use tracing::info;
-use tracing::warn;
+use tracing::{info, instrument, warn};
 
 // Worker limits
 pub const WORKER_MAX_LIMIT: usize = 500;
@@ -796,6 +795,7 @@ impl<S: StorageAPI> ReplicationPool<S> {
     }
 
     /// Load bucket replication resync statuses into memory
+    #[instrument(skip(cancellation_token))]
     async fn load_resync(self: Arc<Self>, buckets: &[String], cancellation_token: CancellationToken) -> Result<(), EcstoreError> {
         // TODO: add leader_lock
         // Make sure only one node running resync on the cluster
@@ -861,17 +861,8 @@ async fn load_bucket_resync_metadata<S: StorageAPI>(
     bucket: &str,
     obj_api: Arc<S>,
 ) -> Result<BucketReplicationResyncStatus, EcstoreError> {
-    use std::convert::TryInto;
-
     let mut brs = BucketReplicationResyncStatus::new();
 
-    // Constants that would be defined elsewhere
-    const REPLICATION_DIR: &str = "replication";
-    const RESYNC_FILE_NAME: &str = "resync.bin";
-    const RESYNC_META_FORMAT: u16 = 1;
-    const RESYNC_META_VERSION: u16 = 1;
-    const RESYNC_META_VERSION_V1: u16 = 1;
-
     let resync_dir_path = format!("{BUCKET_META_PREFIX}/{bucket}/{REPLICATION_DIR}");
     let resync_file_path = format!("{resync_dir_path}/{RESYNC_FILE_NAME}");
 
@@ -886,27 +877,7 @@ async fn load_bucket_resync_metadata<S: StorageAPI>(
         return Ok(brs);
     }
 
-    if data.len() <= 4 {
-        return Err(EcstoreError::CorruptedFormat);
-    }
-
-    // Read resync meta header
-    let format = u16::from_le_bytes(data[0..2].try_into().unwrap());
-    if format != RESYNC_META_FORMAT {
-        return Err(EcstoreError::CorruptedFormat);
-    }
-
-    let version = u16::from_le_bytes(data[2..4].try_into().unwrap());
-    if version != RESYNC_META_VERSION {
-        return Err(EcstoreError::CorruptedFormat);
-    }
-
-    // Parse data
-    brs = BucketReplicationResyncStatus::unmarshal_msg(&data[4..])?;
-
-    if brs.version != RESYNC_META_VERSION_V1 {
-        return Err(EcstoreError::CorruptedFormat);
-    }
+    brs = decode_resync_file(&data)?;
 
     Ok(brs)
 }
@@ -984,10 +955,8 @@ pub fn get_global_replication_pool() -> Option<Arc<DynReplicationPool>> {
 pub async fn schedule_replication<S: StorageAPI>(oi: ObjectInfo, o: Arc<S>, dsc: ReplicateDecision, op_type: ReplicationType) {
     let tgt_statuses = replication_statuses_map(&oi.replication_status_internal.clone().unwrap_or_default());
     let purge_statuses = version_purge_statuses_map(&oi.version_purge_status_internal.clone().unwrap_or_default());
-    let tm = oi
-        .user_defined
-        .get(&format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replication-timestamp"))
-        .map(|v| OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH));
+    let tm = get_str(&oi.user_defined, SUFFIX_REPLICATION_TIMESTAMP)
+        .map(|v| OffsetDateTime::parse(&v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH));
     let mut rstate = oi.replication_state();
     rstate.replicate_decision_str = dsc.to_string();
     let asz = oi.get_actual_size().unwrap_or_default();

crates/ecstore/src/bucket/replication/replication_state.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use crate::error::Error;
+use crate::global::get_global_bucket_monitor;
 use rustfs_filemeta::{ReplicatedTargetInfo, ReplicationStatusType, ReplicationType};
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
@@ -585,6 +586,8 @@ pub struct BucketReplicationStat {
     pub latency: LatencyStats,
     pub xfer_rate_lrg: XferStats,
     pub xfer_rate_sml: XferStats,
+    pub bandwidth_limit_bytes_per_sec: i64,
+    pub current_bandwidth_bytes_per_sec: f64,
 }
 
 impl BucketReplicationStat {
@@ -1019,6 +1022,9 @@ impl ReplicationStats {
                     latency: stat.latency.merge(&old_stat.latency),
                     xfer_rate_lrg: lrg,
                     xfer_rate_sml: sml,
+                    bandwidth_limit_bytes_per_sec: stat.bandwidth_limit_bytes_per_sec,
+                    current_bandwidth_bytes_per_sec: stat.current_bandwidth_bytes_per_sec
+                        + old_stat.current_bandwidth_bytes_per_sec,
                 };
 
                 tot_replicated_size += stat.replicated_size;
@@ -1069,24 +1075,43 @@ impl ReplicationStats {
         // In actual implementation, statistics would be obtained from cluster
         // This is simplified to get from local cache
         let cache = self.cache.read().await;
-        if let Some(stats) = cache.get(bucket) {
-            BucketStats {
-                uptime: SystemTime::now()
-                    .duration_since(SystemTime::UNIX_EPOCH)
-                    .unwrap_or_default()
-                    .as_secs() as i64,
-                replication_stats: stats.clone_stats(),
-                queue_stats: Default::default(),
-                proxy_stats: ProxyMetric::default(),
-            }
+        let mut replication_stats = if let Some(stats) = cache.get(bucket) {
+            stats.clone_stats()
         } else {
-            BucketStats {
-                uptime: 0,
-                replication_stats: BucketReplicationStats::new(),
-                queue_stats: Default::default(),
-                proxy_stats: ProxyMetric::default(),
+            BucketReplicationStats::new()
+        };
+        let uptime = if cache.contains_key(bucket) {
+            SystemTime::now()
+                .duration_since(SystemTime::UNIX_EPOCH)
+                .unwrap_or_default()
+                .as_secs() as i64
+        } else {
+            0
+        };
+        drop(cache);
+
+        if let Some(monitor) = get_global_bucket_monitor() {
+            let bw_report = monitor.get_report(|name| name == bucket);
+            for (opts, bw) in bw_report.bucket_stats {
+                let stat = replication_stats
+                    .stats
+                    .entry(opts.replication_arn)
+                    .or_insert_with(|| BucketReplicationStat {
+                        xfer_rate_lrg: XferStats::new(),
+                        xfer_rate_sml: XferStats::new(),
+                        ..Default::default()
+                    });
+                stat.bandwidth_limit_bytes_per_sec = bw.limit_bytes_per_sec;
+                stat.current_bandwidth_bytes_per_sec = bw.current_bandwidth_bytes_per_sec;
             }
         }
+
+        BucketStats {
+            uptime,
+            replication_stats,
+            queue_stats: Default::default(),
+            proxy_stats: ProxyMetric::default(),
+        }
     }
 
     /// Increase queue statistics

crates/ecstore/src/bucket/utils.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::disk::RUSTFS_META_BUCKET;
+use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result, StorageError};
 use regex::Regex;
 use rustfs_utils::path::SLASH_SEPARATOR;
@@ -20,7 +20,7 @@ use s3s::xml;
 use tracing::instrument;
 
 pub fn is_meta_bucketname(name: &str) -> bool {
-    name.starts_with(RUSTFS_META_BUCKET)
+    name.starts_with(RUSTFS_META_BUCKET) || name.starts_with(MIGRATING_META_BUCKET)
 }
 
 lazy_static::lazy_static! {

crates/ecstore/src/client/transition_api.rs

@@ -139,7 +139,11 @@ pub enum BucketLookupType {
 fn load_root_store_from_tls_path() -> Option<rustls::RootCertStore> {
     // Load the root certificate bundle from the path specified by the
     // RUSTFS_TLS_PATH environment variable.
-    let tp = std::env::var("RUSTFS_TLS_PATH").ok()?;
+    let tp = rustfs_utils::get_env_str(rustfs_config::ENV_RUSTFS_TLS_PATH, rustfs_config::DEFAULT_RUSTFS_TLS_PATH);
+    // If no TLS path is configured, do not fall back to a CA bundle in the current directory.
+    if tp.is_empty() {
+        return None;
+    }
     let ca = std::path::Path::new(&tp).join(rustfs_config::RUSTFS_CA_CERT);
     if !ca.exists() {
         return None;
@@ -155,19 +159,33 @@ fn load_root_store_from_tls_path() -> Option<rustls::RootCertStore> {
     Some(store)
 }
 
-impl TransitionClient {
-    pub async fn new(endpoint: &str, opts: Options, tier_type: &str) -> Result<TransitionClient, std::io::Error> {
-        let clnt = Self::private_new(endpoint, opts, tier_type).await?;
-
-        Ok(clnt)
+fn panic_payload_to_message(payload: Box<dyn std::any::Any + Send>) -> String {
+    if let Some(message) = payload.downcast_ref::<String>() {
+        return message.clone();
     }
 
-    async fn private_new(endpoint: &str, opts: Options, tier_type: &str) -> Result<TransitionClient, std::io::Error> {
-        let endpoint_url = get_endpoint_url(endpoint, opts.secure)?;
+    if let Some(message) = payload.downcast_ref::<&'static str>() {
+        return (*message).to_string();
+    }
 
-        let scheme = endpoint_url.scheme();
-        let client;
-        let tls = if let Some(store) = load_root_store_from_tls_path() {
+    "unknown panic payload".to_string()
+}
+
+fn with_rustls_init_guard<T, F>(build: F) -> Result<T, std::io::Error>
+where
+    F: FnOnce() -> Result<T, std::io::Error>,
+{
+    std::panic::catch_unwind(std::panic::AssertUnwindSafe(build)).unwrap_or_else(|payload| {
+        let panic_message = panic_payload_to_message(payload);
+        Err(std::io::Error::other(format!(
+            "failed to initialize rustls crypto provider: {panic_message}. Ensure exactly one rustls crypto provider feature is enabled (aws-lc-rs or ring), or install one with CryptoProvider::install_default()"
+        )))
+    })
+}
+
+fn build_tls_config() -> Result<rustls::ClientConfig, std::io::Error> {
+    with_rustls_init_guard(|| {
+        let config = if let Some(store) = load_root_store_from_tls_path() {
             rustls::ClientConfig::builder()
                 .with_root_certificates(store)
                 .with_no_client_auth()
@@ -175,20 +193,47 @@ impl TransitionClient {
             rustls::ClientConfig::builder().with_native_roots()?.with_no_client_auth()
         };
 
+        Ok(config)
+    })
+}
+
+impl TransitionClient {
+    pub async fn new(endpoint: &str, opts: Options, tier_type: &str) -> Result<TransitionClient, std::io::Error> {
+        let client = Self::private_new(endpoint, opts, tier_type).await?;
+
+        Ok(client)
+    }
+
+    async fn private_new(endpoint: &str, opts: Options, tier_type: &str) -> Result<TransitionClient, std::io::Error> {
+        if rustls::crypto::CryptoProvider::get_default().is_none() {
+            // No default provider is set yet; try to install aws-lc-rs.
+            // `install_default` can only fail if another thread races us and installs a provider
+            // between our check and this call, which is still safe to ignore.
+            if rustls::crypto::aws_lc_rs::default_provider().install_default().is_err() {
+                debug!("rustls crypto provider was installed concurrently, skipping aws-lc-rs install");
+            }
+        } else {
+            debug!("rustls crypto provider already installed, skipping aws-lc-rs install");
+        }
+
+        let endpoint_url = get_endpoint_url(endpoint, opts.secure)?;
+
+        let tls = build_tls_config()?;
+
         let https = hyper_rustls::HttpsConnectorBuilder::new()
             .with_tls_config(tls)
             .https_or_http()
             .enable_http1()
             .enable_http2()
             .build();
-        client = Client::builder(TokioExecutor::new()).build(https);
+        let http_client = Client::builder(TokioExecutor::new()).build(https);
 
-        let mut clnt = TransitionClient {
+        let mut client = TransitionClient {
             endpoint_url,
             creds_provider: Arc::new(Mutex::new(opts.creds)),
             override_signer_type: SignatureType::SignatureDefault,
             secure: opts.secure,
-            http_client: client,
+            http_client,
             bucket_loc_cache: Arc::new(Mutex::new(BucketLocationCache::new())),
             is_trace_enabled: Arc::new(Mutex::new(false)),
             trace_errors_only: Arc::new(Mutex::new(false)),
@@ -206,23 +251,23 @@ impl TransitionClient {
         };
 
         {
-            let mut md5_hasher = clnt.md5_hasher.lock().unwrap();
+            let mut md5_hasher = client.md5_hasher.lock().unwrap();
             if md5_hasher.is_none() {
                 *md5_hasher = Some(HashAlgorithm::Md5);
             }
         }
-        if clnt.sha256_hasher.is_none() {
-            clnt.sha256_hasher = Some(HashAlgorithm::SHA256);
+        if client.sha256_hasher.is_none() {
+            client.sha256_hasher = Some(HashAlgorithm::SHA256);
         }
 
-        clnt.trailing_header_support = opts.trailing_headers && clnt.override_signer_type == SignatureType::SignatureV4;
+        client.trailing_header_support = opts.trailing_headers && client.override_signer_type == SignatureType::SignatureV4;
 
-        clnt.max_retries = MAX_RETRY;
+        client.max_retries = MAX_RETRY;
         if opts.max_retries > 0 {
-            clnt.max_retries = opts.max_retries;
+            client.max_retries = opts.max_retries;
         }
 
-        Ok(clnt)
+        Ok(client)
     }
 
     fn endpoint_url(&self) -> Url {
@@ -1278,3 +1323,57 @@ pub struct CreateBucketConfiguration {
     #[serde(rename = "LocationConstraint")]
     pub location_constraint: String,
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{build_tls_config, load_root_store_from_tls_path, with_rustls_init_guard};
+
+    #[test]
+    fn rustls_guard_converts_panics_to_io_errors() {
+        let err = with_rustls_init_guard(|| -> Result<(), std::io::Error> { panic!("missing provider") })
+            .expect_err("panic should be converted into an io::Error");
+        assert!(
+            err.to_string().contains("missing provider"),
+            "expected panic message to be preserved, got: {err}"
+        );
+    }
+
+    #[test]
+    fn build_tls_config_returns_result_without_panicking() {
+        let outcome = std::panic::catch_unwind(build_tls_config);
+        assert!(outcome.is_ok(), "TLS config creation should not panic");
+    }
+
+    /// When RUSTFS_TLS_PATH is not set, `load_root_store_from_tls_path` must return `None`
+    /// (i.e. it must not silently look for a CA bundle in the current working directory).
+    #[test]
+    fn tls_path_unset_returns_none() {
+        let result = temp_env::with_var_unset(rustfs_config::ENV_RUSTFS_TLS_PATH, || load_root_store_from_tls_path());
+        assert!(result.is_none(), "expected None when RUSTFS_TLS_PATH is unset, but got a root store");
+    }
+
+    /// When RUSTFS_TLS_PATH is set to an empty string, `load_root_store_from_tls_path` must
+    /// return `None` to avoid accidentally trusting a CA bundle in the current directory.
+    #[test]
+    fn tls_path_empty_returns_none() {
+        let result = temp_env::with_var(rustfs_config::ENV_RUSTFS_TLS_PATH, Some(""), || load_root_store_from_tls_path());
+        assert!(result.is_none(), "expected None when RUSTFS_TLS_PATH is empty, but got a root store");
+    }
+
+    /// Installing the rustls crypto provider when one is already set must not panic or return
+    /// an error that surfaces to callers (the race-safe `get_default` check guards the install).
+    #[test]
+    fn provider_install_is_idempotent() {
+        // Install once (may already be set by another test in this binary — that's fine).
+        let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
+
+        // A second install attempt on an already-set provider must not panic.
+        let outcome = std::panic::catch_unwind(|| {
+            if rustls::crypto::CryptoProvider::get_default().is_none() {
+                let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
+            }
+            // If a default is already present, the branch above is simply skipped.
+        });
+        assert!(outcome.is_ok(), "provider install guard must not panic when a provider is already set");
+    }
+}

crates/ecstore/src/config/com.rs

@@ -13,16 +13,17 @@
 // limitations under the License.
 
 use crate::config::{Config, GLOBAL_STORAGE_CLASS, storageclass};
-use crate::disk::RUSTFS_META_BUCKET;
+use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result};
 use crate::store_api::{ObjectInfo, ObjectOptions, PutObjReader, StorageAPI};
 use http::HeaderMap;
-use rustfs_config::DEFAULT_DELIMITER;
+use rustfs_config::{DEFAULT_DELIMITER, RUSTFS_REGION};
 use rustfs_utils::path::SLASH_SEPARATOR;
-use std::collections::HashSet;
+use serde_json::{Map, Value};
+use std::collections::{HashMap, HashSet};
 use std::sync::Arc;
 use std::sync::LazyLock;
-use tracing::{error, warn};
+use tracing::{debug, error, info, instrument, warn};
 
 pub const CONFIG_PREFIX: &str = "config";
 const CONFIG_FILE: &str = "config.json";
@@ -36,6 +37,8 @@ static SUB_SYSTEMS_DYNAMIC: LazyLock<HashSet<String>> = LazyLock::new(|| {
     h.insert(STORAGE_CLASS_SUB_SYS.to_owned());
     h
 });
+
+#[instrument(skip(api))]
 pub async fn read_config<S: StorageAPI>(api: Arc<S>, file: &str) -> Result<Vec<u8>> {
     let (data, _obj) = read_config_with_metadata(api, file, &ObjectOptions::default()).await?;
     Ok(data)
@@ -68,6 +71,7 @@ pub async fn read_config_with_metadata<S: StorageAPI>(
     Ok((data, rd.object_info))
 }
 
+#[instrument(skip(api, data))]
 pub async fn save_config<S: StorageAPI>(api: Arc<S>, file: &str, data: Vec<u8>) -> Result<()> {
     save_config_with_opts(
         api,
@@ -81,6 +85,7 @@ pub async fn save_config<S: StorageAPI>(api: Arc<S>, file: &str, data: Vec<u8>)
     .await
 }
 
+#[instrument(skip(api))]
 pub async fn delete_config<S: StorageAPI>(api: Arc<S>, file: &str) -> Result<()> {
     match api
         .delete_object(
@@ -132,6 +137,226 @@ fn get_config_file() -> String {
     format!("{CONFIG_PREFIX}{SLASH_SEPARATOR}{CONFIG_FILE}")
 }
 
+fn storage_class_kvs_mut(cfg: &mut Config) -> &mut crate::config::KVS {
+    let sub_cfg = cfg.0.entry(STORAGE_CLASS_SUB_SYS.to_string()).or_insert_with(|| {
+        let mut section = HashMap::new();
+        section.insert(DEFAULT_DELIMITER.to_string(), storageclass::DEFAULT_KVS.clone());
+        section
+    });
+    sub_cfg
+        .entry(DEFAULT_DELIMITER.to_string())
+        .or_insert_with(|| storageclass::DEFAULT_KVS.clone())
+}
+
+fn parse_storage_class_value(value: &Value) -> Option<String> {
+    match value {
+        Value::String(v) => Some(v.trim().to_string()),
+        Value::Object(m) => m
+            .get("parity")
+            .and_then(Value::as_u64)
+            .map(|parity| if parity == 0 { String::new() } else { format!("EC:{parity}") }),
+        _ => None,
+    }
+}
+
+fn parse_inline_block_value(value: &Value) -> Option<String> {
+    match value {
+        Value::String(v) if !v.trim().is_empty() => Some(v.trim().to_string()),
+        Value::Number(v) => Some(v.to_string()),
+        _ => None,
+    }
+}
+
+fn apply_external_storage_class_map(cfg: &mut Config, root: &Map<String, Value>) -> bool {
+    let sc = root.get("storageclass").or_else(|| root.get("storage_class"));
+    let Some(Value::Object(sc_obj)) = sc else {
+        return false;
+    };
+
+    let mut applied = false;
+    let kvs = storage_class_kvs_mut(cfg);
+
+    if let Some(v) = sc_obj.get("standard").and_then(parse_storage_class_value) {
+        kvs.insert(storageclass::CLASS_STANDARD.to_string(), v);
+        applied = true;
+    }
+    if let Some(v) = sc_obj.get("rrs").and_then(parse_storage_class_value) {
+        kvs.insert(storageclass::CLASS_RRS.to_string(), v);
+        applied = true;
+    }
+    if let Some(Value::String(v)) = sc_obj.get("optimize")
+        && !v.trim().is_empty()
+    {
+        kvs.insert(storageclass::OPTIMIZE.to_string(), v.clone());
+        applied = true;
+    }
+    if let Some(v) = sc_obj.get("inline_block").and_then(parse_inline_block_value) {
+        kvs.insert(storageclass::INLINE_BLOCK.to_string(), v);
+        applied = true;
+    }
+
+    applied
+}
+
+fn decode_server_config_blob(data: &[u8]) -> Result<Config> {
+    if let Ok(cfg) = Config::unmarshal(data) {
+        return Ok(cfg);
+    }
+
+    let value: Value = serde_json::from_slice(data)?;
+    let Value::Object(root) = value else {
+        return Err(Error::other("unrecognized external server config shape"));
+    };
+
+    let mut cfg = Config::new();
+    let has_storage = apply_external_storage_class_map(&mut cfg, &root);
+    let has_header = root.contains_key("version") || root.contains_key("region") || root.contains_key("credential");
+    if !has_storage && !has_header {
+        return Err(Error::other("unrecognized external server config shape"));
+    }
+    Ok(cfg)
+}
+
+fn parse_object_seed(data: &[u8]) -> Option<Map<String, Value>> {
+    let value: Value = serde_json::from_slice(data).ok()?;
+    value.as_object().cloned()
+}
+
+fn build_storageclass_object(cfg: &Config) -> Map<String, Value> {
+    let kvs = cfg.get_value(STORAGE_CLASS_SUB_SYS, DEFAULT_DELIMITER).unwrap_or_default();
+    let mut sc_obj = Map::new();
+    sc_obj.insert(
+        "standard".to_string(),
+        Value::String(kvs.lookup(storageclass::CLASS_STANDARD).unwrap_or_default()),
+    );
+    sc_obj.insert("rrs".to_string(), Value::String(kvs.lookup(storageclass::CLASS_RRS).unwrap_or_default()));
+    let optimize = kvs
+        .lookup(storageclass::OPTIMIZE)
+        .filter(|v| !v.trim().is_empty())
+        .unwrap_or_else(|| "availability".to_string());
+    sc_obj.insert("optimize".to_string(), Value::String(optimize));
+    if let Some(v) = kvs.lookup(storageclass::INLINE_BLOCK).filter(|v| !v.trim().is_empty()) {
+        sc_obj.insert("inline_block".to_string(), Value::String(v));
+    }
+    sc_obj
+}
+
+fn encode_server_config_blob(cfg: &Config, seed: Option<&[u8]>) -> Result<Vec<u8>> {
+    let mut root = seed.and_then(parse_object_seed).unwrap_or_default();
+
+    if !matches!(root.get("version"), Some(Value::String(v)) if !v.trim().is_empty()) {
+        root.insert("version".to_string(), Value::String("33".to_string()));
+    }
+    if !matches!(root.get("region"), Some(Value::String(v)) if !v.trim().is_empty()) {
+        root.insert("region".to_string(), Value::String(RUSTFS_REGION.to_string()));
+    }
+
+    let mut sc_obj = match root.remove("storageclass") {
+        Some(Value::Object(v)) => v,
+        _ => Map::new(),
+    };
+    for (k, v) in build_storageclass_object(cfg) {
+        sc_obj.insert(k, v);
+    }
+    root.insert("storageclass".to_string(), Value::Object(sc_obj));
+    root.remove("storage_class");
+
+    Ok(serde_json::to_vec(&Value::Object(root))?)
+}
+
+fn is_standard_object_server_config(data: &[u8]) -> bool {
+    let Ok(value) = serde_json::from_slice::<Value>(data) else {
+        return false;
+    };
+    let Value::Object(root) = value else {
+        return false;
+    };
+    matches!(root.get("version"), Some(Value::String(v)) if !v.trim().is_empty())
+        && matches!(root.get("storageclass"), Some(Value::Object(_)))
+        && !root.contains_key("storage_class")
+}
+
+fn configs_semantically_equal(lhs: &Config, rhs: &Config) -> bool {
+    build_storageclass_object(lhs) == build_storageclass_object(rhs)
+}
+
+fn is_object_not_found(err: &Error) -> bool {
+    *err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _) | Error::BucketNotFound(_))
+}
+
+pub async fn try_migrate_server_config<S: StorageAPI>(api: Arc<S>) {
+    let config_file = get_config_file();
+    match api
+        .get_object_info(RUSTFS_META_BUCKET, &config_file, &ObjectOptions::default())
+        .await
+    {
+        Ok(_) => {
+            debug!("server config already exists in RustFS metadata bucket, skip migration");
+            return;
+        }
+        Err(err) if is_object_not_found(&err) => {}
+        Err(err) => {
+            warn!("check target server config failed, skip migration: {:?}", err);
+            return;
+        }
+    }
+
+    let opts = ObjectOptions {
+        max_parity: true,
+        no_lock: true,
+        ..Default::default()
+    };
+
+    let mut rd = match api
+        .get_object_reader(MIGRATING_META_BUCKET, &config_file, None, HeaderMap::new(), &opts)
+        .await
+    {
+        Ok(v) => v,
+        Err(err) => {
+            if !is_object_not_found(&err) {
+                warn!("read legacy server config failed: {:?}", err);
+            }
+            return;
+        }
+    };
+
+    let data = match rd.read_all().await {
+        Ok(v) if !v.is_empty() => v,
+        Ok(_) => {
+            debug!("legacy server config is empty, skip migration");
+            return;
+        }
+        Err(err) => {
+            warn!("read legacy server config body failed: {:?}", err);
+            return;
+        }
+    };
+
+    let cfg = match decode_server_config_blob(&data) {
+        Ok(v) => v,
+        Err(err) => {
+            warn!("legacy server config format is incompatible, skip migration: {:?}", err);
+            return;
+        }
+    };
+    let normalized = match encode_server_config_blob(&cfg, Some(&data)) {
+        Ok(v) => v,
+        Err(err) => {
+            warn!("serialize migrated server config failed, skip migration: {:?}", err);
+            return;
+        }
+    };
+
+    match save_config(api, &config_file, normalized).await {
+        Ok(()) => {
+            info!("Migrated compatible server config from legacy metadata bucket");
+        }
+        Err(err) => {
+            warn!("write migrated server config failed: {:?}", err);
+        }
+    }
+}
+
 /// Handle the situation where the configuration file does not exist, create and save a new configuration
 async fn handle_missing_config<S: StorageAPI>(api: Arc<S>, context: &str) -> Result<Config> {
     warn!("Configuration not found ({}): Start initializing new configuration", context);
@@ -167,7 +392,7 @@ async fn read_server_config<S: StorageAPI>(api: Arc<S>, data: &[u8]) -> Result<C
         match read_config(api.clone(), &config_file).await {
             Ok(cfg_data) => {
                 // TODO: decrypt
-                let cfg = Config::unmarshal(&cfg_data)?;
+                let cfg = decode_server_config_blob(&cfg_data)?;
                 return Ok(cfg.merge());
             }
             Err(Error::ConfigNotFound) => return handle_missing_config(api, "Read alternate configuration").await,
@@ -176,14 +401,35 @@ async fn read_server_config<S: StorageAPI>(api: Arc<S>, data: &[u8]) -> Result<C
     }
 
     // Process non-empty configuration data
-    let cfg = Config::unmarshal(data)?;
+    let cfg = decode_server_config_blob(data)?;
     Ok(cfg.merge())
 }
 
 pub async fn save_server_config<S: StorageAPI>(api: Arc<S>, cfg: &Config) -> Result<()> {
-    let data = cfg.marshal()?;
-
     let config_file = get_config_file();
+    let existing = match read_config(api.clone(), &config_file).await {
+        Ok(v) => Some(v),
+        Err(Error::ConfigNotFound) => None,
+        Err(err) => {
+            warn!("read existing server config before save failed, continue with clean output: {:?}", err);
+            None
+        }
+    };
+
+    if let Some(current) = existing.as_deref()
+        && is_standard_object_server_config(current)
+        && let Ok(decoded_current) = decode_server_config_blob(current)
+        && configs_semantically_equal(&decoded_current, cfg)
+    {
+        debug!("server config unchanged and already in standard object shape, skip write");
+        return Ok(());
+    }
+
+    let data = encode_server_config_blob(cfg, existing.as_deref())?;
+    if existing.as_deref().is_some_and(|current| current == data.as_slice()) {
+        debug!("server config bytes unchanged after encode, skip write");
+        return Ok(());
+    }
 
     save_config(api, &config_file, data).await
 }
@@ -228,3 +474,84 @@ async fn apply_dynamic_config_for_sub_sys<S: StorageAPI>(cfg: &mut Config, api:
 
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{
+        configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
+        storage_class_kvs_mut,
+    };
+    use crate::config::Config;
+    use serde_json::Value;
+
+    #[test]
+    fn test_decode_server_config_accepts_legacy_hidden_if_empty_alias() {
+        let input = r#"{"storage_class":{"_":[{"key":"standard","value":"EC:2","hiddenIfEmpty":true}]}}"#;
+        let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
+        let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
+        assert!(kvs.0[0].hidden_if_empty);
+    }
+
+    #[test]
+    fn test_decode_server_config_accepts_missing_hidden_if_empty() {
+        let input = r#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"}]}}"#;
+        let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
+        let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
+        assert!(!kvs.0[0].hidden_if_empty);
+    }
+
+    #[test]
+    fn test_decode_server_config_accepts_v33_object_shape() {
+        let input = r#"{
+          "version":"33",
+          "credential":{"accessKey":"test","secretKey":"testtesttest"},
+          "region":"us-east-1",
+          "worm":"off",
+          "storageclass":{"standard":"EC:2","rrs":"EC:1"},
+          "notify":{},
+          "logger":{},
+          "compress":{"enabled":false},
+          "openid":{},
+          "policy":{"opa":{}},
+          "ldapserverconfig":{}
+        }"#;
+
+        let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
+        let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
+        assert_eq!(kvs.get("standard"), "EC:2");
+        assert_eq!(kvs.get("rrs"), "EC:1");
+        assert_eq!(kvs.get("optimize"), "availability");
+    }
+
+    #[test]
+    fn test_encode_server_config_writes_external_object_shape() {
+        let mut cfg = Config::new();
+        let kvs = storage_class_kvs_mut(&mut cfg);
+        kvs.insert("standard".to_string(), "EC:2".to_string());
+        kvs.insert("rrs".to_string(), "EC:1".to_string());
+
+        let out = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+        let v: Value = serde_json::from_slice(&out).expect("output should be json");
+        assert!(v.get("version").is_some(), "external object should have version");
+        assert!(v.get("storageclass").is_some(), "external object should have storageclass");
+        assert!(v.get("storage_class").is_none(), "should not write rustfs map shape");
+    }
+
+    #[test]
+    fn test_is_standard_object_server_config_detection() {
+        let external = br#"{"version":"33","storageclass":{"standard":"EC:2","rrs":"EC:1"}}"#;
+        assert!(is_standard_object_server_config(external));
+
+        let legacy = br#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"}]}}"#;
+        assert!(!is_standard_object_server_config(legacy));
+    }
+
+    #[test]
+    fn test_configs_semantically_equal_for_equivalent_shapes() {
+        let external = br#"{"version":"33","storageclass":{"standard":"EC:2","rrs":"EC:1","optimize":"availability"}}"#;
+        let legacy = br#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"},{"key":"rrs","value":"EC:1"},{"key":"optimize","value":"availability"}]}}"#;
+        let lhs = decode_server_config_blob(external).expect("decode external");
+        let rhs = decode_server_config_blob(legacy).expect("decode legacy");
+        assert!(configs_semantically_equal(&lhs, &rhs));
+    }
+}

crates/ecstore/src/config/mod.rs

@@ -75,10 +75,15 @@ pub async fn init_global_config_sys(api: Arc<ECStore>) -> Result<()> {
     GLOBAL_CONFIG_SYS.init(api).await
 }
 
+pub async fn try_migrate_server_config(api: Arc<ECStore>) {
+    com::try_migrate_server_config(api).await
+}
+
 #[derive(Debug, Deserialize, Serialize, Clone)]
 pub struct KV {
     pub key: String,
     pub value: String,
+    #[serde(default, alias = "hiddenIfEmpty")]
     pub hidden_if_empty: bool,
 }
 

crates/ecstore/src/data_usage.rs

@@ -15,12 +15,8 @@
 pub mod local_snapshot;
 
 use crate::{
-    bucket::metadata_sys::get_replication_config,
-    config::com::read_config,
-    disk::DiskAPI,
-    error::Error,
-    store::ECStore,
-    store_api::{BucketOperations, ListOperations},
+    bucket::metadata_sys::get_replication_config, config::com::read_config, disk::DiskAPI, error::Error, store::ECStore,
+    store_api::ListOperations,
 };
 pub use local_snapshot::{
     DATA_USAGE_DIR, DATA_USAGE_STATE_DIR, LOCAL_USAGE_SNAPSHOT_VERSION, LocalUsageSnapshot, LocalUsageSnapshotMeta,
@@ -38,7 +34,7 @@ use std::{
 };
 use tokio::fs;
 use tokio::sync::RwLock;
-use tracing::{debug, error, info, warn};
+use tracing::{debug, error, info, instrument};
 
 // Data usage storage constants
 pub const DATA_USAGE_ROOT: &str = SLASH_SEPARATOR;
@@ -81,6 +77,7 @@ lazy_static::lazy_static! {
 }
 
 /// Store data usage info to backend storage
+#[instrument(skip(store))]
 pub async fn store_data_usage_in_backend(data_usage_info: DataUsageInfo, store: Arc<ECStore>) -> Result<(), Error> {
     // Prevent older data from overwriting newer persisted stats
     if let Ok(buf) = read_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await
@@ -107,37 +104,30 @@ pub async fn store_data_usage_in_backend(data_usage_info: DataUsageInfo, store:
 }
 
 /// Load data usage info from backend storage
+#[instrument(skip(store))]
 pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
     let buf: Vec<u8> = match read_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await {
         Ok(data) => data,
         Err(e) => {
             error!("Failed to read data usage info from backend: {}", e);
-            if e == Error::ConfigNotFound {
-                info!("Data usage config not found, building basic statistics");
-                return build_basic_data_usage_info(store).await;
+
+            match read_config(store.clone(), format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()).as_str()).await {
+                Ok(data) => data,
+                Err(e) => {
+                    if e == Error::ConfigNotFound {
+                        return Ok(DataUsageInfo::default());
+                    }
+                    error!("Failed to read data usage info from backend: {}", e);
+                    return Err(Error::other(e));
+                }
             }
-            return Err(Error::other(e));
         }
     };
-
     let mut data_usage_info: DataUsageInfo =
         serde_json::from_slice(&buf).map_err(|e| Error::other(format!("Failed to deserialize data usage info: {e}")))?;
 
     info!("Loaded data usage info from backend with {} buckets", data_usage_info.buckets_count);
 
-    // Validate data and supplement if empty
-    if data_usage_info.buckets_count == 0 || data_usage_info.buckets_usage.is_empty() {
-        warn!("Loaded data is empty, supplementing with basic statistics");
-        if let Ok(basic_info) = build_basic_data_usage_info(store.clone()).await {
-            data_usage_info.buckets_count = basic_info.buckets_count;
-            data_usage_info.buckets_usage = basic_info.buckets_usage;
-            data_usage_info.bucket_sizes = basic_info.bucket_sizes;
-            data_usage_info.objects_total_count = basic_info.objects_total_count;
-            data_usage_info.objects_total_size = basic_info.objects_total_size;
-            data_usage_info.last_update = basic_info.last_update;
-        }
-    }
-
     // Handle backward compatibility
     if data_usage_info.buckets_usage.is_empty() {
         data_usage_info.buckets_usage = data_usage_info
@@ -502,57 +492,6 @@ pub async fn sync_memory_cache_with_backend() -> Result<(), Error> {
     Ok(())
 }
 
-/// Build basic data usage info with real object counts
-pub async fn build_basic_data_usage_info(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
-    let mut data_usage_info = DataUsageInfo::default();
-
-    // Get bucket list
-    match store.list_bucket(&crate::store_api::BucketOptions::default()).await {
-        Ok(buckets) => {
-            data_usage_info.buckets_count = buckets.len() as u64;
-            data_usage_info.last_update = Some(SystemTime::now());
-
-            let mut total_objects = 0u64;
-            let mut total_versions = 0u64;
-            let mut total_size = 0u64;
-            let mut total_delete_markers = 0u64;
-
-            for bucket_info in buckets {
-                if bucket_info.name.starts_with('.') {
-                    continue; // Skip system buckets
-                }
-
-                match compute_bucket_usage(store.clone(), &bucket_info.name).await {
-                    Ok(bucket_usage) => {
-                        total_objects = total_objects.saturating_add(bucket_usage.objects_count);
-                        total_versions = total_versions.saturating_add(bucket_usage.versions_count);
-                        total_size = total_size.saturating_add(bucket_usage.size);
-                        total_delete_markers = total_delete_markers.saturating_add(bucket_usage.delete_markers_count);
-
-                        data_usage_info
-                            .buckets_usage
-                            .insert(bucket_info.name.clone(), bucket_usage.clone());
-                        data_usage_info.bucket_sizes.insert(bucket_info.name, bucket_usage.size);
-                    }
-                    Err(e) => {
-                        warn!("Failed to compute bucket usage for {}: {}", bucket_info.name, e);
-                    }
-                }
-            }
-
-            data_usage_info.objects_total_count = total_objects;
-            data_usage_info.versions_total_count = total_versions;
-            data_usage_info.objects_total_size = total_size;
-            data_usage_info.delete_markers_total_count = total_delete_markers;
-        }
-        Err(e) => {
-            warn!("Failed to list buckets for basic data usage info: {}", e);
-        }
-    }
-
-    Ok(data_usage_info)
-}
-
 /// Create a data usage cache entry from size summary
 pub fn create_cache_entry_from_summary(summary: &SizeSummary) -> DataUsageEntry {
     let mut entry = DataUsageEntry::default();
@@ -696,6 +635,7 @@ pub async fn load_data_usage_cache(store: &crate::set_disk::SetDisks, name: &str
     Ok(d)
 }
 
+#[instrument(skip(cache))]
 pub async fn save_data_usage_cache(cache: &DataUsageCache, name: &str) -> crate::error::Result<()> {
     use crate::config::com::save_config;
     use crate::disk::BUCKET_META_PREFIX;

crates/ecstore/src/disk/local.rs

@@ -697,7 +697,6 @@ impl LocalDisk {
                 match self.read_metadata_with_dmtime(meta_path).await {
                     Ok(res) => Ok(res),
                     Err(err) => {
-                        warn!("read_raw: error: {:?}", err);
                         if err == Error::FileNotFound
                             && !skip_access_checks(volume_dir.as_ref().to_string_lossy().to_string().as_str())
                             && let Err(e) = access(volume_dir.as_ref()).await
@@ -1317,39 +1316,34 @@ fn normalize_path_components(path: impl AsRef<Path>) -> PathBuf {
 
 #[async_trait::async_trait]
 impl DiskAPI for LocalDisk {
-    #[tracing::instrument(skip(self))]
     fn to_string(&self) -> String {
         self.root.to_string_lossy().to_string()
     }
-    #[tracing::instrument(skip(self))]
+
     fn is_local(&self) -> bool {
         true
     }
-    #[tracing::instrument(skip(self))]
+
     fn host_name(&self) -> String {
         self.endpoint.host_port()
     }
-    #[tracing::instrument(skip(self))]
+
     async fn is_online(&self) -> bool {
         true
     }
 
-    #[tracing::instrument(skip(self))]
     fn endpoint(&self) -> Endpoint {
         self.endpoint.clone()
     }
 
-    #[tracing::instrument(skip(self))]
     async fn close(&self) -> Result<()> {
         Ok(())
     }
 
-    #[tracing::instrument(skip(self))]
     fn path(&self) -> PathBuf {
         self.root.clone()
     }
 
-    #[tracing::instrument(skip(self))]
     fn get_disk_location(&self) -> DiskLocation {
         DiskLocation {
             pool_idx: {
@@ -1437,7 +1431,6 @@ impl DiskAPI for LocalDisk {
         Ok(Some(disk_id))
     }
 
-    #[tracing::instrument(skip(self))]
     async fn set_disk_id(&self, _id: Option<Uuid>) -> Result<()> {
         // No setup is required locally
         Ok(())
@@ -1499,6 +1492,12 @@ impl DiskAPI for LocalDisk {
         let erasure = &fi.erasure;
         for (i, part) in fi.parts.iter().enumerate() {
             let checksum_info = erasure.get_checksum_info(part.number);
+            let checksum_algo =
+                if fi.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
+                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
+                } else {
+                    checksum_info.algorithm
+                };
             let part_path = self.get_object_path(
                 volume,
                 path_join_buf(&[
@@ -1512,7 +1511,7 @@ impl DiskAPI for LocalDisk {
                 .bitrot_verify(
                     &part_path,
                     erasure.shard_file_size(part.size as i64) as usize,
-                    checksum_info.algorithm,
+                    checksum_algo,
                     &checksum_info.hash,
                     erasure.shard_size(),
                 )
@@ -2064,7 +2063,6 @@ impl DiskAPI for LocalDisk {
         let search_version_id = fi.version_id.or(Some(Uuid::nil()));
 
         // Check if there's an existing version with the same version_id that has a data_dir to clean up
-        // Note: For non-versioned buckets, fi.version_id is None, but in xl.meta it's stored as Some(Uuid::nil())
         let has_old_data_dir = {
             xlmeta.find_version(search_version_id).ok().and_then(|(_, ver)| {
                 // shard_count == 0 means no other version shares this data_dir
@@ -2601,6 +2599,7 @@ impl DiskAPI for LocalDisk {
         ScanGuard(Arc::clone(&self.scanning))
     }
 
+    #[tracing::instrument(skip(self))]
     async fn read_metadata(&self, volume: &str, path: &str) -> Result<Bytes> {
         // Try to use cached file content reading for better performance, with safe fallback
         let file_path = self.get_object_path(volume, path)?;
@@ -2617,6 +2616,7 @@ impl DiskAPI for LocalDisk {
     }
 }
 
+#[tracing::instrument]
 async fn get_disk_info(drive_path: PathBuf) -> Result<(rustfs_utils::os::DiskInfo, bool)> {
     let drive_path = drive_path.to_string_lossy().to_string();
     check_path_length(&drive_path)?;

crates/ecstore/src/disk/mod.rs

@@ -23,6 +23,7 @@ pub mod local;
 pub mod os;
 
 pub const RUSTFS_META_BUCKET: &str = ".rustfs.sys";
+pub const MIGRATING_META_BUCKET: &str = ".minio.sys";
 pub const RUSTFS_META_MULTIPART_BUCKET: &str = ".rustfs.sys/multipart";
 pub const RUSTFS_META_TMP_BUCKET: &str = ".rustfs.sys/tmp";
 pub const RUSTFS_META_TMP_DELETED_BUCKET: &str = ".rustfs.sys/tmp/.trash";
@@ -60,7 +61,6 @@ pub enum Disk {
 
 #[async_trait::async_trait]
 impl DiskAPI for Disk {
-    #[tracing::instrument(skip(self))]
     fn to_string(&self) -> String {
         match self {
             Disk::Local(local_disk) => local_disk.to_string(),
@@ -68,7 +68,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     async fn is_online(&self) -> bool {
         match self {
             Disk::Local(local_disk) => local_disk.is_online().await,
@@ -76,7 +75,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     fn is_local(&self) -> bool {
         match self {
             Disk::Local(local_disk) => local_disk.is_local(),
@@ -84,7 +82,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     fn host_name(&self) -> String {
         match self {
             Disk::Local(local_disk) => local_disk.host_name(),
@@ -92,7 +89,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     fn endpoint(&self) -> Endpoint {
         match self {
             Disk::Local(local_disk) => local_disk.endpoint(),
@@ -100,7 +96,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     async fn close(&self) -> Result<()> {
         match self {
             Disk::Local(local_disk) => local_disk.close().await,
@@ -108,7 +103,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     async fn get_disk_id(&self) -> Result<Option<Uuid>> {
         match self {
             Disk::Local(local_disk) => local_disk.get_disk_id().await,
@@ -116,7 +110,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     async fn set_disk_id(&self, id: Option<Uuid>) -> Result<()> {
         match self {
             Disk::Local(local_disk) => local_disk.set_disk_id(id).await,
@@ -124,7 +117,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     fn path(&self) -> PathBuf {
         match self {
             Disk::Local(local_disk) => local_disk.path(),
@@ -132,7 +124,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     fn get_disk_location(&self) -> DiskLocation {
         match self {
             Disk::Local(local_disk) => local_disk.get_disk_location(),
@@ -164,7 +155,6 @@ impl DiskAPI for Disk {
         }
     }
 
-    #[tracing::instrument(skip(self))]
     async fn stat_volume(&self, volume: &str) -> Result<VolumeInfo> {
         match self {
             Disk::Local(local_disk) => local_disk.stat_volume(volume).await,

crates/ecstore/src/erasure_coding/bitrot.rs

@@ -28,10 +28,7 @@ pin_project! {
         shard_size: usize,
         buf: Vec<u8>,
         hash_buf: Vec<u8>,
-        // hash_read: usize,
-        // data_buf: Vec<u8>,
-        // data_read: usize,
-        // hash_checked: bool,
+        skip_verify: bool,
         id: Uuid,
     }
 }
@@ -41,7 +38,7 @@ where
     R: AsyncRead + Unpin + Send + Sync,
 {
     /// Create a new BitrotReader.
-    pub fn new(inner: R, shard_size: usize, algo: HashAlgorithm) -> Self {
+    pub fn new(inner: R, shard_size: usize, algo: HashAlgorithm, skip_verify: bool) -> Self {
         let hash_size = algo.size();
         Self {
             inner,
@@ -49,10 +46,7 @@ where
             shard_size,
             buf: Vec::new(),
             hash_buf: vec![0u8; hash_size],
-            // hash_read: 0,
-            // data_buf: Vec::new(),
-            // data_read: 0,
-            // hash_checked: false,
+            skip_verify,
             id: Uuid::new_v4(),
         }
     }
@@ -90,7 +84,7 @@ where
             data_len += n;
         }
 
-        if hash_size > 0 {
+        if hash_size > 0 && !self.skip_verify {
             let actual_hash = self.hash_algo.hash_encode(&out[..data_len]);
             if actual_hash.as_ref() != self.hash_buf.as_slice() {
                 error!("bitrot reader hash mismatch, id={} data_len={}, out_len={}", self.id, data_len, out.len());
@@ -179,7 +173,7 @@ where
 }
 
 pub fn bitrot_shard_file_size(size: usize, shard_size: usize, algo: HashAlgorithm) -> usize {
-    if algo != HashAlgorithm::HighwayHash256S {
+    if algo != HashAlgorithm::HighwayHash256S && algo != HashAlgorithm::HighwayHash256SLegacy {
         return size;
     }
     size.div_ceil(shard_size) * algo.size() + size
@@ -388,7 +382,7 @@ mod tests {
         // Read
         let reader = bitrot_writer.into_inner();
         let reader = Cursor::new(reader.into_inner());
-        let mut bitrot_reader = BitrotReader::new(reader, shard_size, HashAlgorithm::HighwayHash256);
+        let mut bitrot_reader = BitrotReader::new(reader, shard_size, HashAlgorithm::HighwayHash256, false);
         let mut out = Vec::new();
         let mut n = 0;
         while n < data_size {
@@ -420,7 +414,7 @@ mod tests {
         let pos = written.len() - 1;
         written[pos] ^= 0xFF;
         let reader = Cursor::new(written);
-        let mut bitrot_reader = BitrotReader::new(reader, shard_size, HashAlgorithm::HighwayHash256);
+        let mut bitrot_reader = BitrotReader::new(reader, shard_size, HashAlgorithm::HighwayHash256, false);
 
         let count = data_size.div_ceil(shard_size);
 
@@ -464,7 +458,7 @@ mod tests {
 
         let reader = bitrot_writer.into_inner();
         let reader = Cursor::new(reader.into_inner());
-        let mut bitrot_reader = BitrotReader::new(reader, shard_size, HashAlgorithm::None);
+        let mut bitrot_reader = BitrotReader::new(reader, shard_size, HashAlgorithm::None, false);
         let mut out = Vec::new();
         let mut n = 0;
         while n < data_size {

crates/ecstore/src/erasure_coding/decode.rs

@@ -454,6 +454,6 @@ mod tests {
         }
 
         let reader_cursor = Cursor::new(buf);
-        BitrotReader::new(reader_cursor, shard_size, hash_algo.clone())
+        BitrotReader::new(reader_cursor, shard_size, hash_algo.clone(), false)
     }
 }

crates/ecstore/src/erasure_coding/erasure.rs

@@ -12,31 +12,12 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Erasure coding implementation using Reed-Solomon SIMD backend.
+//! Erasure coding implementation using reed-solomon-erasure (GF(2^8)).
+//! Supports legacy (reed-solomon-simd) for reading/healing old-version files.
 //!
-//! This module provides erasure coding functionality with high-performance SIMD
-//! Reed-Solomon implementation:
-//!
-//! ## Reed-Solomon Implementation
-//!
-//! ### SIMD Mode (Only)
-//! - **Performance**: Uses SIMD optimization for high-performance encoding/decoding
-//! - **Compatibility**: Works with any shard size through SIMD implementation
-//! - **Reliability**: High-performance SIMD implementation for large data processing
-//! - **Use case**: Optimized for maximum performance in large data processing scenarios
-//!
-//! ## Example
-//!
-//! ```ignore
-//! use rustfs_ecstore::erasure_coding::Erasure;
-//!
-//! let erasure = Erasure::new(4, 2, 1024); // 4 data shards, 2 parity shards, 1KB block size
-//! let data = b"hello world";
-//! let shards = erasure.encode_data(data).unwrap();
-//! // Simulate loss and recovery...
-//! ```
 
 use bytes::{Bytes, BytesMut};
+use reed_solomon_erasure::galois_8::ReedSolomon;
 use reed_solomon_simd;
 use smallvec::SmallVec;
 use std::io;
@@ -44,132 +25,88 @@ use tokio::io::AsyncRead;
 use tracing::warn;
 use uuid::Uuid;
 
-/// Reed-Solomon encoder using SIMD implementation.
-pub struct ReedSolomonEncoder {
+/// Legacy calc_shard_size formula: (block_size.div_ceil(data_shards) + 1) & !1
+/// Matches main branch and filemeta::ErasureInfo for old-version files.
+pub fn calc_shard_size_legacy(block_size: usize, data_shards: usize) -> usize {
+    (block_size.div_ceil(data_shards) + 1) & !1
+}
+
+/// Reed-Solomon encoder for legacy (main branch) format using reed-solomon-simd.
+/// Used when decoding/encoding files with uses_legacy_checksum == true.
+struct LegacyReedSolomonEncoder {
     data_shards: usize,
     parity_shards: usize,
-    // Use RwLock to ensure thread safety, implementing Send + Sync
     encoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
     decoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
 }
 
-impl Clone for ReedSolomonEncoder {
+impl Clone for LegacyReedSolomonEncoder {
     fn clone(&self) -> Self {
         Self {
             data_shards: self.data_shards,
             parity_shards: self.parity_shards,
-            // Create an empty cache for the new instance instead of sharing one
             encoder_cache: std::sync::RwLock::new(None),
             decoder_cache: std::sync::RwLock::new(None),
         }
     }
 }
 
-impl ReedSolomonEncoder {
-    /// Create a new Reed-Solomon encoder with specified data and parity shards.
-    pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
-        Ok(ReedSolomonEncoder {
-            data_shards,
-            parity_shards,
+impl LegacyReedSolomonEncoder {
+    fn new(_data_shards: usize, _parity_shards: usize) -> io::Result<Self> {
+        Ok(Self {
+            data_shards: _data_shards,
+            parity_shards: _parity_shards,
             encoder_cache: std::sync::RwLock::new(None),
             decoder_cache: std::sync::RwLock::new(None),
         })
     }
 
-    /// Encode data shards with parity.
-    pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
+    fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
         let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
         if shards_vec.is_empty() {
             return Ok(());
         }
-
-        let simd_result = self.encode_with_simd(&mut shards_vec);
-
-        match simd_result {
-            Ok(()) => Ok(()),
-            Err(simd_error) => {
-                warn!("SIMD encoding failed: {}", simd_error);
-                Err(simd_error)
-            }
-        }
-    }
-
-    fn encode_with_simd(&self, shards_vec: &mut [&mut [u8]]) -> io::Result<()> {
         let shard_len = shards_vec[0].len();
-
-        // Get or create encoder
         let mut encoder = {
             let mut cache_guard = self
                 .encoder_cache
                 .write()
                 .map_err(|_| io::Error::other("Failed to acquire encoder cache lock"))?;
-
             match cache_guard.take() {
-                Some(mut cached_encoder) => {
-                    // Use reset method to reset existing encoder to adapt to new parameters
-                    if let Err(e) = cached_encoder.reset(self.data_shards, self.parity_shards, shard_len) {
-                        warn!("Failed to reset SIMD encoder: {:?}, creating new one", e);
-                        // If reset fails, create new encoder
+                Some(mut cached) => {
+                    if cached.reset(self.data_shards, self.parity_shards, shard_len).is_err() {
                         reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
                             .map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?
                     } else {
-                        cached_encoder
+                        cached
                     }
                 }
-                None => {
-                    // First use, create new encoder
-                    reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
-                        .map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?
-                }
+                None => reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
+                    .map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?,
             }
         };
-
-        // Add original shards
         for (i, shard) in shards_vec.iter().enumerate().take(self.data_shards) {
             encoder
                 .add_original_shard(shard)
                 .map_err(|e| io::Error::other(format!("Failed to add shard {i}: {e:?}")))?;
         }
-
-        // Encode and get recovery shards
         let result = encoder
             .encode()
             .map_err(|e| io::Error::other(format!("SIMD encoding failed: {e:?}")))?;
-
-        // Copy recovery shards to output buffer
         for (i, recovery_shard) in result.recovery_iter().enumerate() {
             if i + self.data_shards < shards_vec.len() {
                 shards_vec[i + self.data_shards].copy_from_slice(recovery_shard);
             }
         }
-
-        // Return encoder to cache (encoder is automatically reset after result is dropped, can be reused)
-        drop(result); // Explicitly drop result to ensure encoder is reset
-
+        drop(result);
         *self
             .encoder_cache
             .write()
             .map_err(|_| io::Error::other("Failed to return encoder to cache"))? = Some(encoder);
-
         Ok(())
     }
 
-    /// Reconstruct missing shards.
-    pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
-        // Use SIMD for reconstruction
-        let simd_result = self.reconstruct_with_simd(shards);
-
-        match simd_result {
-            Ok(()) => Ok(()),
-            Err(simd_error) => {
-                warn!("SIMD reconstruction failed: {}", simd_error);
-                Err(simd_error)
-            }
-        }
-    }
-
-    fn reconstruct_with_simd(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
-        // Find a valid shard to determine length
+    fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         let shard_len = shards
             .iter()
             .find_map(|s| s.as_ref().map(|v| v.len()))
@@ -185,7 +122,6 @@ impl ReedSolomonEncoder {
                 Some(mut cached_decoder) => {
                     if let Err(e) = cached_decoder.reset(self.data_shards, self.parity_shards, shard_len) {
                         warn!("Failed to reset SIMD decoder: {:?}, creating new one", e);
-
                         reed_solomon_simd::ReedSolomonDecoder::new(self.data_shards, self.parity_shards, shard_len)
                             .map_err(|e| io::Error::other(format!("Failed to create SIMD decoder: {e:?}")))?
                     } else {
@@ -197,7 +133,6 @@ impl ReedSolomonEncoder {
             }
         };
 
-        // Add available shards (both data and parity)
         for (i, shard_opt) in shards.iter().enumerate() {
             if let Some(shard) = shard_opt {
                 if i < self.data_shards {
@@ -217,7 +152,6 @@ impl ReedSolomonEncoder {
             .decode()
             .map_err(|e| io::Error::other(format!("SIMD decode error: {e:?}")))?;
 
-        // Fill in missing data shards from reconstruction result
         for (i, shard_opt) in shards.iter_mut().enumerate() {
             if shard_opt.is_none() && i < self.data_shards {
                 for (restored_index, restored_data) in result.restored_original_iter() {
@@ -240,6 +174,67 @@ impl ReedSolomonEncoder {
     }
 }
 
+/// Reed-Solomon encoder using reed-solomon-erasure
+pub struct ReedSolomonEncoder {
+    data_shards: usize,
+    parity_shards: usize,
+    encoder: Option<ReedSolomon>,
+}
+
+impl Clone for ReedSolomonEncoder {
+    fn clone(&self) -> Self {
+        Self {
+            data_shards: self.data_shards,
+            parity_shards: self.parity_shards,
+            encoder: self.encoder.clone(),
+        }
+    }
+}
+
+impl ReedSolomonEncoder {
+    /// Create a new Reed-Solomon encoder with specified data and parity shards.
+    pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
+        let encoder = if parity_shards > 0 {
+            ReedSolomon::new(data_shards, parity_shards)
+                .map_err(|e| io::Error::other(format!("Failed to create Reed-Solomon encoder: {e:?}")))
+                .map(Some)?
+        } else {
+            None
+        };
+
+        Ok(ReedSolomonEncoder {
+            data_shards,
+            parity_shards,
+            encoder,
+        })
+    }
+
+    /// Encode data shards with parity.
+    pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
+        let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
+        if shards_vec.is_empty() {
+            return Ok(());
+        }
+
+        if let Some(ref rs) = self.encoder {
+            rs.encode(&mut shards_vec)
+                .map_err(|e| io::Error::other(format!("Reed-Solomon encode failed: {e:?}")))
+        } else {
+            Ok(())
+        }
+    }
+
+    /// Reconstruct missing shards.
+    pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        if let Some(ref rs) = self.encoder {
+            rs.reconstruct_data(shards)
+                .map_err(|e| io::Error::other(format!("Reed-Solomon reconstruct failed: {e:?}")))
+        } else {
+            Ok(())
+        }
+    }
+}
+
 /// Erasure coding utility for data reliability using Reed-Solomon codes.
 ///
 /// This struct provides encoding and decoding of data into data and parity shards.
@@ -262,24 +257,41 @@ impl ReedSolomonEncoder {
 /// let shards = erasure.encode_data(data).unwrap();
 /// // Simulate loss and recovery...
 /// ```
-
-#[derive(Default)]
 pub struct Erasure {
     pub data_shards: usize,
     pub parity_shards: usize,
     encoder: Option<ReedSolomonEncoder>,
+    legacy_encoder: Option<LegacyReedSolomonEncoder>,
     pub block_size: usize,
+    uses_legacy: bool,
     _id: Uuid,
     _buf: Vec<u8>,
 }
 
+impl Default for Erasure {
+    fn default() -> Self {
+        Self {
+            data_shards: 0,
+            parity_shards: 0,
+            encoder: None,
+            legacy_encoder: None,
+            block_size: 0,
+            uses_legacy: false,
+            _id: Uuid::nil(),
+            _buf: vec![],
+        }
+    }
+}
+
 impl Clone for Erasure {
     fn clone(&self) -> Self {
         Self {
             data_shards: self.data_shards,
             parity_shards: self.parity_shards,
             encoder: self.encoder.clone(),
+            legacy_encoder: self.legacy_encoder.clone(),
             block_size: self.block_size,
+            uses_legacy: self.uses_legacy,
             _id: Uuid::new_v4(), // Generate new ID for clone
             _buf: vec![0u8; self.block_size],
         }
@@ -287,28 +299,44 @@ impl Clone for Erasure {
 }
 
 pub fn calc_shard_size(block_size: usize, data_shards: usize) -> usize {
-    (block_size.div_ceil(data_shards) + 1) & !1
+    block_size.div_ceil(data_shards)
 }
 
 impl Erasure {
-    /// Create a new Erasure instance.
+    /// Create a new Erasure instance
     ///
     /// # Arguments
     /// * `data_shards` - Number of data shards.
     /// * `parity_shards` - Number of parity shards.
     /// * `block_size` - Block size for each shard.
     pub fn new(data_shards: usize, parity_shards: usize, block_size: usize) -> Self {
-        let encoder = if parity_shards > 0 {
+        Self::new_with_options(data_shards, parity_shards, block_size, false)
+    }
+
+    /// Create a new Erasure instance with legacy format support.
+    ///
+    /// When `uses_legacy` is true, uses main-branch shard_size formula and reed-solomon-simd
+    /// for decode/reconstruct (for reading and healing old-version files).
+    pub fn new_with_options(data_shards: usize, parity_shards: usize, block_size: usize, uses_legacy: bool) -> Self {
+        let encoder = if !uses_legacy && parity_shards > 0 {
             Some(ReedSolomonEncoder::new(data_shards, parity_shards).unwrap())
         } else {
             None
         };
 
+        let legacy_encoder = if uses_legacy && parity_shards > 0 {
+            Some(LegacyReedSolomonEncoder::new(data_shards, parity_shards).unwrap())
+        } else {
+            None
+        };
+
         Erasure {
             data_shards,
             parity_shards,
             block_size,
             encoder,
+            legacy_encoder,
+            uses_legacy,
             _id: Uuid::new_v4(),
             _buf: vec![0u8; block_size],
         }
@@ -321,30 +349,31 @@ impl Erasure {
     ///
     /// # Returns
     /// A vector of encoded shards as `Bytes`.
-    #[tracing::instrument(level = "info", skip_all, fields(data_len=data.len()))]
+    #[tracing::instrument(level = "debug", skip_all, fields(data_len=data.len()))]
     pub fn encode_data(&self, data: &[u8]) -> io::Result<Vec<Bytes>> {
-        // let shard_size = self.shard_size();
-        // let total_size = shard_size * self.total_shard_count();
-
-        // Data shard count
-        let per_shard_size = calc_shard_size(data.len(), self.data_shards);
-        // Total required size
+        let shard_size_fn = if self.uses_legacy {
+            calc_shard_size_legacy
+        } else {
+            calc_shard_size
+        };
+        let per_shard_size = shard_size_fn(data.len(), self.data_shards);
         let need_total_size = per_shard_size * self.total_shard_count();
 
-        // Create a new buffer with the required total length for all shards
         let mut data_buffer = BytesMut::with_capacity(need_total_size);
-
-        // Copy source data
         data_buffer.extend_from_slice(data);
         data_buffer.resize(need_total_size, 0u8);
 
         {
-            // EC encode, the result will be written into data_buffer
             let data_slices: SmallVec<[&mut [u8]; 16]> = data_buffer.chunks_exact_mut(per_shard_size).collect();
 
-            // Only do EC if parity_shards > 0
             if self.parity_shards > 0 {
-                if let Some(encoder) = self.encoder.as_ref() {
+                if self.uses_legacy {
+                    if let Some(encoder) = self.legacy_encoder.as_ref() {
+                        encoder.encode(data_slices)?;
+                    } else {
+                        warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
+                    }
+                } else if let Some(encoder) = self.encoder.as_ref() {
                     encoder.encode(data_slices)?;
                 } else {
                     warn!("parity_shards > 0, but encoder is None");
@@ -372,7 +401,13 @@ impl Erasure {
     /// Ok if reconstruction succeeds, error otherwise.
     pub fn decode_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         if self.parity_shards > 0 {
-            if let Some(encoder) = self.encoder.as_ref() {
+            if self.uses_legacy {
+                if let Some(encoder) = self.legacy_encoder.as_ref() {
+                    encoder.reconstruct(shards)?;
+                } else {
+                    warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
+                }
+            } else if let Some(encoder) = self.encoder.as_ref() {
                 encoder.reconstruct(shards)?;
             } else {
                 warn!("parity_shards > 0, but encoder is None");
@@ -395,7 +430,11 @@ impl Erasure {
 
     /// Calculate the size of each shard.
     pub fn shard_size(&self) -> usize {
-        calc_shard_size(self.block_size, self.data_shards)
+        if self.uses_legacy {
+            calc_shard_size_legacy(self.block_size, self.data_shards)
+        } else {
+            calc_shard_size(self.block_size, self.data_shards)
+        }
     }
     /// Calculate the total erasure file size for a given original size.
     // Returns the final erasure size from the original size
@@ -408,10 +447,15 @@ impl Erasure {
         }
 
         let total_length = total_length as usize;
+        let shard_size_fn = if self.uses_legacy {
+            calc_shard_size_legacy
+        } else {
+            calc_shard_size
+        };
 
         let num_shards = total_length / self.block_size;
         let last_block_size = total_length % self.block_size;
-        let last_shard_size = calc_shard_size(last_block_size, self.data_shards);
+        let last_shard_size = shard_size_fn(last_block_size, self.data_shards);
         (num_shards * self.shard_size() + last_shard_size) as i64
     }
 
@@ -494,8 +538,7 @@ mod tests {
     #[test]
     fn test_shard_file_size_cases2() {
         let erasure = Erasure::new(12, 4, 1024 * 1024);
-
-        assert_eq!(erasure.shard_file_size(1572864), 131074);
+        assert_eq!(erasure.shard_file_size(1572864), 131073);
     }
 
     #[test]
@@ -517,11 +560,14 @@ mod tests {
         // Case 5: total_length > block_size, aligned
         assert_eq!(erasure.shard_file_size(16), 4); // 16/8=2, last=0, 2*2+0=4
 
-        assert_eq!(erasure.shard_file_size(1248739), 312186); // 1248739/8=156092, last=3, 3 div_ceil 4=1, 156092*2+1=312185
+        // MinIO-compatible: 1248739/8=156092, last=3, ceil(3/4)=1, 156092*2+1=312185
+        assert_eq!(erasure.shard_file_size(1248739), 312185);
 
-        assert_eq!(erasure.shard_file_size(43), 12); // 43/8=5, last=3, 3 div_ceil 4=1, 5*2+1=11
+        // MinIO-compatible: 43/8=5, last=3, ceil(3/4)=1, 5*2+1=11
+        assert_eq!(erasure.shard_file_size(43), 11);
 
-        assert_eq!(erasure.shard_file_size(1572864), 393216); // 43/8=5, last=3, 3 div_ceil 4=1, 5*2+1=11
+        // 1572864 with block_size=8: 196608 full blocks, last=0, 196608*2+0=393216
+        assert_eq!(erasure.shard_file_size(1572864), 393216);
     }
 
     #[test]
@@ -601,10 +647,70 @@ mod tests {
     #[test]
     fn test_shard_size_and_file_size() {
         let erasure = Erasure::new(4, 2, 8);
+        assert_eq!(erasure.shard_file_size(33), 9);
+        assert_eq!(erasure.shard_file_size(0), 0);
+    }
+
+    #[test]
+    fn test_legacy_shard_size_and_file_size() {
+        let erasure = Erasure::new_with_options(4, 2, 8, true);
+        assert_eq!(erasure.shard_size(), 2);
+        assert_eq!(calc_shard_size_legacy(8, 4), 2);
+        assert_eq!(calc_shard_size_legacy(1, 4), 2);
         assert_eq!(erasure.shard_file_size(33), 10);
         assert_eq!(erasure.shard_file_size(0), 0);
     }
 
+    #[test]
+    fn test_legacy_encode_decode_roundtrip() {
+        let data_shards = 4;
+        let parity_shards = 2;
+        let block_size = 1024;
+        let erasure = Erasure::new_with_options(data_shards, parity_shards, block_size, true);
+
+        let data = b"Legacy encode/decode roundtrip test data with sufficient length.".repeat(20);
+        let encoded_shards = erasure.encode_data(&data).unwrap();
+        assert_eq!(encoded_shards.len(), data_shards + parity_shards);
+
+        let mut decode_input: Vec<Option<Vec<u8>>> = vec![None; data_shards + parity_shards];
+        for i in 0..data_shards {
+            decode_input[i] = Some(encoded_shards[i].to_vec());
+        }
+
+        erasure.decode_data(&mut decode_input).unwrap();
+
+        let mut recovered = Vec::new();
+        for shard in decode_input.iter().take(data_shards) {
+            recovered.extend_from_slice(shard.as_ref().unwrap());
+        }
+        recovered.truncate(data.len());
+        assert_eq!(&recovered, &data);
+    }
+
+    #[test]
+    fn test_legacy_decode_with_missing_shards() {
+        let data_shards = 4;
+        let parity_shards = 2;
+        let block_size = 256;
+        let erasure = Erasure::new_with_options(data_shards, parity_shards, block_size, true);
+
+        let data = b"Legacy decode with missing shards test.".repeat(10);
+        let encoded_shards = erasure.encode_data(&data).unwrap();
+
+        let mut shards_opt: Vec<Option<Vec<u8>>> = encoded_shards.iter().map(|s| Some(s.to_vec())).collect();
+        shards_opt[1] = None;
+        shards_opt[5] = None;
+
+        erasure.decode_data(&mut shards_opt).unwrap();
+
+        let mut recovered = Vec::new();
+        for shard in shards_opt.iter().take(data_shards) {
+            recovered.extend_from_slice(shard.as_ref().unwrap());
+        }
+        recovered.truncate(data.len());
+        assert_eq!(&recovered, &data);
+    }
+
     #[test]
     fn test_shard_file_offset() {
         let erasure = Erasure::new(8, 8, 1024 * 1024);
@@ -887,6 +993,57 @@ mod tests {
             assert_eq!(&recovered, &data);
         }
 
+        /// Generates 7557 bytes identical to MinIO generateCompatTestData.
+        fn generate_compat_test_data(size: usize) -> Vec<u8> {
+            (0..size).map(|i| ((i * 7 + 13) % 256) as u8).collect()
+        }
+
+        /// Verifies reed-solomon-simd produces same shards.
+        /// Data shards (0-3) must match for MinIO to read RustFS part files.
+        /// Parity shards (4-5) differ: reed-solomon-simd vs klauspost use different RS encoding.
+        /// Run: cargo test -p rustfs-ecstore test_reed_solomon_compat
+        #[test]
+        fn test_reed_solomon_compat() {
+            let data = generate_compat_test_data(7557);
+            let erasure = Erasure::new(4, 2, 7557);
+            let shards = erasure.encode_data(&data).unwrap();
+            assert_eq!(shards.len(), 6, "expected 6 shards (4 data + 2 parity)");
+
+            // Per-shard HighwayHash
+            let expected_hashes: [&str; 6] = [
+                "fb3db9338e610cec541504ddae4b0bfd54445bcbd45318cf21f35f024240914d", // data 0
+                "a545269a3196e18e77ef9f5ec6e735a4f4ebe82d342db666b11a5256eb305720", // data 1
+                "2adbf0058f36c4cbcb5c9c16c38a6530c54198dfe504179a6f92d2349f245318", // data 2
+                "898e6d060b0cb4f0e830add7e1f936bc8b78442bf582283ee244a3a058602db8", // data 3
+                "4a20460bca044b3a777b26f2b0bcd371e3eab2f156f84778be3ccd8edd521ef2", // parity 4
+                "eb8ba4c0db15ca910d58d031f74e4601ba2fed62ad03ec29cadde3367ab0d415", // parity 5
+            ];
+
+            let mut data_shards_match = true;
+            let mut parity_shards_match = true;
+            for (i, shard) in shards.iter().enumerate() {
+                let hash = rustfs_utils::HashAlgorithm::HighwayHash256S.hash_encode(shard);
+                let got = hex_simd::encode_to_string(hash.as_ref(), hex_simd::AsciiCase::Lower);
+                let matches = got == expected_hashes[i];
+                if i < 4 {
+                    data_shards_match &= matches;
+                } else {
+                    parity_shards_match &= matches;
+                }
+                if !matches {
+                    eprintln!(
+                        "Shard {} ({}): got {} want {}",
+                        i,
+                        if i < 4 { "data" } else { "parity" },
+                        got,
+                        expected_hashes[i]
+                    );
+                }
+            }
+            assert!(data_shards_match, "Data shards (0-3) must match");
+            assert!(parity_shards_match, "Parity shards (4-5): reed-solomon-simd differs");
+        }
+
         #[test]
         fn test_simd_small_data_handling() {
             let data_shards = 4;

crates/ecstore/src/erasure_coding/mod.rs

@@ -19,4 +19,4 @@ pub mod erasure;
 pub mod heal;
 pub use bitrot::*;
 
-pub use erasure::{Erasure, ReedSolomonEncoder, calc_shard_size};
+pub use erasure::{Erasure, ReedSolomonEncoder, calc_shard_size, calc_shard_size_legacy};

crates/ecstore/src/event/name.rs

@@ -13,6 +13,12 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+//! Defines the EventName enum which represents the various S3 event types that can trigger notifications.
+//! This enum includes both specific event types (e.g., ObjectCreated:Put) and aggregate types (e.g., ObjectCreated:*). Each variant has methods to expand into its constituent event types and to compute a bitmask for efficient filtering.
+//! The EventName enum is used in the event notification system to determine which events should trigger notifications based on the configured rules.
+//!
+//! @Deprecated: This module is currently not fully implemented and serves as a placeholder for future development of the event notification system. The EventName enum and its associated methods are defined, but the actual logic for handling events and sending notifications is not yet implemented.
+
 #[derive(Default, Clone)]
 pub enum EventName {
     ObjectAccessedGet,

crates/ecstore/src/event_notification.rs

@@ -15,7 +15,7 @@
 #![allow(unused_variables)]
 
 use crate::bucket::metadata::BucketMetadata;
-use crate::event::name::EventName;
+// use crate::event::name::EventName;
 use crate::event::targetlist::TargetList;
 use crate::store::ECStore;
 use crate::store_api::ObjectInfo;

crates/ecstore/src/pools.rs

@@ -13,6 +13,17 @@
 // limitations under the License.
 
 use crate::bucket::versioning_sys::BucketVersioningSys;
+use crate::bucket::{
+    lifecycle::{
+        bucket_lifecycle_audit::LcEventSrc,
+        bucket_lifecycle_ops::{
+            LifecycleOps, apply_expiry_on_transitioned_object, apply_expiry_rule, eval_action_from_lifecycle,
+        },
+        lifecycle::IlmAction,
+    },
+    metadata_sys,
+    object_lock::objectlock_sys::BucketObjectLockSys,
+};
 use crate::cache_value::metacache_set::{ListPathRawOptions, list_path_raw};
 use crate::config::com::{CONFIG_PREFIX, read_config, save_config};
 use crate::data_usage::DATA_USAGE_CACHE_NAME;
@@ -30,25 +41,32 @@ use crate::store_api::{
     BucketOperations, BucketOptions, CompletePart, GetObjectReader, HealOperations, MakeBucketOptions, MultipartOperations,
     ObjectIO, ObjectOperations, ObjectOptions, PutObjReader, StorageAPI,
 };
-use crate::{sets::Sets, store::ECStore};
+use crate::{global::GLOBAL_LifecycleSys, sets::Sets, store::ECStore};
 use byteorder::{ByteOrder, LittleEndian, WriteBytesExt};
+use bytes::Bytes;
 use futures::future::BoxFuture;
 use http::HeaderMap;
 use rmp_serde::{Deserializer, Serializer};
 use rustfs_common::defer;
 use rustfs_common::heal_channel::HealOpts;
-use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
-use rustfs_rio::{HashReader, WarpReader};
+use rustfs_filemeta::{FileInfoVersions, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
+use rustfs_rio::{EtagResolvable, HashReader, HashReaderDetector, Index, Reader, TryGetIndex, WarpReader};
 use rustfs_utils::path::{SLASH_SEPARATOR, encode_dir_object, path_join};
 use rustfs_workers::workers::Workers;
+use s3s::dto::{BucketLifecycleConfiguration, DefaultRetention, ReplicationConfiguration};
 use serde::{Deserialize, Serialize};
 use std::collections::{HashMap, HashSet};
 use std::fmt::Display;
 use std::io::{Cursor, Write};
 use std::path::PathBuf;
-use std::sync::Arc;
+use std::pin::Pin;
+use std::sync::{
+    Arc,
+    atomic::{AtomicUsize, Ordering},
+};
+use std::task::{Context, Poll};
 use time::{Duration, OffsetDateTime};
-use tokio::io::{AsyncReadExt, BufReader};
+use tokio::io::{AsyncRead, AsyncReadExt, BufReader, ReadBuf};
 use tokio_util::sync::CancellationToken;
 use tracing::{debug, error, info, warn};
 
@@ -75,6 +93,147 @@ pub struct PoolMeta {
     pub dont_save: bool,
 }
 
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct PersistedPoolMeta {
+    pub version: u16,
+    pub pools: Vec<PersistedPoolStatus>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct PersistedPoolStatus {
+    #[serde(rename = "id")]
+    pub id: usize,
+    #[serde(rename = "cmdline")]
+    pub cmd_line: String,
+    #[serde(rename = "lastUpdate", with = "time::serde::rfc3339")]
+    pub last_update: OffsetDateTime,
+    #[serde(rename = "decommissionInfo")]
+    pub decommission: Option<PersistedPoolDecommissionInfo>,
+}
+
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+struct PersistedPoolDecommissionInfo {
+    #[serde(rename = "startTime", with = "time::serde::rfc3339::option")]
+    pub start_time: Option<OffsetDateTime>,
+    #[serde(rename = "startSize")]
+    pub start_size: usize,
+    #[serde(rename = "totalSize")]
+    pub total_size: usize,
+    #[serde(rename = "currentSize")]
+    pub current_size: usize,
+    #[serde(rename = "complete")]
+    pub complete: bool,
+    #[serde(rename = "failed")]
+    pub failed: bool,
+    #[serde(rename = "canceled")]
+    pub canceled: bool,
+    #[serde(rename = "queuedBuckets", default)]
+    pub queued_buckets: Vec<String>,
+    #[serde(rename = "decommissionedBuckets", default)]
+    pub decommissioned_buckets: Vec<String>,
+    #[serde(rename = "bucket", default)]
+    pub bucket: String,
+    #[serde(rename = "prefix", default)]
+    pub prefix: String,
+    #[serde(rename = "object", default)]
+    pub object: String,
+    #[serde(rename = "objectsDecommissioned")]
+    pub items_decommissioned: usize,
+    #[serde(rename = "objectsDecommissionedFailed")]
+    pub items_decommission_failed: usize,
+    #[serde(rename = "bytesDecommissioned")]
+    pub bytes_done: usize,
+    #[serde(rename = "bytesDecommissionedFailed")]
+    pub bytes_failed: usize,
+}
+
+impl From<PersistedPoolMeta> for PoolMeta {
+    fn from(value: PersistedPoolMeta) -> Self {
+        Self {
+            version: value.version,
+            pools: value.pools.into_iter().map(Into::into).collect(),
+            dont_save: false,
+        }
+    }
+}
+
+impl From<PersistedPoolStatus> for PoolStatus {
+    fn from(value: PersistedPoolStatus) -> Self {
+        Self {
+            id: value.id,
+            cmd_line: value.cmd_line,
+            last_update: value.last_update,
+            decommission: value.decommission.map(Into::into),
+        }
+    }
+}
+
+impl From<PersistedPoolDecommissionInfo> for PoolDecommissionInfo {
+    fn from(value: PersistedPoolDecommissionInfo) -> Self {
+        Self {
+            start_time: value.start_time,
+            start_size: value.start_size,
+            total_size: value.total_size,
+            current_size: value.current_size,
+            complete: value.complete,
+            failed: value.failed,
+            canceled: value.canceled,
+            queued_buckets: value.queued_buckets,
+            decommissioned_buckets: value.decommissioned_buckets,
+            bucket: value.bucket,
+            prefix: value.prefix,
+            object: value.object,
+            items_decommissioned: value.items_decommissioned,
+            items_decommission_failed: value.items_decommission_failed,
+            bytes_done: value.bytes_done,
+            bytes_failed: value.bytes_failed,
+        }
+    }
+}
+
+impl From<&PoolMeta> for PersistedPoolMeta {
+    fn from(value: &PoolMeta) -> Self {
+        Self {
+            version: value.version,
+            pools: value.pools.iter().map(Into::into).collect(),
+        }
+    }
+}
+
+impl From<&PoolStatus> for PersistedPoolStatus {
+    fn from(value: &PoolStatus) -> Self {
+        Self {
+            id: value.id,
+            cmd_line: value.cmd_line.clone(),
+            last_update: value.last_update,
+            decommission: value.decommission.as_ref().map(Into::into),
+        }
+    }
+}
+
+impl From<&PoolDecommissionInfo> for PersistedPoolDecommissionInfo {
+    fn from(value: &PoolDecommissionInfo) -> Self {
+        Self {
+            start_time: value.start_time,
+            start_size: value.start_size,
+            total_size: value.total_size,
+            current_size: value.current_size,
+            complete: value.complete,
+            failed: value.failed,
+            canceled: value.canceled,
+            queued_buckets: value.queued_buckets.clone(),
+            decommissioned_buckets: value.decommissioned_buckets.clone(),
+            bucket: value.bucket.clone(),
+            prefix: value.prefix.clone(),
+            object: value.object.clone(),
+            items_decommissioned: value.items_decommissioned,
+            items_decommission_failed: value.items_decommission_failed,
+            bytes_done: value.bytes_done,
+            bytes_failed: value.bytes_failed,
+        }
+    }
+}
+
 impl PoolMeta {
     pub fn new(pools: &[Arc<Sets>], prev_meta: &PoolMeta) -> Self {
         let mut new_meta = Self {
@@ -144,8 +303,8 @@ impl PoolMeta {
         }
 
         let mut buf = Deserializer::new(Cursor::new(&data[4..]));
-        let meta: PoolMeta = Deserialize::deserialize(&mut buf)?;
-        *self = meta;
+        let meta: PersistedPoolMeta = Deserialize::deserialize(&mut buf)?;
+        *self = meta.into();
 
         if self.version != POOL_META_VERSION {
             return Err(Error::other(format!("unexpected PoolMeta version: {}", self.version)));
@@ -161,7 +320,7 @@ impl PoolMeta {
         data.write_u16::<LittleEndian>(POOL_META_FORMAT)?;
         data.write_u16::<LittleEndian>(POOL_META_VERSION)?;
         let mut buf = Vec::new();
-        self.serialize(&mut Serializer::new(&mut buf))?;
+        PersistedPoolMeta::from(self).serialize(&mut Serializer::new(&mut buf))?;
         data.write_all(&buf)?;
 
         for pool in pools {
@@ -178,7 +337,6 @@ impl PoolMeta {
                     stats.last_update = OffsetDateTime::now_utc();
 
                     let mut pd = d.clone();
-                    pd.start_time = None;
                     pd.canceled = true;
                     pd.failed = false;
                     pd.complete = false;
@@ -202,7 +360,6 @@ impl PoolMeta {
                     stats.last_update = OffsetDateTime::now_utc();
 
                     let mut pd = d.clone();
-                    pd.start_time = None;
                     pd.canceled = false;
                     pd.failed = true;
                     pd.complete = false;
@@ -226,7 +383,6 @@ impl PoolMeta {
                     stats.last_update = OffsetDateTime::now_utc();
 
                     let mut pd = d.clone();
-                    pd.start_time = None;
                     pd.canceled = false;
                     pd.failed = false;
                     pd.complete = true;
@@ -576,6 +732,132 @@ impl Display for DecomBucketInfo {
     }
 }
 
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum DecommissionFinalState {
+    Complete,
+    Failed,
+}
+
+fn determine_decommission_final_state(items_failed: usize, was_cancelled: bool) -> DecommissionFinalState {
+    if items_failed > 0 || was_cancelled {
+        DecommissionFinalState::Failed
+    } else {
+        DecommissionFinalState::Complete
+    }
+}
+
+fn remaining_versions_after_decommission(fivs: &FileInfoVersions) -> usize {
+    fivs.versions.iter().filter(|version| !version.deleted).count()
+}
+
+fn decommission_delete_marker_opts(
+    version: &rustfs_filemeta::FileInfo,
+    version_id: Option<String>,
+    src_pool_idx: usize,
+) -> ObjectOptions {
+    ObjectOptions {
+        versioned: true,
+        version_id,
+        mod_time: version.mod_time,
+        src_pool_idx,
+        data_movement: true,
+        delete_marker: true,
+        skip_decommissioned: true,
+        delete_replication: version.replication_state_internal.clone(),
+        ..Default::default()
+    }
+}
+
+async fn should_skip_lifecycle_for_decommission(
+    store: Arc<ECStore>,
+    bucket: &str,
+    version: &rustfs_filemeta::FileInfo,
+    lifecycle_config: Option<&BucketLifecycleConfiguration>,
+    lock_retention: Option<DefaultRetention>,
+    replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
+    apply_actions: bool,
+) -> bool {
+    let Some(lifecycle_config) = lifecycle_config else {
+        return false;
+    };
+
+    let versioned = BucketVersioningSys::prefix_enabled(bucket, &version.name).await;
+    let object_info = crate::store_api::ObjectInfo::from_file_info(version, bucket, &version.name, versioned);
+    let event = eval_action_from_lifecycle(lifecycle_config, lock_retention, replication_config, &object_info).await;
+
+    match event.action {
+        IlmAction::DeleteRestoredAction | IlmAction::DeleteRestoredVersionAction => {
+            if apply_actions && object_info.is_remote() {
+                let _ = apply_expiry_on_transitioned_object(store, &object_info, &event, &LcEventSrc::Decom).await;
+            }
+            false
+        }
+        IlmAction::DeleteAction
+        | IlmAction::DeleteVersionAction
+        | IlmAction::DeleteAllVersionsAction
+        | IlmAction::DelMarkerDeleteAllVersionsAction => {
+            if apply_actions {
+                let _ = apply_expiry_rule(&event, &LcEventSrc::Decom, &object_info).await;
+            }
+            true
+        }
+        _ => false,
+    }
+}
+
+struct IndexedDecommissionReader<R> {
+    inner: R,
+    index: Option<Index>,
+}
+
+impl<R> IndexedDecommissionReader<R> {
+    fn new(inner: R, index: Option<Index>) -> Self {
+        Self { inner, index }
+    }
+}
+
+impl<R: AsyncRead + Unpin + Send + Sync> AsyncRead for IndexedDecommissionReader<R> {
+    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
+        Pin::new(&mut self.inner).poll_read(cx, buf)
+    }
+}
+
+impl<R: AsyncRead + Unpin + Send + Sync> EtagResolvable for IndexedDecommissionReader<R> {}
+
+impl<R: AsyncRead + Unpin + Send + Sync> HashReaderDetector for IndexedDecommissionReader<R> {}
+
+impl<R: AsyncRead + Unpin + Send + Sync> TryGetIndex for IndexedDecommissionReader<R> {
+    fn try_get_index(&self) -> Option<&Index> {
+        self.index.as_ref()
+    }
+}
+
+impl<R: AsyncRead + Unpin + Send + Sync> Reader for IndexedDecommissionReader<R> {}
+
+fn decode_part_index(index: Option<&Bytes>) -> Option<Index> {
+    let bytes = index?;
+    let mut decoded = Index::new();
+    if decoded.load(bytes.as_ref()).is_ok() {
+        Some(decoded)
+    } else {
+        None
+    }
+}
+
+fn put_obj_reader_from_chunk(chunk: Vec<u8>, size: i64, actual_size: i64, index: Option<Index>) -> Result<PutObjReader> {
+    use sha2::{Digest, Sha256};
+
+    let sha256hex = if !chunk.is_empty() {
+        Some(hex_simd::encode_to_string(Sha256::digest(&chunk), hex_simd::AsciiCase::Lower))
+    } else {
+        None
+    };
+
+    let reader = IndexedDecommissionReader::new(WarpReader::new(Cursor::new(chunk)), index);
+    let hash_reader = HashReader::new(Box::new(reader), size, actual_size, None, sha256hex, false)?;
+    Ok(PutObjReader::new(hash_reader))
+}
+
 impl ECStore {
     pub async fn status(&self, idx: usize) -> Result<PoolStatus> {
         let space_info = self.get_decommission_pool_space_info(idx).await?;
@@ -621,13 +903,18 @@ impl ECStore {
             return Err(Error::other("InvalidArgument"));
         }
 
-        let Some(has_canceler) = self.decommission_cancelers.get(idx) else {
-            return Err(Error::other("InvalidArgument"));
-        };
+        let canceler = {
+            let mut cancelers = self.decommission_cancelers.write().await;
+            let Some(slot) = cancelers.get_mut(idx) else {
+                return Err(Error::other("InvalidArgument"));
+            };
 
-        if has_canceler.is_none() {
-            return Err(StorageError::DecommissionNotStarted);
-        }
+            let Some(canceler) = slot.take() else {
+                return Err(StorageError::DecommissionNotStarted);
+            };
+
+            canceler
+        };
 
         let mut lock = self.pool_meta.write().await;
         if lock.decommission_cancel(idx) {
@@ -640,6 +927,8 @@ impl ECStore {
             }
         }
 
+        canceler.cancel();
+
         Ok(())
     }
     pub async fn is_decommission_running(&self) -> bool {
@@ -684,8 +973,8 @@ impl ECStore {
         Ok(())
     }
 
-    #[allow(unused_assignments)]
-    #[tracing::instrument(skip(self, set, wk, rcfg))]
+    #[allow(unused_assignments, clippy::too_many_arguments)]
+    #[tracing::instrument(skip(self, set, wk, lifecycle_config, lock_retention, replication_config))]
     async fn decommission_entry(
         self: &Arc<Self>,
         idx: usize,
@@ -693,7 +982,9 @@ impl ECStore {
         bucket: String,
         set: Arc<SetDisks>,
         wk: Arc<Workers>,
-        rcfg: Option<String>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
+        lock_retention: Option<DefaultRetention>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
     ) {
         warn!("decommission_entry: {} {}", &bucket, &entry.name);
         wk.give().await;
@@ -713,12 +1004,27 @@ impl ECStore {
         fivs.versions.sort_by(|a, b| b.mod_time.cmp(&a.mod_time));
 
         let mut decommissioned: usize = 0;
-        let expired: usize = 0;
+        let mut expired: usize = 0;
 
         for version in fivs.versions.iter() {
-            // TODO: filterLifecycle
+            if should_skip_lifecycle_for_decommission(
+                self.clone(),
+                &bucket,
+                version,
+                lifecycle_config.as_ref(),
+                lock_retention.clone(),
+                replication_config.clone(),
+                true,
+            )
+            .await
+            {
+                expired += 1;
+                decommissioned += 1;
+                continue;
+            }
+
             let remaining_versions = fivs.versions.len() - expired;
-            if version.deleted && remaining_versions == 1 && rcfg.is_none() {
+            if version.deleted && remaining_versions == 1 && replication_config.is_none() {
                 //
                 decommissioned += 1;
                 info!("decommission_pool: DELETE marked object with no other non-current versions will be skipped");
@@ -731,25 +1037,19 @@ impl ECStore {
             let mut failure = false;
             let mut error = None;
             if version.deleted {
-                // TODO: other params
                 if let Err(err) = self
                     .delete_object(
                         bucket.as_str(),
                         &version.name,
-                        ObjectOptions {
-                            versioned: true,
-                            version_id: version_id.clone(),
-                            mod_time: version.mod_time,
-                            src_pool_idx: idx,
-                            data_movement: true,
-                            delete_marker: true,
-                            skip_decommissioned: true,
-                            ..Default::default()
-                        },
+                        decommission_delete_marker_opts(version, version_id.clone(), idx),
                     )
                     .await
                 {
                     if is_err_object_not_found(&err) || is_err_version_not_found(&err) || is_err_data_movement_overwrite(&err) {
+                        warn!(
+                            "decommission_pool: ignore delete-marker copy for {}/{} version {:?}: {:?}",
+                            &bucket, &version.name, &version_id, &err
+                        );
                         ignore = true;
                         continue;
                     }
@@ -776,7 +1076,33 @@ impl ECStore {
 
             for _i in 0..3 {
                 if version.is_remote() {
-                    // TODO: DecomTieredObject
+                    if let Err(err) = self
+                        .decommission_tiered_object(
+                            bucket.as_str(),
+                            &version.name,
+                            version,
+                            &ObjectOptions {
+                                version_id: version_id.clone(),
+                                mod_time: version.mod_time,
+                                user_defined: version.metadata.clone(),
+                                src_pool_idx: idx,
+                                data_movement: true,
+                                ..Default::default()
+                            },
+                        )
+                        .await
+                    {
+                        if is_err_object_not_found(&err) || is_err_version_not_found(&err) || is_err_data_movement_overwrite(&err)
+                        {
+                            ignore = true;
+                            break;
+                        }
+
+                        failure = true;
+                        error!("decommission_pool: decommission_tiered_object err {:?}", &err);
+                        error = Some(err);
+                    }
+                    break;
                 }
 
                 let bucket = bucket.clone();
@@ -847,7 +1173,8 @@ impl ECStore {
             }
 
             {
-                self.pool_meta.write().await.count_item(idx, decommissioned, failure);
+                let size = usize::try_from(version.size).unwrap_or_default();
+                self.pool_meta.write().await.count_item(idx, size, failure);
             }
 
             if failure {
@@ -872,6 +1199,14 @@ impl ECStore {
                 .await
         {
             error!("decommission_pool: delete_object err {:?}", &err);
+        } else if decommissioned != fivs.versions.len() {
+            warn!(
+                "decommission_pool: source object retained for {}/{} because only {}/{} versions were decommissioned",
+                &bucket,
+                &entry.name,
+                decommissioned,
+                fivs.versions.len()
+            );
         }
 
         {
@@ -903,16 +1238,19 @@ impl ECStore {
     ) -> Result<()> {
         let wk = Workers::new(pool.disk_set.len() * 2).map_err(Error::other)?;
 
-        // let mut vc = None;
-        // replication
-        let rcfg: Option<String> = None;
+        let mut lifecycle_config = None;
+        let mut lock_retention = None;
+        let mut replication_config = None;
 
         if bi.name != RUSTFS_META_BUCKET {
-            let _versioning = BucketVersioningSys::get(&bi.name).await?;
-            // vc = Some(versioning);
-            // TODO: LifecycleSys
-            // TODO: BucketObjectLockSys
-            // TODO: ReplicationConfig
+            let _ = BucketVersioningSys::get(&bi.name).await?;
+            lifecycle_config = GLOBAL_LifecycleSys.get(&bi.name).await;
+            lock_retention = BucketObjectLockSys::get(&bi.name).await;
+            replication_config = match metadata_sys::get_replication_config(&bi.name).await {
+                Ok(config) => Some(config),
+                Err(Error::ConfigNotFound) => None,
+                Err(err) => return Err(err),
+            };
         }
 
         for (set_idx, set) in pool.disk_set.iter().enumerate() {
@@ -925,17 +1263,22 @@ impl ECStore {
                 let bucket = bi.name.clone();
                 let wk = wk.clone();
                 let set = set.clone();
-                let rcfg = rcfg.clone();
+                let lifecycle_config = lifecycle_config.clone();
+                let lock_retention = lock_retention.clone();
+                let replication_config = replication_config.clone();
                 move |entry: MetaCacheEntry| {
                     let this = this.clone();
                     let bucket = bucket.clone();
                     let wk = wk.clone();
                     let set = set.clone();
-                    let rcfg = rcfg.clone();
+                    let lifecycle_config = lifecycle_config.clone();
+                    let lock_retention = lock_retention.clone();
+                    let replication_config = replication_config.clone();
 
                     Box::pin(async move {
                         wk.take().await;
-                        this.decommission_entry(idx, entry, bucket, set, wk, rcfg).await
+                        this.decommission_entry(idx, entry, bucket, set, wk, lifecycle_config, lock_retention, replication_config)
+                            .await
                     })
                 }
             });
@@ -988,7 +1331,15 @@ impl ECStore {
 
     #[tracing::instrument(skip(self, rx))]
     pub async fn do_decommission_in_routine(self: &Arc<Self>, rx: CancellationToken, idx: usize) {
-        if let Err(err) = self.decommission_in_background(rx, idx).await {
+        let decommission_token = rx.child_token();
+        {
+            let mut cancelers = self.decommission_cancelers.write().await;
+            if let Some(slot) = cancelers.get_mut(idx) {
+                *slot = Some(decommission_token.clone());
+            }
+        }
+
+        if let Err(err) = self.decommission_in_background(decommission_token.clone(), idx).await {
             error!("decom err {:?}", &err);
             if let Err(er) = self.decommission_failed(idx).await {
                 error!("decom failed err {:?}", &er);
@@ -1001,29 +1352,49 @@ impl ECStore {
 
         warn!("decommission: decommission_in_background complete {}", idx);
 
-        let (failed, cmd_line) = {
+        let (final_state, cmd_line) = {
             let pool_meta = self.pool_meta.read().await;
-            let failed = {
+            let final_state = {
                 if let Some(info) = &pool_meta.pools[idx].decommission {
-                    info.items_decommission_failed > 0
+                    determine_decommission_final_state(info.items_decommission_failed, info.canceled)
                 } else {
-                    false
+                    DecommissionFinalState::Failed
                 }
             };
             let cmd_line = pool_meta.pools[idx].cmd_line.clone();
-            (failed, cmd_line)
+            (final_state, cmd_line)
         };
 
-        if !failed {
+        let mut completed_successfully = false;
+
+        if final_state == DecommissionFinalState::Complete {
             warn!("Decommissioning complete for pool {}, verifying for any pending objects", cmd_line);
-            if let Err(er) = self.decommission_failed(idx).await {
-                error!("decom failed err {:?}", &er);
+            if let Err(err) = self.check_after_decommission(idx).await {
+                error!("decom post-check err {:?}", &err);
+                if let Err(er) = self.decommission_failed(idx).await {
+                    error!("decom failed err {:?}", &er);
+                }
+            } else if let Err(er) = self.complete_decommission(idx).await {
+                error!("decom complete err {:?}", &er);
+            } else {
+                completed_successfully = true;
             }
-        } else if let Err(er) = self.complete_decommission(idx).await {
-            error!("decom complete err {:?}", &er);
+        } else if let Err(er) = self.decommission_failed(idx).await {
+            error!("decom failed err {:?}", &er);
         }
 
-        warn!("Decommissioning complete for pool {}", cmd_line);
+        {
+            let mut cancelers = self.decommission_cancelers.write().await;
+            if let Some(slot) = cancelers.get_mut(idx) {
+                *slot = None;
+            }
+        }
+
+        if completed_successfully {
+            warn!("Decommissioning complete for pool {}", cmd_line);
+        } else {
+            warn!("Decommissioning finished in failed state for pool {}", cmd_line);
+        }
     }
 
     #[tracing::instrument(skip(self))]
@@ -1043,6 +1414,14 @@ impl ECStore {
             }
         }
 
+        let canceler = {
+            let mut cancelers = self.decommission_cancelers.write().await;
+            cancelers.get_mut(idx).and_then(Option::take)
+        };
+        if let Some(canceler) = canceler {
+            canceler.cancel();
+        }
+
         Ok(())
     }
 
@@ -1061,6 +1440,14 @@ impl ECStore {
             }
         }
 
+        let canceler = {
+            let mut cancelers = self.decommission_cancelers.write().await;
+            cancelers.get_mut(idx).and_then(Option::take)
+        };
+        if let Some(canceler) = canceler {
+            canceler.cancel();
+        }
+
         Ok(())
     }
 
@@ -1190,6 +1577,94 @@ impl ECStore {
         Ok(ret)
     }
 
+    async fn check_after_decommission(self: &Arc<Self>, idx: usize) -> Result<()> {
+        let buckets = self.get_buckets_to_decommission().await?;
+        let pool = self.pools[idx].clone();
+
+        for set in &pool.disk_set {
+            for bucket_info in &buckets {
+                let mut lifecycle_config = None;
+                let mut lock_retention = None;
+                let mut replication_config = None;
+                if bucket_info.name != RUSTFS_META_BUCKET {
+                    lifecycle_config = GLOBAL_LifecycleSys.get(&bucket_info.name).await;
+                    lock_retention = BucketObjectLockSys::get(&bucket_info.name).await;
+                    replication_config = match metadata_sys::get_replication_config(&bucket_info.name).await {
+                        Ok(config) => Some(config),
+                        Err(Error::ConfigNotFound) => None,
+                        Err(err) => return Err(err),
+                    };
+                }
+
+                let versions_found = Arc::new(AtomicUsize::new(0));
+                let versions_found_cb = versions_found.clone();
+                let bucket_name = bucket_info.name.clone();
+                let lifecycle_config_cb = lifecycle_config.clone();
+                let lock_retention_cb = lock_retention.clone();
+                let replication_config_cb = replication_config.clone();
+                let store = Arc::clone(self);
+
+                let callback: ListCallback = Arc::new(move |entry: MetaCacheEntry| {
+                    let versions_found = versions_found_cb.clone();
+                    let bucket_name = bucket_name.clone();
+                    let lifecycle_config = lifecycle_config_cb.clone();
+                    let lock_retention = lock_retention_cb.clone();
+                    let replication_config = replication_config_cb.clone();
+                    let store = Arc::clone(&store);
+                    Box::pin(async move {
+                        if !entry.is_object() {
+                            return;
+                        }
+
+                        if bucket_name == RUSTFS_META_BUCKET && entry.name.contains(DATA_USAGE_CACHE_NAME) {
+                            return;
+                        }
+
+                        let Ok(fivs) = entry.file_info_versions(&bucket_name) else {
+                            return;
+                        };
+
+                        let mut remaining = 0;
+                        for version in &fivs.versions {
+                            if version.deleted {
+                                continue;
+                            }
+                            if should_skip_lifecycle_for_decommission(
+                                Arc::clone(&store),
+                                &bucket_name,
+                                version,
+                                lifecycle_config.as_ref(),
+                                lock_retention.clone(),
+                                replication_config.clone(),
+                                false,
+                            )
+                            .await
+                            {
+                                continue;
+                            }
+                            remaining += 1;
+                        }
+
+                        versions_found.fetch_add(remaining, Ordering::Relaxed);
+                    })
+                });
+
+                set.list_objects_to_decommission(CancellationToken::new(), bucket_info.clone(), callback)
+                    .await?;
+
+                let versions_found = versions_found.load(Ordering::Relaxed);
+                if versions_found > 0 {
+                    return Err(Error::other(format!(
+                        "at least {versions_found} object(s)/version(s) were found in bucket `{}` after decommissioning",
+                        bucket_info.name
+                    )));
+                }
+            }
+        }
+
+        Ok(())
+    }
+
     #[tracing::instrument(skip(self, rd))]
     async fn decommission_object(self: Arc<Self>, pool_idx: usize, bucket: String, rd: GetObjectReader) -> Result<()> {
         warn!("decommission_object: start {} {}", &bucket, &rd.object_info.name);
@@ -1238,7 +1713,10 @@ impl ECStore {
 
                 reader.read_exact(&mut chunk).await?;
 
-                let mut data = PutObjReader::from_vec(chunk);
+                let part_size = i64::try_from(part.size).map_err(|_| Error::other("part size overflow"))?;
+                let part_actual_size = if part.actual_size > 0 { part.actual_size } else { part_size };
+                let index = decode_part_index(part.index.as_ref());
+                let mut data = put_obj_reader_from_chunk(chunk, part_size, part_actual_size, index)?;
 
                 let pi = match self
                     .put_object_part(
@@ -1294,8 +1772,13 @@ impl ECStore {
             return Ok(());
         }
 
-        let reader = BufReader::new(rd.stream);
-        let hrd = HashReader::new(Box::new(WarpReader::new(reader)), object_info.size, object_info.size, None, None, false)?;
+        let actual_size = object_info.get_actual_size()?;
+        let index = object_info
+            .parts
+            .first()
+            .and_then(|part| decode_part_index(part.index.as_ref()));
+        let reader = IndexedDecommissionReader::new(WarpReader::new(BufReader::new(rd.stream)), index);
+        let hrd = HashReader::new(Box::new(reader), object_info.size, actual_size, object_info.etag.clone(), None, false)?;
         let mut data = PutObjReader::new(hrd);
 
         if let Err(err) = self
@@ -1325,6 +1808,159 @@ impl ECStore {
     }
 }
 
+#[cfg(test)]
+#[allow(clippy::items_after_test_module)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn determine_decommission_final_state_marks_failures_and_cancellations() {
+        assert_eq!(determine_decommission_final_state(0, false), DecommissionFinalState::Complete);
+        assert_eq!(determine_decommission_final_state(1, false), DecommissionFinalState::Failed);
+        assert_eq!(determine_decommission_final_state(0, true), DecommissionFinalState::Failed);
+    }
+
+    #[test]
+    fn remaining_versions_after_decommission_ignores_delete_markers() {
+        let fivs = FileInfoVersions {
+            versions: vec![
+                rustfs_filemeta::FileInfo {
+                    deleted: false,
+                    size: 128,
+                    ..Default::default()
+                },
+                rustfs_filemeta::FileInfo {
+                    deleted: true,
+                    size: 0,
+                    ..Default::default()
+                },
+            ],
+            ..Default::default()
+        };
+
+        assert_eq!(remaining_versions_after_decommission(&fivs), 1);
+    }
+
+    #[test]
+    fn decommission_delete_marker_opts_preserves_replication_state() {
+        let mod_time = OffsetDateTime::now_utc();
+        let version = rustfs_filemeta::FileInfo {
+            mod_time: Some(mod_time),
+            replication_state_internal: Some(rustfs_filemeta::ReplicationState {
+                replica_status: rustfs_filemeta::ReplicationStatusType::Replica,
+                delete_marker: true,
+                replicate_decision_str: "existing".to_string(),
+                ..Default::default()
+            }),
+            ..Default::default()
+        };
+
+        let opts = decommission_delete_marker_opts(&version, Some("version-id".to_string()), 7);
+        let replication = opts.delete_replication.expect("replication state should be preserved");
+
+        assert!(opts.versioned);
+        assert!(opts.data_movement);
+        assert!(opts.delete_marker);
+        assert!(opts.skip_decommissioned);
+        assert_eq!(opts.src_pool_idx, 7);
+        assert_eq!(opts.version_id.as_deref(), Some("version-id"));
+        assert_eq!(opts.mod_time, Some(mod_time));
+        assert_eq!(replication.replica_status, rustfs_filemeta::ReplicationStatusType::Replica);
+        assert!(replication.delete_marker);
+        assert_eq!(replication.replicate_decision_str, "existing");
+    }
+
+    #[test]
+    fn decommission_state_transitions_preserve_start_time() {
+        let start_time = OffsetDateTime::now_utc();
+        let mut pool_meta = PoolMeta {
+            version: POOL_META_VERSION,
+            pools: vec![PoolStatus {
+                id: 0,
+                cmd_line: "/tmp/pool".to_string(),
+                last_update: start_time,
+                decommission: Some(PoolDecommissionInfo {
+                    start_time: Some(start_time),
+                    ..Default::default()
+                }),
+            }],
+            dont_save: true,
+        };
+
+        assert!(pool_meta.decommission_failed(0));
+        assert_eq!(
+            pool_meta.pools[0].decommission.as_ref().and_then(|info| info.start_time),
+            Some(start_time)
+        );
+
+        assert!(pool_meta.decommission_complete(0));
+        assert_eq!(
+            pool_meta.pools[0].decommission.as_ref().and_then(|info| info.start_time),
+            Some(start_time)
+        );
+
+        assert!(pool_meta.decommission_cancel(0));
+        assert_eq!(
+            pool_meta.pools[0].decommission.as_ref().and_then(|info| info.start_time),
+            Some(start_time)
+        );
+    }
+
+    #[test]
+    fn pool_meta_persists_decommission_resume_queues() {
+        let start_time = OffsetDateTime::now_utc();
+        let pool_meta = PoolMeta {
+            version: POOL_META_VERSION,
+            pools: vec![PoolStatus {
+                id: 1,
+                cmd_line: "/data/pool1/disk{1...4}".to_string(),
+                last_update: start_time,
+                decommission: Some(PoolDecommissionInfo {
+                    start_time: Some(start_time),
+                    queued_buckets: vec!["bucket-a".to_string(), "bucket-b/prefix".to_string()],
+                    decommissioned_buckets: vec!["bucket-done".to_string()],
+                    bucket: "bucket-b".to_string(),
+                    prefix: "prefix".to_string(),
+                    object: "object.txt".to_string(),
+                    items_decommissioned: 7,
+                    items_decommission_failed: 1,
+                    bytes_done: 1024,
+                    bytes_failed: 128,
+                    ..Default::default()
+                }),
+            }],
+            dont_save: false,
+        };
+
+        let mut buf = Vec::new();
+        PersistedPoolMeta::from(&pool_meta)
+            .serialize(&mut Serializer::new(&mut buf))
+            .expect("pool meta should serialize");
+
+        let mut deserializer = Deserializer::new(Cursor::new(&buf));
+        let restored: PoolMeta = PersistedPoolMeta::deserialize(&mut deserializer)
+            .expect("pool meta should deserialize")
+            .into();
+
+        let restored_decommission = restored.pools[0]
+            .decommission
+            .as_ref()
+            .expect("decommission info should survive round-trip");
+        assert_eq!(
+            restored_decommission.queued_buckets,
+            vec!["bucket-a".to_string(), "bucket-b/prefix".to_string()]
+        );
+        assert_eq!(restored_decommission.decommissioned_buckets, vec!["bucket-done".to_string()]);
+        assert_eq!(restored_decommission.bucket, "bucket-b");
+        assert_eq!(restored_decommission.prefix, "prefix");
+        assert_eq!(restored_decommission.object, "object.txt");
+        assert_eq!(restored_decommission.items_decommissioned, 7);
+        assert_eq!(restored_decommission.items_decommission_failed, 1);
+        assert_eq!(restored_decommission.bytes_done, 1024);
+        assert_eq!(restored_decommission.bytes_failed, 128);
+    }
+}
+
 // impl Fn(MetaCacheEntry) -> impl Future<Output = Result<(), Error>>
 
 pub type ListCallback = Arc<dyn Fn(MetaCacheEntry) -> BoxFuture<'static, ()> + Send + Sync + 'static>;

crates/ecstore/src/set_disk.rs

@@ -48,7 +48,7 @@ use crate::{
         UpdateMetadataOpts, endpoint::Endpoint, error::DiskError, format::FormatV3, new_disk,
     },
     error::{StorageError, to_object_err},
-    event::name::EventName,
+    // event::name::EventName,
     event_notification::{EventArgs, send_event},
     global::{GLOBAL_LOCAL_DISK_MAP, GLOBAL_LOCAL_DISK_SET_DRIVES, get_global_deployment_id, is_dist_erasure},
     store_api::{
@@ -79,9 +79,13 @@ use rustfs_lock::local_lock::LocalLock;
 use rustfs_lock::{FastLockGuard, NamespaceLock, NamespaceLockGuard, NamespaceLockWrapper, ObjectKey};
 use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem};
 use rustfs_rio::{EtagResolvable, HashReader, HashReaderMut, TryGetIndex as _, WarpReader};
-use rustfs_utils::http::RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM;
+use rustfs_s3_common::EventName;
+use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
 use rustfs_utils::http::headers::AMZ_STORAGE_CLASS;
-use rustfs_utils::http::headers::{AMZ_OBJECT_TAGGING, RESERVED_METADATA_PREFIX, RESERVED_METADATA_PREFIX_LOWER};
+use rustfs_utils::http::{
+    SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
+    contains_key_str, get_header_map, get_str, insert_str, remove_header_map,
+};
 use rustfs_utils::{
     HashAlgorithm,
     crypto::hex,
@@ -135,6 +139,30 @@ pub fn get_lock_acquire_timeout() -> Duration {
     Duration::from_secs(rustfs_utils::get_env_u64("RUSTFS_LOCK_ACQUIRE_TIMEOUT", 5))
 }
 
+fn build_tiered_decommission_file_info(
+    bucket: &str,
+    object: &str,
+    fi: &FileInfo,
+    disk_count: usize,
+    default_parity_count: usize,
+    storage_class: Option<&str>,
+) -> (FileInfo, usize) {
+    let parity_drives = GLOBAL_STORAGE_CLASS
+        .get()
+        .and_then(|sc| sc.get_parity_for_sc(storage_class.unwrap_or_default()))
+        .unwrap_or(default_parity_count);
+    let data_drives = disk_count - parity_drives;
+    let mut write_quorum = data_drives;
+    if data_drives == parity_drives {
+        write_quorum += 1;
+    }
+
+    let mut updated = fi.clone();
+    updated.erasure = FileInfo::new([bucket, object].join("/").as_str(), data_drives, parity_drives).erasure;
+
+    (updated, write_quorum)
+}
+
 #[derive(Clone, Debug)]
 pub struct SetDisks {
     pub locker_owner: String,
@@ -493,6 +521,7 @@ impl ObjectIO for SetDisks {
         let object = object.to_owned();
         let set_index = self.set_index;
         let pool_index = self.pool_index;
+        let skip_verify = opts.skip_verify_bitrot;
         // Move the read-lock guard into the task so it lives for the duration of the read
         // let _guard_to_hold = _read_lock_guard; // moved into closure below
         tokio::spawn(async move {
@@ -509,6 +538,7 @@ impl ObjectIO for SetDisks {
                 &disks,
                 set_index,
                 pool_index,
+                skip_verify,
             )
             .await
             {
@@ -519,7 +549,7 @@ impl ObjectIO for SetDisks {
         Ok(reader)
     }
 
-    #[tracing::instrument(level = "debug", skip(self, data,))]
+    #[tracing::instrument(skip(self, data,))]
     async fn put_object(&self, bucket: &str, object: &str, data: &mut PutObjReader, opts: &ObjectOptions) -> Result<ObjectInfo> {
         let disks = self.get_disks_internal().await;
 
@@ -617,7 +647,7 @@ impl ObjectIO for SetDisks {
                     &tmp_object,
                     erasure.shard_file_size(data.size()),
                     erasure.shard_size(),
-                    HashAlgorithm::HighwayHash256,
+                    HashAlgorithm::HighwayHash256S,
                 )
                 .await
                 {
@@ -675,8 +705,8 @@ impl ObjectIO for SetDisks {
             )));
         }
 
-        if user_defined.contains_key(&format!("{RESERVED_METADATA_PREFIX_LOWER}compression")) {
-            user_defined.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression-size"), w_size.to_string());
+        if contains_key_str(&user_defined, SUFFIX_COMPRESSION) {
+            insert_str(&mut user_defined, SUFFIX_COMPRESSION_SIZE, w_size.to_string());
         }
 
         let index_op = data.stream.try_get_index().map(|v| v.clone().into_vec());
@@ -762,7 +792,7 @@ impl ObjectIO for SetDisks {
         .await?;
 
         if let Some(old_dir) = op_old_dir {
-            self.commit_rename_data_dir(&shuffle_disks, bucket, object, &old_dir.to_string(), write_quorum)
+            self.commit_rename_data_dir(&online_disks, bucket, object, &old_dir.to_string(), write_quorum)
                 .await?;
         }
 
@@ -1333,6 +1363,12 @@ impl ObjectOperations for SetDisks {
         let mut delete_marker = opts.versioned;
 
         if opts.version_id.is_some() {
+            // Decommission/rebalance may recreate a delete marker on a new pool before that
+            // exact version exists there, so we must still treat it as a mark-delete write.
+            if opts.data_movement && opts.delete_marker && !version_found {
+                mark_delete = true;
+            }
+
             if version_found && opts.delete_marker_replication_status() == ReplicationStatusType::Replica {
                 mark_delete = false;
             }
@@ -1653,6 +1689,7 @@ impl ObjectOperations for SetDisks {
         let cloned_fi = fi.clone();
         let set_index = self.set_index;
         let pool_index = self.pool_index;
+        let skip_verify = opts.skip_verify_bitrot;
         tokio::spawn(async move {
             if let Err(e) = Self::get_object_with_fileinfo(
                 &cloned_bucket,
@@ -1665,6 +1702,7 @@ impl ObjectOperations for SetDisks {
                 &online_disks,
                 set_index,
                 pool_index,
+                skip_verify,
             )
             .await
             {
@@ -1682,17 +1720,17 @@ impl ObjectOperations for SetDisks {
         if let Err(err) = rv {
             return Err(StorageError::Io(err));
         }
-        let rv = rv.unwrap();
+        let rv = rv?;
         fi.transition_status = TRANSITION_COMPLETE.to_string();
         fi.transitioned_objname = dest_obj;
         fi.transition_tier = opts.transition.tier.clone();
         fi.transition_version_id = if rv.is_empty() { None } else { Some(Uuid::parse_str(&rv)?) };
-        let mut event_name = EventName::ObjectTransitionComplete.as_ref();
+        let mut event_name = EventName::ObjectTransitionComplete.as_str();
 
         let disks = self.get_disks(0, 0).await?;
 
         if let Err(err) = self.delete_object_version(bucket, object, &fi, false).await {
-            event_name = EventName::ObjectTransitionFailed.as_ref();
+            event_name = EventName::ObjectTransitionFailed.as_str();
         }
 
         for disk in disks.iter() {
@@ -1884,6 +1922,68 @@ impl ObjectOperations for SetDisks {
     }
 }
 
+impl SetDisks {
+    #[tracing::instrument(skip(self, fi, opts))]
+    pub(crate) async fn decommission_tiered_object(
+        &self,
+        bucket: &str,
+        object: &str,
+        fi: &FileInfo,
+        opts: &ObjectOptions,
+    ) -> Result<()> {
+        let _lock_guard = if !opts.no_lock {
+            Some(
+                self.new_ns_lock(bucket, object)
+                    .await?
+                    .get_write_lock(get_lock_acquire_timeout())
+                    .await
+                    .map_err(|e| {
+                        Error::other(format!(
+                            "Failed to acquire write lock: {}",
+                            self.format_lock_error_from_error(bucket, object, "write", &e)
+                        ))
+                    })?,
+            )
+        } else {
+            None
+        };
+
+        let disks = self.disks.read().await.clone();
+        let storage_class = opts.user_defined.get(AMZ_STORAGE_CLASS).map(String::as_str);
+        let (fi, write_quorum) =
+            build_tiered_decommission_file_info(bucket, object, fi, disks.len(), self.default_parity_count, storage_class);
+        let parts_metadata = vec![fi.clone(); disks.len()];
+        let (shuffle_disks, parts_metadata) = Self::shuffle_disks_and_parts_metadata(&disks, &parts_metadata, &fi);
+
+        let mut errs = Vec::with_capacity(shuffle_disks.len());
+        let mut futures = Vec::with_capacity(shuffle_disks.len());
+        for (index, disk) in shuffle_disks.iter().enumerate() {
+            let mut file_info = parts_metadata[index].clone();
+            file_info.erasure.index = index + 1;
+            futures.push(async move {
+                if let Some(disk) = disk {
+                    disk.write_metadata("", bucket, object, file_info).await
+                } else {
+                    Err(DiskError::DiskNotFound)
+                }
+            });
+        }
+
+        for result in join_all(futures).await {
+            match result {
+                Ok(_) => errs.push(None),
+                Err(err) => errs.push(Some(err)),
+            }
+        }
+
+        if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, write_quorum) {
+            return Err(to_object_err(err.into(), vec![bucket, object]));
+        }
+
+        Ok(())
+    }
+}
+
 #[async_trait::async_trait]
 impl ListOperations for SetDisks {
     #[tracing::instrument(skip(self))]
@@ -2002,7 +2102,7 @@ impl MultipartOperations for SetDisks {
                     &tmp_part_path,
                     erasure.shard_file_size(data.size()),
                     erasure.shard_size(),
-                    HashAlgorithm::HighwayHash256,
+                    HashAlgorithm::HighwayHash256S,
                 )
                 .await
                 {
@@ -2449,11 +2549,11 @@ impl MultipartOperations for SetDisks {
 
         fi.data_dir = Some(Uuid::new_v4());
 
-        if let Some(cssum) = user_defined.get(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM)
+        if let Some(cssum) = get_header_map(&user_defined, SUFFIX_REPLICATION_SSEC_CRC)
             && !cssum.is_empty()
         {
-            fi.checksum = base64_simd::STANDARD.decode_to_vec(cssum).ok().map(Bytes::from);
-            user_defined.remove(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM);
+            fi.checksum = base64_simd::STANDARD.decode_to_vec(&cssum).ok().map(Bytes::from);
+            remove_header_map(&mut user_defined, SUFFIX_REPLICATION_SSEC_CRC);
         }
 
         let parts_metadata = vec![fi.clone(); disks.len()];
@@ -2804,8 +2904,8 @@ impl MultipartOperations for SetDisks {
             }
         }
 
-        if let Some(rc_crc) = opts.user_defined.get(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM) {
-            if let Ok(rc_crc_bytes) = base64_simd::STANDARD.decode_to_vec(rc_crc) {
+        if let Some(rc_crc) = get_header_map(&opts.user_defined, SUFFIX_REPLICATION_SSEC_CRC) {
+            if let Ok(rc_crc_bytes) = base64_simd::STANDARD.decode_to_vec(&rc_crc) {
                 fi.checksum = Some(Bytes::from(rc_crc_bytes));
             } else {
                 error!("complete_multipart_upload decode rc_crc failed rc_crc={}", rc_crc);
@@ -2844,25 +2944,19 @@ impl MultipartOperations for SetDisks {
         fi.metadata.insert("etag".to_owned(), etag);
 
         if opts.replication_request {
-            if let Some(actual_size) = opts
-                .user_defined
-                .get(format!("{RESERVED_METADATA_PREFIX_LOWER}Actual-Object-Size").as_str())
-            {
+            if let Some(actual_size) = get_str(&opts.user_defined, SUFFIX_ACTUAL_OBJECT_SIZE_CAP) {
+                insert_str(&mut fi.metadata, SUFFIX_ACTUAL_SIZE, actual_size.clone());
                 fi.metadata
-                    .insert(format!("{RESERVED_METADATA_PREFIX}actual-size"), actual_size.clone());
-                fi.metadata
-                    .insert("x-rustfs-encryption-original-size".to_string(), actual_size.to_string());
+                    .insert("x-rustfs-encryption-original-size".to_string(), actual_size);
             }
         } else {
-            fi.metadata
-                .insert(format!("{RESERVED_METADATA_PREFIX}actual-size"), object_actual_size.to_string());
+            insert_str(&mut fi.metadata, SUFFIX_ACTUAL_SIZE, object_actual_size.to_string());
             fi.metadata
                 .insert("x-rustfs-encryption-original-size".to_string(), object_actual_size.to_string());
         }
 
         if fi.is_compressed() {
-            fi.metadata
-                .insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression-size"), object_size.to_string());
+            insert_str(&mut fi.metadata, SUFFIX_COMPRESSION_SIZE, object_size.to_string());
         }
 
         if opts.data_movement {
@@ -2922,7 +3016,7 @@ impl MultipartOperations for SetDisks {
         .await?;
 
         if let Some(old_dir) = op_old_dir {
-            self.commit_rename_data_dir(&shuffle_disks, bucket, object, &old_dir.to_string(), write_quorum)
+            self.commit_rename_data_dir(&online_disks, bucket, object, &old_dir.to_string(), write_quorum)
                 .await?;
         }
 
@@ -3351,12 +3445,18 @@ async fn disks_with_all_parts(
         if (meta.data.is_some() || meta.size == 0) && !meta.parts.is_empty() {
             if let Some(data) = &meta.data {
                 let checksum_info = meta.erasure.get_checksum_info(meta.parts[0].number);
+                let checksum_algo =
+                    if meta.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
+                        rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
+                    } else {
+                        checksum_info.algorithm
+                    };
                 let data_len = data.len();
                 let verify_err = bitrot_verify(
                     Box::new(Cursor::new(data.clone())),
                     data_len,
                     meta.erasure.shard_file_size(meta.size) as usize,
-                    checksum_info.algorithm,
+                    checksum_algo,
                     checksum_info.hash,
                     meta.erasure.shard_size(),
                 )
@@ -4156,6 +4256,33 @@ mod tests {
         assert!(e_tag_matches("\"abc\"", "*"));
     }
 
+    #[test]
+    fn test_build_tiered_decommission_file_info_preserves_transition_metadata() {
+        let version_id = Uuid::new_v4();
+        let transition_version_id = Uuid::new_v4();
+        let original = FileInfo {
+            version_id: Some(version_id),
+            transition_status: TRANSITION_COMPLETE.to_string(),
+            transitioned_objname: "remote/object".to_string(),
+            transition_tier: "WARM-TIER".to_string(),
+            transition_version_id: Some(transition_version_id),
+            erasure: FileInfo::new("old-bucket/old-object", 8, 8).erasure,
+            ..Default::default()
+        };
+
+        let (updated, write_quorum) = build_tiered_decommission_file_info("bucket", "object", &original, 16, 4, None);
+
+        assert_eq!(updated.version_id, original.version_id);
+        assert_eq!(updated.transition_status, original.transition_status);
+        assert_eq!(updated.transitioned_objname, original.transitioned_objname);
+        assert_eq!(updated.transition_tier, original.transition_tier);
+        assert_eq!(updated.transition_version_id, original.transition_version_id);
+        assert_eq!(updated.erasure.data_blocks, 12);
+        assert_eq!(updated.erasure.parity_blocks, 4);
+        assert_eq!(write_quorum, 12);
+        assert_ne!(updated.erasure.distribution, original.erasure.distribution);
+    }
+
     #[test]
     fn test_should_prevent_write() {
         let oi = ObjectInfo {

crates/ecstore/src/set_disk/heal.rs

@@ -124,11 +124,12 @@ impl SetDisks {
                         );
 
                         let erasure = if !latest_meta.deleted && !latest_meta.is_remote() {
-                            // Initialize erasure coding
-                            erasure_coding::Erasure::new(
+                            // Initialize erasure coding; use legacy mode for old-version files
+                            erasure_coding::Erasure::new_with_options(
                                 latest_meta.erasure.data_blocks,
                                 latest_meta.erasure.parity_blocks,
                                 latest_meta.erasure.block_size,
+                                latest_meta.uses_legacy_checksum,
                             )
                         } else {
                             erasure_coding::Erasure::default()
@@ -347,7 +348,14 @@ impl SetDisks {
 
                             for (part_index, part) in latest_meta.parts.iter().enumerate() {
                                 let till_offset = erasure.shard_file_offset(0, part.size, part.size);
-                                let checksum_algo = erasure_info.get_checksum_info(part.number).algorithm;
+                                let checksum_info = erasure_info.get_checksum_info(part.number);
+                                let checksum_algo = if latest_meta.uses_legacy_checksum
+                                    && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S
+                                {
+                                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
+                                } else {
+                                    checksum_info.algorithm
+                                };
                                 let mut readers = Vec::with_capacity(latest_disks.len());
                                 let mut writers = Vec::with_capacity(out_dated_disks.len());
                                 // let mut errors = Vec::with_capacity(out_dated_disks.len());
@@ -375,6 +383,7 @@ impl SetDisks {
                                             till_offset,
                                             erasure.shard_size(),
                                             checksum_algo.clone(),
+                                            false,
                                         )
                                         .await
                                         {
@@ -419,7 +428,7 @@ impl SetDisks {
                                             ]),
                                             erasure.shard_file_size(part.size as i64),
                                             erasure.shard_size(),
-                                            HashAlgorithm::HighwayHash256,
+                                            HashAlgorithm::HighwayHash256S,
                                         )
                                         .await
                                         {
@@ -680,6 +689,7 @@ impl SetDisks {
         Ok((result, None))
     }
 
+    #[tracing::instrument(skip(self))]
     pub(super) async fn heal_object_dir(
         &self,
         bucket: &str,

crates/ecstore/src/set_disk/read.rs

@@ -568,6 +568,7 @@ impl SetDisks {
         disks: &[Option<DiskStore>],
         set_index: usize,
         pool_index: usize,
+        skip_verify_bitrot: bool,
     ) -> Result<()>
     where
         W: AsyncWrite + Send + Sync + Unpin + 'static,
@@ -602,7 +603,12 @@ impl SetDisks {
             object, offset, length, end_offset, part_index, last_part_index, last_part_relative_offset, "Multipart read bounds"
         );
 
-        let erasure = erasure_coding::Erasure::new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size);
+        let erasure = erasure_coding::Erasure::new_with_options(
+            fi.erasure.data_blocks,
+            fi.erasure.parity_blocks,
+            fi.erasure.block_size,
+            fi.uses_legacy_checksum,
+        );
 
         let part_indices: Vec<usize> = (part_index..=last_part_index).collect();
         debug!(bucket, object, ?part_indices, "Multipart part indices to stream");
@@ -647,6 +653,14 @@ impl SetDisks {
                 "Streaming multipart part"
             );
 
+            let checksum_info = fi.erasure.get_checksum_info(part_number);
+            let checksum_algo =
+                if fi.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
+                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
+                } else {
+                    checksum_info.algorithm
+                };
+
             let mut readers = Vec::with_capacity(disks.len());
             let mut errors = Vec::with_capacity(disks.len());
             for (idx, disk_op) in disks.iter().enumerate() {
@@ -658,7 +672,8 @@ impl SetDisks {
                     read_offset,
                     till_offset,
                     erasure.shard_size(),
-                    HashAlgorithm::HighwayHash256,
+                    checksum_algo.clone(),
+                    skip_verify_bitrot,
                 )
                 .await
                 {

crates/ecstore/src/sets.rs

@@ -381,7 +381,6 @@ impl BucketOperations for Sets {
 
 #[async_trait::async_trait]
 impl ObjectOperations for Sets {
-    #[tracing::instrument(skip(self))]
     async fn get_object_info(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<ObjectInfo> {
         self.get_disks_by_key(object).get_object_info(bucket, object, opts).await
     }
@@ -826,7 +825,6 @@ impl HealOperations for Sets {
 
 #[async_trait::async_trait]
 impl StorageAPI for Sets {
-    #[tracing::instrument(skip(self))]
     async fn new_ns_lock(&self, bucket: &str, object: &str) -> Result<NamespaceLockWrapper> {
         self.disk_set[0].new_ns_lock(bucket, object).await
     }