feat(admin): add persisted OIDC config APIs (#2267)

Co-authored-by: heihutu <heihutu@gmail.com>

.agents/skills/code-change-verification/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: code-change-verification
+description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
+---
+
+# Code Change Verification
+
+Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
+
+## Quick Start
+
+1. Read the scope: commit, PR, patch, or file list.
+2. Map each changed area by risk and user impact.
+3. Inspect each risky change in context.
+4. Report findings first, ordered by severity.
+5. Close with residual risks and verification recommendations.
+
+## Core Workflow
+
+### 1) Scope and assumptions
+- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
+- If context is missing, state assumptions before deeper analysis.
+- Focus only on requested scope; avoid reviewing unrelated files.
+
+### 2) Risk map
+- Prioritize in this order:
+  - Data correctness and user-visible behavior
+  - API/contract compatibility
+  - Security and authz/authn boundaries
+  - Concurrency and lifecycle correctness
+  - Performance and resource usage
+- Give higher priority to stateful paths, migration logic, defaults, and error handling.
+
+### 3) Evidence-based inspection
+- Read each modified hunk with neighboring context.
+- Trace call paths and call-site expectations.
+- Check for:
+  - invariant breaks and missing guards
+  - unchecked assumptions and null/empty/error-path handling
+  - stale tests, fixtures, and configs
+  - hidden coupling to shared helpers/constants/features
+- If a point is uncertain, mark it as an open question instead of guessing.
+
+### 4) Findings-first output
+- Order findings by severity:
+  - P0: critical failure, security breach, or data loss risk
+  - P1: high-impact regression
+  - P2: medium risk correctness gap
+  - P3: low risk/quality debt
+- For each finding include:
+  - Severity
+  - `path:line` reference
+  - concise issue statement
+  - impact and likely failure mode
+  - specific fix or mitigation
+  - validation step to confirm
+- If no issues exist, explicitly state `No findings` and why.
+
+### 5) Close
+- Report assumptions and unknowns.
+- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
+
+## Output Template
+
+1. Findings
+2. No findings (if applicable)
+3. Assumptions / Unknowns
+4. Recommended verification steps
+
+## Finding Template
+
+- `[P1] Missing timeout for downstream call`
+  - Location: `path/to/file.rs:123`
+  - Issue: ...
+  - Impact: ...
+  - Fix suggestion: ...
+  - Validation: ...
+

.agents/skills/code-change-verification/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Code Change Verification"
+  short_description: "Prioritize risks and verify code changes before merge."
+  default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

.agents/skills/pr-creation-checker/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: pr-creation-checker
+description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
+---
+
+# PR Creation Checker
+
+Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
+
+## Read sources of truth first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Use `Makefile` and `.config/make/` for local quality commands.
+- Use `.github/workflows/ci.yml` for CI expectations.
+- Do not restate long command matrices or template sections from memory when the files exist.
+
+## Workflow
+
+1. Collect PR context
+- Confirm base branch, current branch, change goal, and scope.
+- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
+- Confirm whether the branch includes only intended changes.
+
+2. Inspect change scope
+- Review the diff and summarize what changed.
+- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
+- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+
+3. Verify readiness requirements
+- Require `make pre-commit` before marking the PR ready.
+- If `make` is unavailable, use the equivalent commands from `.config/make/`.
+- Add scope-specific verification commands when the changed area needs more than the baseline.
+- If required checks fail, stop and return `BLOCKED`.
+
+4. Draft PR metadata
+- Write the PR title in English using Conventional Commits and keep it within 72 characters.
+- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
+- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
+- Keep the PR body in English.
+- Use the exact section headings from `.github/pull_request_template.md`.
+- Fill non-applicable sections with `N/A`.
+- Include verification commands in the PR description.
+- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
+- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
+
+5. Prepare reviewer context
+- Summarize why the change exists.
+- Summarize what was verified.
+- Call out risks, rollout notes, config impact, and rollback notes when applicable.
+- Mention assumptions or missing context instead of guessing.
+
+6. Prepare CLI-safe output
+- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
+- Return a ready-to-save PR body plus a short title.
+- If not ready, return blockers first and list the minimum steps needed to unblock.
+
+## Output format
+
+### Status
+- `READY` or `BLOCKED`
+
+### Title
+- `<type>(<scope>): <summary>`
+
+### PR Body
+- Reproduce the repository template headings exactly.
+- Fill every section.
+- Omit local absolute paths unless explicitly required.
+
+### Verification
+- List each command run.
+- State pass/fail.
+
+### Risks
+- List breaking changes, config changes, migration impact, or `N/A`.
+
+## Blocker rules
+
+- Return `BLOCKED` if `make pre-commit` has not passed.
+- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
+- Return `BLOCKED` if required template sections are missing.
+- Return `BLOCKED` if the title/body is not in English.
+- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
+
+## Reference
+
+- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "PR Creation Checker"
+  short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
+  default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -0,0 +1,14 @@
+# PR Readiness Checklist
+
+- Confirm the branch is based on current `main`.
+- Confirm the diff matches the stated scope.
+- Confirm no secrets, logs, temp files, or unrelated refactors are included.
+- Confirm `make pre-commit` passed, or document why it could not run.
+- Confirm extra verification commands are listed for risky changes.
+- Confirm the PR title uses Conventional Commits and stays within 72 characters.
+- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+- Confirm the PR body is in English.
+- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
+- Confirm non-applicable sections are filled with `N/A`.
+- Confirm the PR body does not include local absolute paths unless explicitly required.
+- Confirm multiline GitHub CLI commands use `--body-file`.

.agents/skills/test-coverage-improver/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: test-coverage-improver
+description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
+---
+
+# Test Coverage Improver
+
+Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
+
+## Usage assumptions
+- Focus scope is either changed lines/files, a module, or the whole repository.
+- Coverage artifact must be generated or provided in a supported format.
+- If required context is missing, call out assumptions explicitly before proposing work.
+
+## Workflow
+
+1. Define scope and baseline
+   - Confirm target language, framework, and branch.
+   - Confirm whether the scope is changed files only or full-repo.
+
+2. Produce coverage snapshot
+   - Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
+   - JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
+   - Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
+   - Collect total, per-file, and changed-line coverage.
+
+3. Rank highest-risk gaps
+   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
+   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
+   - Keep shortlist to 5–8 gaps.
+   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
+
+4. Propose high-impact tests
+   - For each shortlisted gap, output:
+     - Intent and expected behavior.
+     - Normal, edge, and failure scenarios.
+     - Assertions and side effects to verify.
+     - Setup needs (fixtures, mocks, integration dependencies).
+     - Estimated effort (`S/M/L`).
+
+5. Close with validation plan
+   - State which gaps remain after proposals.
+   - Provide concrete verification command and acceptance threshold.
+   - List assumptions or blockers (environment, fixtures, flaky dependencies).
+
+## Output template
+
+### Coverage Snapshot
+- total / branch coverage
+- changed-file coverage
+- top missing regions by size
+
+### Top Gaps (ranked)
+- `path:line-range` | risk score | why critical
+
+### Test Proposals
+- `path:line-range`
+  - Test name
+  - scenarios
+  - assertions
+  - effort
+
+### Validation Plan
+- command
+- pass criteria
+- remaining risk

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Test Coverage Improver"
+  short_description: "Find top uncovered risk areas and propose high-impact tests."
+  default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -0,0 +1,25 @@
+# Coverage Gap Prioritization Guide
+
+Use this rubric for each uncovered area.
+
+Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
+
+- Criticality:
+  - 5: authz/authn, data-loss, payment/consistency path
+  - 4: state mutation, cache invalidation, scheduling
+  - 3: error handling + fallbacks in user-visible flows
+  - 2: parsing/format conversion paths
+  - 1: logging-only or low-impact utilities
+
+- CoverageDebt:
+  - 0: 0–5 uncovered lines
+  - 1: 6–20 uncovered lines
+  - 2: 21–40 uncovered lines
+  - 3: 41+ uncovered lines
+
+- Volatility:
+  - 1: stable legacy code with few recent edits
+  - 2: changed in last 2 releases
+  - 3: touched in last 30 days or currently in active PR
+
+Sort by score descending, then by business impact.

.docker/observability/docker-compose.yml

@@ -15,10 +15,11 @@
 services:
 
   # --- Tracing ---
-
   tempo:
-    image: grafana/tempo:latest
+    image: grafana/tempo:2.10.3
     container_name: tempo
+    depends_on:
+      - redpanda
     command: [ "-config.file=/etc/tempo.yaml" ]
     volumes:
       - ./tempo.yaml:/etc/tempo.yaml:ro
@@ -37,6 +38,38 @@ services:
       timeout: 5s
       retries: 3
       start_period: 15s
+  redpanda:
+    image: redpandadata/redpanda:latest
+    ports:
+      - "9092:9092" # Kafka API for clients
+    command: >
+      redpanda start --overprovisioned
+      --mode=dev-container
+      --kafka-addr=PLAINTEXT://0.0.0.0:9092
+      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
+
+  redpanda-console:
+    image: docker.redpanda.com/redpandadata/console:latest
+    environment:
+      - CONFIG_FILEPATH=/etc/redpanda/redpanda-console-config.yaml
+    volumes:
+      - ./redpanda-console.yaml:/etc/redpanda/redpanda-console-config.yaml
+    ports:
+      - "8080:8080"
+    depends_on:
+      - redpanda
+
+  vulture:
+    image: grafana/tempo-vulture:latest
+    restart: always
+    command:
+      [
+        "-prometheus-listen-address=:8080",
+        "-tempo-query-url=http://tempo:3200",
+        "-tempo-push-url=http://tempo:4317",
+      ]
+    depends_on:
+      - tempo
 
   jaeger:
     image: jaegertracing/jaeger:latest

.docker/observability/prometheus.yml

@@ -50,6 +50,11 @@ scrape_configs:
     static_configs:
       - targets: [ 'localhost:9090' ]
 
+  - job_name: 'vulture'
+    static_configs:
+      - targets:
+          - 'vulture:8080'
+
 otlp:
   promote_resource_attributes:
     - service.instance.id

.docker/observability/redpanda-console.yaml

@@ -0,0 +1,3 @@
+kafka:
+  brokers:
+    - redpanda:9092

.docker/observability/tempo.yaml

@@ -25,32 +25,18 @@ memberlist:
   join_members:
     - tempo:7946
 
-# Distributor configuration - receives traces and writes directly to ingesters
 distributor:
-  ingester_write_path_enabled: true
-  kafka_write_path_enabled: false
   receivers:
     otlp:
       protocols:
         grpc:
-          endpoint: "tempo:4317"
+          endpoint: "0.0.0.0:4317"
         http:
-          endpoint: "tempo:4318"
-  ring:
-    kvstore:
-      store: memberlist
-
-# Ingester configuration - consumes from Kafka and stores traces
-ingester:
-  lifecycler:
-    ring:
-      kvstore:
-        store: memberlist
-      replication_factor: 1
-    tokens_file_path: /var/tempo/tokens.json
-  trace_idle_period: 10s
-  max_block_bytes: 1_000_000
-  max_block_duration: 5m
+          endpoint: "0.0.0.0:4318"
+  #log_received_spans:
+  #    enabled: true
+  #  log_discarded_spans:
+  #    enabled: true
 
 backend_scheduler:
   provider:
@@ -67,8 +53,7 @@ backend_worker:
       store: memberlist
 
 querier:
-  frontend_worker:
-    frontend_address: tempo:3200
+  query_live_store: true
 
 metrics_generator:
   registry:
@@ -90,9 +75,9 @@ storage:
   trace:
     backend: local
     wal:
-      path: /var/tempo/wal
+      path: /var/tempo/wal # where to store the wal locally
     local:
-      path: /var/tempo/blocks
+      path: /var/tempo/blocks # where to store the traces locally
 
 overrides:
   defaults:
@@ -101,24 +86,14 @@ overrides:
       generate_native_histograms: both
 
 ingest:
-  enabled: false
-  # Disabled because using direct ingester write path
-  # If you want Kafka path, enable this and set:
-  # kafka:
-  #   brokers: [redpanda:9092]
-  #   topic: tempo-ingest
-  #   encoding: protobuf
-  #   consumer_group: tempo-ingest-consumer
+  enabled: true
+  kafka:
+    address: redpanda:9092
+    topic: tempo-ingest
 
 block_builder:
   consume_cycle_duration: 30s
 
-compactor:
-  compaction:
-    block_retention: 168h  # 7 days
-  ring:
-    kvstore:
-      store: memberlist
-
 usage_report:
   reporting_enabled: false
+

.github/dependabot.yml

@@ -26,11 +26,19 @@ updates:
       day: "monday"
       timezone: "Asia/Shanghai"
       time: "08:00"
+    assignees:
+      - "heihutu"
+    reviewers:
+      - "houseme"
+      - "overtrue"
+      - "majinghe"
     ignore:
       - dependency-name: "object_store"
         versions: [ "0.13.x" ]
       - dependency-name: "libunftp"
         versions: [ "0.23.x" ]
+      - dependency-name: "ratelimit"
+        versions: [ "1.x" ]
     groups:
       s3s:
         update-types:

.github/workflows/build.yml

@@ -23,6 +23,7 @@
 #
 # Manual Parameters:
 # - build_docker: Build and push Docker images (default: true)
+# - platforms: Comma-separated platform IDs or 'all' (default: all)
 
 name: Build and Release
 
@@ -53,6 +54,11 @@ on:
         required: false
         default: true
         type: boolean
+      platforms:
+        description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
+        required: false
+        default: "all"
+        type: string
 
 permissions:
   contents: read
@@ -138,63 +144,70 @@ jobs:
           echo "  - Is prerelease: $is_prerelease"
 
   # Build RustFS binaries
+  prepare-platform-matrix:
+    name: Prepare Platform Matrix
+    runs-on: ubicloud-standard-2
+    outputs:
+      matrix: ${{ steps.select.outputs.matrix }}
+      selected: ${{ steps.select.outputs.selected }}
+    steps:
+      - name: Select target platforms
+        id: select
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
+          selected="$(echo "${selected}" | tr -d '[:space:]')"
+          if [[ -z "${selected}" ]]; then
+            selected="all"
+          fi
+
+          all='{"include":[
+            {"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
+          ]}'
+
+          if [[ "${selected}" == "all" ]]; then
+            matrix="$(jq -c . <<<"${all}")"
+          else
+            unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | ($all.include | map(.target_id)) as $known
+              | [$req[] | select(( $known | index(.) ) == null)]
+            ')"
+            if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
+              echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
+              echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
+              exit 1
+            fi
+
+            matrix="$(jq -c --arg selected "${selected}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | .include |= map(select(.target_id as $id | ($req | index($id))))
+            ' <<<"${all}")"
+          fi
+
+          echo "selected=${selected}" >> "$GITHUB_OUTPUT"
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
+          echo "Selected platforms: ${selected}"
+
   build-rustfs:
     name: Build RustFS
-    needs: [ build-check ]
-    if: needs.build-check.outputs.should_build == 'true'
+    needs: [ build-check, prepare-platform-matrix ]
+    if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
     timeout-minutes: 60
     env:
       RUSTFLAGS: ${{ matrix.rustflags }}
     strategy:
       fail-fast: false
-      matrix:
-        include:
-          # Linux builds
-          # Use x86-64-v2 (SSE4.2 baseline) instead of native to ensure distributed
-          # binaries run on older x86_64 CPUs (e.g. Intel Celeron/Atom, Synology NAS).
-          # See: https://github.com/rustfs/rustfs/issues/1838
-          - os: ubicloud-standard-2
-            target: x86_64-unknown-linux-musl
-            cross: false
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: aarch64-unknown-linux-musl
-            cross: true
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: x86_64-unknown-linux-gnu
-            cross: false
-            platform: linux
-            rustflags: ''
-          - os: ubicloud-standard-2
-            target: aarch64-unknown-linux-gnu
-            cross: true
-            platform: linux
-            rustflags: ''
-          # macOS builds
-          - os: macos-latest
-            target: aarch64-apple-darwin
-            cross: false
-            platform: macos
-            rustflags: ''
-          - os: macos-latest
-            target: x86_64-apple-darwin
-            cross: false
-            platform: macos
-            rustflags: ''
-          # Windows builds (temporarily disabled)
-          - os: windows-latest
-            target: x86_64-pc-windows-msvc
-            cross: false
-            platform: windows
-            rustflags: ''
-          #- os: windows-latest
-          #  target: aarch64-pc-windows-msvc
-          #  cross: true
-          #  platform: windows
+      matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6

.github/workflows/docker.yml

@@ -163,7 +163,14 @@ jobs:
                 if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
                   build_type="prerelease"
                   is_prerelease=true
-                  echo "🧪 Building Docker image for prerelease: $version"
+                  # TODO: Temporary change - currently allows alpha versions to also create latest tags
+                  # After the version is stable, you need to remove the following line and restore the original logic (latest is created only for stable versions)
+                  if [[ "$version" == *"alpha"* ]]; then
+                    create_latest=true
+                     echo "🧪 Building Docker image for prerelease: $version (temporarily allowing creation of latest tag)"
+                  else
+                    echo "🧪 Building Docker image for prerelease: $version"
+                  fi
                 else
                   build_type="release"
                   create_latest=true
@@ -209,7 +216,14 @@ jobs:
               v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                 build_type="prerelease"
                 is_prerelease=true
-                echo "🧪 Building with prerelease version: $input_version"
+                # TODO: Temporary change - currently allows alpha versions to also create latest tags
+                # After the version is stable, you need to remove the if block below and restore the original logic.
+                if [[ "$input_version" == *"alpha"* ]]; then
+                  create_latest=true
+                   echo "🧪 Building with prerelease version: $input_version (temporarily allowing creation of latest tag)"
+                else
+                  echo "🧪 Building with prerelease version: $input_version"
+                fi
                 ;;
               # Release versions (match after prereleases, more general)
               v[0-9]*|[0-9]*.*.*)
@@ -436,8 +450,10 @@ jobs:
             "prerelease")
               echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
               echo "⚠️  This is a prerelease image - use with caution"
-              if [[ "$CREATE_LATEST" == "true" ]]; then
-                echo "🏷️  Latest tag has been explicitly created for prerelease"
+              # TODO: Temporary change - alpha versions currently create the latest tag
+              # After the version is stable, you need to restore the following prompt information
+              if [[ "$VERSION" == *"alpha"* ]] && [[ "$CREATE_LATEST" == "true" ]]; then
+                echo "🏷️  Latest tag has been created for alpha version (temporary measures)"
               else
                 echo "🚫 Latest tag NOT created for prerelease"
               fi

.github/workflows/nix-flake-update.yml

@@ -35,7 +35,7 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
-
+        
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@v3
 
@@ -46,14 +46,18 @@ jobs:
         id: update
         uses: DeterminateSystems/update-flake-lock@main
         with:
+          git-author-name: heihutu
+          git-author-email: heihutu@gmail.com
+          git-committer-name: heihutu
+          git-committer-email: heihutu@gmail.com
           pr-title: "chore(deps): update flake.lock"
           pr-labels: |
             dependencies
             nix
             automated
           commit-msg: "chore(deps): update flake.lock"
-          pr-reviewers: houseme, heihutu
-          token: ${{ secrets.GITHUB_TOKEN }}
+          pr-reviewers: houseme, overtrue, majinghe
+          token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
 
       - name: Log PR details
         if: steps.update.outputs.pull-request-number

.gitignore

@@ -16,6 +16,7 @@ vendor
 cli/rustfs-gui/embedded-rustfs/rustfs
 *.log
 deploy/certs/*
+deploy/data/*
 *jsonl
 .env
 .rustfs.sys
@@ -46,3 +47,6 @@ docs
 result*
 *.gz
 rustfs-webdav.code-workspace
+
+.aiexclude
+*.bak

Cargo.toml

@@ -154,8 +154,8 @@ schemars = "1.2.1"
 
 # Cryptography and Security
 aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
-argon2 = { version = "0.6.0-rc.7" }
-blake3 = { version = "1.8.3", features = ["rayon", "mmap"] }
+argon2 = { version = "0.6.0-rc.8" }
+blake2 = "0.11.0-rc.5"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
 crc-fast = "1.9.0"
 hmac = { version = "0.13.0-rc.5" }
@@ -178,15 +178,15 @@ time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros
 
 # Utilities and Tools
 anyhow = "1.0.102"
-arc-swap = "1.8.2"
-astral-tokio-tar = "0.5.6"
+arc-swap = "1.9.0"
+astral-tokio-tar = "0.6.0"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.8.15" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.126.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-sdk-s3 = { version = "1.127.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.6" }
+aws-smithy-types = { version = "1.4.7" }
 backtrace = "0.3.76"
 base64 = "0.22.1"
 base64-simd = "0.8.0"
@@ -197,7 +197,10 @@ const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
 criterion = { version = "0.8", features = ["html_reports"] }
 crossbeam-queue = "0.3.12"
-datafusion = "52.3.0"
+crossbeam-channel = "0.5.15"
+crossbeam-deque = "0.8.6"
+crossbeam-utils = "0.8.21"
+datafusion = "52.4.0"
 derive_builder = "0.20.2"
 enumset = "1.1.10"
 faster-hex = "0.10.0"
@@ -219,7 +222,7 @@ matchit = "0.9.1"
 md-5 = "0.11.0-rc.5"
 md5 = "0.8.0"
 mime_guess = "2.0.5"
-moka = { version = "0.12.14", features = ["future"] }
+moka = { version = "0.12.15", features = ["future"] }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.0"
@@ -231,15 +234,16 @@ percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.0", features = ["serde"] }
-ratelimit = "0.10.0"
+ratelimit = "0.10.1"
 rayon = "1.11.0"
-reed-solomon-simd = { version = "3.1.0" }
+reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
+reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
 rumqttc = { version = "0.25.1" }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.1" }
-s3s = { git = "https://github.com/s3s-project/s3s", rev = "c2dc7b16535659904d4efff52c558fc039be1ef3", features = ["minio"] }
+s3s = { git = "https://github.com/rustfs/s3s", rev = "d9556e3c0036bd3f2b330966009cbaa5aebf19a3", features = ["minio"] }
 serial_test = "3.4.0"
 shadow-rs = { version = "1.7.1", default-features = false }
 siphasher = "1.0.2"
@@ -263,19 +267,19 @@ transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
 uuid = { version = "1.22.0", features = ["v4", "fast-rng", "macro-diagnostics"] }
-vaultrs = { version = "0.7.4" }
+vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
 windows = { version = "0.62.2" }
 xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
-zip = "8.2.0"
+zip = "8.4.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
 metrics = "0.24.3"
 opentelemetry = { version = "0.31.0" }
 opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
-opentelemetry-otlp = { version = "0.31.0", features = ["gzip-http", "reqwest-rustls"] }
+opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rustls"] }
 opentelemetry_sdk = { version = "0.31.0" }
 opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.31.0" }

Dockerfile

@@ -72,7 +72,8 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-RUN apk add --no-cache ca-certificates coreutils curl
+RUN apk update && \
+    apk add --no-cache ca-certificates coreutils curl "zlib>=1.3.2-r0"
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs
@@ -86,12 +87,7 @@ RUN addgroup -g 10001 -S rustfs && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_CONSOLE_ADDRESS=":9001" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_CORS_ALLOWED_ORIGINS="*" \
+ENV RUSTFS_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \

Dockerfile.decommission-local

@@ -0,0 +1,65 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM rust:1.91-trixie
+
+RUN set -eux; \
+    export DEBIAN_FRONTEND=noninteractive; \
+    apt-get update; \
+    apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    curl \
+    git \
+    pkg-config \
+    libssl-dev \
+    lld \
+    protobuf-compiler \
+    flatbuffers-compiler; \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /usr/src/rustfs
+
+COPY . .
+
+RUN ./scripts/static.sh
+RUN cargo run --bin gproto
+RUN cargo build --release --locked --bin rustfs
+
+RUN set -eux; \
+    groupadd -g 10001 rustfs; \
+    useradd -u 10001 -g rustfs -M -s /usr/sbin/nologin rustfs
+
+WORKDIR /app
+
+RUN set -eux; \
+    mkdir -p /data /logs; \
+    chown -R rustfs:rustfs /data /logs /app; \
+    chmod 0750 /data /logs
+
+COPY entrypoint.sh /entrypoint.sh
+RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && chmod +x /entrypoint.sh
+
+ENV RUSTFS_VOLUMES="/data" \
+    RUST_LOG="warn" \
+    RUSTFS_OBS_LOG_DIRECTORY="/logs" \
+    RUSTFS_USERNAME="rustfs" \
+    RUSTFS_GROUPNAME="rustfs" \
+    RUSTFS_UID="10001" \
+    RUSTFS_GID="10001"
+
+EXPOSE 9000 9001
+
+ENTRYPOINT ["/entrypoint.sh"]
+CMD ["/usr/bin/rustfs"]

Dockerfile.source

@@ -157,11 +157,7 @@ WORKDIR /app
 ENV CARGO_INCREMENTAL=1
 
 # Ensure we have the same default env vars available
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_VOLUMES="/data" \
+ENV RUSTFS_VOLUMES="/data" \
     RUST_LOG="warn" \
     RUSTFS_OBS_LOG_DIRECTORY="/logs" \
     RUSTFS_USERNAME="rustfs" \
@@ -222,11 +218,7 @@ COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
 # Default environment (override in docker run/compose as needed)
-ENV RUSTFS_ADDRESS=":9000" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
-    RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_VOLUMES="/data" \
+ENV RUSTFS_VOLUMES="/data" \
     RUST_LOG="warn" \
     RUSTFS_USERNAME="rustfs" \
     RUSTFS_GROUPNAME="rustfs" \

crates/appauth/Cargo.toml

@@ -33,3 +33,6 @@ rand.workspace = true
 
 [lints]
 workspace = true
+
+[lib]
+doctest = false

crates/audit/Cargo.toml

@@ -47,3 +47,7 @@ rumqttc = { workspace = true }
 
 [lints]
 workspace = true
+
+[lib]
+test = false
+doctest = false

crates/audit/src/registry.rs

@@ -313,6 +313,12 @@ impl AuditRegistry {
                 }
             }
 
+            if &new_config == config {
+                info!("Audit target configuration unchanged, skip persisting server config");
+                info!(count = successful_targets.len(), "All target processing completed");
+                return Ok(successful_targets);
+            }
+
             let Some(store) = rustfs_ecstore::global::new_object_layer_fn() else {
                 return Err(AuditError::StorageNotAvailable(
                     "Failed to save target configuration: server storage not initialized".to_string(),

crates/audit/tests/performance_test.rs

@@ -159,7 +159,8 @@ async fn test_audit_log_dispatch_performance() {
 
     let start = Instant::now();
 
-    // Dispatch audit log (should be fast since no targets are configured)
+    // Dispatch audit log against an unstarted system state. Empty config keeps
+    // the audit system stopped, so dispatch should fail fast without targets.
     let result = system.dispatch(Arc::new(audit_entry)).await;
     let elapsed = start.elapsed();
 
@@ -168,8 +169,10 @@ async fn test_audit_log_dispatch_performance() {
     // Should be very fast (sub-millisecond for no targets)
     assert!(elapsed < Duration::from_millis(100), "Dispatch took too long: {elapsed:?}");
 
-    // Should succeed even with no targets
-    assert!(result.is_ok(), "Dispatch should succeed with no targets");
+    assert!(
+        matches!(result, Err(AuditError::NotInitialized(_))),
+        "Dispatch on a stopped system should return NotInitialized, got: {result:?}"
+    );
 
     // Clean up
     let _ = system.close().await;
@@ -186,11 +189,11 @@ async fn test_system_state_transitions() {
     let config = rustfs_ecstore::config::Config(std::collections::HashMap::new());
     let start_result = system.start(config).await;
 
-    // Should be running (or failed due to server storage)
+    // Empty config keeps the audit system stopped even when start() succeeds.
     let state = system.get_state().await;
     match start_result {
         Ok(_) => {
-            assert_eq!(state, rustfs_audit::system::AuditSystemState::Running);
+            assert_eq!(state, rustfs_audit::system::AuditSystemState::Stopped);
         }
         Err(_) => {
             // Expected in test environment due to server storage not being initialized

crates/audit/tests/system_integration_test.rs

@@ -29,27 +29,21 @@ async fn test_complete_audit_system_lifecycle() {
     assert_eq!(system.get_state().await, system::AuditSystemState::Stopped);
     assert!(!system.is_running().await);
 
-    // 2. Start with empty config (will fail due to no server storage in test)
+    // 2. Start with empty config. The current implementation returns Ok(())
+    // but keeps the system stopped when no audit targets are enabled.
     let config = Config(HashMap::new());
     let start_result = system.start(config).await;
 
-    // Should fail in test environment but state handling should work
+    // State handling should remain consistent for both empty-config success and
+    // storage-unavailable failure paths.
     match start_result {
         Err(AuditError::StorageNotAvailable(_)) => {
             // Expected in test environment
             assert_eq!(system.get_state().await, system::AuditSystemState::Stopped);
         }
         Ok(_) => {
-            // If it somehow succeeds, verify running state
-            assert_eq!(system.get_state().await, system::AuditSystemState::Running);
-            assert!(system.is_running().await);
-
-            // Test pause/resume
-            system.pause().await.expect("Should pause successfully");
-            assert_eq!(system.get_state().await, system::AuditSystemState::Paused);
-
-            system.resume().await.expect("Should resume successfully");
-            assert_eq!(system.get_state().await, system::AuditSystemState::Running);
+            assert_eq!(system.get_state().await, system::AuditSystemState::Stopped);
+            assert!(!system.is_running().await);
         }
         Err(e) => {
             panic!("Unexpected error: {e}");

crates/checksums/Cargo.toml

@@ -36,3 +36,6 @@ sha2 = { workspace = true }
 
 [dev-dependencies]
 pretty_assertions = { workspace = true }
+
+[lib]
+doctest = false

crates/common/Cargo.toml

@@ -41,3 +41,6 @@ rmp-serde = { workspace = true }
 async-trait = { workspace = true }
 s3s = { workspace = true }
 tracing = { workspace = true }
+
+[lib]
+doctest = false

crates/common/src/internode_metrics.rs

@@ -0,0 +1,170 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use metrics::{counter, gauge};
+use std::sync::{
+    Arc, LazyLock,
+    atomic::{AtomicU64, Ordering},
+};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct InternodeMetricsSnapshot {
+    pub sent_bytes_total: u64,
+    pub recv_bytes_total: u64,
+    pub outgoing_requests_total: u64,
+    pub incoming_requests_total: u64,
+    pub errors_total: u64,
+    pub dial_errors_total: u64,
+    pub dial_avg_time_nanos: u64,
+    pub last_dial_unix_millis: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct InternodeMetrics {
+    sent_bytes_total: AtomicU64,
+    recv_bytes_total: AtomicU64,
+    outgoing_requests_total: AtomicU64,
+    incoming_requests_total: AtomicU64,
+    errors_total: AtomicU64,
+    dial_errors_total: AtomicU64,
+    dial_total_time_nanos: AtomicU64,
+    dial_samples_total: AtomicU64,
+    last_dial_unix_millis: AtomicU64,
+}
+
+impl InternodeMetrics {
+    pub fn record_sent_bytes(&self, bytes: usize) {
+        let bytes = bytes as u64;
+        if bytes == 0 {
+            return;
+        }
+        self.sent_bytes_total.fetch_add(bytes, Ordering::Relaxed);
+        counter!("rustfs.internode.sent.bytes.total").increment(bytes);
+    }
+
+    pub fn record_recv_bytes(&self, bytes: usize) {
+        let bytes = bytes as u64;
+        if bytes == 0 {
+            return;
+        }
+        self.recv_bytes_total.fetch_add(bytes, Ordering::Relaxed);
+        counter!("rustfs.internode.recv.bytes.total").increment(bytes);
+    }
+
+    pub fn record_outgoing_request(&self) {
+        self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
+        counter!("rustfs.internode.requests.outgoing.total").increment(1);
+    }
+
+    pub fn record_incoming_request(&self) {
+        self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
+        counter!("rustfs.internode.requests.incoming.total").increment(1);
+    }
+
+    pub fn record_error(&self) {
+        self.errors_total.fetch_add(1, Ordering::Relaxed);
+        counter!("rustfs.internode.errors.total").increment(1);
+    }
+
+    pub fn record_dial_result(&self, duration: Duration, success: bool) {
+        let elapsed_nanos = duration.as_nanos().min(u128::from(u64::MAX)) as u64;
+        self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
+        let samples = self.dial_samples_total.fetch_add(1, Ordering::Relaxed) + 1;
+        let total = self.dial_total_time_nanos.load(Ordering::Relaxed);
+        gauge!("rustfs.internode.dial.avg_time.nanos").set(total as f64 / samples as f64);
+
+        if !success {
+            self.dial_errors_total.fetch_add(1, Ordering::Relaxed);
+            counter!("rustfs.internode.dial.errors.total").increment(1);
+        }
+
+        let now_ms = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_millis()
+            .min(u128::from(u64::MAX)) as u64;
+        self.last_dial_unix_millis.store(now_ms, Ordering::Relaxed);
+    }
+
+    pub fn snapshot(&self) -> InternodeMetricsSnapshot {
+        let dial_samples_total = self.dial_samples_total.load(Ordering::Relaxed);
+        let dial_total_time_nanos = self.dial_total_time_nanos.load(Ordering::Relaxed);
+        let dial_avg_time_nanos = if dial_samples_total == 0 {
+            0
+        } else {
+            dial_total_time_nanos / dial_samples_total
+        };
+
+        InternodeMetricsSnapshot {
+            sent_bytes_total: self.sent_bytes_total.load(Ordering::Relaxed),
+            recv_bytes_total: self.recv_bytes_total.load(Ordering::Relaxed),
+            outgoing_requests_total: self.outgoing_requests_total.load(Ordering::Relaxed),
+            incoming_requests_total: self.incoming_requests_total.load(Ordering::Relaxed),
+            errors_total: self.errors_total.load(Ordering::Relaxed),
+            dial_errors_total: self.dial_errors_total.load(Ordering::Relaxed),
+            dial_avg_time_nanos,
+            last_dial_unix_millis: self.last_dial_unix_millis.load(Ordering::Relaxed),
+        }
+    }
+
+    #[doc(hidden)]
+    pub fn reset_for_test(&self) {
+        self.sent_bytes_total.store(0, Ordering::Relaxed);
+        self.recv_bytes_total.store(0, Ordering::Relaxed);
+        self.outgoing_requests_total.store(0, Ordering::Relaxed);
+        self.incoming_requests_total.store(0, Ordering::Relaxed);
+        self.errors_total.store(0, Ordering::Relaxed);
+        self.dial_errors_total.store(0, Ordering::Relaxed);
+        self.dial_total_time_nanos.store(0, Ordering::Relaxed);
+        self.dial_samples_total.store(0, Ordering::Relaxed);
+        self.last_dial_unix_millis.store(0, Ordering::Relaxed);
+    }
+}
+
+pub fn global_internode_metrics() -> &'static Arc<InternodeMetrics> {
+    static GLOBAL_INTERNODE_METRICS: LazyLock<Arc<InternodeMetrics>> = LazyLock::new(|| Arc::new(InternodeMetrics::default()));
+    &GLOBAL_INTERNODE_METRICS
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn snapshot_reports_recorded_values() {
+        let metrics = global_internode_metrics();
+        metrics.reset_for_test();
+
+        metrics.record_sent_bytes(64);
+        metrics.record_recv_bytes(32);
+        metrics.record_outgoing_request();
+        metrics.record_incoming_request();
+        metrics.record_error();
+        metrics.record_dial_result(Duration::from_millis(9), true);
+        metrics.record_dial_result(Duration::from_millis(3), false);
+
+        let snapshot = metrics.snapshot();
+        assert_eq!(snapshot.sent_bytes_total, 64);
+        assert_eq!(snapshot.recv_bytes_total, 32);
+        assert_eq!(snapshot.outgoing_requests_total, 1);
+        assert_eq!(snapshot.incoming_requests_total, 1);
+        assert_eq!(snapshot.errors_total, 1);
+        assert_eq!(snapshot.dial_errors_total, 1);
+        assert_eq!(snapshot.dial_avg_time_nanos, 6_000_000);
+        assert!(snapshot.last_dial_unix_millis > 0);
+
+        metrics.reset_for_test();
+    }
+}

crates/common/src/lib.rs

@@ -17,6 +17,7 @@ pub mod bucket_stats;
 pub mod data_usage;
 pub mod globals;
 pub mod heal_channel;
+pub mod internode_metrics;
 pub mod last_minute;
 pub mod metrics;
 mod readiness;

crates/common/src/readiness.rs

@@ -60,6 +60,23 @@ impl GlobalReadiness {
     pub fn is_ready(&self) -> bool {
         self.status.load(Ordering::SeqCst) == SystemStage::FullReady as u8
     }
+
+    /// Get the current system stage
+    /// # Returns
+    /// The current SystemStage of the service
+    pub fn current_stage(&self) -> SystemStage {
+        match self.status.load(Ordering::SeqCst) {
+            0 => SystemStage::Booting,
+            1 => SystemStage::StorageReady,
+            2 => SystemStage::IamReady,
+            3 => SystemStage::FullReady,
+            invalid => {
+                debug_assert!(false, "GlobalReadiness::current_stage: invalid status value {}", invalid);
+                // Fallback to the most conservative stage on invalid values
+                SystemStage::Booting
+            }
+        }
+    }
 }
 
 #[cfg(test)]

crates/config/Cargo.toml

@@ -37,3 +37,6 @@ constants = ["dep:const-str"]
 notify = ["dep:const-str", "constants"]
 observability = ["constants"]
 opa = ["constants"]
+
+[lib]
+doctest = false

crates/config/src/constants/app.rs

@@ -131,9 +131,61 @@ pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
 /// Environment variable for server volumes.
 pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
 
+/// Environment variable for server access key.
+pub const ENV_RUSTFS_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";
+
+/// Environment variable for server access key file.
+pub const ENV_RUSTFS_ACCESS_KEY_FILE: &str = "RUSTFS_ACCESS_KEY_FILE";
+
+/// Environment variable for server root user.
+pub const ENV_RUSTFS_ROOT_USER: &str = "RUSTFS_ROOT_USER";
+
+/// Environment variable for server secret key.
+pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
+
+/// Environment variable for server secret key file.
+pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
+
+/// Environment variable for server root password.
+pub const ENV_RUSTFS_ROOT_PASSWORD: &str = "RUSTFS_ROOT_PASSWORD";
+
+/// Environment variable for server OBS endpoint.
+pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
+
+/// Environment variable for console server enable.
+pub const ENV_RUSTFS_CONSOLE_ENABLE: &str = "RUSTFS_CONSOLE_ENABLE";
+
+/// Environment variable for console server address.
+pub const ENV_RUSTFS_CONSOLE_ADDRESS: &str = "RUSTFS_CONSOLE_ADDRESS";
+
 /// Environment variable for server tls path.
 pub const ENV_RUSTFS_TLS_PATH: &str = "RUSTFS_TLS_PATH";
 
+/// Environment variable for server KMS enable.
+pub const ENV_RUSTFS_KMS_ENABLE: &str = "RUSTFS_KMS_ENABLE";
+
+/// Default KMS enable for server-side encryption
+/// This is the default value for enabling KMS encryption for server-side encryption.
+/// Default value: false
+pub const DEFAULT_KMS_ENABLE: bool = false;
+
+/// Environment variable for server KMS backend.
+pub const ENV_RUSTFS_KMS_BACKEND: &str = "RUSTFS_KMS_BACKEND";
+
+/// Default KMS backend for server-side encryption
+/// This is the default KMS backend for server-side encryption.
+/// Default value: local
+pub const DEFAULT_KMS_BACKEND: &str = "local";
+
+/// Environment variable for selecting the buffer profile used for adaptive buffer sizing.
+pub const ENV_RUSTFS_BUFFER_PROFILE: &str = "RUSTFS_BUFFER_PROFILE";
+
+/// Default buffer profile for adaptive buffer sizing
+/// This is the default buffer profile for adaptive buffer sizing.
+/// It is used to identify the workload profile for adaptive buffer sizing.
+/// Default value: GeneralPurpose
+pub const DEFAULT_BUFFER_PROFILE: &str = "GeneralPurpose";
+
 /// Default value for the server TLS path if `ENV_RUSTFS_TLS_PATH` is not set.
 pub const DEFAULT_RUSTFS_TLS_PATH: &str = "";
 

crates/config/src/constants/mod.rs

@@ -28,3 +28,4 @@ pub(crate) mod runtime;
 pub(crate) mod scanner;
 pub(crate) mod targets;
 pub(crate) mod tls;
+pub(crate) mod workload;

crates/config/src/constants/object.rs

@@ -180,3 +180,265 @@ pub const ENV_OBJECT_GET_SKIP_BITROT_VERIFY: &str = "RUSTFS_OBJECT_GET_SKIP_BITR
 
 /// Default: bitrot verification is enabled on GetObject reads (do not skip).
 pub const DEFAULT_OBJECT_GET_SKIP_BITROT_VERIFY: bool = false;
+
+// =============================================================================
+// Concurrent Request Fix - Timeout and Backpressure Configuration
+// =============================================================================
+
+/// Environment variable for GetObject request timeout in seconds.
+///
+/// When a GetObject request exceeds this duration, it will be cancelled
+/// and return a 504 Gateway Timeout error. This prevents requests from
+/// hanging indefinitely due to deadlocks or resource exhaustion.
+///
+/// Default: 30 seconds (can be overridden by `RUSTFS_OBJECT_GET_TIMEOUT`).
+/// Set to 0 to disable timeout (not recommended for production).
+pub const ENV_OBJECT_GET_TIMEOUT: &str = "RUSTFS_OBJECT_GET_TIMEOUT";
+
+/// Default GetObject request timeout in seconds.
+///
+/// This value balances between allowing large object transfers to complete
+/// and preventing indefinite hangs. For 20-26MB objects with concurrent
+/// range reads, 30 seconds should be sufficient under normal conditions.
+pub const DEFAULT_OBJECT_GET_TIMEOUT: u64 = 30;
+
+/// Environment variable for disk read operation timeout in seconds.
+///
+/// Individual disk read operations that exceed this duration will be
+/// cancelled and treated as failures. This helps detect slow or hung
+/// disks without waiting indefinitely.
+///
+/// Default: 10 seconds (can be overridden by `RUSTFS_OBJECT_DISK_READ_TIMEOUT`).
+pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT";
+
+/// Default disk read timeout in seconds.
+pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;
+
+/// Environment variable for duplex pipe buffer size in bytes.
+///
+/// The duplex pipe connects the disk read task to the HTTP response stream.
+/// A larger buffer reduces backpressure but increases memory usage.
+/// For large objects (20-26MB), a 4MB buffer provides good throughput.
+///
+/// Default: 4194304 (4 MB, can be overridden by `RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE`).
+/// Minimum recommended: 1048576 (1 MB).
+pub const ENV_OBJECT_DUPLEX_BUFFER_SIZE: &str = "RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE";
+
+/// Default duplex buffer size: 4 MB.
+///
+/// This is 4x larger than the original 1 MB buffer, providing better
+/// handling of large objects and reducing backpressure-related hangs.
+pub const DEFAULT_OBJECT_DUPLEX_BUFFER_SIZE: usize = 4 * 1024 * 1024;
+
+/// Environment variable for I/O buffer size in bytes.
+///
+/// This controls the buffer size used for individual I/O operations.
+/// A larger buffer improves throughput for sequential reads but may
+/// increase latency for small random reads.
+///
+/// Default: 131072 (128 KB, can be overridden by `RUSTFS_OBJECT_IO_BUFFER_SIZE`).
+pub const ENV_OBJECT_IO_BUFFER_SIZE: &str = "RUSTFS_OBJECT_IO_BUFFER_SIZE";
+
+/// Default I/O buffer size: 128 KB.
+pub const DEFAULT_OBJECT_IO_BUFFER_SIZE: usize = 128 * 1024;
+
+/// Environment variable to enable/disable lock optimization.
+///
+/// When enabled, read locks are released immediately after metadata
+/// is read, rather than being held for the entire data transfer.
+/// This significantly reduces lock contention under high concurrency.
+///
+/// Default: true (enabled, can be overridden by `RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE`).
+pub const ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE: &str = "RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE";
+
+/// Default: lock optimization is enabled.
+pub const DEFAULT_OBJECT_LOCK_OPTIMIZATION_ENABLE: bool = true;
+
+/// Environment variable to enable/disable priority-based I/O scheduling.
+///
+/// When enabled, smaller requests (< 1MB) are given higher priority
+/// than larger requests (> 10MB), preventing "starvation" of small
+/// requests by large ones.
+///
+/// Default: true (enabled, can be overridden by `RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE`).
+pub const ENV_OBJECT_PRIORITY_SCHEDULING_ENABLE: &str = "RUSTFS_OBJECT_PRIORITY_SCHEDULING_ENABLE";
+
+/// Default: priority scheduling is enabled.
+pub const DEFAULT_OBJECT_PRIORITY_SCHEDULING_ENABLE: bool = true;
+
+/// Environment variable to enable/disable deadlock detection.
+///
+/// When enabled, the system monitors active requests and detects
+/// potential deadlock situations (circular lock wait chains).
+/// This has some performance overhead and is intended for debugging.
+///
+/// Default: false (disabled, can be overridden by `RUSTFS_OBJECT_DEADLOCK_DETECTION_ENABLE`).
+pub const ENV_OBJECT_DEADLOCK_DETECTION_ENABLE: &str = "RUSTFS_OBJECT_DEADLOCK_DETECTION_ENABLE";
+
+/// Default: deadlock detection is disabled for performance.
+pub const DEFAULT_OBJECT_DEADLOCK_DETECTION_ENABLE: bool = false;
+
+/// Environment variable for deadlock detection check interval in seconds.
+///
+/// How often the deadlock detector analyzes the lock wait graph.
+/// More frequent checks detect deadlocks faster but use more CPU.
+///
+/// Default: 5 seconds (can be overridden by `RUSTFS_OBJECT_DEADLOCK_CHECK_INTERVAL`).
+pub const ENV_OBJECT_DEADLOCK_CHECK_INTERVAL: &str = "RUSTFS_OBJECT_DEADLOCK_CHECK_INTERVAL";
+
+/// Default deadlock check interval: 5 seconds.
+pub const DEFAULT_OBJECT_DEADLOCK_CHECK_INTERVAL: u64 = 5;
+
+/// Environment variable for deadlock detection hang threshold in seconds.
+///
+/// Requests that have been running longer than this threshold are
+/// considered "potentially hung" and included in deadlock analysis.
+///
+/// Default: 10 seconds (can be overridden by `RUSTFS_OBJECT_DEADLOCK_HANG_THRESHOLD`).
+pub const ENV_OBJECT_DEADLOCK_HANG_THRESHOLD: &str = "RUSTFS_OBJECT_DEADLOCK_HANG_THRESHOLD";
+
+/// Default hang threshold: 10 seconds.
+pub const DEFAULT_OBJECT_DEADLOCK_HANG_THRESHOLD: u64 = 10;
+
+/// Environment variable for backpressure high watermark percentage.
+///
+/// When buffer usage exceeds this percentage, the system enters
+/// "high watermark" state and may apply backpressure to producers.
+///
+/// Default: 80 (80%, can be overridden by `RUSTFS_OBJECT_BACKPRESSURE_HIGH_WATERMARK`).
+pub const ENV_OBJECT_BACKPRESSURE_HIGH_WATERMARK: &str = "RUSTFS_OBJECT_BACKPRESSURE_HIGH_WATERMARK";
+
+/// Default high watermark: 80%.
+pub const DEFAULT_OBJECT_BACKPRESSURE_HIGH_WATERMARK: u32 = 80;
+
+/// Environment variable for backpressure low watermark percentage.
+///
+/// When buffer usage drops below this percentage after being in
+/// high watermark state, backpressure is released.
+///
+/// Default: 50 (50%, can be overridden by `RUSTFS_OBJECT_BACKPRESSURE_LOW_WATERMARK`).
+pub const ENV_OBJECT_BACKPRESSURE_LOW_WATERMARK: &str = "RUSTFS_OBJECT_BACKPRESSURE_LOW_WATERMARK";
+
+/// Default low watermark: 50%.
+pub const DEFAULT_OBJECT_BACKPRESSURE_LOW_WATERMARK: u32 = 50;
+
+/// Environment variable for lock acquisition timeout in seconds.
+///
+/// When a lock cannot be acquired within this duration, the operation
+/// will fail with a timeout error. This prevents indefinite waiting
+/// for locks that may never be released due to deadlocks.
+///
+/// Default: 5 seconds (can be overridden by `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT`).
+pub const ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT: &str = "RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT";
+
+/// Default lock acquisition timeout: 5 seconds.
+pub const DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT: u64 = 5;
+
+// ============================================================================
+// I/O priority scheduling configuration
+// ============================================================================
+
+/// Environment variable for I/O high priority size threshold in bytes.
+///
+/// Requests smaller than this threshold are classified as high priority.
+/// High priority requests are processed first to prevent starvation of small requests.
+///
+/// Default: 1048576 (1 MB, can be overridden by `RUSTFS_OBJECT_IO_HIGH_PRIORITY_SIZE_THRESHOLD`).
+pub const ENV_OBJECT_IO_HIGH_PRIORITY_SIZE_THRESHOLD: &str = "RUSTFS_OBJECT_IO_HIGH_PRIORITY_SIZE_THRESHOLD";
+
+/// Default high priority size threshold: 1 MB.
+pub const DEFAULT_OBJECT_IO_HIGH_PRIORITY_SIZE_THRESHOLD: usize = 1024 * 1024;
+
+/// Environment variable for I/O low priority size threshold in bytes.
+///
+/// Requests larger than this threshold are classified as low priority.
+/// Low priority requests are processed last to avoid blocking small requests.
+///
+/// Default: 104857600 (100 MB, can be overridden by `RUSTFS_OBJECT_IO_LOW_PRIORITY_SIZE_THRESHOLD`).
+pub const ENV_OBJECT_IO_LOW_PRIORITY_SIZE_THRESHOLD: &str = "RUSTFS_OBJECT_IO_LOW_PRIORITY_SIZE_THRESHOLD";
+
+/// Default low priority size threshold: 100 MB.
+pub const DEFAULT_OBJECT_IO_LOW_PRIORITY_SIZE_THRESHOLD: usize = 100 * 1024 * 1024;
+
+/// Environment variable for high priority queue capacity.
+///
+/// Maximum number of requests that can be queued in the high priority queue.
+///
+/// Default: 32 (can be overridden by `RUSTFS_OBJECT_IO_QUEUE_HIGH_CAPACITY`).
+pub const ENV_OBJECT_IO_QUEUE_HIGH_CAPACITY: &str = "RUSTFS_OBJECT_IO_QUEUE_HIGH_CAPACITY";
+
+/// Default high priority queue capacity: 32.
+pub const DEFAULT_OBJECT_IO_QUEUE_HIGH_CAPACITY: usize = 32;
+
+/// Environment variable for normal priority queue capacity.
+///
+/// Maximum number of requests that can be queued in the normal priority queue.
+///
+/// Default: 64 (can be overridden by `RUSTFS_OBJECT_IO_QUEUE_NORMAL_CAPACITY`).
+pub const ENV_OBJECT_IO_QUEUE_NORMAL_CAPACITY: &str = "RUSTFS_OBJECT_IO_QUEUE_NORMAL_CAPACITY";
+
+/// Default normal priority queue capacity: 64.
+pub const DEFAULT_OBJECT_IO_QUEUE_NORMAL_CAPACITY: usize = 64;
+
+/// Environment variable for low priority queue capacity.
+///
+/// Maximum number of requests that can be queued in the low priority queue.
+///
+/// Default: 16 (can be overridden by `RUSTFS_OBJECT_IO_QUEUE_LOW_CAPACITY`).
+pub const ENV_OBJECT_IO_QUEUE_LOW_CAPACITY: &str = "RUSTFS_OBJECT_IO_QUEUE_LOW_CAPACITY";
+
+/// Default low priority queue capacity: 16.
+pub const DEFAULT_OBJECT_IO_QUEUE_LOW_CAPACITY: usize = 16;
+
+/// Environment variable for starvation prevention check interval in milliseconds.
+///
+/// How often the system checks for starving low-priority requests.
+/// When a low-priority request has been waiting longer than the starvation threshold,
+/// it is promoted to normal priority.
+///
+/// Default: 100 ms (can be overridden by `RUSTFS_OBJECT_IO_STARVATION_PREVENTION_INTERVAL`).
+pub const ENV_OBJECT_IO_STARVATION_PREVENTION_INTERVAL: &str = "RUSTFS_OBJECT_IO_STARVATION_PREVENTION_INTERVAL";
+
+/// Default starvation prevention interval: 100 ms.
+pub const DEFAULT_OBJECT_IO_STARVATION_PREVENTION_INTERVAL: u64 = 100;
+
+/// Environment variable for starvation threshold in seconds.
+///
+/// Maximum time a low-priority request can wait before being promoted to normal priority.
+/// This prevents indefinite starvation of low-priority requests.
+///
+/// Default: 5 seconds (can be overridden by `RUSTFS_OBJECT_IO_STARVATION_THRESHOLD_SECS`).
+pub const ENV_OBJECT_IO_STARVATION_THRESHOLD_SECS: &str = "RUSTFS_OBJECT_IO_STARVATION_THRESHOLD_SECS";
+
+/// Default starvation threshold: 5 seconds.
+pub const DEFAULT_OBJECT_IO_STARVATION_THRESHOLD_SECS: u64 = 5;
+
+/// Environment variable for load sampling window size.
+///
+/// Number of recent samples used to calculate I/O load metrics.
+///
+/// Default: 100 samples (can be overridden by `RUSTFS_OBJECT_IO_LOAD_SAMPLE_WINDOW`).
+pub const ENV_OBJECT_IO_LOAD_SAMPLE_WINDOW: &str = "RUSTFS_OBJECT_IO_LOAD_SAMPLE_WINDOW";
+
+/// Default load sampling window: 100 samples.
+pub const DEFAULT_OBJECT_IO_LOAD_SAMPLE_WINDOW: usize = 100;
+
+/// Environment variable for high load wait time threshold in milliseconds.
+///
+/// When average wait time exceeds this threshold, the system is considered to be under high load.
+///
+/// Default: 50 ms (can be overridden by `RUSTFS_OBJECT_IO_LOAD_HIGH_THRESHOLD_MS`).
+pub const ENV_OBJECT_IO_LOAD_HIGH_THRESHOLD_MS: &str = "RUSTFS_OBJECT_IO_LOAD_HIGH_THRESHOLD_MS";
+
+/// Default high load threshold: 50 ms.
+pub const DEFAULT_OBJECT_IO_LOAD_HIGH_THRESHOLD_MS: u64 = 50;
+
+/// Environment variable for low load wait time threshold in milliseconds.
+///
+/// When average wait time is below this threshold, the system is considered to be under low load.
+///
+/// Default: 10 ms (can be overridden by `RUSTFS_OBJECT_IO_LOAD_LOW_THRESHOLD_MS`).
+pub const ENV_OBJECT_IO_LOAD_LOW_THRESHOLD_MS: &str = "RUSTFS_OBJECT_IO_LOAD_LOW_THRESHOLD_MS";
+
+/// Default low load threshold: 10 ms.
+pub const DEFAULT_OBJECT_IO_LOAD_LOW_THRESHOLD_MS: u64 = 10;

crates/config/src/constants/workload.rs

@@ -0,0 +1,70 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Workload profile buffer configuration constants.
+//!
+//! This module defines environment variable keys and default values for
+//! custom buffer profile configuration.
+
+use crate::{KI_B, MI_B};
+
+/// Environment variable for minimum buffer size
+/// Default: 64KB (65536 bytes)
+pub const ENV_RUSTFS_BUFFER_MIN_SIZE: &str = "RUSTFS_BUFFER_MIN_SIZE";
+
+/// Environment variable for maximum buffer size
+/// Default: 1MB (1048576 bytes)
+pub const ENV_RUSTFS_BUFFER_MAX_SIZE: &str = "RUSTFS_BUFFER_MAX_SIZE";
+
+/// Environment variable for default buffer size (used when file size is unknown)
+/// Default: 256KB (262144 bytes)
+pub const ENV_RUSTFS_BUFFER_DEFAULT_SIZE: &str = "RUSTFS_BUFFER_DEFAULT_SIZE";
+
+/// Default minimum buffer size: 64KB
+pub const DEFAULT_BUFFER_MIN_SIZE: usize = 64 * KI_B;
+
+/// Default maximum buffer size: 1MB
+pub const DEFAULT_BUFFER_MAX_SIZE: usize = MI_B;
+
+/// Default buffer size for unknown file size: 256KB
+pub const DEFAULT_BUFFER_UNKNOWN_SIZE: usize = 256 * KI_B;
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::MI_B;
+
+    #[test]
+    fn test_default_values() {
+        assert_eq!(DEFAULT_BUFFER_MIN_SIZE, 65536); // 64KB
+        assert_eq!(DEFAULT_BUFFER_MAX_SIZE, 1048576); // 1MB
+        assert_eq!(DEFAULT_BUFFER_UNKNOWN_SIZE, 262144); // 256KB
+    }
+
+    #[test]
+    fn test_constants() {
+        assert_eq!(KI_B, 1024);
+        assert_eq!(MI_B, 1024 * 1024);
+        assert_eq!(64 * KI_B, DEFAULT_BUFFER_MIN_SIZE);
+        assert_eq!(MI_B, DEFAULT_BUFFER_MAX_SIZE);
+        assert_eq!(256 * KI_B, DEFAULT_BUFFER_UNKNOWN_SIZE);
+    }
+
+    #[test]
+    fn test_env_var_names() {
+        assert_eq!(ENV_RUSTFS_BUFFER_MIN_SIZE, "RUSTFS_BUFFER_MIN_SIZE");
+        assert_eq!(ENV_RUSTFS_BUFFER_MAX_SIZE, "RUSTFS_BUFFER_MAX_SIZE");
+        assert_eq!(ENV_RUSTFS_BUFFER_DEFAULT_SIZE, "RUSTFS_BUFFER_DEFAULT_SIZE");
+    }
+}

crates/config/src/lib.rs

@@ -45,6 +45,8 @@ pub use constants::targets::*;
 #[cfg(feature = "constants")]
 pub use constants::tls::*;
 #[cfg(feature = "constants")]
+pub use constants::workload::*;
+#[cfg(feature = "constants")]
 pub mod oidc {
     pub use super::constants::oidc::*;
 }

crates/config/src/observability/mod.rs

@@ -48,6 +48,12 @@ pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZ
 pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
 pub const ENV_OBS_LOG_COMPRESS_OLD_FILES: &str = "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES";
 pub const ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL: &str = "RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL";
+pub const ENV_OBS_LOG_COMPRESSION_ALGORITHM: &str = "RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM";
+pub const ENV_OBS_LOG_PARALLEL_COMPRESS: &str = "RUSTFS_OBS_LOG_PARALLEL_COMPRESS";
+pub const ENV_OBS_LOG_PARALLEL_WORKERS: &str = "RUSTFS_OBS_LOG_PARALLEL_WORKERS";
+pub const ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL: &str = "RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL";
+pub const ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP: &str = "RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP";
+pub const ENV_OBS_LOG_ZSTD_WORKERS: &str = "RUSTFS_OBS_LOG_ZSTD_WORKERS";
 pub const ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: &str = "RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS";
 pub const ENV_OBS_LOG_EXCLUDE_PATTERNS: &str = "RUSTFS_OBS_LOG_EXCLUDE_PATTERNS";
 pub const ENV_OBS_LOG_DELETE_EMPTY_FILES: &str = "RUSTFS_OBS_LOG_DELETE_EMPTY_FILES";
@@ -61,13 +67,24 @@ pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; //
 pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
 pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP: &str = "gzip";
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD: &str = "zstd";
+pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM: &str = DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD;
+pub const DEFAULT_OBS_LOG_PARALLEL_COMPRESS: bool = true;
+pub const DEFAULT_OBS_LOG_PARALLEL_WORKERS: usize = 6;
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL: i32 = 8;
+pub const DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP: bool = true;
+pub const DEFAULT_OBS_LOG_ZSTD_WORKERS: usize = 1;
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION: &str = "gz";
 pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION);
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_EXTENSION: &str = "zst";
+pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_ZSTD_COMPRESSION_EXTENSION);
 pub const DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: u64 = 30; // Retain compressed files for 30 days
 pub const DEFAULT_OBS_LOG_DELETE_EMPTY_FILES: bool = true;
 pub const DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS: u64 = 3600; // 1 hour
 pub const DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS: u64 = 1800; // 0.5 hours
 pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
+pub const DEFAULT_OBS_LOG_MATCH_MODE_PREFIX: &str = "prefix";
 pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";
 
 /// Default values for observability configuration
@@ -113,6 +130,12 @@ mod tests {
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_COMPRESS_OLD_FILES, "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES");
         assert_eq!(ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL, "RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL");
+        assert_eq!(ENV_OBS_LOG_COMPRESSION_ALGORITHM, "RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM");
+        assert_eq!(ENV_OBS_LOG_PARALLEL_COMPRESS, "RUSTFS_OBS_LOG_PARALLEL_COMPRESS");
+        assert_eq!(ENV_OBS_LOG_PARALLEL_WORKERS, "RUSTFS_OBS_LOG_PARALLEL_WORKERS");
+        assert_eq!(ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL, "RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL");
+        assert_eq!(ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, "RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP");
+        assert_eq!(ENV_OBS_LOG_ZSTD_WORKERS, "RUSTFS_OBS_LOG_ZSTD_WORKERS");
         assert_eq!(
             ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS,
             "RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS"
@@ -131,6 +154,10 @@ mod tests {
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_DEVELOPMENT, "development");
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_TEST, "test");
         assert_eq!(DEFAULT_OBS_ENVIRONMENT_STAGING, "staging");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP, "gzip");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD, "zstd");
+        assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE_PREFIX, "prefix");
         assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE, "suffix");
+        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, "zstd");
     }
 }

crates/credentials/Cargo.toml

@@ -15,7 +15,7 @@ base64-simd = { workspace = true }
 rand = { workspace = true }
 serde = { workspace = true }
 serde_json.workspace = true
-time = { workspace = true, features = ["serde-human-readable"] }
+time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }
 
 [lints]
 workspace = true

crates/credentials/src/credentials.rs

@@ -273,15 +273,25 @@ impl<'a> fmt::Display for Masked<'a> {
 ///
 #[derive(Serialize, Deserialize, Clone, Default)]
 pub struct Credentials {
+    #[serde(rename = "accessKey", alias = "access_key", default)]
     pub access_key: String,
+    #[serde(rename = "secretKey", alias = "secret_key", default)]
     pub secret_key: String,
+    #[serde(rename = "sessionToken", alias = "session_token", default)]
     pub session_token: String,
+    #[serde(default, with = "crate::serde_datetime::option")]
     pub expiration: Option<OffsetDateTime>,
+    #[serde(default)]
     pub status: String,
+    #[serde(rename = "parentUser", alias = "parent_user", default)]
     pub parent_user: String,
+    #[serde(default)]
     pub groups: Option<Vec<String>>,
+    #[serde(default)]
     pub claims: Option<HashMap<String, Value>>,
+    #[serde(default)]
     pub name: Option<String>,
+    #[serde(default)]
     pub description: Option<String>,
 }
 
@@ -491,4 +501,31 @@ mod tests {
         assert_eq!(format!("{:?}", Masked(Some("中文"))), "中***|2");
         assert_eq!(format!("{:?}", Masked(Some("中文测试"))), "中***试|4");
     }
+
+    #[test]
+    fn test_credentials_expiration_serialize_as_rfc3339() {
+        use time::OffsetDateTime;
+
+        let c = Credentials {
+            access_key: "ak".to_string(),
+            secret_key: "sk12345678".to_string(),
+            expiration: Some(OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+
+        let json = serde_json::to_string(&c).expect("serialize");
+        assert!(
+            json.contains('T') && (json.contains('Z') || json.contains("+00:00")),
+            "Credentials expiration should be RFC3339; got: {}",
+            json
+        );
+    }
+
+    #[test]
+    fn test_credentials_deserialize_minio_style_rfc3339_expiration() {
+        let minio_style = r#"{"accessKey":"ak","secretKey":"sk12345678","expiration":"2025-03-07T12:00:00Z"}"#;
+        let c: Credentials = serde_json::from_str(minio_style).expect("deserialize");
+        assert_eq!(c.access_key, "ak");
+        assert!(c.expiration.is_some());
+    }
 }

crates/credentials/src/lib.rs

@@ -14,6 +14,7 @@
 
 mod constants;
 mod credentials;
+mod serde_datetime;
 
 pub use constants::*;
 pub use credentials::*;

crates/credentials/src/serde_datetime.rs

@@ -0,0 +1,68 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Serde helpers for expiration timestamp: serialize as RFC3339 (MinIO-compatible),
+//! deserialize from RFC3339 or legacy RustFS human-readable format.
+
+use time::OffsetDateTime;
+use time::format_description;
+use time::format_description::well_known::Rfc3339;
+
+static LEGACY_FORMAT: std::sync::OnceLock<time::format_description::OwnedFormatItem> = std::sync::OnceLock::new();
+
+fn legacy_format() -> &'static time::format_description::OwnedFormatItem {
+    LEGACY_FORMAT.get_or_init(|| {
+        format_description::parse_owned::<2>(
+            "[year]-[month]-[day] [hour]:[minute]:[second].[subsecond] [offset_hour sign:mandatory]:[offset_minute]:[offset_second]",
+        )
+        .expect("legacy format description is valid")
+    });
+    LEGACY_FORMAT.get().expect("initialized above")
+}
+
+fn parse_rfc3339_or_legacy(s: &str) -> Result<OffsetDateTime, time::Error> {
+    OffsetDateTime::parse(s, &Rfc3339).or_else(|_| OffsetDateTime::parse(s, legacy_format()).map_err(Into::into))
+}
+
+/// Option<OffsetDateTime>: serialize as RFC3339; deserialize from RFC3339 or legacy.
+pub mod option {
+    use serde::{Deserialize, Deserializer, Serializer};
+    use time::OffsetDateTime;
+
+    use super::{Rfc3339, parse_rfc3339_or_legacy};
+
+    pub fn serialize<S>(opt: &Option<OffsetDateTime>, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        match opt {
+            Some(dt) => {
+                let s = dt.format(&Rfc3339).map_err(serde::ser::Error::custom)?;
+                serializer.serialize_some(&s)
+            }
+            None => serializer.serialize_none(),
+        }
+    }
+
+    pub fn deserialize<'de, D>(deserializer: D) -> Result<Option<OffsetDateTime>, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let opt: Option<&str> = Option::deserialize(deserializer)?;
+        match opt {
+            None => Ok(None),
+            Some(s) => parse_rfc3339_or_legacy(s).map(Some).map_err(serde::de::Error::custom),
+        }
+    }
+}

crates/crypto/Cargo.toml

@@ -56,3 +56,6 @@ crypto = [
     "dep:rand",
     "dep:sha2",
 ]
+
+[lib]
+doctest = false

crates/crypto/src/encdec.rs

@@ -21,5 +21,8 @@ pub(crate) mod id;
 pub(crate) mod decrypt;
 pub(crate) mod encrypt;
 
+#[cfg(any(test, feature = "crypto"))]
+pub(crate) mod stream_io;
+
 #[cfg(test)]
 mod tests;

crates/crypto/src/encdec/stream_io.rs

@@ -0,0 +1,219 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! sio-go compatible stream encryption for IAM config.
+//! Header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes.
+//! Body: DARE-style fragmented AEAD (bufSize=16384, per-fragment nonce).
+
+#![allow(deprecated)] // AeadInPlace deprecated in favor of AeadInOut; keep for aead 0.6 compatibility
+
+use crate::encdec::id::ID;
+use crate::error::Error;
+use aes_gcm::{
+    Aes256Gcm,
+    aead::{AeadCore, AeadInPlace, KeyInit as _, array::Array},
+};
+use chacha20poly1305::ChaCha20Poly1305;
+
+const STREAM_IO_HEADER_LEN: usize = 41;
+const SIO_BUF_SIZE: usize = 16384;
+const SIO_NONCE_PREFIX_LEN: usize = 8;
+const AES_GCM_OVERHEAD: usize = 16;
+const CHACHA_OVERHEAD: usize = 16;
+
+/// Decrypt data in stream_io (sio-go) format.
+pub fn decrypt_stream_io(password: &[u8], data: &[u8]) -> Result<Vec<u8>, Error> {
+    if data.len() < STREAM_IO_HEADER_LEN {
+        return Err(Error::ErrUnexpectedHeader);
+    }
+    let salt = &data[0..32];
+    let id = ID::try_from(data[32])?;
+    let nonce_prefix = &data[33..41];
+    let body = &data[STREAM_IO_HEADER_LEN..];
+
+    let key = id.get_key(password, salt)?;
+
+    match id {
+        ID::Argon2idChaCHa20Poly1305 => decrypt_stream(
+            ChaCha20Poly1305::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            nonce_prefix,
+            body,
+            CHACHA_OVERHEAD,
+        ),
+        _ => decrypt_stream(
+            Aes256Gcm::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            nonce_prefix,
+            body,
+            AES_GCM_OVERHEAD,
+        ),
+    }
+}
+
+fn decrypt_stream<A>(aead: A, nonce_prefix: &[u8], body: &[u8], overhead: usize) -> Result<Vec<u8>, Error>
+where
+    A: AeadInPlace,
+{
+    let ciphertext_len = SIO_BUF_SIZE + overhead;
+    let ad = build_associated_data(&aead, nonce_prefix)?;
+    let mut plain = Vec::with_capacity(body.len());
+
+    let mut seq_num: u32 = 1;
+    let mut pos = 0;
+
+    while pos < body.len() {
+        let remaining = body.len() - pos;
+        let frag_len = remaining.min(ciphertext_len);
+        let is_last = (pos + frag_len) == body.len();
+
+        if frag_len < overhead {
+            return Err(Error::ErrDecryptFailed(aes_gcm::aead::Error));
+        }
+
+        let mut nonce = [0u8; 12];
+        nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+        nonce[8..12].copy_from_slice(&seq_num.to_le_bytes());
+
+        let mut ad_mut = ad.clone();
+        ad_mut[0] = if is_last { 0x80 } else { 0x00 };
+
+        let fragment = &body[pos..pos + frag_len];
+        let tag_len = overhead;
+        let (ct, tag) = fragment.split_at(frag_len - tag_len);
+
+        let mut buffer = ct.to_vec();
+        let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+            .map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        let tag_arr =
+            Array::<u8, <A as AeadCore>::TagSize>::try_from(tag).map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        aead.decrypt_in_place_detached(&nonce_arr, &ad_mut, &mut buffer, &tag_arr)
+            .map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
+        plain.extend_from_slice(&buffer);
+
+        pos += frag_len;
+        seq_num += 1;
+
+        if is_last {
+            break;
+        }
+    }
+
+    Ok(plain)
+}
+
+fn build_associated_data<A>(aead: &A, nonce_prefix: &[u8]) -> Result<Vec<u8>, Error>
+where
+    A: AeadInPlace,
+{
+    let mut nonce = [0u8; 12];
+    nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+    nonce[8..12].copy_from_slice(&0u32.to_le_bytes());
+
+    let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+        .map_err(|_| Error::ErrEncryptFailed(aes_gcm::aead::Error))?;
+    let mut empty: [u8; 0] = [];
+    let tag = aead
+        .encrypt_in_place_detached(&nonce_arr, &[] as &[u8], &mut empty)
+        .map_err(Error::ErrEncryptFailed)?;
+
+    let mut ad = vec![0u8; 1 + tag.len()];
+    ad[0] = 0x00;
+    ad[1..].copy_from_slice(tag.as_slice());
+    Ok(ad)
+}
+
+/// Encrypt data in stream_io (sio-go) format.
+pub fn encrypt_stream_io(password: &[u8], data: &[u8]) -> Result<Vec<u8>, Error> {
+    let salt: [u8; 32] = rand::random();
+
+    #[cfg(feature = "fips")]
+    let id = ID::Pbkdf2AESGCM;
+
+    #[cfg(not(feature = "fips"))]
+    let id = if crate::encdec::encrypt::native_aes() {
+        ID::Argon2idAESGCM
+    } else {
+        ID::Argon2idChaCHa20Poly1305
+    };
+
+    let key = id.get_key(password, &salt)?;
+    let nonce_prefix: [u8; SIO_NONCE_PREFIX_LEN] = rand::random();
+
+    let mut out = Vec::with_capacity(STREAM_IO_HEADER_LEN + data.len() + 32);
+    out.extend_from_slice(&salt);
+    out.push(id as u8);
+    out.extend_from_slice(&nonce_prefix);
+
+    match id {
+        ID::Argon2idChaCHa20Poly1305 => encrypt_stream(
+            ChaCha20Poly1305::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            &nonce_prefix,
+            data,
+            &mut out,
+            CHACHA_OVERHEAD,
+        )?,
+        _ => encrypt_stream(
+            Aes256Gcm::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
+            &nonce_prefix,
+            data,
+            &mut out,
+            AES_GCM_OVERHEAD,
+        )?,
+    }
+
+    Ok(out)
+}
+
+fn encrypt_stream<A>(
+    aead: A,
+    nonce_prefix: &[u8; SIO_NONCE_PREFIX_LEN],
+    data: &[u8],
+    out: &mut Vec<u8>,
+    _overhead: usize,
+) -> Result<(), Error>
+where
+    A: AeadInPlace,
+{
+    let ad = build_associated_data(&aead, nonce_prefix)?;
+    let mut seq_num: u32 = 1;
+    let mut pos = 0;
+
+    while pos < data.len() {
+        let remaining = data.len() - pos;
+        let is_last = remaining <= SIO_BUF_SIZE;
+
+        let chunk_len = if is_last { remaining } else { SIO_BUF_SIZE };
+        let chunk = &data[pos..pos + chunk_len];
+
+        let mut nonce = [0u8; 12];
+        nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
+        nonce[8..12].copy_from_slice(&seq_num.to_le_bytes());
+
+        let mut ad_mut = ad.clone();
+        ad_mut[0] = if is_last { 0x80 } else { 0x00 };
+
+        let mut buffer = chunk.to_vec();
+        let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
+            .map_err(|_| Error::ErrEncryptFailed(aes_gcm::aead::Error))?;
+        let tag = aead
+            .encrypt_in_place_detached(&nonce_arr, &ad_mut, &mut buffer)
+            .map_err(Error::ErrEncryptFailed)?;
+        out.extend_from_slice(&buffer);
+        out.extend_from_slice(tag.as_slice());
+
+        pos += chunk_len;
+        seq_num += 1;
+    }
+
+    Ok(())
+}

crates/crypto/src/encdec/tests.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::{decrypt_data, encrypt_data};
+use crate::{decrypt_data, decrypt_stream_io, encrypt_data, encrypt_stream_io};
 
 const PASSWORD: &[u8] = "test_password".as_bytes();
 const LONG_PASSWORD: &[u8] = "very_long_password_with_many_characters_for_testing_purposes_123456789".as_bytes();
@@ -317,3 +317,67 @@ fn test_concurrent_encryption_safety() -> Result<(), crate::Error> {
 
     Ok(())
 }
+
+#[test]
+fn test_stream_io_roundtrip() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = br#"{"Version":1,"policy":"readonly"}"#;
+    let encrypted = encrypt_stream_io(password, data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert_eq!(data, decrypted.as_slice());
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_large_data_roundtrip() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = vec![0xAB; 32 * 1024]; // > SIO_BUF_SIZE to test fragmentation
+    let encrypted = encrypt_stream_io(password, &data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert_eq!(data, decrypted);
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_wrong_password_fails() {
+    let password = b"access:secret";
+    let data = br#"{"Version":1}"#;
+    let encrypted = encrypt_stream_io(password, data).expect("encrypt should succeed");
+    let result = decrypt_stream_io(b"wrong:password", &encrypted);
+    assert!(result.is_err(), "decrypt with wrong password should fail");
+}
+
+#[test]
+fn test_stream_io_empty_data() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data: &[u8] = &[];
+    let encrypted = encrypt_stream_io(password, data)?;
+    let decrypted = decrypt_stream_io(password, &encrypted)?;
+    assert!(decrypted.is_empty());
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_header_format() -> Result<(), crate::Error> {
+    let password = b"access:secret";
+    let data = b"test";
+    let encrypted = encrypt_stream_io(password, data)?;
+    // stream_io header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes
+    const STREAM_IO_HEADER_LEN: usize = 41;
+    assert!(encrypted.len() >= STREAM_IO_HEADER_LEN, "encrypted should have at least 41-byte header");
+    assert!(
+        encrypted[32] == 0x00 || encrypted[32] == 0x01 || encrypted[32] == 0x02,
+        "alg_id should be 0x00, 0x01, or 0x02"
+    );
+    Ok(())
+}
+
+#[test]
+fn test_stream_io_truncated_data_fails() {
+    let password = b"access:secret";
+    let data = b"test";
+    let encrypted = encrypt_stream_io(password, data).expect("encrypt should succeed");
+    let truncated = &encrypted[..40]; // less than 41-byte header
+    let result = decrypt_stream_io(password, truncated);
+    assert!(result.is_err(), "truncated data should fail decrypt");
+}

crates/crypto/src/lib.rs

@@ -19,6 +19,9 @@ mod jwt;
 
 pub use encdec::decrypt::decrypt_data;
 pub use encdec::encrypt::encrypt_data;
+
+#[cfg(feature = "crypto")]
+pub use encdec::stream_io::{decrypt_stream_io, encrypt_stream_io};
 pub use error::Error;
 pub use jwt::decode::decode as jwt_decode;
 pub use jwt::encode::encode as jwt_encode;

crates/e2e_test/src/delete_objects_versioning_test.rs

@@ -16,15 +16,13 @@
 //! "In a versioned Bucket, DeleteMarkers are not appearing straight after
 //! a delete_objects is called."
 //!
-//! Root cause: `delete_versions_internal` wrote new xl.meta to disk via
-//! `write_all_private` without invalidating the `GlobalFileCache`. Subsequent
-//! calls to `read_metadata` returned the stale cached xl.meta (without the
-//! delete marker), making `list_object_versions` show the old version as
-//! `IsLatest=true` rather than the new delete marker.
+//! Root cause: metadata updates could become temporarily invisible to
+//! `list_object_versions`, so the old version was still reported as
+//! `IsLatest=true` instead of the newly-created delete marker.
 //!
-//! Fix: `write_all_private` now calls `get_global_file_cache().invalidate()`
-//! after every successful write, and `rename_data` also invalidates the cache
-//! for the destination path after the atomic rename.
+//! Fix: metadata write, delete, and rename paths now make the updated
+//! `xl.meta` immediately visible, and the old file-cache shortcut has been
+//! removed from the read path.
 
 #[cfg(test)]
 mod tests {

crates/e2e_test/src/lib.rs

@@ -66,6 +66,11 @@ mod compression_test;
 // Regression test for Issue #1878: DeleteMarkers not visible immediately after delete_objects
 #[cfg(test)]
 mod delete_objects_versioning_test;
+
+// Regression test for Issue #2252: ListObjectVersions misses newest version after put -> delete -> put
+#[cfg(test)]
+mod list_object_versions_regression_test;
+
 #[cfg(test)]
 mod protocols;
 

crates/e2e_test/src/list_object_versions_regression_test.rs

@@ -0,0 +1,182 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Regression test for Issue #2252:
+//! "ListObjectVersions misses the newest version after create -> delete -> create."
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::Client;
+    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
+    use serial_test::serial;
+    use tracing::info;
+
+    fn create_s3_client(env: &RustFSTestEnvironment) -> Client {
+        env.create_s3_client()
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_list_object_versions_immediately_returns_latest_put_after_delete_marker() {
+        init_logging();
+        info!("🧪 TEST: ListObjectVersions returns the newest version immediately after put -> delete -> put");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-list-object-versions-2252";
+        let key = "test-prefix/test-object.txt";
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("Failed to create bucket");
+
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await
+            .expect("Failed to enable versioning");
+
+        let first_put = client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(aws_sdk_s3::primitives::ByteStream::from_static(b"first version"))
+            .send()
+            .await
+            .expect("Failed to put first object version");
+        let first_version_id = first_put
+            .version_id()
+            .map(str::to_string)
+            .expect("First put should return a version_id");
+
+        let delete_resp = client
+            .delete_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("Failed to create delete marker");
+        let delete_marker_version_id = delete_resp
+            .version_id()
+            .map(str::to_string)
+            .expect("DeleteObject should return a delete marker version_id");
+
+        let second_put = client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(aws_sdk_s3::primitives::ByteStream::from_static(b"second version"))
+            .send()
+            .await
+            .expect("Failed to put second object version");
+        let second_version_id = second_put
+            .version_id()
+            .map(str::to_string)
+            .expect("Second put should return a version_id");
+
+        let first_listing = client
+            .list_object_versions()
+            .bucket(bucket)
+            .prefix(key)
+            .send()
+            .await
+            .expect("Failed to list object versions immediately after second put");
+        let second_listing = client
+            .list_object_versions()
+            .bucket(bucket)
+            .prefix(key)
+            .send()
+            .await
+            .expect("Failed to list object versions a second time");
+
+        let first_versions = first_listing.versions().to_vec();
+        let first_delete_markers = first_listing.delete_markers().to_vec();
+        let second_versions = second_listing.versions().to_vec();
+        let second_delete_markers = second_listing.delete_markers().to_vec();
+
+        info!(
+            "First listing: {} versions, {} delete markers",
+            first_versions.len(),
+            first_delete_markers.len()
+        );
+        info!(
+            "Second listing: {} versions, {} delete markers",
+            second_versions.len(),
+            second_delete_markers.len()
+        );
+
+        assert_eq!(
+            first_versions.len(),
+            2,
+            "First ListObjectVersions call should return both object versions immediately (regression #2252)"
+        );
+        assert_eq!(
+            first_delete_markers.len(),
+            1,
+            "First ListObjectVersions call should return the delete marker immediately (regression #2252)"
+        );
+        assert_eq!(
+            second_versions.len(),
+            2,
+            "Second ListObjectVersions call should still return both object versions"
+        );
+        assert_eq!(
+            second_delete_markers.len(),
+            1,
+            "Second ListObjectVersions call should still return the delete marker"
+        );
+
+        let first_latest_version = first_versions
+            .iter()
+            .find(|version| version.version_id() == Some(second_version_id.as_str()))
+            .expect("First listing should include the newest object version");
+        assert_eq!(
+            first_latest_version.is_latest(),
+            Some(true),
+            "Newest object version should be latest on the first listing"
+        );
+
+        let first_original_version = first_versions
+            .iter()
+            .find(|version| version.version_id() == Some(first_version_id.as_str()))
+            .expect("First listing should include the original object version");
+        assert_eq!(
+            first_original_version.is_latest(),
+            Some(false),
+            "Original object version should no longer be latest"
+        );
+
+        let first_delete_marker = first_delete_markers
+            .iter()
+            .find(|marker| marker.version_id() == Some(delete_marker_version_id.as_str()))
+            .expect("First listing should include the delete marker");
+        assert_eq!(
+            first_delete_marker.is_latest(),
+            Some(false),
+            "Delete marker should no longer be latest after the second put"
+        );
+    }
+}

crates/e2e_test/src/reliant/grpc_lock_client.rs

@@ -61,6 +61,7 @@ impl GrpcLockClient {
             metadata: LockMetadata::default(),
             priority: LockPriority::Normal,
             deadlock_detection: false,
+            suppress_contention_logs: false,
         }
     }
 }

crates/e2e_test/src/reliant/lifecycle.rs

@@ -19,7 +19,6 @@ use aws_sdk_s3::config::{Credentials, Region};
 use bytes::Bytes;
 use serial_test::serial;
 use std::error::Error;
-use tokio::time::sleep;
 
 const ENDPOINT: &str = "http://localhost:9000";
 const ACCESS_KEY: &str = "rustfsadmin";
@@ -62,6 +61,7 @@ async fn setup_test_bucket(client: &Client) -> Result<(), Box<dyn Error>> {
 #[ignore = "requires running RustFS server at localhost:9000"]
 async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error::Error>> {
     use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
+    use chrono::{Duration as ChronoDuration, Utc};
     use tokio::time::Duration;
 
     let client = create_aws_s3_client().await?;
@@ -70,6 +70,7 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
     // Upload test object first
     let test_content = "Test object for lifecycle expiration";
     let lifecycle_object_key = "lifecycle-test-object.txt";
+    let untouched_object_key = "keep-object.txt";
     client
         .put_object()
         .bucket(BUCKET)
@@ -77,13 +78,29 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
         .body(Bytes::from(test_content.as_bytes()).into())
         .send()
         .await?;
+    client
+        .put_object()
+        .bucket(BUCKET)
+        .key(untouched_object_key)
+        .body(Bytes::from("should-stay".as_bytes()).into())
+        .send()
+        .await?;
 
     // Verify object exists initially
     let resp = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await?;
     assert!(resp.content_length().unwrap_or(0) > 0);
+    let untouched_resp = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
+    assert!(untouched_resp.content_length().unwrap_or(0) > 0);
 
-    // Configure lifecycle rule: expire after current time + 3 seconds
-    let expiration = LifecycleExpiration::builder().days(0).build();
+    // Use a past midnight UTC date to trigger immediate lifecycle expiry without requiring days=0.
+    let yesterday_midnight_utc = Utc::now()
+        .date_naive()
+        .and_hms_opt(0, 0, 0)
+        .expect("midnight should always be valid")
+        - ChronoDuration::days(1);
+    let expiration = LifecycleExpiration::builder()
+        .date(aws_sdk_s3::primitives::DateTime::from_secs(yesterday_midnight_utc.and_utc().timestamp()))
+        .build();
     let filter = LifecycleRuleFilter::builder().prefix(lifecycle_object_key).build();
     let rule = LifecycleRule::builder()
         .id("expire-test-object")
@@ -105,29 +122,73 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
     let rules = resp.rules();
     assert!(rules.iter().any(|r| r.id().unwrap_or("") == "expire-test-object"));
 
-    // Wait for lifecycle processing (scanner runs every 1 second)
-    sleep(Duration::from_secs(3)).await;
-
-    // After lifecycle processing, the object should be deleted by the lifecycle rule
-    let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
-
-    match get_result {
-        Ok(_) => {
-            panic!("Expected object to be deleted by lifecycle rule, but it still exists");
-        }
-        Err(e) => {
-            if let Some(service_error) = e.as_service_error() {
-                if service_error.is_no_such_key() {
-                    println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
-                } else {
-                    panic!("Expected NoSuchKey error, but got: {e:?}");
+    // Poll for deletion instead of using a fixed sleep to keep the test deterministic.
+    // Default scanner cycle interval is 60s with jitter, so allow enough time for one full cycle.
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(150);
+    loop {
+        let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
+        match get_result {
+            Ok(_) => {
+                if tokio::time::Instant::now() >= deadline {
+                    panic!("Expected object to be deleted by lifecycle rule within 150s, but it still exists");
+                }
+                tokio::time::sleep(Duration::from_secs(1)).await;
+            }
+            Err(e) => {
+                if let Some(service_error) = e.as_service_error() {
+                    if service_error.is_no_such_key() {
+                        println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
+                        break;
+                    }
+                    panic!("Expected NoSuchKey error, but got: {e:?}");
+                } else {
+                    panic!("Expected service error, but got: {e:?}");
                 }
-            } else {
-                panic!("Expected service error, but got: {e:?}");
             }
         }
     }
 
     println!("Lifecycle configuration test completed.");
+
+    // Non-matching prefix object should remain available.
+    let untouched_after = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
+    assert!(untouched_after.content_length().unwrap_or(0) > 0);
+
+    Ok(())
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+#[serial]
+#[ignore = "requires running RustFS server at localhost:9000"]
+async fn test_bucket_lifecycle_rejects_zero_days() -> Result<(), Box<dyn std::error::Error>> {
+    use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
+
+    let client = create_aws_s3_client().await?;
+    setup_test_bucket(&client).await?;
+
+    let expiration = LifecycleExpiration::builder().days(0).build();
+    let filter = LifecycleRuleFilter::builder().prefix("zero-days/").build();
+    let rule = LifecycleRule::builder()
+        .id("expire-zero-days")
+        .filter(filter)
+        .expiration(expiration)
+        .status(aws_sdk_s3::types::ExpirationStatus::Enabled)
+        .build()?;
+    let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
+
+    let err = client
+        .put_bucket_lifecycle_configuration()
+        .bucket(BUCKET)
+        .lifecycle_configuration(lifecycle)
+        .send()
+        .await
+        .expect_err("zero-day lifecycle expiration should be rejected");
+
+    let err_msg = format!("{err:?}");
+    assert!(
+        err_msg.contains("InvalidArgument") && err_msg.contains("greater than 0"),
+        "unexpected error: {err_msg}"
+    );
+
     Ok(())
 }

crates/ecstore/Cargo.toml

@@ -66,6 +66,7 @@ http-body = { workspace = true }
 http-body-util.workspace = true
 url.workspace = true
 uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
+reed-solomon-erasure = { workspace = true }
 reed-solomon-simd = { workspace = true }
 lazy_static.workspace = true
 rustfs-lock.workspace = true
@@ -115,11 +116,15 @@ faster-hex = { workspace = true }
 ratelimit = { workspace = true }
 aws-smithy-http-client.workspace = true
 
+# Observability and Metrics
+metrics = { workspace = true }
+
 [dev-dependencies]
 tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true }
 tracing-subscriber = { workspace = true }
+serial_test = { workspace = true }
 
 [build-dependencies]
 shadow-rs = { workspace = true, features = ["build", "metadata"] }
@@ -131,3 +136,6 @@ harness = false
 [[bench]]
 name = "comparison_benchmark"
 harness = false
+
+[lib]
+doctest = false

crates/ecstore/benches/erasure_benchmark.rs

@@ -118,41 +118,38 @@ fn bench_encode_performance(c: &mut Criterion) {
         });
         group.finish();
 
-        // Test direct SIMD implementation for large shards (>= 512 bytes)
+        // Test direct reed-solomon-erasure implementation for large shards (>= 512 bytes)
         let shard_size = calc_shard_size(config.data_size, config.data_shards);
-        if shard_size >= 512 {
-            let mut simd_group = c.benchmark_group("encode_simd_direct");
-            simd_group.throughput(Throughput::Bytes(config.data_size as u64));
-            simd_group.sample_size(10);
-            simd_group.measurement_time(Duration::from_secs(5));
+        if shard_size >= 512 && config.parity_shards > 0 {
+            use reed_solomon_erasure::galois_8::ReedSolomon;
 
-            simd_group.bench_with_input(BenchmarkId::new("simd_direct", &config.name), &(&data, &config), |b, (data, config)| {
-                b.iter(|| {
-                    // Direct SIMD implementation
-                    let per_shard_size = calc_shard_size(data.len(), config.data_shards);
-                    match reed_solomon_simd::ReedSolomonEncoder::new(config.data_shards, config.parity_shards, per_shard_size) {
-                        Ok(mut encoder) => {
-                            // Create properly sized buffer and fill with data
-                            let mut buffer = vec![0u8; per_shard_size * config.data_shards];
+            let mut rse_group = c.benchmark_group("encode_rse_direct");
+            rse_group.throughput(Throughput::Bytes(config.data_size as u64));
+            rse_group.sample_size(10);
+            rse_group.measurement_time(Duration::from_secs(5));
+
+            if let Ok(rs) = ReedSolomon::new(config.data_shards, config.parity_shards) {
+                let total_shards = config.data_shards + config.parity_shards;
+                let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
+                let need_total = per_shard_size * total_shards;
+
+                rse_group.bench_with_input(
+                    BenchmarkId::new("rse_direct", &config.name),
+                    &(&data, need_total, per_shard_size),
+                    |b, (data, need_total, per_shard_size)| {
+                        b.iter(|| {
+                            let mut buffer = vec![0u8; *need_total];
                             let copy_len = data.len().min(buffer.len());
                             buffer[..copy_len].copy_from_slice(&data[..copy_len]);
 
-                            // Add data shards with correct shard size
-                            for chunk in buffer.chunks_exact(per_shard_size) {
-                                encoder.add_original_shard(black_box(chunk)).unwrap();
-                            }
-
-                            let result = encoder.encode().unwrap();
-                            black_box(result);
-                        }
-                        Err(_) => {
-                            // SIMD doesn't support this configuration, skip
-                            black_box(());
-                        }
-                    }
-                });
-            });
-            simd_group.finish();
+                            let mut slices: Vec<&mut [u8]> = buffer.chunks_exact_mut(*per_shard_size).collect();
+                            rs.encode(&mut slices).unwrap();
+                            black_box(buffer);
+                        });
+                    },
+                );
+            }
+            rse_group.finish();
         }
     }
 }
@@ -203,47 +200,33 @@ fn bench_decode_performance(c: &mut Criterion) {
         );
         group.finish();
 
-        // Test direct SIMD decoding for large shards
+        // Test direct reed-solomon-erasure decoding for large shards
         let shard_size = calc_shard_size(config.data_size, config.data_shards);
-        if shard_size >= 512 {
-            let mut simd_group = c.benchmark_group("decode_simd_direct");
-            simd_group.throughput(Throughput::Bytes(config.data_size as u64));
-            simd_group.sample_size(10);
-            simd_group.measurement_time(Duration::from_secs(5));
+        if shard_size >= 512 && config.parity_shards > 0 {
+            use reed_solomon_erasure::galois_8::ReedSolomon;
 
-            simd_group.bench_with_input(
-                BenchmarkId::new("simd_direct", &config.name),
-                &(&encoded_shards, &config),
-                |b, (shards, config)| {
-                    b.iter(|| {
-                        let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
-                        match reed_solomon_simd::ReedSolomonDecoder::new(config.data_shards, config.parity_shards, per_shard_size)
-                        {
-                            Ok(mut decoder) => {
-                                // Add available shards (except lost ones)
-                                for (i, shard) in shards.iter().enumerate() {
-                                    if i != config.data_shards - 1 && i != config.data_shards {
-                                        if i < config.data_shards {
-                                            decoder.add_original_shard(i, black_box(shard)).unwrap();
-                                        } else {
-                                            let recovery_idx = i - config.data_shards;
-                                            decoder.add_recovery_shard(recovery_idx, black_box(shard)).unwrap();
-                                        }
-                                    }
-                                }
+            if let Ok(rs) = ReedSolomon::new(config.data_shards, config.parity_shards) {
+                let mut rse_group = c.benchmark_group("decode_rse_direct");
+                rse_group.throughput(Throughput::Bytes(config.data_size as u64));
+                rse_group.sample_size(10);
+                rse_group.measurement_time(Duration::from_secs(5));
 
-                                let result = decoder.decode().unwrap();
-                                black_box(result);
-                            }
-                            Err(_) => {
-                                // SIMD doesn't support this configuration, skip
-                                black_box(());
-                            }
-                        }
-                    });
-                },
-            );
-            simd_group.finish();
+                rse_group.bench_with_input(
+                    BenchmarkId::new("rse_direct", &config.name),
+                    &(&encoded_shards, &config),
+                    |b, (shards, config)| {
+                        b.iter(|| {
+                            let mut shards_opt: Vec<Option<Vec<u8>>> = shards.iter().map(|s| Some(s.to_vec())).collect();
+                            shards_opt[config.data_shards - 1] = None;
+                            shards_opt[config.data_shards] = None;
+
+                            rs.reconstruct_data(&mut shards_opt).unwrap();
+                            black_box(shards_opt);
+                        });
+                    },
+                );
+                rse_group.finish();
+            }
         }
     }
 }

crates/ecstore/src/bitrot.rs

@@ -119,7 +119,7 @@ mod tests {
     async fn test_create_bitrot_reader_with_inline_data() {
         let test_data = b"hello world test data";
         let shard_size = 16;
-        let checksum_algo = HashAlgorithm::HighwayHash256;
+        let checksum_algo = HashAlgorithm::HighwayHash256S;
 
         let result =
             create_bitrot_reader(Some(test_data), None, "test-bucket", "test-path", 0, 0, shard_size, checksum_algo, false).await;
@@ -131,7 +131,7 @@ mod tests {
     #[tokio::test]
     async fn test_create_bitrot_reader_without_data_or_disk() {
         let shard_size = 16;
-        let checksum_algo = HashAlgorithm::HighwayHash256;
+        let checksum_algo = HashAlgorithm::HighwayHash256S;
 
         let result =
             create_bitrot_reader(None, None, "test-bucket", "test-path", 0, 1024, shard_size, checksum_algo, false).await;
@@ -151,7 +151,7 @@ mod tests {
             "test-path",
             1024, // length
             1024, // shard_size
-            HashAlgorithm::HighwayHash256,
+            HashAlgorithm::HighwayHash256S,
         )
         .await;
 
@@ -183,7 +183,7 @@ mod tests {
             "test-path",
             1024, // length
             1024, // shard_size
-            HashAlgorithm::HighwayHash256,
+            HashAlgorithm::HighwayHash256S,
         )
         .await;
 

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -43,11 +43,13 @@ use rustfs_config::{DEFAULT_TRUST_LEAF_CERT_AS_CA, ENV_TRUST_LEAF_CERT_AS_CA, RU
 use rustfs_filemeta::{ReplicationStatusType, ReplicationType};
 use rustfs_utils::http::{
     AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE,
-    AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, RUSTFS_BUCKET_REPLICATION_CHECK,
-    RUSTFS_BUCKET_REPLICATION_DELETE_MARKER, RUSTFS_BUCKET_REPLICATION_REQUEST, RUSTFS_BUCKET_SOURCE_ETAG,
-    RUSTFS_BUCKET_SOURCE_MTIME, RUSTFS_BUCKET_SOURCE_VERSION_ID, RUSTFS_FORCE_DELETE, is_amz_header, is_minio_header,
+    AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, is_amz_header, is_minio_header,
     is_rustfs_header, is_standard_header, is_storageclass_header,
 };
+use rustfs_utils::http::{
+    SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
+    SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, insert_header,
+};
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::error::Error;
@@ -68,8 +70,6 @@ use uuid::Uuid;
 const DEFAULT_HEALTH_CHECK_DURATION: Duration = Duration::from_secs(5);
 const DEFAULT_HEALTH_CHECK_RELOAD_DURATION: Duration = Duration::from_secs(30 * 60);
 
-const REPLICATION_REQUEST_TRUE: HeaderValue = HeaderValue::from_static("true");
-
 pub static GLOBAL_BUCKET_TARGET_SYS: OnceLock<BucketTargetSys> = OnceLock::new();
 
 #[derive(Debug, Clone)]
@@ -1081,23 +1081,21 @@ impl PutObjectOptions {
         }
 
         if !self.internal.source_version_id.is_empty() {
-            header.insert(
-                RUSTFS_BUCKET_SOURCE_VERSION_ID,
-                HeaderValue::from_str(&self.internal.source_version_id).expect("err"),
-            );
+            insert_header(&mut header, SUFFIX_SOURCE_VERSION_ID, &self.internal.source_version_id);
         }
         if self.internal.source_etag.is_empty() {
-            header.insert(RUSTFS_BUCKET_SOURCE_ETAG, HeaderValue::from_str(&self.internal.source_etag).expect("err"));
+            insert_header(&mut header, SUFFIX_SOURCE_ETAG, &self.internal.source_etag);
         }
         if self.internal.source_mtime.unix_timestamp() != 0 {
-            header.insert(
-                RUSTFS_BUCKET_SOURCE_MTIME,
-                HeaderValue::from_str(&self.internal.source_mtime.format(&Rfc3339).unwrap_or_default()).expect("err"),
+            insert_header(
+                &mut header,
+                SUFFIX_SOURCE_MTIME,
+                self.internal.source_mtime.format(&Rfc3339).unwrap_or_default(),
             );
         }
 
         if self.internal.replication_request {
-            header.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, REPLICATION_REQUEST_TRUE);
+            insert_header(&mut header, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
 
         header
@@ -1266,10 +1264,8 @@ impl TargetClient {
         let builder = self.client.put_object();
 
         let version_id = opts.internal.source_version_id.clone();
-        if !version_id.is_empty()
-            && let Ok(header_value) = HeaderValue::from_str(&version_id)
-        {
-            headers.insert(RUSTFS_BUCKET_SOURCE_VERSION_ID, header_value);
+        if !version_id.is_empty() {
+            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
         }
 
         match builder
@@ -1303,13 +1299,11 @@ impl TargetClient {
     ) -> Result<String, S3ClientError> {
         let mut headers = HeaderMap::new();
         let version_id = opts.internal.source_version_id.clone();
-        if !version_id.is_empty()
-            && let Ok(header_value) = HeaderValue::from_str(&version_id)
-        {
-            headers.insert(RUSTFS_BUCKET_SOURCE_VERSION_ID, header_value);
+        if !version_id.is_empty() {
+            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
         }
         if opts.internal.replication_request {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, REPLICATION_REQUEST_TRUE);
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
 
         match self
@@ -1418,21 +1412,18 @@ impl TargetClient {
     ) -> Result<(), S3ClientError> {
         let mut headers = HeaderMap::new();
         if opts.force_delete {
-            headers.insert(RUSTFS_FORCE_DELETE, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_FORCE_DELETE, "true");
         }
         if opts.governance_bypass {
             headers.insert(AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, "true".parse().unwrap());
         }
 
         if opts.replication_delete_marker {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_DELETE_MARKER, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_DELETEMARKER, "true");
         }
 
         if let Some(t) = opts.replication_mtime {
-            headers.insert(
-                RUSTFS_BUCKET_SOURCE_MTIME,
-                t.format(&Rfc3339).unwrap_or_default().as_str().parse().unwrap(),
-            );
+            insert_header(&mut headers, SUFFIX_SOURCE_MTIME, t.format(&Rfc3339).unwrap_or_default());
         }
 
         if !opts.replication_status.is_empty() {
@@ -1440,10 +1431,10 @@ impl TargetClient {
         }
 
         if opts.replication_request {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
         }
         if opts.replication_validity_check {
-            headers.insert(RUSTFS_BUCKET_REPLICATION_CHECK, "true".parse().unwrap());
+            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
         }
 
         match self

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -33,7 +33,9 @@ use crate::global::GLOBAL_LocalNodeName;
 use crate::global::{GLOBAL_LifecycleSys, GLOBAL_TierConfigMgr, get_global_deployment_id};
 use crate::store::ECStore;
 use crate::store_api::StorageAPI;
-use crate::store_api::{GetObjectReader, HTTPRangeSpec, ObjectInfo, ObjectOperations, ObjectOptions, ObjectToDelete};
+use crate::store_api::{
+    GetObjectReader, HTTPRangeSpec, ListOperations, ObjectInfo, ObjectOperations, ObjectOptions, ObjectToDelete,
+};
 use crate::tier::warm_backend::WarmBackendGetOpts;
 use async_channel::{Receiver as A_Receiver, Sender as A_Sender, bounded};
 use bytes::BytesMut;
@@ -45,8 +47,7 @@ use rustfs_common::heal_channel::rep_has_active_rules;
 use rustfs_common::metrics::{IlmAction, Metrics};
 use rustfs_filemeta::{NULL_VERSION_ID, RestoreStatusOps, is_restored_object_on_disk};
 use rustfs_s3_common::EventName;
-use rustfs_utils::path::encode_dir_object;
-use rustfs_utils::string::strings_has_prefix_fold;
+use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::strings_has_prefix_fold};
 use s3s::Body;
 use s3s::dto::{
     BucketLifecycleConfiguration, DefaultRetention, ReplicationConfiguration, RestoreRequest, RestoreRequestType, RestoreStatus,
@@ -97,8 +98,14 @@ impl LifecycleSys {
     }
 
     pub async fn get(&self, bucket: &str) -> Option<BucketLifecycleConfiguration> {
-        let lc = get_lifecycle_config(bucket).await.expect("get_lifecycle_config err!").0;
-        Some(lc)
+        match get_lifecycle_config(bucket).await {
+            Ok((lc, _)) => Some(lc),
+            Err(err) if err == Error::ConfigNotFound => None,
+            Err(err) => {
+                warn!(bucket, error = ?err, "failed to load lifecycle config");
+                None
+            }
+        }
     }
 
     pub fn trace(_oi: &ObjectInfo) -> TraceFn {
@@ -471,10 +478,7 @@ impl TransitionState {
     }
 
     pub async fn init(api: Arc<ECStore>) {
-        let max_workers = env::var("RUSTFS_MAX_TRANSITION_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<i64>().ok())
-            .unwrap_or_else(|| std::cmp::min(num_cpus::get() as i64, 16));
+        let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
         let mut n = max_workers;
         let tw = 8; //globalILMConfig.getTransitionWorkers();
         if tw > 0 {
@@ -569,17 +573,11 @@ impl TransitionState {
     pub async fn update_workers_inner(api: Arc<ECStore>, n: i64) {
         let mut n = n;
         if n == 0 {
-            let max_workers = env::var("RUSTFS_MAX_TRANSITION_WORKERS")
-                .ok()
-                .and_then(|s| s.parse::<i64>().ok())
-                .unwrap_or_else(|| std::cmp::min(num_cpus::get() as i64, 16));
+            let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
             n = max_workers;
         }
         // Allow environment override of maximum workers
-        let absolute_max = env::var("RUSTFS_ABSOLUTE_MAX_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<i64>().ok())
-            .unwrap_or(32);
+        let absolute_max = get_env_i64("RUSTFS_ABSOLUTE_MAX_WORKERS", 32);
         n = std::cmp::min(n, absolute_max);
 
         let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
@@ -603,10 +601,7 @@ impl TransitionState {
 }
 
 pub async fn init_background_expiry(api: Arc<ECStore>) {
-    let mut workers = env::var("RUSTFS_MAX_EXPIRY_WORKERS")
-        .ok()
-        .and_then(|s| s.parse::<usize>().ok())
-        .unwrap_or_else(|| std::cmp::min(num_cpus::get(), 16));
+    let mut workers = get_env_usize("RUSTFS_MAX_EXPIRY_WORKERS", std::cmp::min(num_cpus::get(), 16));
     //globalILMConfig.getExpirationWorkers()
     if let Ok(env_expiration_workers) = env::var("_RUSTFS_ILM_EXPIRATION_WORKERS") {
         if let Ok(num_expirations) = env_expiration_workers.parse::<usize>() {
@@ -615,10 +610,7 @@ pub async fn init_background_expiry(api: Arc<ECStore>) {
     }
 
     if workers == 0 {
-        workers = env::var("RUSTFS_DEFAULT_EXPIRY_WORKERS")
-            .ok()
-            .and_then(|s| s.parse::<usize>().ok())
-            .unwrap_or(8);
+        workers = get_env_usize("RUSTFS_DEFAULT_EXPIRY_WORKERS", 8);
     }
 
     //let expiry_state = GLOBAL_ExpiryStSate.write().await;
@@ -656,18 +648,48 @@ pub async fn validate_transition_tier(lc: &BucketLifecycleConfiguration) -> Resu
 }
 
 pub async fn enqueue_transition_immediate(oi: &ObjectInfo, src: LcEventSrc) {
-    let lc = GLOBAL_LifecycleSys.get(&oi.bucket).await;
-    if !lc.is_none() {
-        let event = lc.expect("err").eval(&oi.to_lifecycle_opts()).await;
-        match event.action {
-            IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
-                if oi.delete_marker || oi.is_dir {
-                    return;
-                }
-                GLOBAL_TransitionState.queue_transition_task(oi, &event, &src).await;
-            }
-            _ => (),
+    if let Some(lc) = GLOBAL_LifecycleSys.get(&oi.bucket).await {
+        enqueue_transition_with_lifecycle(oi, &lc, &src).await;
+    }
+}
+
+pub async fn enqueue_transition_for_existing_objects(api: Arc<ECStore>, bucket: &str) -> Result<(), Error> {
+    let Some(lc) = GLOBAL_LifecycleSys.get(bucket).await else {
+        return Ok(());
+    };
+    let mut marker = None;
+    let mut version_marker = None;
+    let src = LcEventSrc::Scanner;
+
+    loop {
+        let page = api
+            .clone()
+            .list_object_versions(bucket, "", marker.clone(), version_marker.clone(), None, 1000)
+            .await?;
+
+        for object in &page.objects {
+            enqueue_transition_with_lifecycle(object, &lc, &src).await;
         }
+
+        if !page.is_truncated {
+            return Ok(());
+        }
+
+        marker = page.next_marker;
+        version_marker = page.next_version_idmarker;
+    }
+}
+
+async fn enqueue_transition_with_lifecycle(oi: &ObjectInfo, lc: &BucketLifecycleConfiguration, src: &LcEventSrc) {
+    let event = lc.eval(&oi.to_lifecycle_opts()).await;
+    match event.action {
+        IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
+            if oi.delete_marker || oi.is_dir {
+                return;
+            }
+            GLOBAL_TransitionState.queue_transition_task(oi, &event, src).await;
+        }
+        _ => (),
     }
 }
 

crates/ecstore/src/bucket/lifecycle/lifecycle.rs

@@ -20,8 +20,8 @@
 
 use rustfs_filemeta::{ReplicationStatusType, VersionPurgeStatusType};
 use s3s::dto::{
-    BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, NoncurrentVersionTransition,
-    ObjectLockConfiguration, ObjectLockEnabled, RestoreRequest, Transition, TransitionStorageClass,
+    BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleAndOperator,
+    NoncurrentVersionTransition, ObjectLockConfiguration, ObjectLockEnabled, RestoreRequest, Transition, TransitionStorageClass,
 };
 use std::cmp::Ordering;
 use std::collections::HashMap;
@@ -49,6 +49,8 @@ const ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS: &str = "Lifecycle expiration days m
 const ERR_LIFECYCLE_INVALID_EXPIRATION_DATE_NOT_MIDNIGHT: &str = "Expiration.Date must be at midnight UTC";
 const ERR_LIFECYCLE_INVALID_RULE_ID_TOO_LONG: &str = "Rule ID must be at most 255 characters";
 const ERR_LIFECYCLE_INVALID_RULE_STATUS: &str = "Rule status must be either Enabled or Disabled";
+const ERR_LIFECYCLE_DEL_MARKER_WITH_TAGS: &str = "Rule with DelMarkerExpiration cannot have tags based filtering";
+const ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION: &str = "Rule must have at least one of Expiration, Transition, NoncurrentVersionExpiration, NoncurrentVersionTransition, or DelMarkerExpiration";
 
 pub use rustfs_common::metrics::IlmAction;
 
@@ -117,23 +119,66 @@ impl RuleValidate for LifecycleRule {
     }*/
 
     fn validate(&self) -> Result<(), std::io::Error> {
-        /*self.validate_id()?;
-        self.validate_status()?;
-        self.validate_expiration()?;
-        self.validate_noncurrent_expiration()?;
-        self.validate_prefix_and_filter()?;
-        self.validate_transition()?;
-        self.validate_noncurrent_transition()?;
-        if (!self.Filter.Tag.IsEmpty() || len(self.Filter.And.Tags) != 0) && !self.delmarker_expiration.Empty() {
-          return errInvalidRuleDelMarkerExpiration
+        // Rule with DelMarkerExpiration cannot have tags based filtering
+        let has_tag_filter = self
+            .filter
+            .as_ref()
+            .map_or(false, |f| f.tag.is_some() || f.and.as_ref().and_then(|a| a.tags.as_ref()).is_some());
+        if has_tag_filter && self.del_marker_expiration.is_some() {
+            return Err(std::io::Error::other(ERR_LIFECYCLE_DEL_MARKER_WITH_TAGS));
+        }
+        // Rule must have at least one action
+        let has_expiration = self.expiration.is_some();
+        let has_transition = self.transitions.as_ref().map_or(false, |t| !t.is_empty());
+        let has_noncurrent_expiration = self
+            .noncurrent_version_expiration
+            .as_ref()
+            .and_then(|e| e.noncurrent_days)
+            .map_or(false, |d| d != 0);
+        let has_noncurrent_transition = self
+            .noncurrent_version_transitions
+            .as_ref()
+            .and_then(|t| t.first())
+            .and_then(|t| t.storage_class.as_ref())
+            .is_some();
+        let has_abort_incomplete_multipart_upload = self.abort_incomplete_multipart_upload.is_some();
+        let has_del_marker_expiration = self
+            .del_marker_expiration
+            .as_ref()
+            .and_then(|d| d.days)
+            .map_or(false, |d| d > 0);
+        if !has_expiration
+            && !has_transition
+            && !has_noncurrent_expiration
+            && !has_noncurrent_transition
+            && !has_abort_incomplete_multipart_upload
+            && !has_del_marker_expiration
+        {
+            return Err(std::io::Error::other(ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION));
         }
-        if !self.expiration.set && !self.transition.set && !self.noncurrent_version_expiration.set && !self.noncurrent_version_transitions.unwrap()[0].set && self.delmarker_expiration.Empty() {
-          return errXMLNotWellFormed
-        }*/
         Ok(())
     }
 }
 
+fn lifecycle_rule_prefix(rule: &LifecycleRule) -> Option<&str> {
+    // Prefer a non-empty legacy prefix; treat an empty legacy prefix as if it were not set
+    if let Some(p) = rule.prefix.as_deref() {
+        if !p.is_empty() {
+            return Some(p);
+        }
+    }
+
+    let Some(filter) = rule.filter.as_ref() else {
+        return None;
+    };
+
+    if let Some(p) = filter.prefix.as_deref() {
+        return Some(p);
+    }
+
+    filter.and.as_ref().and_then(|and| and.prefix.as_deref())
+}
+
 #[async_trait::async_trait]
 pub trait Lifecycle {
     async fn has_transition(&self) -> bool;
@@ -177,8 +222,11 @@ impl Lifecycle for BucketLifecycleConfiguration {
                 continue;
             }
 
-            let rule_prefix = &rule.prefix.clone().unwrap_or_default();
-            if prefix.len() > 0 && rule_prefix.len() > 0 && !prefix.starts_with(rule_prefix) && !rule_prefix.starts_with(&prefix)
+            let rule_prefix = lifecycle_rule_prefix(rule).unwrap_or("");
+            if !prefix.is_empty()
+                && !rule_prefix.is_empty()
+                && !prefix.starts_with(rule_prefix)
+                && !rule_prefix.starts_with(prefix)
             {
                 continue;
             }
@@ -297,8 +345,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
             if rule.status.as_str() == ExpirationStatus::DISABLED {
                 continue;
             }
-            if let Some(prefix) = rule.prefix.clone() {
-                if !obj.name.starts_with(prefix.as_str()) {
+            if let Some(rule_prefix) = lifecycle_rule_prefix(rule) {
+                if !obj.name.starts_with(rule_prefix) {
                     continue;
                 }
             }
@@ -414,43 +462,32 @@ impl Lifecycle for BucketLifecycleConfiguration {
 
         if let Some(ref lc_rules) = self.filter_rules(obj).await {
             for rule in lc_rules.iter() {
-                if obj.expired_object_deletemarker() {
+                if obj.is_latest && obj.expired_object_deletemarker() {
                     if let Some(expiration) = rule.expiration.as_ref() {
-                        if let Some(expired_object_delete_marker) = expiration.expired_object_delete_marker {
-                            events.push(Event {
-                                action: IlmAction::DeleteVersionAction,
-                                rule_id: rule.id.clone().unwrap_or_default(),
-                                due: Some(now),
-                                noncurrent_days: 0,
-                                newer_noncurrent_versions: 0,
-                                storage_class: "".into(),
-                            });
-                            break;
-                        }
-
-                        if let Some(days) = expiration.days {
-                            let expected_expiry = expected_expiry_time(mod_time, days /*, date*/);
-                            if now.unix_timestamp() >= expected_expiry.unix_timestamp() {
+                        if expiration.expired_object_delete_marker.is_some_and(|v| v) {
+                            // Preserve explicit date/days scheduling when configured.
+                            // If only ExpiredObjectDeleteMarker=true is set, delete immediately.
+                            let due = expiration.next_due(obj).unwrap_or(now);
+                            if now.unix_timestamp() >= due.unix_timestamp() {
                                 events.push(Event {
                                     action: IlmAction::DeleteVersionAction,
                                     rule_id: rule.id.clone().unwrap_or_default(),
-                                    due: Some(expected_expiry),
+                                    due: Some(due),
                                     noncurrent_days: 0,
                                     newer_noncurrent_versions: 0,
                                     storage_class: "".into(),
                                 });
+                                // Stop after scheduling an expired delete-marker event.
                                 break;
                             }
                         }
                     }
-                }
-
-                if obj.is_latest {
-                    if let Some(ref expiration) = rule.expiration {
-                        if let Some(expired_object_delete_marker) = expiration.expired_object_delete_marker {
-                            if obj.delete_marker && expired_object_delete_marker {
-                                let due = expiration.next_due(obj);
-                                if let Some(due) = due {
+                    // DelMarkerExpiration: expire delete marker after N days from mod_time
+                    if obj.delete_marker {
+                        if let Some(ref dme) = rule.del_marker_expiration {
+                            if let Some(days) = dme.days {
+                                if days > 0 {
+                                    let due = expected_expiry_time(mod_time, days);
                                     if now.unix_timestamp() >= due.unix_timestamp() {
                                         events.push(Event {
                                             action: IlmAction::DelMarkerDeleteAllVersionsAction,
@@ -461,8 +498,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
                                             storage_class: "".into(),
                                         });
                                     }
+                                    continue;
                                 }
-                                continue;
                             }
                         }
                     }
@@ -694,8 +731,16 @@ impl LifecycleCalculate for LifecycleExpiration {
         if !obj.is_latest || !obj.delete_marker {
             return None;
         }
+        // Check date first (date-based expiration takes priority over days).
+        // A zero unix timestamp means "not set" (default value) and is skipped.
+        if let Some(ref date) = self.date {
+            let expiry_date = OffsetDateTime::from(date.clone());
+            if expiry_date.unix_timestamp() != 0 {
+                return Some(expiry_date);
+            }
+        }
         match self.days {
-            Some(days) => Some(expected_expiry_time(obj.mod_time.unwrap(), days)),
+            Some(days) => obj.mod_time.map(|mod_time| expected_expiry_time(mod_time, days)),
             None => None,
         }
     }
@@ -723,7 +768,7 @@ impl LifecycleCalculate for NoncurrentVersionTransition {
 #[async_trait::async_trait]
 impl LifecycleCalculate for Transition {
     fn next_due(&self, obj: &ObjectOpts) -> Option<OffsetDateTime> {
-        if !obj.is_latest || self.days.is_none() {
+        if !obj.is_latest {
             return None;
         }
 
@@ -860,10 +905,15 @@ impl Default for TransitionOptions {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use s3s::dto::LifecycleRuleFilter;
+    use serial_test::serial;
+    use std::sync::Arc;
 
     #[tokio::test]
+    #[serial]
     async fn validate_rejects_non_positive_expiration_days() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -871,6 +921,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -889,8 +940,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_accepts_positive_expiration_days() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -898,6 +951,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -913,8 +967,36 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
+    async fn validate_accepts_abort_incomplete_multipart_upload_only_rule() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: Some(s3s::dto::AbortIncompleteMultipartUpload {
+                    days_after_initiation: Some(2),
+                }),
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("abort-only".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: Some("test/".to_string()),
+                transitions: None,
+            }],
+        };
+
+        lc.validate(&ObjectLockConfiguration::default())
+            .await
+            .expect("expected validation to pass");
+    }
+
+    #[tokio::test]
+    #[serial]
     async fn validate_rejects_non_midnight_expiration_date() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -922,6 +1004,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -937,9 +1020,11 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn predict_expiration_selects_closest_expiry_for_put_object() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![
                 LifecycleRule {
                     status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
@@ -948,6 +1033,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: Some("rule-days".to_string()),
                     noncurrent_version_expiration: None,
@@ -962,6 +1048,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: Some("rule-date".to_string()),
                     noncurrent_version_expiration: None,
@@ -988,8 +1075,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_accepts_multiple_rules_without_ids() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![
                 LifecycleRule {
                     status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
@@ -998,6 +1087,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: None,
                     noncurrent_version_expiration: None,
@@ -1012,6 +1102,7 @@ mod tests {
                         ..Default::default()
                     }),
                     abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
                     filter: None,
                     id: None,
                     noncurrent_version_expiration: None,
@@ -1028,8 +1119,10 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn validate_rejects_rule_id_too_long() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
                 expiration: Some(LifecycleExpiration {
@@ -1037,6 +1130,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: Some("a".repeat(256)),
                 noncurrent_version_expiration: None,
@@ -1052,8 +1146,315 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
+    async fn validate_rejects_duplicate_rule_ids() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![
+                LifecycleRule {
+                    status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                    expiration: Some(LifecycleExpiration {
+                        days: Some(1),
+                        ..Default::default()
+                    }),
+                    abort_incomplete_multipart_upload: None,
+                    del_marker_expiration: None,
+                    filter: None,
+                    id: Some("dup-rule".to_string()),
+                    noncurrent_version_expiration: None,
+                    noncurrent_version_transitions: None,
+                    prefix: None,
+                    transitions: None,
+                },
+                LifecycleRule {
+                    status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                    expiration: Some(LifecycleExpiration {
+                        days: Some(2),
+                        ..Default::default()
+                    }),
+                    abort_incomplete_multipart_upload: None,
+                    filter: None,
+                    id: Some("dup-rule".to_string()),
+                    noncurrent_version_expiration: None,
+                    noncurrent_version_transitions: None,
+                    prefix: None,
+                    transitions: None,
+                    del_marker_expiration: None,
+                },
+            ],
+        };
+
+        let err = lc.validate(&ObjectLockConfiguration::default()).await.unwrap_err();
+        assert_eq!(err.to_string(), ERR_LIFECYCLE_DUPLICATE_ID);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_expires_latest_object_after_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("expire-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteAction);
+        assert_eq!(event.rule_id, "expire-days");
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_keeps_latest_object_before_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(2),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("expire-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::hours(12), 0).await;
+
+        assert_eq!(event.action, IlmAction::NoneAction);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_transitions_latest_object_after_days_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("transition-days".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: Some(vec![Transition {
+                    days: Some(1),
+                    date: None,
+                    storage_class: Some(TransitionStorageClass::from_static("COLDTIER44")),
+                }]),
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            transition_status: "".to_string(),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::TransitionAction);
+        assert_eq!(event.rule_id, "transition-days");
+        assert_eq!(event.storage_class, "COLDTIER44");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_transitions_latest_object_after_date_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let transition_date = base_time - Duration::days(1);
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("transition-date".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: Some(vec![Transition {
+                    days: None,
+                    date: Some(transition_date.into()),
+                    storage_class: Some(TransitionStorageClass::from_static("WARM")),
+                }]),
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            transition_status: "".to_string(),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(1), 0).await;
+
+        assert_eq!(event.action, IlmAction::TransitionAction);
+        assert_eq!(event.rule_id, "transition-date");
+        assert_eq!(event.storage_class, "WARM");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_expires_noncurrent_version_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-expire".to_string()),
+                noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
+                    noncurrent_days: Some(1),
+                    newer_noncurrent_versions: None,
+                }),
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            successor_mod_time: Some(base_time),
+            is_latest: false,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-expire");
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn eval_inner_transitions_noncurrent_version_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-transition".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
+                    noncurrent_days: Some(1),
+                    newer_noncurrent_versions: None,
+                    storage_class: Some(TransitionStorageClass::from_static("COLDTIER44")),
+                }]),
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            successor_mod_time: Some(base_time),
+            is_latest: false,
+            transition_status: "".to_string(),
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+
+        assert_eq!(event.action, IlmAction::TransitionVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-transition");
+        assert_eq!(event.storage_class, "COLDTIER44");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn noncurrent_versions_expiration_limit_returns_configured_limits() {
+        let lc = Arc::new(BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: None,
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
+                filter: None,
+                id: Some("noncurrent-limit".to_string()),
+                noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
+                    noncurrent_days: Some(7),
+                    newer_noncurrent_versions: Some(3),
+                }),
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        });
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: false,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let event = lc.noncurrent_versions_expiration_limit(&opts).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.rule_id, "noncurrent-limit");
+        assert_eq!(event.noncurrent_days, 7);
+        assert_eq!(event.newer_noncurrent_versions, 3);
+    }
+
+    #[tokio::test]
+    #[serial]
     async fn validate_rejects_invalid_status_case_sensitive() {
         let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
             rules: vec![LifecycleRule {
                 status: ExpirationStatus::from_static("enabled"),
                 expiration: Some(LifecycleExpiration {
@@ -1061,6 +1462,7 @@ mod tests {
                     ..Default::default()
                 }),
                 abort_incomplete_multipart_upload: None,
+                del_marker_expiration: None,
                 filter: None,
                 id: None,
                 noncurrent_version_expiration: None,
@@ -1074,4 +1476,260 @@ mod tests {
 
         assert_eq!(err.to_string(), ERR_LIFECYCLE_INVALID_RULE_STATUS);
     }
+
+    #[tokio::test]
+    #[serial]
+    async fn filter_rules_respects_filter_prefix() {
+        let mut filter = LifecycleRuleFilter::default();
+        filter.prefix = Some("prefix".to_string());
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: Some(filter),
+                id: Some("rule".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let match_obj = ObjectOpts {
+            name: "prefix/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let matched = lc.filter_rules(&match_obj).await.unwrap();
+        assert_eq!(matched.len(), 1);
+
+        let non_match_obj = ObjectOpts {
+            name: "other/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let not_matched = lc.filter_rules(&non_match_obj).await.unwrap();
+        assert_eq!(not_matched.len(), 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn filter_rules_respects_filter_and_prefix() {
+        let mut filter = LifecycleRuleFilter::default();
+
+        let mut and = LifecycleRuleAndOperator::default();
+        and.prefix = Some("prefix".to_string());
+        filter.and = Some(and);
+
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: Some(filter),
+                id: Some("rule-and-prefix".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let match_obj = ObjectOpts {
+            name: "prefix/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let matched = lc.filter_rules(&match_obj).await.unwrap();
+        assert_eq!(matched.len(), 1);
+
+        let non_match_obj = ObjectOpts {
+            name: "other/file".to_string(),
+            mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
+            is_latest: true,
+            ..Default::default()
+        };
+        let not_matched = lc.filter_rules(&non_match_obj).await.unwrap();
+        assert_eq!(not_matched.len(), 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expired_object_delete_marker_requires_single_version() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 2,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+        assert_eq!(event.action, IlmAction::NoneAction);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expired_object_delete_marker_deletes_only_delete_marker_after_due() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(1),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+    }
+
+    #[tokio::test]
+    async fn expired_object_delete_marker_without_date_or_days_deletes_immediately() {
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-expired-del-marker-immediate".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        let now = base_time + Duration::days(2);
+        let event = lc.eval_inner(&opts, now, 0).await;
+        assert_eq!(event.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event.due, Some(now));
+    }
+
+    #[tokio::test]
+    async fn expired_object_delete_marker_date_based_not_yet_due() {
+        // A date-based rule that has not yet reached its expiry date must not
+        // trigger immediate deletion (unwrap_or(now) must not override the date).
+        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let future_date = base_time + Duration::days(10);
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    date: Some(future_date.into()),
+                    expired_object_delete_marker: Some(true),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                filter: None,
+                id: Some("rule-date-del-marker".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+                del_marker_expiration: None,
+            }],
+        };
+
+        let opts = ObjectOpts {
+            name: "obj".to_string(),
+            mod_time: Some(base_time),
+            is_latest: true,
+            delete_marker: true,
+            num_versions: 1,
+            version_id: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+
+        // now is before the configured date — must not schedule deletion
+        let now_before = base_time + Duration::days(5);
+        let event_before = lc.eval_inner(&opts, now_before, 0).await;
+        assert_eq!(event_before.action, IlmAction::NoneAction);
+
+        // now is after the configured date — must schedule deletion
+        let now_after = base_time + Duration::days(11);
+        let event_after = lc.eval_inner(&opts, now_after, 0).await;
+        assert_eq!(event_after.action, IlmAction::DeleteVersionAction);
+        assert_eq!(event_after.due, Some(future_date));
+    }
 }

crates/ecstore/src/bucket/lifecycle/tier_last_day_stats.rs

@@ -51,6 +51,10 @@ impl LastDayTierStats {
         self.bins[now_idx] = self.bins[now_idx].add(&ts);
     }
 
+    pub fn total(&self) -> TierStats {
+        self.bins.iter().fold(TierStats::default(), |acc, bin| acc.add(bin))
+    }
+
     fn forward_to(&mut self, t: &mut OffsetDateTime) {
         if t.unix_timestamp() == 0 {
             *t = OffsetDateTime::now_utc();
@@ -99,4 +103,30 @@ impl LastDayTierStats {
 }
 
 #[cfg(test)]
-mod test {}
+mod test {
+    use super::*;
+
+    #[test]
+    fn total_sums_all_recorded_stats() {
+        let mut stats = LastDayTierStats::default();
+        stats.add_stats(TierStats {
+            total_size: 10,
+            num_versions: 1,
+            num_objects: 1,
+        });
+        stats.add_stats(TierStats {
+            total_size: 20,
+            num_versions: 2,
+            num_objects: 0,
+        });
+
+        assert_eq!(
+            stats.total(),
+            TierStats {
+                total_size: 30,
+                num_versions: 3,
+                num_objects: 1,
+            }
+        );
+    }
+}

crates/ecstore/src/bucket/metadata.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
 use super::object_lock::ObjectLockApi;
 use super::versioning::VersioningApi;
 use super::{quota::BucketQuota, target::BucketTargets};
@@ -22,7 +23,6 @@ use crate::error::{Error, Result};
 use crate::new_object_layer_fn;
 use crate::store::ECStore;
 use byteorder::{BigEndian, ByteOrder, LittleEndian};
-use rmp_serde::Serializer as rmpSerializer;
 use rustfs_policy::policy::BucketPolicy;
 use s3s::dto::{
     BucketLifecycleConfiguration, CORSConfiguration, NotificationConfiguration, ObjectLockConfiguration,
@@ -30,12 +30,199 @@ use s3s::dto::{
     VersioningConfiguration,
 };
 use serde::Serializer;
-use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
+use std::io::{Read, Write};
 use std::sync::Arc;
-use time::OffsetDateTime;
+use time::{Date, OffsetDateTime, PrimitiveDateTime, Time as CivilTime, UtcOffset};
 use tracing::error;
 
+fn read_msgp_str<R: Read>(rd: &mut R) -> Result<String> {
+    let len = rmp::decode::read_str_len(rd)? as usize;
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    Ok(String::from_utf8(buf)?)
+}
+
+fn read_msgp_bool<R: Read>(rd: &mut R) -> Result<bool> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    match marker {
+        rmp::Marker::True => Ok(true),
+        rmp::Marker::False => Ok(false),
+        rmp::Marker::FixPos(v) => Ok(v != 0),
+        rmp::Marker::U8 => Ok(read_u8(rd)? != 0),
+        rmp::Marker::U16 => Ok(read_u16_raw(rd)? != 0),
+        rmp::Marker::U32 => Ok(read_u32_raw(rd)? != 0),
+        rmp::Marker::U64 => Ok(read_u64_raw(rd)? != 0),
+        rmp::Marker::I8 => Ok(read_i8_raw(rd)? != 0),
+        rmp::Marker::I16 => Ok(read_i16_raw(rd)? != 0),
+        rmp::Marker::I32 => Ok(read_i32_raw(rd)? != 0),
+        rmp::Marker::I64 => Ok(read_i64_raw(rd)? != 0),
+        rmp::Marker::FixNeg(v) => Ok(v != 0),
+        _ => Err(Error::other(format!("expected bool or int-like bool, got marker: {marker:?}"))),
+    }
+}
+
+fn read_msgp_time_value<R: Read>(rd: &mut R) -> Result<OffsetDateTime> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    match marker {
+        rmp::Marker::Null => Ok(OffsetDateTime::UNIX_EPOCH),
+        rmp::Marker::Ext8 => read_msgp_ext8_time(rd),
+        rmp::Marker::FixArray(len) => read_msgp_legacy_compact_time(rd, u32::from(len)),
+        rmp::Marker::Array16 => {
+            let len = read_u16_raw(rd)?;
+            read_msgp_legacy_compact_time(rd, u32::from(len))
+        }
+        rmp::Marker::Array32 => {
+            let len = read_u32_raw(rd)?;
+            read_msgp_legacy_compact_time(rd, len)
+        }
+        rmp::Marker::Bin8 => {
+            let len = usize::from(read_u8(rd)?);
+            read_msgp_time_value_from_embedded_bin(rd, len)
+        }
+        rmp::Marker::Bin16 => {
+            let len = usize::from(read_u16_raw(rd)?);
+            read_msgp_time_value_from_embedded_bin(rd, len)
+        }
+        rmp::Marker::Bin32 => {
+            let len = read_u32_raw(rd)? as usize;
+            read_msgp_time_value_from_embedded_bin(rd, len)
+        }
+        _ => Err(Error::other(format!("expected time ext or nil, got marker: {marker:?}"))),
+    }
+}
+
+fn read_u8<R: Read>(rd: &mut R) -> Result<u8> {
+    let mut buf = [0u8; 1];
+    rd.read_exact(&mut buf)?;
+    Ok(buf[0])
+}
+
+fn read_u16_raw<R: Read>(rd: &mut R) -> Result<u16> {
+    let mut buf = [0u8; 2];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_u16(&buf))
+}
+
+fn read_u32_raw<R: Read>(rd: &mut R) -> Result<u32> {
+    let mut buf = [0u8; 4];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_u32(&buf))
+}
+
+fn read_u64_raw<R: Read>(rd: &mut R) -> Result<u64> {
+    let mut buf = [0u8; 8];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_u64(&buf))
+}
+
+fn read_i8_raw<R: Read>(rd: &mut R) -> Result<i8> {
+    Ok(read_u8(rd)? as i8)
+}
+
+fn read_i16_raw<R: Read>(rd: &mut R) -> Result<i16> {
+    let mut buf = [0u8; 2];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_i16(&buf))
+}
+
+fn read_i32_raw<R: Read>(rd: &mut R) -> Result<i32> {
+    let mut buf = [0u8; 4];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_i32(&buf))
+}
+
+fn read_i64_raw<R: Read>(rd: &mut R) -> Result<i64> {
+    let mut buf = [0u8; 8];
+    rd.read_exact(&mut buf)?;
+    Ok(BigEndian::read_i64(&buf))
+}
+
+fn read_msgp_time_value_from_embedded_bin<R: Read>(rd: &mut R, len: usize) -> Result<OffsetDateTime> {
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    let mut cur = std::io::Cursor::new(buf);
+    read_msgp_time_value(&mut cur)
+}
+
+fn read_msgp_legacy_compact_time<R: Read>(rd: &mut R, len: u32) -> Result<OffsetDateTime> {
+    if len != 9 {
+        return Err(Error::other(format!("invalid legacy compact time len: {len}")));
+    }
+
+    let year: i32 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let ordinal: u16 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let hour: u8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let minute: u8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let second: u8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let nanosecond: u32 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let offset_hour: i8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let offset_minute: i8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let offset_second: i8 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+
+    let date =
+        Date::from_ordinal_date(year, ordinal).map_err(|e| Error::other(format!("invalid legacy compact time date: {e}")))?;
+    let time = CivilTime::from_hms_nano(hour, minute, second, nanosecond)
+        .map_err(|e| Error::other(format!("invalid legacy compact time time: {e}")))?;
+    let offset = UtcOffset::from_hms(offset_hour, offset_minute, offset_second)
+        .map_err(|e| Error::other(format!("invalid legacy compact time offset: {e}")))?;
+
+    Ok(PrimitiveDateTime::new(date, time)
+        .assume_offset(offset)
+        .to_offset(UtcOffset::UTC))
+}
+
+fn read_msgp_bin<R: Read>(rd: &mut R) -> Result<Vec<u8>> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    match marker {
+        rmp::Marker::Null => Ok(Vec::new()),
+        rmp::Marker::Bin8 => {
+            let len = usize::from(read_u8(rd)?);
+            read_exact_bytes(rd, len)
+        }
+        rmp::Marker::Bin16 => {
+            let len = usize::from(read_u16_raw(rd)?);
+            read_exact_bytes(rd, len)
+        }
+        rmp::Marker::Bin32 => {
+            let len = read_u32_raw(rd)? as usize;
+            read_exact_bytes(rd, len)
+        }
+        rmp::Marker::FixArray(len) => read_msgp_legacy_byte_array(rd, u32::from(len)),
+        rmp::Marker::Array16 => {
+            let len = read_u16_raw(rd)?;
+            read_msgp_legacy_byte_array(rd, u32::from(len))
+        }
+        rmp::Marker::Array32 => {
+            let len = read_u32_raw(rd)?;
+            read_msgp_legacy_byte_array(rd, len)
+        }
+        _ => Err(Error::other(format!("expected bin or byte array, got marker: {marker:?}"))),
+    }
+}
+
+fn read_exact_bytes<R: Read>(rd: &mut R, len: usize) -> Result<Vec<u8>> {
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    Ok(buf)
+}
+
+fn read_msgp_legacy_byte_array<R: Read>(rd: &mut R, len: u32) -> Result<Vec<u8>> {
+    let mut buf = Vec::with_capacity(len as usize);
+    for _ in 0..len {
+        let value: i64 = rmp::decode::read_int(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+        let byte = u8::try_from(value).map_err(|_| Error::other(format!("byte value out of range: {value}")))?;
+        buf.push(byte);
+    }
+    Ok(buf)
+}
+
+fn write_bin_field<W: Write>(wr: &mut W, key: &str, val: &[u8]) -> Result<()> {
+    rmp::encode::write_str(wr, key)?;
+    rmp::encode::write_bin(wr, val)?;
+    Ok(())
+}
+
 pub const BUCKET_METADATA_FILE: &str = ".metadata.bin";
 pub const BUCKET_METADATA_FORMAT: u16 = 1;
 pub const BUCKET_METADATA_VERSION: u16 = 1;
@@ -54,8 +241,7 @@ pub const BUCKET_CORS_CONFIG: &str = "cors.xml";
 pub const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = "public-access-block.xml";
 pub const BUCKET_ACL_CONFIG: &str = "bucket-acl.json";
 
-#[derive(Debug, Deserialize, Serialize, Clone)]
-#[serde(rename_all = "PascalCase", default)]
+#[derive(Debug, Clone)]
 pub struct BucketMetadata {
     pub name: String,
     pub created: OffsetDateTime,
@@ -90,36 +276,21 @@ pub struct BucketMetadata {
     pub public_access_block_config_updated_at: OffsetDateTime,
     pub bucket_acl_config_updated_at: OffsetDateTime,
 
-    #[serde(skip)]
     pub new_field_updated_at: OffsetDateTime,
 
-    #[serde(skip)]
     pub policy_config: Option<BucketPolicy>,
-    #[serde(skip)]
     pub notification_config: Option<NotificationConfiguration>,
-    #[serde(skip)]
     pub lifecycle_config: Option<BucketLifecycleConfiguration>,
-    #[serde(skip)]
     pub object_lock_config: Option<ObjectLockConfiguration>,
-    #[serde(skip)]
     pub versioning_config: Option<VersioningConfiguration>,
-    #[serde(skip)]
     pub sse_config: Option<ServerSideEncryptionConfiguration>,
-    #[serde(skip)]
     pub tagging_config: Option<Tagging>,
-    #[serde(skip)]
     pub quota_config: Option<BucketQuota>,
-    #[serde(skip)]
     pub replication_config: Option<ReplicationConfiguration>,
-    #[serde(skip)]
     pub bucket_target_config: Option<BucketTargets>,
-    #[serde(skip)]
     pub bucket_target_config_meta: Option<HashMap<String, String>>,
-    #[serde(skip)]
     pub cors_config: Option<CORSConfiguration>,
-    #[serde(skip)]
     pub public_access_block_config: Option<PublicAccessBlockConfiguration>,
-    #[serde(skip)]
     pub bucket_acl_config: Option<String>,
 }
 
@@ -198,17 +369,141 @@ impl BucketMetadata {
         self.lock_enabled || (self.versioning_config.as_ref().is_some_and(|v| v.enabled()))
     }
 
+    /// Decode from msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
+    pub fn decode_from<R: Read>(&mut self, rd: &mut R) -> Result<()> {
+        let mut fields = rmp::decode::read_map_len(rd)?;
+        *self = Self::default();
+
+        while fields > 0 {
+            fields -= 1;
+
+            let key_len = rmp::decode::read_str_len(rd)?;
+            let mut key_buf = vec![0u8; key_len as usize];
+            rd.read_exact(&mut key_buf)?;
+            let key = String::from_utf8(key_buf)?;
+
+            match key.as_str() {
+                "Name" => self.name = read_msgp_str(rd)?,
+                "Created" => self.created = read_msgp_time_value(rd)?,
+                "LockEnabled" => self.lock_enabled = read_msgp_bool(rd)?,
+                "PolicyConfigJSON" | "PolicyConfigJson" => self.policy_config_json = read_msgp_bin(rd)?,
+                "NotificationConfigXML" | "NotificationConfigXml" => self.notification_config_xml = read_msgp_bin(rd)?,
+                "LifecycleConfigXML" | "LifecycleConfigXml" => self.lifecycle_config_xml = read_msgp_bin(rd)?,
+                "ObjectLockConfigXML" | "ObjectLockConfigXml" => self.object_lock_config_xml = read_msgp_bin(rd)?,
+                "VersioningConfigXML" | "VersioningConfigXml" => self.versioning_config_xml = read_msgp_bin(rd)?,
+                "EncryptionConfigXML" | "EncryptionConfigXml" => self.encryption_config_xml = read_msgp_bin(rd)?,
+                "TaggingConfigXML" | "TaggingConfigXml" => self.tagging_config_xml = read_msgp_bin(rd)?,
+                "QuotaConfigJSON" | "QuotaConfigJson" => self.quota_config_json = read_msgp_bin(rd)?,
+                "ReplicationConfigXML" | "ReplicationConfigXml" => self.replication_config_xml = read_msgp_bin(rd)?,
+                "BucketTargetsConfigJSON" | "BucketTargetsConfigJson" => self.bucket_targets_config_json = read_msgp_bin(rd)?,
+                "BucketTargetsConfigMetaJSON" | "BucketTargetsConfigMetaJson" => {
+                    self.bucket_targets_config_meta_json = read_msgp_bin(rd)?
+                }
+                "PolicyConfigUpdatedAt" => self.policy_config_updated_at = read_msgp_time_value(rd)?,
+                "ObjectLockConfigUpdatedAt" => self.object_lock_config_updated_at = read_msgp_time_value(rd)?,
+                "EncryptionConfigUpdatedAt" => self.encryption_config_updated_at = read_msgp_time_value(rd)?,
+                "TaggingConfigUpdatedAt" => self.tagging_config_updated_at = read_msgp_time_value(rd)?,
+                "QuotaConfigUpdatedAt" => self.quota_config_updated_at = read_msgp_time_value(rd)?,
+                "ReplicationConfigUpdatedAt" => self.replication_config_updated_at = read_msgp_time_value(rd)?,
+                "VersioningConfigUpdatedAt" => self.versioning_config_updated_at = read_msgp_time_value(rd)?,
+                "LifecycleConfigUpdatedAt" => self.lifecycle_config_updated_at = read_msgp_time_value(rd)?,
+                "NotificationConfigUpdatedAt" => self.notification_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketTargetsConfigUpdatedAt" => self.bucket_targets_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketTargetsConfigMetaUpdatedAt" => self.bucket_targets_config_meta_updated_at = read_msgp_time_value(rd)?,
+                "CorsConfigXML" | "CorsConfigXml" => self.cors_config_xml = read_msgp_bin(rd)?,
+                "PublicAccessBlockConfigXML" | "PublicAccessBlockConfigXml" => {
+                    self.public_access_block_config_xml = read_msgp_bin(rd)?
+                }
+                "BucketAclConfigJSON" | "BucketAclConfigJson" => self.bucket_acl_config_json = read_msgp_bin(rd)?,
+                "CorsConfigUpdatedAt" => self.cors_config_updated_at = read_msgp_time_value(rd)?,
+                "PublicAccessBlockConfigUpdatedAt" => self.public_access_block_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketAclConfigUpdatedAt" => self.bucket_acl_config_updated_at = read_msgp_time_value(rd)?,
+                other => {
+                    tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
+                    skip_msgp_value(rd)?;
+                }
+            }
+        }
+
+        Ok(())
+    }
+
+    /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
+    pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
+        // Map size: MinIO fields (25) + RustFS extensions (6)
+        let map_len: u32 = 31;
+        rmp::encode::write_map_len(wr, map_len)?;
+
+        // MinIO field order (same as Go struct)
+        rmp::encode::write_str(wr, "Name")?;
+        rmp::encode::write_str(wr, &self.name)?;
+
+        rmp::encode::write_str(wr, "Created")?;
+        write_msgp_time(wr, self.created)?;
+
+        rmp::encode::write_str(wr, "LockEnabled")?;
+        rmp::encode::write_bool(wr, self.lock_enabled)?;
+
+        write_bin_field(wr, "PolicyConfigJSON", &self.policy_config_json)?;
+        write_bin_field(wr, "NotificationConfigXML", &self.notification_config_xml)?;
+        write_bin_field(wr, "LifecycleConfigXML", &self.lifecycle_config_xml)?;
+        write_bin_field(wr, "ObjectLockConfigXML", &self.object_lock_config_xml)?;
+        write_bin_field(wr, "VersioningConfigXML", &self.versioning_config_xml)?;
+        write_bin_field(wr, "EncryptionConfigXML", &self.encryption_config_xml)?;
+        write_bin_field(wr, "TaggingConfigXML", &self.tagging_config_xml)?;
+        write_bin_field(wr, "QuotaConfigJSON", &self.quota_config_json)?;
+        write_bin_field(wr, "ReplicationConfigXML", &self.replication_config_xml)?;
+        write_bin_field(wr, "BucketTargetsConfigJSON", &self.bucket_targets_config_json)?;
+        write_bin_field(wr, "BucketTargetsConfigMetaJSON", &self.bucket_targets_config_meta_json)?;
+
+        rmp::encode::write_str(wr, "PolicyConfigUpdatedAt")?;
+        write_msgp_time(wr, self.policy_config_updated_at)?;
+        rmp::encode::write_str(wr, "ObjectLockConfigUpdatedAt")?;
+        write_msgp_time(wr, self.object_lock_config_updated_at)?;
+        rmp::encode::write_str(wr, "EncryptionConfigUpdatedAt")?;
+        write_msgp_time(wr, self.encryption_config_updated_at)?;
+        rmp::encode::write_str(wr, "TaggingConfigUpdatedAt")?;
+        write_msgp_time(wr, self.tagging_config_updated_at)?;
+        rmp::encode::write_str(wr, "QuotaConfigUpdatedAt")?;
+        write_msgp_time(wr, self.quota_config_updated_at)?;
+        rmp::encode::write_str(wr, "ReplicationConfigUpdatedAt")?;
+        write_msgp_time(wr, self.replication_config_updated_at)?;
+        rmp::encode::write_str(wr, "VersioningConfigUpdatedAt")?;
+        write_msgp_time(wr, self.versioning_config_updated_at)?;
+        rmp::encode::write_str(wr, "LifecycleConfigUpdatedAt")?;
+        write_msgp_time(wr, self.lifecycle_config_updated_at)?;
+        rmp::encode::write_str(wr, "NotificationConfigUpdatedAt")?;
+        write_msgp_time(wr, self.notification_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketTargetsConfigUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_targets_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketTargetsConfigMetaUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_targets_config_meta_updated_at)?;
+
+        // RustFS extensions
+        write_bin_field(wr, "CorsConfigXML", &self.cors_config_xml)?;
+        write_bin_field(wr, "PublicAccessBlockConfigXML", &self.public_access_block_config_xml)?;
+        write_bin_field(wr, "BucketAclConfigJSON", &self.bucket_acl_config_json)?;
+        rmp::encode::write_str(wr, "CorsConfigUpdatedAt")?;
+        write_msgp_time(wr, self.cors_config_updated_at)?;
+        rmp::encode::write_str(wr, "PublicAccessBlockConfigUpdatedAt")?;
+        write_msgp_time(wr, self.public_access_block_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketAclConfigUpdatedAt")?;
+        write_msgp_time(wr, self.bucket_acl_config_updated_at)?;
+
+        Ok(())
+    }
+
     pub fn marshal_msg(&self) -> Result<Vec<u8>> {
         let mut buf = Vec::new();
-
-        self.serialize(&mut rmpSerializer::new(&mut buf).with_struct_map())?;
-
+        self.encode_to(&mut buf)?;
         Ok(buf)
     }
 
     pub fn unmarshal(buf: &[u8]) -> Result<Self> {
-        let t: BucketMetadata = rmp_serde::from_slice(buf)?;
-        Ok(t)
+        let mut bm = Self::default();
+        let mut cur = std::io::Cursor::new(buf);
+        bm.decode_from(&mut cur)?;
+        Ok(bm)
     }
 
     pub fn check_header(buf: &[u8]) -> Result<()> {
@@ -382,50 +677,80 @@ impl BucketMetadata {
     }
 
     fn parse_all_configs(&mut self, _api: Arc<ECStore>) -> Result<()> {
-        self.parse_policy_config()?;
-        if !self.notification_config_xml.is_empty() {
-            self.notification_config = Some(deserialize::<NotificationConfiguration>(&self.notification_config_xml)?);
+        if let Err(e) = self.parse_policy_config() {
+            tracing::warn!(bucket = %self.name, config = "policy", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.lifecycle_config_xml.is_empty() {
-            self.lifecycle_config = Some(deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml)?);
+        if !self.notification_config_xml.is_empty()
+            && let Err(e) = deserialize::<NotificationConfiguration>(&self.notification_config_xml)
+                .map(|c| self.notification_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "notification", error = %e, "parse_all_configs: failed to parse");
         }
-
-        if !self.object_lock_config_xml.is_empty() {
-            self.object_lock_config = Some(deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml)?);
+        if !self.lifecycle_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "lifecycle", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.versioning_config_xml.is_empty() {
-            self.versioning_config = Some(deserialize::<VersioningConfiguration>(&self.versioning_config_xml)?);
+        if !self.object_lock_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml).map(|c| self.object_lock_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "object_lock", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.encryption_config_xml.is_empty() {
-            self.sse_config = Some(deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml)?);
+        if !self.versioning_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<VersioningConfiguration>(&self.versioning_config_xml).map(|c| self.versioning_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "versioning", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.tagging_config_xml.is_empty() {
-            self.tagging_config = Some(deserialize::<Tagging>(&self.tagging_config_xml)?);
+        if !self.encryption_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml).map(|c| self.sse_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "encryption", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.quota_config_json.is_empty() {
-            self.quota_config = Some(serde_json::from_slice(&self.quota_config_json)?);
+        if !self.tagging_config_xml.is_empty()
+            && let Err(e) = deserialize::<Tagging>(&self.tagging_config_xml).map(|c| self.tagging_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "tagging", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.replication_config_xml.is_empty() {
-            self.replication_config = Some(deserialize::<ReplicationConfiguration>(&self.replication_config_xml)?);
+        if !self.quota_config_json.is_empty()
+            && let Err(e) = serde_json::from_slice(&self.quota_config_json).map(|c| self.quota_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "quota", error = %e, "parse_all_configs: failed to parse");
+        }
+        if !self.replication_config_xml.is_empty()
+            && let Err(e) =
+                deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "replication", error = %e, "parse_all_configs: failed to parse");
         }
-        //let temp = self.bucket_targets_config_json.clone();
         if !self.bucket_targets_config_json.is_empty() {
-            let bucket_targets: BucketTargets = serde_json::from_slice(&self.bucket_targets_config_json)?;
-            self.bucket_target_config = Some(bucket_targets);
+            if let Err(e) = serde_json::from_slice::<BucketTargets>(&self.bucket_targets_config_json)
+                .map(|t| self.bucket_target_config = Some(t))
+            {
+                tracing::warn!(bucket = %self.name, config = "bucket_targets", error = %e, "parse_all_configs: failed to parse");
+                self.bucket_target_config = Some(BucketTargets::default());
+            }
         } else {
-            self.bucket_target_config = Some(BucketTargets::default())
+            self.bucket_target_config = Some(BucketTargets::default());
         }
-        if !self.cors_config_xml.is_empty() {
-            self.cors_config = Some(deserialize::<CORSConfiguration>(&self.cors_config_xml)?);
+        if !self.cors_config_xml.is_empty()
+            && let Err(e) = deserialize::<CORSConfiguration>(&self.cors_config_xml).map(|c| self.cors_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "cors", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.public_access_block_config_xml.is_empty() {
-            self.public_access_block_config =
-                Some(deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)?);
+        if !self.public_access_block_config_xml.is_empty()
+            && let Err(e) = deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)
+                .map(|c| self.public_access_block_config = Some(c))
+        {
+            tracing::warn!(bucket = %self.name, config = "public_access_block", error = %e, "parse_all_configs: failed to parse");
         }
-        if !self.bucket_acl_config_json.is_empty() {
-            let acl = String::from_utf8(self.bucket_acl_config_json.clone())
-                .map_err(|e| Error::other(format!("invalid UTF-8 in bucket ACL: {}", e)))?;
-            self.bucket_acl_config = Some(acl);
+        if !self.bucket_acl_config_json.is_empty()
+            && let Err(e) = String::from_utf8(self.bucket_acl_config_json.clone()).map(|acl| self.bucket_acl_config = Some(acl))
+        {
+            tracing::warn!(bucket = %self.name, config = "bucket_acl", error = %e, "parse_all_configs: failed to parse");
         }
 
         Ok(())
@@ -478,7 +803,6 @@ async fn read_bucket_metadata(api: Arc<ECStore>, bucket: &str) -> Result<BucketM
 
     Ok(bm)
 }
-
 fn _write_time<S>(t: &OffsetDateTime, s: S) -> std::result::Result<S::Ok, S::Error>
 where
     S: Serializer,

crates/ecstore/src/bucket/metadata_sys.rs

@@ -360,12 +360,14 @@ impl BucketMetadataSys {
             };
 
             if !meta.lifecycle_config_xml.is_empty() {
-                let cfg = deserialize::<BucketLifecycleConfiguration>(&meta.lifecycle_config_xml)?;
-                // TODO: FIXME:
-                // for _v in cfg.rules.iter() {
-                //     break;
-                // }
-                if let Some(_v) = cfg.rules.first() {}
+                if let Ok(cfg) = deserialize::<BucketLifecycleConfiguration>(&meta.lifecycle_config_xml) {
+                    if let Some(_v) = cfg.rules.first() {}
+                } else {
+                    tracing::warn!(
+                        bucket = %bucket,
+                        "delete: failed to parse lifecycle config XML"
+                    );
+                }
             }
 
             // TODO: other lifecycle handle

crates/ecstore/src/bucket/metadata_test.rs

@@ -0,0 +1,398 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::metadata::BucketMetadata;
+use time::OffsetDateTime;
+
+/// Full BucketMetadata hex (all fields populated).
+const TEST_BUCKET_METADATA_HEX: &str = "de0019a44e616d65b27275737466732d636f6d7061742d74657374a743726561746564c70c050000000065920080075bcd15ab4c6f636b456e61626c6564c3b0506f6c696379436f6e6669674a534f4ec4907b2256657273696f6e223a22323031322d31302d3137222c2253746174656d656e74223a5b7b22456666656374223a22416c6c6f77222c225072696e636970616c223a222a222c22416374696f6e223a2273333a4765744f626a656374222c225265736f75726365223a2261726e3a6177733a73333a3a3a7275737466732d636f6d7061742d746573742f2a227d5d7db54e6f74696669636174696f6e436f6e666967584d4cc4963c4e6f74696669636174696f6e436f6e66696775726174696f6e3e3c436c6f75645761746368436f6e66696775726174696f6e3e3c49643e6e313c2f49643e3c4576656e743e73333a4f626a656374437265617465643a2a3c2f4576656e743e3c2f436c6f75645761746368436f6e66696775726174696f6e3e3c2f4e6f74696669636174696f6e436f6e66696775726174696f6e3eb24c6966656379636c65436f6e666967584d4cc48c3c4c6966656379636c65436f6e66696775726174696f6e3e3c52756c653e3c49443e72756c65313c2f49443e3c5374617475733e456e61626c65643c2f5374617475733e3c45787069726174696f6e3e3c446179733e33303c2f446179733e3c2f45787069726174696f6e3e3c2f52756c653e3c2f4c6966656379636c65436f6e66696775726174696f6e3eb34f626a6563744c6f636b436f6e666967584d4cc4b83c4f626a6563744c6f636b436f6e66696775726174696f6e3e3c4f626a6563744c6f636b456e61626c65643e456e61626c65643c2f4f626a6563744c6f636b456e61626c65643e3c52756c653e3c44656661756c74526574656e74696f6e3e3c4d6f64653e474f5645524e414e43453c2f4d6f64653e3c446179733e373c2f446179733e3c2f44656661756c74526574656e74696f6e3e3c2f52756c653e3c2f4f626a6563744c6f636b436f6e66696775726174696f6e3eb356657273696f6e696e67436f6e666967584d4cc44b3c56657273696f6e696e67436f6e66696775726174696f6e3e3c5374617475733e456e61626c65643c2f5374617475733e3c2f56657273696f6e696e67436f6e66696775726174696f6e3eb3456e6372797074696f6e436f6e666967584d4cc4c03c53657276657253696465456e6372797074696f6e436f6e66696775726174696f6e3e3c52756c653e3c4170706c7953657276657253696465456e6372797074696f6e427944656661756c743e3c535345416c676f726974686d3e4145533235363c2f535345416c676f726974686d3e3c2f4170706c7953657276657253696465456e6372797074696f6e427944656661756c743e3c2f52756c653e3c2f53657276657253696465456e6372797074696f6e436f6e66696775726174696f6e3eb054616767696e67436f6e666967584d4cc4503c54616767696e673e3c5461675365743e3c5461673e3c4b65793e456e763c2f4b65793e3c56616c75653e546573743c2f56616c75653e3c2f5461673e3c2f5461675365743e3c2f54616767696e673eaf51756f7461436f6e6669674a534f4ec4707b2271756f7461223a313037333734313832342c2271756f74615f74797065223a2248617264222c22637265617465645f6174223a22323032342d30312d30315430303a30303a30305a222c22757064617465645f6174223a22323032342d30312d30315430303a30303a30305a227db45265706c69636174696f6e436f6e666967584d4cc4e73c5265706c69636174696f6e436f6e66696775726174696f6e3e3c526f6c653e61726e3a6177733a69616d3a3a3132333435363738393031323a726f6c652f7265706c3c2f526f6c653e3c52756c653e3c49443e72313c2f49443e3c5374617475733e456e61626c65643c2f5374617475733e3c5072656669783e646f632f3c2f5072656669783e3c44657374696e6174696f6e3e3c4275636b65743e61726e3a6177733a73333a3a3a646573743c2f4275636b65743e3c2f44657374696e6174696f6e3e3c2f52756c653e3c2f5265706c69636174696f6e436f6e66696775726174696f6e3eb74275636b657454617267657473436f6e6669674a534f4ec4535b7b22656e64706f696e74223a22687474703a2f2f7461726765742e6578616d706c652e636f6d222c227461726765744275636b6574223a227462222c22726567696f6e223a2275732d656173742d31227d5dbb4275636b657454617267657473436f6e6669674d6574614a534f4ec42d7b227265706c69636174696f6e4964223a227265706c2d31222c2273796e634d6f6465223a226173796e63227db5506f6c696379436f6e666967557064617465644174c70c050000000065a5022000000000b94f626a6563744c6f636b436f6e666967557064617465644174c70c050000000065a5022000000000b9456e6372797074696f6e436f6e666967557064617465644174c70c050000000065a5022000000000b654616767696e67436f6e666967557064617465644174c70c050000000065a5022000000000b451756f7461436f6e666967557064617465644174c70c050000000065a5022000000000ba5265706c69636174696f6e436f6e666967557064617465644174c70c050000000065a5022000000000b956657273696f6e696e67436f6e666967557064617465644174c70c050000000065a5022000000000b84c6966656379636c65436f6e666967557064617465644174c70c050000000065a5022000000000bb4e6f74696669636174696f6e436f6e666967557064617465644174c70c050000000065a5022000000000bc4275636b657454617267657473436f6e666967557064617465644174c70c050000000065a5022000000000d9204275636b657454617267657473436f6e6669674d657461557064617465644174c70c050000000065a5022000000000";
+
+#[tokio::test]
+async fn marshal_msg() {
+    let bm = BucketMetadata::new("dada");
+
+    let buf = bm.marshal_msg().unwrap();
+
+    let new = BucketMetadata::unmarshal(&buf).unwrap();
+
+    assert_eq!(bm.name, new.name);
+}
+
+/// Verifies that serialized time uses msgp ext type 5.
+#[tokio::test]
+async fn marshal_msg_uses_time_format() {
+    let mut bm = BucketMetadata::new("test-bucket");
+    bm.created = OffsetDateTime::from_unix_timestamp(1704067200).unwrap(); // 2024-01-01 00:00:00 UTC
+
+    let buf = bm.marshal_msg().unwrap();
+
+    // msgp uses ext8 (0xc7), len 12, type 5 for time
+    assert!(
+        buf.windows(3).any(|w| w == [0xc7, 0x0c, 0x05]),
+        "serialized data should contain msgp time ext (0xc7 0x0c 0x05)"
+    );
+}
+
+#[tokio::test]
+async fn unmarshal_test_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let mut bytes = vec![0u8; TEST_BUCKET_METADATA_HEX.len() / 2];
+    hex_decode(TEST_BUCKET_METADATA_HEX.as_bytes(), &mut bytes).expect("valid hex");
+    let bm = BucketMetadata::unmarshal(&bytes).expect("RustFS must unmarshal MinIO format");
+
+    assert_eq!(bm.name, "rustfs-compat-test");
+    assert_eq!(bm.created.unix_timestamp(), 1704067200);
+    assert_eq!(bm.created.nanosecond(), 123456789);
+    assert!(bm.lock_enabled);
+
+    assert!(!bm.policy_config_json.is_empty());
+    assert!(bm.policy_config_json.starts_with(b"{\"Version\""));
+    assert!(!bm.notification_config_xml.is_empty());
+    assert!(bm.notification_config_xml.starts_with(b"<Notification"));
+    assert!(!bm.lifecycle_config_xml.is_empty());
+    assert!(bm.lifecycle_config_xml.starts_with(b"<Lifecycle"));
+    assert!(!bm.object_lock_config_xml.is_empty());
+    assert!(bm.object_lock_config_xml.starts_with(b"<ObjectLock"));
+    assert!(!bm.versioning_config_xml.is_empty());
+    assert!(bm.versioning_config_xml.starts_with(b"<Versioning"));
+    assert!(!bm.encryption_config_xml.is_empty());
+    assert!(bm.encryption_config_xml.starts_with(b"<ServerSide"));
+    assert!(!bm.tagging_config_xml.is_empty());
+    assert!(bm.tagging_config_xml.starts_with(b"<Tagging"));
+    assert!(!bm.quota_config_json.is_empty());
+    assert!(bm.quota_config_json.starts_with(b"{\"quota\""));
+    assert!(!bm.replication_config_xml.is_empty());
+    assert!(bm.replication_config_xml.starts_with(b"<Replication"));
+    assert!(!bm.bucket_targets_config_json.is_empty());
+    assert!(bm.bucket_targets_config_json.starts_with(b"[{"));
+    assert!(!bm.bucket_targets_config_meta_json.is_empty());
+    assert!(bm.bucket_targets_config_meta_json.starts_with(b"{\"replication"));
+
+    let updated_sec = 1705312800; // 2024-01-15 12:00:00 UTC
+    assert_eq!(bm.policy_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.object_lock_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.encryption_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.tagging_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.quota_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.replication_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.versioning_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.lifecycle_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.notification_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.bucket_targets_config_updated_at.unix_timestamp(), updated_sec);
+    assert_eq!(bm.bucket_targets_config_meta_updated_at.unix_timestamp(), updated_sec);
+
+    assert!(bm.cors_config_xml.is_empty());
+    assert!(bm.public_access_block_config_xml.is_empty());
+    assert!(bm.bucket_acl_config_json.is_empty());
+}
+
+#[test]
+fn unmarshal_legacy_compact_time_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let legacy_hex = concat!(
+        "83",
+        "a44e616d65",
+        "a474657374",
+        "a743726561746564",
+        "99cd07e9cd01100c1021ce2026b1fa000000",
+        "ab4c6f636b456e61626c6564",
+        "c2"
+    );
+
+    let mut bytes = vec![0u8; legacy_hex.len() / 2];
+    hex_decode(legacy_hex.as_bytes(), &mut bytes).expect("valid hex");
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("legacy compact time should decode");
+
+    assert_eq!(bm.name, "test");
+    assert_eq!(bm.created.unix_timestamp(), 1759148193);
+    assert_eq!(bm.created.nanosecond(), 539406842);
+    assert!(!bm.lock_enabled);
+}
+
+#[test]
+fn unmarshal_bin_wrapped_ext_time_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let wrapped_hex = concat!(
+        "83",
+        "a44e616d65",
+        "a464616461",
+        "a743726561746564",
+        "c40fc70c05fffffff1886e090000000000",
+        "ab4c6f636b456e61626c6564",
+        "c2"
+    );
+
+    let mut bytes = vec![0u8; wrapped_hex.len() / 2];
+    hex_decode(wrapped_hex.as_bytes(), &mut bytes).expect("valid hex");
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("bin-wrapped ext time should decode");
+
+    assert_eq!(bm.name, "dada");
+    assert_eq!(bm.created.unix_timestamp(), -62135596800);
+    assert!(!bm.lock_enabled);
+}
+
+#[test]
+fn unmarshal_legacy_rmp_serde_field_aliases_and_byte_arrays() {
+    use faster_hex::hex_decode;
+
+    let legacy_hex = concat!(
+        "85",
+        "a44e616d65",
+        "a474657374",
+        "a743726561746564",
+        "99cd07e9cd01100c1021ce2026b1fa000000",
+        "ab4c6f636b456e61626c6564",
+        "c2",
+        "b0506f6c696379436f6e6669674a736f6e",
+        "93010203",
+        "bb4275636b657454617267657473436f6e6669674d6574614a736f6e",
+        "920405"
+    );
+
+    let mut bytes = vec![0u8; legacy_hex.len() / 2];
+    hex_decode(legacy_hex.as_bytes(), &mut bytes).expect("valid hex");
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("legacy field aliases and byte arrays should decode");
+
+    assert_eq!(bm.name, "test");
+    assert_eq!(bm.created.unix_timestamp(), 1759148193);
+    assert_eq!(bm.created.nanosecond(), 539406842);
+    assert_eq!(bm.policy_config_json, vec![1, 2, 3]);
+    assert_eq!(bm.bucket_targets_config_meta_json, vec![4, 5]);
+    assert!(!bm.lock_enabled);
+}
+
+#[test]
+fn unmarshal_legacy_bin16_and_array16_bucket_metadata() {
+    let policy = vec![b'x'; 257];
+    let targets_meta: Vec<u8> = (0u8..=16).collect();
+
+    let mut bytes = Vec::new();
+    rmp::encode::write_map_len(&mut bytes, 5).unwrap();
+    rmp::encode::write_str(&mut bytes, "Name").unwrap();
+    rmp::encode::write_str(&mut bytes, "test-bucket").unwrap();
+    rmp::encode::write_str(&mut bytes, "Created").unwrap();
+    bytes.extend_from_slice(&[
+        0xc7, 0x0c, 0x05, 0x00, 0x00, 0x00, 0x00, 0x65, 0x92, 0x00, 0x80, 0x00, 0x00, 0x00, 0x00,
+    ]);
+    rmp::encode::write_str(&mut bytes, "LockEnabled").unwrap();
+    bytes.push(0x01);
+    rmp::encode::write_str(&mut bytes, "PolicyConfigJson").unwrap();
+    rmp::encode::write_bin(&mut bytes, &policy).unwrap();
+    rmp::encode::write_str(&mut bytes, "BucketTargetsConfigMetaJson").unwrap();
+    rmp::encode::write_array_len(&mut bytes, targets_meta.len() as u32).unwrap();
+    for byte in &targets_meta {
+        rmp::encode::write_uint(&mut bytes, u64::from(*byte)).unwrap();
+    }
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("legacy bin16 and array16 should decode");
+
+    assert_eq!(bm.name, "test-bucket");
+    assert_eq!(bm.created.unix_timestamp(), 1704067200);
+    assert!(bm.lock_enabled);
+    assert_eq!(bm.policy_config_json, policy);
+    assert_eq!(bm.bucket_targets_config_meta_json, targets_meta);
+}
+
+#[test]
+fn unmarshal_legacy_numeric_bool_bucket_metadata() {
+    use faster_hex::hex_decode;
+
+    let legacy_hex = concat!(
+        "83",
+        "a44e616d65",
+        "ab746573742d6275636b6574",
+        "a743726561746564",
+        "c70c05000000006592008000000000",
+        "ab4c6f636b456e61626c6564",
+        "01"
+    );
+
+    let mut bytes = vec![0u8; legacy_hex.len() / 2];
+    hex_decode(legacy_hex.as_bytes(), &mut bytes).expect("valid hex");
+
+    let bm = BucketMetadata::unmarshal(&bytes).expect("legacy numeric bool should decode");
+
+    assert_eq!(bm.name, "test-bucket");
+    assert_eq!(bm.created.unix_timestamp(), 1704067200);
+    assert_eq!(bm.created.nanosecond(), 0);
+    assert!(bm.lock_enabled);
+}
+
+#[tokio::test]
+async fn marshal_msg_complete_example() {
+    // Create a complete BucketMetadata with various configurations
+    let mut bm = BucketMetadata::new("test-bucket");
+
+    // Set creation time to current time
+    bm.created = OffsetDateTime::now_utc();
+    bm.lock_enabled = true;
+
+    // Add policy configuration
+    let policy_json = r#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::test-bucket/*"}]}"#;
+    bm.policy_config_json = policy_json.as_bytes().to_vec();
+    bm.policy_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add lifecycle configuration
+    let lifecycle_xml = r#"<LifecycleConfiguration><Rule><ID>rule1</ID><Status>Enabled</Status><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>"#;
+    bm.lifecycle_config_xml = lifecycle_xml.as_bytes().to_vec();
+    bm.lifecycle_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add versioning configuration
+    let versioning_xml = r#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#;
+    bm.versioning_config_xml = versioning_xml.as_bytes().to_vec();
+    bm.versioning_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add encryption configuration
+    let encryption_xml = r#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
+    bm.encryption_config_xml = encryption_xml.as_bytes().to_vec();
+    bm.encryption_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add tagging configuration
+    let tagging_xml = r#"<Tagging><TagSet><Tag><Key>Environment</Key><Value>Test</Value></Tag><Tag><Key>Owner</Key><Value>RustFS</Value></Tag></TagSet></Tagging>"#;
+    bm.tagging_config_xml = tagging_xml.as_bytes().to_vec();
+    bm.tagging_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add quota configuration
+    let quota_json =
+        r#"{"quota":1073741824,"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#; // 1GB quota
+    bm.quota_config_json = quota_json.as_bytes().to_vec();
+    bm.quota_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add object lock configuration
+    let object_lock_xml = r#"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled><Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>7</Days></DefaultRetention></Rule></ObjectLockConfiguration>"#;
+    bm.object_lock_config_xml = object_lock_xml.as_bytes().to_vec();
+    bm.object_lock_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add notification configuration
+    let notification_xml = r#"<NotificationConfiguration><CloudWatchConfiguration><Id>notification1</Id><Event>s3:ObjectCreated:*</Event><CloudWatchConfiguration><LogGroupName>test-log-group</LogGroupName></CloudWatchConfiguration></CloudWatchConfiguration></NotificationConfiguration>"#;
+    bm.notification_config_xml = notification_xml.as_bytes().to_vec();
+    bm.notification_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add replication configuration
+    let replication_xml = r#"<ReplicationConfiguration><Role>arn:aws:iam::123456789012:role/replication-role</Role><Rule><ID>rule1</ID><Status>Enabled</Status><Prefix>documents/</Prefix><Destination><Bucket>arn:aws:s3:::destination-bucket</Bucket></Destination></Rule></ReplicationConfiguration>"#;
+    bm.replication_config_xml = replication_xml.as_bytes().to_vec();
+    bm.replication_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add bucket targets configuration
+    let bucket_targets_json = r#"[{"endpoint":"http://target1.example.com","credentials":{"accessKey":"key1","secretKey":"secret1"},"targetBucket":"target-bucket-1","region":"us-east-1"},{"endpoint":"http://target2.example.com","credentials":{"accessKey":"key2","secretKey":"secret2"},"targetBucket":"target-bucket-2","region":"us-west-2"}]"#;
+    bm.bucket_targets_config_json = bucket_targets_json.as_bytes().to_vec();
+    bm.bucket_targets_config_updated_at = OffsetDateTime::now_utc();
+
+    // Add bucket targets meta configuration
+    let bucket_targets_meta_json = r#"{"replicationId":"repl-123","syncMode":"async","bandwidth":"100MB"}"#;
+    bm.bucket_targets_config_meta_json = bucket_targets_meta_json.as_bytes().to_vec();
+    bm.bucket_targets_config_meta_updated_at = OffsetDateTime::now_utc();
+
+    // Add public access block configuration
+    let public_access_block_xml = r#"<PublicAccessBlockConfiguration><BlockPublicAcls>true</BlockPublicAcls><IgnorePublicAcls>true</IgnorePublicAcls><BlockPublicPolicy>true</BlockPublicPolicy><RestrictPublicBuckets>false</RestrictPublicBuckets></PublicAccessBlockConfiguration>"#;
+    bm.public_access_block_config_xml = public_access_block_xml.as_bytes().to_vec();
+    bm.public_access_block_config_updated_at = OffsetDateTime::now_utc();
+
+    let bucket_acl = r#"{"owner":{"id":"rustfsadmin","display_name":"RustFS Tester"},"grants":[{"grantee":{"grantee_type":"CanonicalUser","id":"rustfsadmin","display_name":"RustFS Tester","uri":null,"email_address":null},"permission":"FULL_CONTROL"}]}"#;
+    bm.bucket_acl_config_json = bucket_acl.as_bytes().to_vec();
+    bm.bucket_acl_config_updated_at = OffsetDateTime::now_utc();
+
+    // Test serialization
+    let buf = bm.marshal_msg().unwrap();
+    assert!(!buf.is_empty(), "Serialized buffer should not be empty");
+
+    // Test deserialization
+    let deserialized_bm = BucketMetadata::unmarshal(&buf).unwrap();
+
+    // Verify all fields are correctly serialized and deserialized
+    assert_eq!(bm.name, deserialized_bm.name);
+    assert_eq!(bm.created.unix_timestamp(), deserialized_bm.created.unix_timestamp());
+    assert_eq!(bm.lock_enabled, deserialized_bm.lock_enabled);
+
+    // Verify configuration data
+    assert_eq!(bm.policy_config_json, deserialized_bm.policy_config_json);
+    assert_eq!(bm.lifecycle_config_xml, deserialized_bm.lifecycle_config_xml);
+    assert_eq!(bm.versioning_config_xml, deserialized_bm.versioning_config_xml);
+    assert_eq!(bm.encryption_config_xml, deserialized_bm.encryption_config_xml);
+    assert_eq!(bm.tagging_config_xml, deserialized_bm.tagging_config_xml);
+    assert_eq!(bm.quota_config_json, deserialized_bm.quota_config_json);
+    assert_eq!(bm.public_access_block_config_xml, deserialized_bm.public_access_block_config_xml);
+    assert_eq!(bm.bucket_acl_config_json, deserialized_bm.bucket_acl_config_json);
+    assert_eq!(bm.object_lock_config_xml, deserialized_bm.object_lock_config_xml);
+    assert_eq!(bm.notification_config_xml, deserialized_bm.notification_config_xml);
+    assert_eq!(bm.replication_config_xml, deserialized_bm.replication_config_xml);
+    assert_eq!(bm.bucket_targets_config_json, deserialized_bm.bucket_targets_config_json);
+    assert_eq!(bm.bucket_targets_config_meta_json, deserialized_bm.bucket_targets_config_meta_json);
+
+    // Verify timestamps (comparing unix timestamps to avoid precision issues)
+    assert_eq!(
+        bm.policy_config_updated_at.unix_timestamp(),
+        deserialized_bm.policy_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.lifecycle_config_updated_at.unix_timestamp(),
+        deserialized_bm.lifecycle_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.versioning_config_updated_at.unix_timestamp(),
+        deserialized_bm.versioning_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.encryption_config_updated_at.unix_timestamp(),
+        deserialized_bm.encryption_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.tagging_config_updated_at.unix_timestamp(),
+        deserialized_bm.tagging_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.quota_config_updated_at.unix_timestamp(),
+        deserialized_bm.quota_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.object_lock_config_updated_at.unix_timestamp(),
+        deserialized_bm.object_lock_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.notification_config_updated_at.unix_timestamp(),
+        deserialized_bm.notification_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.replication_config_updated_at.unix_timestamp(),
+        deserialized_bm.replication_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.bucket_targets_config_updated_at.unix_timestamp(),
+        deserialized_bm.bucket_targets_config_updated_at.unix_timestamp()
+    );
+    assert_eq!(
+        bm.bucket_targets_config_meta_updated_at.unix_timestamp(),
+        deserialized_bm.bucket_targets_config_meta_updated_at.unix_timestamp()
+    );
+
+    // Test that the serialized data contains expected content
+    let buf_str = String::from_utf8_lossy(&buf);
+    assert!(buf_str.contains("test-bucket"), "Serialized data should contain bucket name");
+
+    // Verify the buffer size is reasonable (should be larger due to all the config data)
+    assert!(buf.len() > 1000, "Buffer should be substantial in size due to all configurations");
+
+    println!("✅ Complete BucketMetadata serialization test passed");
+    println!("   - Bucket name: {}", deserialized_bm.name);
+    println!("   - Lock enabled: {}", deserialized_bm.lock_enabled);
+    println!("   - Policy config size: {} bytes", deserialized_bm.policy_config_json.len());
+    println!("   - Lifecycle config size: {} bytes", deserialized_bm.lifecycle_config_xml.len());
+    println!("   - Serialized buffer size: {} bytes", buf.len());
+}

crates/ecstore/src/bucket/migration.rs

@@ -0,0 +1,420 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Migration of bucket metadata and IAM config from legacy format to RustFS format.
+
+use crate::bucket::metadata::BUCKET_METADATA_FILE;
+use crate::bucket::replication::{decode_resync_file, encode_resync_file};
+use crate::disk::{BUCKET_META_PREFIX, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
+use crate::store_api::{BucketOptions, ObjectOptions, PutObjReader, StorageAPI};
+use http::HeaderMap;
+use rustfs_policy::auth::UserIdentity;
+use rustfs_policy::policy::PolicyDoc;
+use rustfs_utils::path::SLASH_SEPARATOR;
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use time::OffsetDateTime;
+use tracing::{debug, info, warn};
+
+/// IAM config prefix under meta bucket (e.g. config/iam/).
+const IAM_CONFIG_PREFIX: &str = "config/iam";
+const IAM_FORMAT_FILE_PATH: &str = "config/iam/format.json";
+const IAM_USERS_PREFIX: &str = "config/iam/users/";
+const IAM_SERVICE_ACCOUNTS_PREFIX: &str = "config/iam/service-accounts/";
+const IAM_STS_PREFIX: &str = "config/iam/sts/";
+const IAM_GROUPS_PREFIX: &str = "config/iam/groups/";
+const IAM_POLICIES_PREFIX: &str = "config/iam/policies/";
+const IAM_POLICY_DB_PREFIX: &str = "config/iam/policydb/";
+const REPLICATION_META_DIR: &str = ".replication";
+const RESYNC_META_FILE: &str = "resync.bin";
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatIamFormat {
+    #[serde(default)]
+    version: i64,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatGroupInfo {
+    #[serde(default)]
+    version: i64,
+    #[serde(default = "default_group_status")]
+    status: String,
+    #[serde(default)]
+    members: Vec<String>,
+    #[serde(
+        rename = "updatedAt",
+        alias = "update_at",
+        default,
+        with = "rustfs_policy::serde_datetime::option"
+    )]
+    update_at: Option<OffsetDateTime>,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct CompatMappedPolicy {
+    #[serde(default)]
+    version: i64,
+    #[serde(rename = "policy", alias = "policies", default)]
+    policy: String,
+    #[serde(
+        rename = "updatedAt",
+        alias = "update_at",
+        default,
+        with = "rustfs_policy::serde_datetime::option"
+    )]
+    update_at: Option<OffsetDateTime>,
+}
+
+fn default_group_status() -> String {
+    "enabled".to_string()
+}
+
+fn normalize_iam_config_blob(path: &str, data: &[u8]) -> std::result::Result<Option<Vec<u8>>, String> {
+    if path == IAM_FORMAT_FILE_PATH {
+        let mut format: CompatIamFormat =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM format failed: {err}"))?;
+        if format.version <= 0 {
+            format.version = 1;
+        }
+        return serde_json::to_vec(&format)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM format failed: {err}"));
+    }
+
+    if is_identity_path(path) {
+        let mut identity: UserIdentity =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM identity failed: {err}"))?;
+        if identity.update_at.is_none() {
+            identity.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&identity)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM identity failed: {err}"));
+    }
+
+    if is_group_path(path) {
+        let mut group: CompatGroupInfo = serde_json::from_slice(data).map_err(|err| format!("parse IAM group failed: {err}"))?;
+        if group.update_at.is_none() {
+            group.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&group)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM group failed: {err}"));
+    }
+
+    if is_policy_doc_path(path) {
+        let mut doc = PolicyDoc::try_from(data.to_vec()).map_err(|err| format!("parse IAM policy doc failed: {err}"))?;
+        if doc.create_date.is_none() {
+            doc.create_date = doc.update_date;
+        }
+        if doc.update_date.is_none() {
+            doc.update_date = doc.create_date;
+        }
+        return serde_json::to_vec(&doc)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM policy doc failed: {err}"));
+    }
+
+    if is_policy_mapping_path(path) {
+        let mut mapped: CompatMappedPolicy =
+            serde_json::from_slice(data).map_err(|err| format!("parse IAM policy mapping failed: {err}"))?;
+        if mapped.update_at.is_none() {
+            mapped.update_at = Some(OffsetDateTime::now_utc());
+        }
+        return serde_json::to_vec(&mapped)
+            .map(Some)
+            .map_err(|err| format!("serialize IAM policy mapping failed: {err}"));
+    }
+
+    Ok(None)
+}
+
+fn is_identity_path(path: &str) -> bool {
+    (path.starts_with(IAM_USERS_PREFIX) || path.starts_with(IAM_SERVICE_ACCOUNTS_PREFIX) || path.starts_with(IAM_STS_PREFIX))
+        && path.ends_with("/identity.json")
+}
+
+fn is_group_path(path: &str) -> bool {
+    path.starts_with(IAM_GROUPS_PREFIX) && path.ends_with("/members.json")
+}
+
+fn is_policy_doc_path(path: &str) -> bool {
+    path.starts_with(IAM_POLICIES_PREFIX) && path.ends_with("/policy.json")
+}
+
+fn is_policy_mapping_path(path: &str) -> bool {
+    path.starts_with(IAM_POLICY_DB_PREFIX) && path.ends_with(".json")
+}
+
+fn is_resync_meta_path(path: &str) -> bool {
+    path.ends_with(&format!("{REPLICATION_META_DIR}/{RESYNC_META_FILE}"))
+}
+
+fn normalize_bucket_meta_blob(path: &str, data: &[u8]) -> std::result::Result<Option<Vec<u8>>, String> {
+    if !is_resync_meta_path(path) {
+        return Ok(None);
+    }
+    let status = decode_resync_file(data).map_err(|err| format!("decode resync meta failed: {err}"))?;
+    encode_resync_file(&status)
+        .map(Some)
+        .map_err(|err| format!("encode resync meta failed: {err}"))
+}
+
+/// Migrates bucket metadata from legacy format to RustFS.
+/// Uses list_bucket (from disk volumes) to get bucket names, since list_objects_v2 on the legacy
+/// meta bucket may not work (legacy format differs from object layer expectations).
+/// Skips buckets that already exist in RustFS (idempotent).
+pub async fn try_migrate_bucket_metadata<S: StorageAPI>(store: Arc<S>) {
+    let buckets_list = match store
+        .list_bucket(&BucketOptions {
+            no_metadata: true,
+            ..Default::default()
+        })
+        .await
+    {
+        Ok(b) => b,
+        Err(e) => {
+            warn!("list buckets failed (skip migration): {e}");
+            return;
+        }
+    };
+
+    let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
+
+    if buckets.is_empty() {
+        debug!("No migrating bucket metadata found");
+        return;
+    }
+
+    debug!("Found {} migrating bucket metadata, migrating...", buckets.len());
+
+    let opts = ObjectOptions {
+        max_parity: true,
+        no_lock: true,
+        ..Default::default()
+    };
+    let h = HeaderMap::new();
+
+    for bucket in buckets {
+        let meta_path = format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{bucket}{SLASH_SEPARATOR}{BUCKET_METADATA_FILE}");
+        migrate_one_if_missing(store.clone(), &opts, &h, &meta_path, &format!("bucket metadata: {bucket}")).await;
+
+        let resync_path = format!(
+            "{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{bucket}{SLASH_SEPARATOR}{REPLICATION_META_DIR}{SLASH_SEPARATOR}{RESYNC_META_FILE}"
+        );
+        migrate_one_if_missing(store.clone(), &opts, &h, &resync_path, &format!("bucket replication resync: {bucket}")).await;
+    }
+}
+
+async fn migrate_one_if_missing<S: StorageAPI>(
+    store: Arc<S>,
+    opts: &ObjectOptions,
+    headers: &HeaderMap,
+    path: &str,
+    label: &str,
+) {
+    if store
+        .get_object_info(RUSTFS_META_BUCKET, path, &ObjectOptions::default())
+        .await
+        .is_ok()
+    {
+        debug!("{label} already exists in RustFS, skip");
+        return;
+    }
+
+    let mut rd = match store
+        .get_object_reader(MIGRATING_META_BUCKET, path, None, headers.clone(), opts)
+        .await
+    {
+        Ok(r) => r,
+        Err(e) => {
+            debug!("read migrating {label}: {e}");
+            return;
+        }
+    };
+
+    let data = match rd.read_all().await {
+        Ok(d) if !d.is_empty() => d,
+        Ok(_) => return,
+        Err(e) => {
+            debug!("read migrating {label} body: {e}");
+            return;
+        }
+    };
+
+    let data = match normalize_bucket_meta_blob(path, &data) {
+        Ok(Some(normalized)) => normalized,
+        Ok(None) => data,
+        Err(e) => {
+            warn!("skip {label} migration due to incompatible format: {e}");
+            return;
+        }
+    };
+
+    if let Err(e) = store
+        .put_object(RUSTFS_META_BUCKET, path, &mut PutObjReader::from_vec(data), opts)
+        .await
+    {
+        warn!("write {label}: {e}");
+    } else {
+        info!("Migrated {label}");
+    }
+}
+
+/// Migrates IAM config from legacy meta bucket `config/iam/` to RustFS meta bucket.
+/// Lists all objects under the IAM prefix in the source, copies each to the target if not present.
+/// Skips objects that already exist in RustFS (idempotent).
+/// If list_objects_v2 on the legacy bucket fails (e.g. format differs), migration is skipped.
+pub async fn try_migrate_iam_config<S: StorageAPI>(store: Arc<S>) {
+    let opts = ObjectOptions {
+        max_parity: true,
+        no_lock: true,
+        ..Default::default()
+    };
+    let h = HeaderMap::new();
+    let prefix = format!("{IAM_CONFIG_PREFIX}/");
+    let mut continuation: Option<String> = None;
+    let mut total_migrated = 0usize;
+
+    loop {
+        let list_result = match store
+            .clone()
+            .list_objects_v2(MIGRATING_META_BUCKET, &prefix, continuation, None, 500, false, None, false)
+            .await
+        {
+            Ok(r) => r,
+            Err(e) => {
+                debug!("list IAM config from legacy bucket failed (skip migration): {e}");
+                return;
+            }
+        };
+
+        for obj in list_result.objects {
+            let path = &obj.name;
+            if path.is_empty() || path.ends_with('/') {
+                continue;
+            }
+            if store
+                .get_object_info(RUSTFS_META_BUCKET, path, &ObjectOptions::default())
+                .await
+                .is_ok()
+            {
+                debug!("IAM config already exists in RustFS, skip: {path}");
+                continue;
+            }
+            let mut rd = match store
+                .get_object_reader(MIGRATING_META_BUCKET, path, None, h.clone(), &opts)
+                .await
+            {
+                Ok(r) => r,
+                Err(e) => {
+                    debug!("read migrating IAM config {path}: {e}");
+                    continue;
+                }
+            };
+            let data = match rd.read_all().await {
+                Ok(d) if !d.is_empty() => d,
+                Ok(_) => continue,
+                Err(e) => {
+                    debug!("read migrating IAM config {path} body: {e}");
+                    continue;
+                }
+            };
+            let data = match normalize_iam_config_blob(path, &data) {
+                Ok(Some(normalized)) => normalized,
+                Ok(None) => {
+                    debug!("skip unsupported IAM config path during migration: {path}");
+                    continue;
+                }
+                Err(e) => {
+                    warn!("skip IAM config migration due to incompatible format, path: {path}, err: {e}");
+                    continue;
+                }
+            };
+            if let Err(e) = store
+                .put_object(RUSTFS_META_BUCKET, path, &mut PutObjReader::from_vec(data), &opts)
+                .await
+            {
+                warn!("write IAM config {path}: {e}");
+            } else {
+                info!("Migrated IAM config: {path}");
+                total_migrated += 1;
+            }
+        }
+
+        continuation = list_result.next_continuation_token.or(list_result.continuation_token);
+        if !list_result.is_truncated || continuation.is_none() {
+            break;
+        }
+    }
+
+    if total_migrated > 0 {
+        info!("IAM migration complete: {} object(s) migrated", total_migrated);
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{normalize_bucket_meta_blob, normalize_iam_config_blob};
+    use crate::bucket::replication::{
+        BucketReplicationResyncStatus, ResyncStatusType, TargetReplicationResyncStatus, decode_resync_file, encode_resync_file,
+    };
+    use std::collections::HashMap;
+
+    #[test]
+    fn test_normalize_policy_mapping_legacy_timestamp_and_fields() {
+        let path = "config/iam/policydb/users/alice.json";
+        let input = r#"{"version":1,"policies":"readwrite","update_at":"2026-03-09 02:22:44.998954 +00:00:00"}"#;
+
+        let output = normalize_iam_config_blob(path, input.as_bytes())
+            .expect("normalize should succeed")
+            .expect("path should be supported");
+
+        let v: serde_json::Value = serde_json::from_slice(&output).expect("output should be valid JSON");
+        assert_eq!(v.get("policy").and_then(|x| x.as_str()), Some("readwrite"));
+        assert!(v.get("policies").is_none(), "legacy field should be normalized");
+
+        let updated_at = v
+            .get("updatedAt")
+            .and_then(|x| x.as_str())
+            .expect("updatedAt should exist as string");
+        assert!(updated_at.contains('T'), "updatedAt should be RFC3339-like");
+    }
+
+    #[test]
+    fn test_normalize_bucket_meta_blob_resync_reencode() {
+        let path = ".buckets/test/.replication/resync.bin";
+        let mut status = BucketReplicationResyncStatus::new();
+        status.id = 123;
+        status.targets_map = HashMap::from([(
+            "arn:replication::1:dest".to_string(),
+            TargetReplicationResyncStatus {
+                resync_id: "reset-1".to_string(),
+                resync_status: ResyncStatusType::ResyncStarted,
+                replicated_count: 1,
+                ..Default::default()
+            },
+        )]);
+
+        let input = encode_resync_file(&status).expect("encode should succeed");
+        let output = normalize_bucket_meta_blob(path, &input)
+            .expect("normalize should succeed")
+            .expect("resync path should be normalized");
+
+        let decoded = decode_resync_file(&output).expect("decode should succeed");
+        assert_eq!(decoded.id, 123);
+        assert_eq!(decoded.targets_map["arn:replication::1:dest"].resync_id, "reset-1");
+    }
+}

crates/ecstore/src/bucket/mod.rs

@@ -18,6 +18,10 @@ pub mod error;
 pub mod lifecycle;
 pub mod metadata;
 pub mod metadata_sys;
+#[cfg(test)]
+mod metadata_test;
+pub mod migration;
+mod msgp_decode;
 pub mod object_lock;
 pub mod policy_sys;
 pub mod quota;

crates/ecstore/src/bucket/msgp_decode.rs

@@ -0,0 +1,189 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! MessagePack decode helpers for bucket metadata, aligned with msgp format.
+
+use crate::error::{Error, Result};
+use byteorder::{BigEndian, ByteOrder};
+use rmp::Marker;
+use std::io::{Read, Write};
+use time::OffsetDateTime;
+
+/// Skip a single MessagePack value. Used for unknown map keys.
+pub(crate) fn skip_msgp_value<R: Read>(rd: &mut R) -> Result<()> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    let skip_len: usize = match marker {
+        Marker::Null | Marker::False | Marker::True => 0,
+        Marker::FixPos(_) | Marker::FixNeg(_) => 0,
+        Marker::U8 => 1,
+        Marker::U16 => 2,
+        Marker::U32 => 4,
+        Marker::U64 => 8,
+        Marker::I8 => 1,
+        Marker::I16 => 2,
+        Marker::I32 => 4,
+        Marker::I64 => 8,
+        Marker::F32 => 4,
+        Marker::F64 => 8,
+        Marker::FixStr(n) => n as usize,
+        Marker::Str8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            b[0] as usize
+        }
+        Marker::Str16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u16::from_be_bytes(b) as usize
+        }
+        Marker::Str32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u32::from_be_bytes(b) as usize
+        }
+        Marker::Bin8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            b[0] as usize
+        }
+        Marker::Bin16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u16::from_be_bytes(b) as usize
+        }
+        Marker::Bin32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            u32::from_be_bytes(b) as usize
+        }
+        Marker::FixArray(n) => {
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Array16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u16::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Array32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u32::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::FixMap(n) => {
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Map16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u16::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::Map32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let n = u32::from_be_bytes(b);
+            for _ in 0..n {
+                skip_msgp_value(rd)?;
+                skip_msgp_value(rd)?;
+            }
+            return Ok(());
+        }
+        Marker::FixExt1 => 1,
+        Marker::FixExt2 => 2,
+        Marker::FixExt4 => 4,
+        Marker::FixExt8 => 8,
+        Marker::FixExt16 => 16,
+        Marker::Ext8 => {
+            let mut b = [0u8; 1];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = b[0] as usize;
+            1 + len // type byte + data
+        }
+        Marker::Ext16 => {
+            let mut b = [0u8; 2];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = u16::from_be_bytes(b) as usize;
+            2 + len
+        }
+        Marker::Ext32 => {
+            let mut b = [0u8; 4];
+            rd.read_exact(&mut b).map_err(Error::other)?;
+            let len = u32::from_be_bytes(b) as usize;
+            4 + len
+        }
+        Marker::Reserved => 0,
+    };
+    if skip_len > 0 {
+        let mut buf = vec![0u8; skip_len];
+        rd.read_exact(&mut buf).map_err(Error::other)?;
+    }
+    Ok(())
+}
+
+/// msgp time format: ext8 (0xc7), len 12, type 5, 8 bytes sec (BE) + 4 bytes nsec (BE).
+pub(crate) const MSGP_TIME_EXT_TYPE: i8 = 5;
+pub(crate) const MSGP_TIME_LEN: u8 = 12;
+
+/// Read msgp ext8 time - caller must have already read the marker and verified it's ext8.
+/// Ext8 format: 1 byte len, 1 byte type, then data bytes.
+pub(crate) fn read_msgp_ext8_time<R: Read>(rd: &mut R) -> Result<OffsetDateTime> {
+    let mut len_buf = [0u8; 1];
+    rd.read_exact(&mut len_buf).map_err(Error::other)?;
+    let len = len_buf[0] as usize;
+    if len != MSGP_TIME_LEN as usize {
+        return Err(Error::other(format!("invalid msgp time len: {len}")));
+    }
+    let mut type_buf = [0u8; 1];
+    rd.read_exact(&mut type_buf).map_err(Error::other)?;
+    if type_buf[0] != MSGP_TIME_EXT_TYPE as u8 {
+        return Err(Error::other(format!("invalid msgp time type: {}", type_buf[0])));
+    }
+    let mut buf = [0u8; 12];
+    rd.read_exact(&mut buf).map_err(Error::other)?;
+    let sec = BigEndian::read_i64(&buf[0..8]);
+    let nsec = BigEndian::read_u32(&buf[8..12]);
+    OffsetDateTime::from_unix_timestamp(sec)
+        .map_err(|_| Error::other("invalid timestamp"))?
+        .replace_nanosecond(nsec)
+        .map_err(|_| Error::other("invalid nanosecond"))
+}
+
+/// Write msgp time as ext8 (0xc7), len 12, type 5. Always uses ext format (never nil).
+pub(crate) fn write_msgp_time<W: Write>(wr: &mut W, t: OffsetDateTime) -> Result<()> {
+    wr.write_all(&[0xc7, MSGP_TIME_LEN, MSGP_TIME_EXT_TYPE as u8])
+        .map_err(Error::other)?;
+    let mut buf = [0u8; 12];
+    BigEndian::write_i64(&mut buf[0..8], t.unix_timestamp());
+    BigEndian::write_u32(&mut buf[8..12], t.nanosecond());
+    wr.write_all(&buf).map_err(Error::other)
+}

crates/ecstore/src/bucket/quota/mod.rs

@@ -15,7 +15,6 @@
 pub mod checker;
 
 use crate::error::Result;
-use rmp_serde::Serializer as rmpSerializer;
 use rustfs_config::{
     QUOTA_API_PATH, QUOTA_EXCEEDED_ERROR_CODE, QUOTA_INTERNAL_ERROR_CODE, QUOTA_INVALID_CONFIG_ERROR_CODE,
     QUOTA_NOT_FOUND_ERROR_CODE,
@@ -28,27 +27,35 @@ use time::OffsetDateTime;
 pub enum QuotaType {
     /// Hard quota: reject immediately when exceeded
     #[default]
+    #[serde(alias = "HARD", alias = "hard")]
     Hard,
 }
 
+/// Bucket quota configuration. quota_type defaults to Hard when omitted.
 #[derive(Debug, Deserialize, Serialize, Default, Clone, PartialEq)]
 pub struct BucketQuota {
+    #[serde(default)]
     pub quota: Option<u64>,
+    /// Defaults to Hard when missing.
+    #[serde(default)]
     pub quota_type: QuotaType,
     /// Timestamp when this quota configuration was set (for audit purposes)
+    #[serde(default, with = "time::serde::rfc3339::option")]
     pub created_at: Option<OffsetDateTime>,
+    /// Accept updated_at for compatibility; not used.
+    #[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
+    pub updated_at: Option<OffsetDateTime>,
 }
 
 impl BucketQuota {
+    /// Serialize to JSON bytes. Same format as parse_all_configs.
     pub fn marshal_msg(&self) -> Result<Vec<u8>> {
-        let mut buf = Vec::new();
-        self.serialize(&mut rmpSerializer::new(&mut buf).with_struct_map())?;
-        Ok(buf)
+        serde_json::to_vec(self).map_err(Into::into)
     }
 
+    /// Deserialize from JSON bytes. Same format as parse_all_configs.
     pub fn unmarshal(buf: &[u8]) -> Result<Self> {
-        let t: BucketQuota = rmp_serde::from_slice(buf)?;
-        Ok(t)
+        serde_json::from_slice(buf).map_err(Into::into)
     }
 
     pub fn new(quota: Option<u64>) -> Self {
@@ -57,6 +64,7 @@ impl BucketQuota {
             quota,
             quota_type: QuotaType::Hard,
             created_at: Some(now),
+            updated_at: None,
         }
     }
 
@@ -156,3 +164,57 @@ impl QuotaErrorResponse {
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// Legacy format: quota, created_at, updated_at (no quota_type)
+    #[test]
+    fn deserialize_format_without_quota_type() {
+        let json = r#"{"quota":1073741824,"created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+        assert!(q.created_at.is_some());
+        assert!(q.updated_at.is_some());
+    }
+
+    /// RustFS format: quota, quota_type, created_at
+    #[test]
+    fn deserialize_rustfs_format() {
+        let json = r#"{"quota":1073741824,"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+        assert!(q.created_at.is_some());
+        assert!(q.created_at.is_some_and(|t| t.unix_timestamp() == 1704067200));
+    }
+
+    /// E2E format uses "HARD" (uppercase)
+    #[test]
+    fn deserialize_quota_type_hard_uppercase() {
+        let json = r#"{"quota":2048,"quota_type":"HARD"}"#;
+        let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota_type, QuotaType::Hard);
+    }
+
+    /// marshal_msg/unmarshal use JSON, same as parse_all_configs
+    #[test]
+    fn marshal_unmarshal_roundtrip() {
+        let q = BucketQuota::new(Some(1073741824));
+        let buf = q.marshal_msg().expect("marshal");
+        let restored = BucketQuota::unmarshal(&buf).expect("unmarshal");
+        assert_eq!(q.quota, restored.quota);
+        assert_eq!(q.quota_type, restored.quota_type);
+    }
+
+    /// unmarshal accepts format without quota_type
+    #[test]
+    fn unmarshal_format_without_quota_type() {
+        let json = r#"{"quota":1073741824,"created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#;
+        let q = BucketQuota::unmarshal(json.as_bytes()).expect("should parse");
+        assert_eq!(q.quota, Some(1073741824));
+        assert_eq!(q.quota_type, QuotaType::Hard);
+    }
+}

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -20,8 +20,8 @@ use crate::bucket::replication::ResyncStatusType;
 use crate::bucket::replication::replicate_delete;
 use crate::bucket::replication::replicate_object;
 use crate::bucket::replication::replication_resyncer::{
-    BucketReplicationResyncStatus, DeletedObjectReplicationInfo, ReplicationConfig, ReplicationResyncer,
-    get_heal_replicate_object_info,
+    BucketReplicationResyncStatus, DeletedObjectReplicationInfo, REPLICATION_DIR, RESYNC_FILE_NAME, ReplicationConfig,
+    ReplicationResyncer, decode_resync_file, get_heal_replicate_object_info,
 };
 use crate::bucket::replication::replication_state::ReplicationStats;
 use crate::config::com::read_config;
@@ -41,7 +41,7 @@ use rustfs_filemeta::VersionPurgeStatusType;
 use rustfs_filemeta::replication_statuses_map;
 use rustfs_filemeta::version_purge_statuses_map;
 use rustfs_filemeta::{REPLICATE_EXISTING, REPLICATE_HEAL, REPLICATE_HEAL_DELETE};
-use rustfs_utils::http::RESERVED_METADATA_PREFIX_LOWER;
+use rustfs_utils::http::{SUFFIX_REPLICATION_TIMESTAMP, get_str};
 use std::any::Any;
 use std::sync::Arc;
 use std::sync::atomic::AtomicI32;
@@ -861,17 +861,8 @@ async fn load_bucket_resync_metadata<S: StorageAPI>(
     bucket: &str,
     obj_api: Arc<S>,
 ) -> Result<BucketReplicationResyncStatus, EcstoreError> {
-    use std::convert::TryInto;
-
     let mut brs = BucketReplicationResyncStatus::new();
 
-    // Constants that would be defined elsewhere
-    const REPLICATION_DIR: &str = "replication";
-    const RESYNC_FILE_NAME: &str = "resync.bin";
-    const RESYNC_META_FORMAT: u16 = 1;
-    const RESYNC_META_VERSION: u16 = 1;
-    const RESYNC_META_VERSION_V1: u16 = 1;
-
     let resync_dir_path = format!("{BUCKET_META_PREFIX}/{bucket}/{REPLICATION_DIR}");
     let resync_file_path = format!("{resync_dir_path}/{RESYNC_FILE_NAME}");
 
@@ -886,27 +877,7 @@ async fn load_bucket_resync_metadata<S: StorageAPI>(
         return Ok(brs);
     }
 
-    if data.len() <= 4 {
-        return Err(EcstoreError::CorruptedFormat);
-    }
-
-    // Read resync meta header
-    let format = u16::from_le_bytes(data[0..2].try_into().unwrap());
-    if format != RESYNC_META_FORMAT {
-        return Err(EcstoreError::CorruptedFormat);
-    }
-
-    let version = u16::from_le_bytes(data[2..4].try_into().unwrap());
-    if version != RESYNC_META_VERSION {
-        return Err(EcstoreError::CorruptedFormat);
-    }
-
-    // Parse data
-    brs = BucketReplicationResyncStatus::unmarshal_msg(&data[4..])?;
-
-    if brs.version != RESYNC_META_VERSION_V1 {
-        return Err(EcstoreError::CorruptedFormat);
-    }
+    brs = decode_resync_file(&data)?;
 
     Ok(brs)
 }
@@ -984,10 +955,8 @@ pub fn get_global_replication_pool() -> Option<Arc<DynReplicationPool>> {
 pub async fn schedule_replication<S: StorageAPI>(oi: ObjectInfo, o: Arc<S>, dsc: ReplicateDecision, op_type: ReplicationType) {
     let tgt_statuses = replication_statuses_map(&oi.replication_status_internal.clone().unwrap_or_default());
     let purge_statuses = version_purge_statuses_map(&oi.version_purge_status_internal.clone().unwrap_or_default());
-    let tm = oi
-        .user_defined
-        .get(&format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replication-timestamp"))
-        .map(|v| OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH));
+    let tm = get_str(&oi.user_defined, SUFFIX_REPLICATION_TIMESTAMP)
+        .map(|v| OffsetDateTime::parse(&v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH));
     let mut rstate = oi.replication_state();
     rstate.replicate_decision_str = dsc.to_string();
     let asz = oi.get_actual_size().unwrap_or_default();

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -17,6 +17,7 @@ use crate::bucket::bucket_target_sys::{
     AdvancedPutOptions, BucketTargetSys, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
 };
 use crate::bucket::metadata_sys;
+use crate::bucket::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
 use crate::bucket::replication::ResyncStatusType;
 use crate::bucket::replication::replication_pool::GLOBAL_REPLICATION_STATS;
 use crate::bucket::replication::{ObjectOpts, ReplicationConfigurationExt as _};
@@ -50,7 +51,7 @@ use http_body::Frame;
 use http_body_util::StreamBody;
 use regex::Regex;
 use rustfs_filemeta::{
-    MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATION_RESET, ReplicateDecision, ReplicateObjectInfo,
+    MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, ReplicateDecision, ReplicateObjectInfo,
     ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType,
     ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision, VersionPurgeStatusType,
     get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
@@ -58,8 +59,13 @@ use rustfs_filemeta::{
 use rustfs_s3_common::EventName;
 use rustfs_utils::http::{
     AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_TAGGING, AMZ_TAGGING_DIRECTIVE, CONTENT_ENCODING, HeaderExt as _,
-    RESERVED_METADATA_PREFIX, RESERVED_METADATA_PREFIX_LOWER, RUSTFS_REPLICATION_ACTUAL_OBJECT_SIZE,
-    RUSTFS_REPLICATION_RESET_STATUS, SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, headers,
+    SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP,
+    SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_RESET_ARN_PREFIX,
+    SUFFIX_REPLICATION_STATUS, SUFFIX_TAGGING_TIMESTAMP, headers,
+};
+use rustfs_utils::http::{
+    SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_RESET_STATUS, SUFFIX_REPLICATION_SSEC_CRC, get_header_map, get_str,
+    has_internal_suffix, insert_header_map, insert_str, internal_key_strip_suffix_prefix, is_internal_key,
 };
 use rustfs_utils::path::path_join_buf;
 use rustfs_utils::string::strings_has_prefix_fold;
@@ -69,7 +75,8 @@ use serde::Deserialize;
 use serde::Serialize;
 use std::any::Any;
 use std::collections::HashMap;
-use std::sync::Arc;
+use std::io::{Cursor, Read};
+use std::sync::{Arc, LazyLock};
 use time::OffsetDateTime;
 use time::format_description::well_known::Rfc3339;
 use tokio::io::AsyncRead;
@@ -80,14 +87,29 @@ use tokio_util::io::ReaderStream;
 use tokio_util::sync::CancellationToken;
 use tracing::{error, info, instrument, warn};
 
-const REPLICATION_DIR: &str = ".replication";
-const RESYNC_FILE_NAME: &str = "resync.bin";
-const RESYNC_META_FORMAT: u16 = 1;
-const RESYNC_META_VERSION: u16 = 1;
+pub(crate) const REPLICATION_DIR: &str = ".replication";
+pub(crate) const RESYNC_FILE_NAME: &str = "resync.bin";
+pub(crate) const RESYNC_META_FORMAT: u16 = 1;
+pub(crate) const RESYNC_META_VERSION: u16 = 1;
 const RESYNC_TIME_INTERVAL: TokioDuration = TokioDuration::from_secs(60);
+const WIRE_ZERO_TIME_UNIX: i64 = -62_135_596_800;
+
+static WIRE_ZERO_TIME: LazyLock<OffsetDateTime> =
+    LazyLock::new(|| OffsetDateTime::from_unix_timestamp(WIRE_ZERO_TIME_UNIX).unwrap_or(OffsetDateTime::UNIX_EPOCH));
 
 static WARNED_MONITOR_UNINIT: std::sync::Once = std::sync::Once::new();
 
+fn wire_time_or_default(value: Option<OffsetDateTime>) -> OffsetDateTime {
+    value.unwrap_or(*WIRE_ZERO_TIME)
+}
+
+fn normalize_wire_time(value: Option<OffsetDateTime>) -> Option<OffsetDateTime> {
+    match value {
+        Some(v) if v == *WIRE_ZERO_TIME || v == OffsetDateTime::UNIX_EPOCH => None,
+        other => other,
+    }
+}
+
 #[derive(Debug, Clone, Default)]
 pub struct ResyncOpts {
     pub bucket: String,
@@ -139,14 +161,199 @@ impl BucketReplicationResyncStatus {
     }
 
     pub fn marshal_msg(&self) -> Result<Vec<u8>> {
-        Ok(rmp_serde::to_vec(&self)?)
+        let mut wr = Vec::new();
+        rmp::encode::write_map_len(&mut wr, 4)?;
+        rmp::encode::write_str(&mut wr, "v")?;
+        rmp::encode::write_i32(&mut wr, i32::from(self.version))?;
+        rmp::encode::write_str(&mut wr, "brs")?;
+        rmp::encode::write_map_len(&mut wr, self.targets_map.len() as u32)?;
+        for (arn, status) in &self.targets_map {
+            rmp::encode::write_str(&mut wr, arn)?;
+            status.marshal_wire_msg(&mut wr)?;
+        }
+        rmp::encode::write_str(&mut wr, "id")?;
+        rmp::encode::write_i32(&mut wr, self.id)?;
+        rmp::encode::write_str(&mut wr, "lu")?;
+        write_msgp_time(&mut wr, wire_time_or_default(self.last_update))?;
+        Ok(wr)
     }
 
     pub fn unmarshal_msg(data: &[u8]) -> Result<Self> {
+        let mut rd = Cursor::new(data);
+        let mut out = Self::new();
+        let mut fields = rmp::decode::read_map_len(&mut rd)?;
+
+        while fields > 0 {
+            fields -= 1;
+            let key = read_msgp_str(&mut rd)?;
+            match key.as_str() {
+                "v" => {
+                    let v: i32 = rmp::decode::read_int(&mut rd)?;
+                    out.version = u16::try_from(v).map_err(|_| Error::other("invalid resync version"))?;
+                }
+                "brs" => {
+                    let map_len = rmp::decode::read_map_len(&mut rd)?;
+                    let mut targets = HashMap::with_capacity(map_len as usize);
+                    for _ in 0..map_len {
+                        let arn = read_msgp_str(&mut rd)?;
+                        let status = TargetReplicationResyncStatus::unmarshal_wire_msg(&mut rd)?;
+                        targets.insert(arn, status);
+                    }
+                    out.targets_map = targets;
+                }
+                "id" => {
+                    out.id = rmp::decode::read_int::<i32, _>(&mut rd)?;
+                }
+                "lu" => {
+                    out.last_update = normalize_wire_time(read_msgp_time_or_nil(&mut rd)?);
+                }
+                _ => skip_msgp_value(&mut rd)?,
+            }
+        }
+        Ok(out)
+    }
+
+    pub fn unmarshal_legacy_msg(data: &[u8]) -> Result<Self> {
         Ok(rmp_serde::from_slice(data)?)
     }
 }
 
+pub(crate) fn encode_resync_file(status: &BucketReplicationResyncStatus) -> Result<Vec<u8>> {
+    let payload = status.marshal_msg()?;
+    let mut data = Vec::with_capacity(4 + payload.len());
+    let mut major = [0u8; 2];
+    byteorder::LittleEndian::write_u16(&mut major, RESYNC_META_FORMAT);
+    data.extend_from_slice(&major);
+    let mut minor = [0u8; 2];
+    byteorder::LittleEndian::write_u16(&mut minor, RESYNC_META_VERSION);
+    data.extend_from_slice(&minor);
+    data.extend_from_slice(&payload);
+    Ok(data)
+}
+
+pub(crate) fn decode_resync_file(data: &[u8]) -> Result<BucketReplicationResyncStatus> {
+    if data.len() <= 4 {
+        return Err(Error::CorruptedFormat);
+    }
+
+    let mut major = [0u8; 2];
+    major.copy_from_slice(&data[0..2]);
+    if byteorder::LittleEndian::read_u16(&major) != RESYNC_META_FORMAT {
+        return Err(Error::CorruptedFormat);
+    }
+
+    let mut minor = [0u8; 2];
+    minor.copy_from_slice(&data[2..4]);
+    if byteorder::LittleEndian::read_u16(&minor) != RESYNC_META_VERSION {
+        return Err(Error::CorruptedFormat);
+    }
+
+    let status = match BucketReplicationResyncStatus::unmarshal_msg(&data[4..]) {
+        Ok(v) => v,
+        Err(_) => BucketReplicationResyncStatus::unmarshal_legacy_msg(&data[4..])?,
+    };
+    if status.version != RESYNC_META_VERSION {
+        return Err(Error::CorruptedFormat);
+    }
+    Ok(status)
+}
+
+impl TargetReplicationResyncStatus {
+    fn marshal_wire_msg(&self, wr: &mut Vec<u8>) -> Result<()> {
+        rmp::encode::write_map_len(wr, 11)?;
+        rmp::encode::write_str(wr, "st")?;
+        write_msgp_time(wr, wire_time_or_default(self.start_time))?;
+        rmp::encode::write_str(wr, "lst")?;
+        write_msgp_time(wr, wire_time_or_default(self.last_update))?;
+        rmp::encode::write_str(wr, "id")?;
+        rmp::encode::write_str(wr, &self.resync_id)?;
+        rmp::encode::write_str(wr, "rdt")?;
+        write_msgp_time(wr, wire_time_or_default(self.resync_before_date))?;
+        rmp::encode::write_str(wr, "rst")?;
+        rmp::encode::write_i32(wr, resync_status_to_i32(self.resync_status))?;
+        rmp::encode::write_str(wr, "fs")?;
+        rmp::encode::write_i64(wr, self.failed_size)?;
+        rmp::encode::write_str(wr, "frc")?;
+        rmp::encode::write_i64(wr, self.failed_count)?;
+        rmp::encode::write_str(wr, "rs")?;
+        rmp::encode::write_i64(wr, self.replicated_size)?;
+        rmp::encode::write_str(wr, "rrc")?;
+        rmp::encode::write_i64(wr, self.replicated_count)?;
+        rmp::encode::write_str(wr, "bkt")?;
+        rmp::encode::write_str(wr, &self.bucket)?;
+        rmp::encode::write_str(wr, "obj")?;
+        rmp::encode::write_str(wr, &self.object)?;
+        Ok(())
+    }
+
+    fn unmarshal_wire_msg<R: Read>(rd: &mut R) -> Result<Self> {
+        let mut out = Self::new();
+        let mut fields = rmp::decode::read_map_len(rd)?;
+
+        while fields > 0 {
+            fields -= 1;
+            let key = read_msgp_str(rd)?;
+            match key.as_str() {
+                "st" => out.start_time = normalize_wire_time(read_msgp_time_or_nil(rd)?),
+                "lst" => out.last_update = normalize_wire_time(read_msgp_time_or_nil(rd)?),
+                "id" => out.resync_id = read_msgp_str(rd)?,
+                "rdt" => out.resync_before_date = normalize_wire_time(read_msgp_time_or_nil(rd)?),
+                "rst" => {
+                    let v: i32 = rmp::decode::read_int(rd)?;
+                    out.resync_status = resync_status_from_i32(v)?;
+                }
+                "fs" => out.failed_size = rmp::decode::read_int(rd)?,
+                "frc" => out.failed_count = rmp::decode::read_int(rd)?,
+                "rs" => out.replicated_size = rmp::decode::read_int(rd)?,
+                "rrc" => out.replicated_count = rmp::decode::read_int(rd)?,
+                "bkt" => out.bucket = read_msgp_str(rd)?,
+                "obj" => out.object = read_msgp_str(rd)?,
+                _ => skip_msgp_value(rd)?,
+            }
+        }
+        Ok(out)
+    }
+}
+
+fn read_msgp_str<R: Read>(rd: &mut R) -> Result<String> {
+    let len = rmp::decode::read_str_len(rd)? as usize;
+    let mut buf = vec![0u8; len];
+    rd.read_exact(&mut buf)?;
+    Ok(String::from_utf8(buf)?)
+}
+
+fn read_msgp_time_or_nil<R: Read>(rd: &mut R) -> Result<Option<OffsetDateTime>> {
+    let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
+    match marker {
+        rmp::Marker::Null => Ok(None),
+        rmp::Marker::Ext8 => Ok(Some(read_msgp_ext8_time(rd)?)),
+        other => Err(Error::other(format!("expected time ext or nil, got marker: {other:?}"))),
+    }
+}
+
+fn resync_status_to_i32(status: ResyncStatusType) -> i32 {
+    match status {
+        ResyncStatusType::NoResync => 0,
+        ResyncStatusType::ResyncPending => 1,
+        ResyncStatusType::ResyncCanceled => 2,
+        ResyncStatusType::ResyncStarted => 3,
+        ResyncStatusType::ResyncCompleted => 4,
+        ResyncStatusType::ResyncFailed => 5,
+    }
+}
+
+fn resync_status_from_i32(code: i32) -> Result<ResyncStatusType> {
+    match code {
+        0 => Ok(ResyncStatusType::NoResync),
+        1 => Ok(ResyncStatusType::ResyncPending),
+        2 => Ok(ResyncStatusType::ResyncCanceled),
+        3 => Ok(ResyncStatusType::ResyncStarted),
+        4 => Ok(ResyncStatusType::ResyncCompleted),
+        5 => Ok(ResyncStatusType::ResyncFailed),
+        _ => Err(Error::other(format!("invalid resync status code: {code}"))),
+    }
+}
+
 static RESYNC_WORKER_COUNT: usize = 10;
 
 #[derive(Debug)]
@@ -617,7 +824,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
 
         let keys_to_update: Vec<_> = user_defined
             .iter()
-            .filter(|(k, _)| k.eq_ignore_ascii_case(format!("{RESERVED_METADATA_PREFIX_LOWER}{REPLICATION_RESET}").as_str()))
+            .filter(|(k, _)| has_internal_suffix(k, SUFFIX_REPLICATION_RESET))
             .map(|(k, v)| (k.clone(), v.clone()))
             .collect();
 
@@ -695,19 +902,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
 }
 
 async fn save_resync_status<S: StorageAPI>(bucket: &str, status: &BucketReplicationResyncStatus, api: Arc<S>) -> Result<()> {
-    let buf = status.marshal_msg()?;
-
-    let mut data = Vec::new();
-
-    let mut major = [0u8; 2];
-    byteorder::LittleEndian::write_u16(&mut major, RESYNC_META_FORMAT);
-    data.extend_from_slice(&major);
-
-    let mut minor = [0u8; 2];
-    byteorder::LittleEndian::write_u16(&mut minor, RESYNC_META_VERSION);
-    data.extend_from_slice(&minor);
-
-    data.extend_from_slice(&buf);
+    let data = encode_resync_file(status)?;
 
     let config_file = path_join_buf(&[BUCKET_META_PREFIX, bucket, REPLICATION_DIR, RESYNC_FILE_NAME]);
     save_config(api, &config_file, data).await?;
@@ -900,8 +1095,8 @@ pub fn resync_target(
     let rs = oi
         .user_defined
         .get(target_reset_header(arn).as_str())
-        .or(oi.user_defined.get(RUSTFS_REPLICATION_RESET_STATUS))
-        .map(|s| s.to_string());
+        .cloned()
+        .or_else(|| get_header_map(&oi.user_defined, SUFFIX_REPLICATION_RESET_STATUS));
 
     let mut dec = ResyncTargetDecision::default();
 
@@ -1132,15 +1327,7 @@ impl ObjectInfoExt for ObjectInfo {
                 .user_defined
                 .iter()
                 .filter_map(|(k, v)| {
-                    if k.starts_with(&format!("{RESERVED_METADATA_PREFIX_LOWER}-{REPLICATION_RESET}")) {
-                        Some((
-                            k.trim_start_matches(&format!("{RESERVED_METADATA_PREFIX_LOWER}-{REPLICATION_RESET}"))
-                                .to_string(),
-                            v.clone(),
-                        ))
-                    } else {
-                        None
-                    }
+                    internal_key_strip_suffix_prefix(k, SUFFIX_REPLICATION_RESET_ARN_PREFIX).map(|arn| (arn, v.clone()))
                 })
                 .collect(),
             ..Default::default()
@@ -1893,7 +2080,7 @@ pub async fn replicate_object<S: StorageAPI>(roi: ReplicateObjectInfo, storage:
     if roi.replication_status_internal != new_replication_internal || rinfos.replication_resynced() {
         let mut eval_metadata = HashMap::new();
         if let Some(ref s) = new_replication_internal {
-            eval_metadata.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}replication-status"), s.clone());
+            insert_str(&mut eval_metadata, SUFFIX_REPLICATION_STATUS, s.clone());
         }
         let popts = ObjectOptions {
             version_id: roi.version_id.map(|v| v.to_string()),
@@ -2266,7 +2453,7 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             return rinfo;
         }
 
-        let sopts = StatObjectOptions {
+        let mut sopts = StatObjectOptions {
             version_id: object_info.version_id.map(|v| v.to_string()).unwrap_or_default(),
             internal: AdvancedGetOptions {
                 replication_proxy_request: "false".to_string(),
@@ -2275,7 +2462,9 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             ..Default::default()
         };
 
-        sopts.set(AMZ_TAGGING_DIRECTIVE, "ACCESS");
+        if let Err(err) = sopts.set(AMZ_TAGGING_DIRECTIVE, "ACCESS") {
+            warn!("failed to set replication tagging directive header: {err}");
+        }
 
         match tgt_client
             .head_object(&tgt_client.bucket, &object, self.version_id.map(|v| v.to_string()))
@@ -2549,8 +2738,6 @@ static VALID_SSE_REPLICATION_HEADERS: &[(&str, &str)] = &[
     ("X-Rustfs-Internal-Actual-Object-Size", "X-Rustfs-Replication-Actual-Object-Size"),
 ];
 
-const REPLICATION_SSEC_CHECKSUM_HEADER: &str = "X-Rustfs-Replication-Ssec-Crc";
-
 fn is_valid_sse_header(k: &str) -> Option<&str> {
     VALID_SSE_REPLICATION_HEADERS
         .iter()
@@ -2574,7 +2761,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
 
         // In case of SSE-C objects copy the allowed internal headers as well
         if !is_ssec || !has_valid_sse_header {
-            if strings_has_prefix_fold(k, RESERVED_METADATA_PREFIX) {
+            if is_internal_key(k) {
                 continue;
             }
             if is_standard_header(k) {
@@ -2598,7 +2785,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
         // Add encrypted CRC to metadata for SSE-C objects
         if is_ssec {
             let encoded = BASE64_STANDARD.encode(checksum_data);
-            meta.insert(REPLICATION_SSEC_CHECKSUM_HEADER.to_string(), encoded);
+            insert_header_map(&mut meta, SUFFIX_REPLICATION_SSEC_CRC, encoded);
         } else {
             // Get checksum metadata for non-SSE-C objects
             let (cs_meta, is_mp) = object_info.decrypt_checksums(0, &HeaderMap::new())?;
@@ -2658,11 +2845,8 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
         if !tags.is_empty() {
             put_op.user_tags = tags;
             // set tag timestamp in opts
-            put_op.internal.tagging_timestamp = if let Some(ts) = object_info
-                .user_defined
-                .get(&format!("{RESERVED_METADATA_PREFIX_LOWER}tagging-timestamp"))
-            {
-                OffsetDateTime::parse(ts, &Rfc3339)
+            put_op.internal.tagging_timestamp = if let Some(ts) = get_str(&object_info.user_defined, SUFFIX_TAGGING_TIMESTAMP) {
+                OffsetDateTime::parse(&ts, &Rfc3339)
                     .map_err(|e| Error::other(format!("Failed to parse tagging timestamp: {}", e)))?
             } else {
                 object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
@@ -2694,28 +2878,24 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
         put_op.retain_until_date =
             OffsetDateTime::parse(v, &Rfc3339).map_err(|e| Error::other(format!("Failed to parse retain until date: {}", e)))?;
         // set retention timestamp in opts
-        put_op.internal.retention_timestamp = if let Some(v) = object_info
-            .user_defined
-            .get(&format!("{RESERVED_METADATA_PREFIX_LOWER}objectlock-retention-timestamp"))
-        {
-            OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
-        } else {
-            object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
-        };
+        put_op.internal.retention_timestamp =
+            if let Some(v) = get_str(&object_info.user_defined, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP) {
+                OffsetDateTime::parse(&v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
+            } else {
+                object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
+            };
     }
 
     if let Some(v) = lk_map.lookup(AMZ_OBJECT_LOCK_LEGAL_HOLD) {
         let hold = v.to_uppercase();
         put_op.legalhold = Some(ObjectLockLegalHoldStatus::from(hold.as_str()));
         // set legalhold timestamp in opts
-        put_op.internal.legalhold_timestamp = if let Some(v) = object_info
-            .user_defined
-            .get(&format!("{RESERVED_METADATA_PREFIX_LOWER}objectlock-legalhold-timestamp"))
-        {
-            OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
-        } else {
-            object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
-        };
+        put_op.internal.legalhold_timestamp =
+            if let Some(v) = get_str(&object_info.user_defined, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP) {
+                OffsetDateTime::parse(&v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
+            } else {
+                object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
+            };
     }
 
     // Handle SSE-S3 encryption
@@ -2736,7 +2916,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
         // If KMS key ID replication is enabled (as by default)
         // we include the object's KMS key ID. In any case, we
         // always set the SSE-KMS header. If no KMS key ID is
-        // specified, MinIO is supposed to use whatever default
+        // specified, the server uses the default applicable
         // config applies on the site or bucket.
         // TODO: Implement SSE-KMS support with key ID replication
         // let key_id = if kms::replicate_key_id() {
@@ -2859,13 +3039,10 @@ async fn replicate_object_with_multipart<S: StorageAPI>(ctx: MultipartReplicatio
 
     let mut user_metadata = HashMap::new();
 
-    user_metadata.insert(
-        RUSTFS_REPLICATION_ACTUAL_OBJECT_SIZE.to_string(),
-        object_info
-            .user_defined
-            .get(&format!("{RESERVED_METADATA_PREFIX}actual-size"))
-            .map(|v| v.to_string())
-            .unwrap_or_default(),
+    insert_header_map(
+        &mut user_metadata,
+        SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE,
+        rustfs_utils::http::get_str(&object_info.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE).unwrap_or_default(),
     );
 
     cli.complete_multipart_upload(
@@ -2994,6 +3171,9 @@ fn get_replication_action(oi1: &ObjectInfo, oi2: &HeadObjectOutput, op_type: Rep
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::bucket::msgp_decode::write_msgp_time;
+    use std::collections::HashMap;
+    use time::OffsetDateTime;
     use uuid::Uuid;
 
     #[test]
@@ -3010,6 +3190,176 @@ mod tests {
         assert!(part_range_spec_from_actual_size(0, -1).is_err());
     }
 
+    #[test]
+    fn test_unmarshal_resync_payload() {
+        let start = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid ts");
+        let last = OffsetDateTime::from_unix_timestamp(1_700_000_123).expect("valid ts");
+        let before = OffsetDateTime::from_unix_timestamp(1_699_000_000).expect("valid ts");
+        let bucket_last = OffsetDateTime::from_unix_timestamp(1_700_111_111).expect("valid ts");
+
+        let mut payload = Vec::new();
+        rmp::encode::write_map_len(&mut payload, 4).expect("write map");
+        rmp::encode::write_str(&mut payload, "v").expect("write key");
+        rmp::encode::write_i32(&mut payload, 1).expect("write version");
+        rmp::encode::write_str(&mut payload, "brs").expect("write key");
+        rmp::encode::write_map_len(&mut payload, 1).expect("write target map");
+        rmp::encode::write_str(&mut payload, "arn:replication::1:dest").expect("write arn");
+        rmp::encode::write_map_len(&mut payload, 11).expect("write target");
+        rmp::encode::write_str(&mut payload, "st").expect("write key");
+        write_msgp_time(&mut payload, start).expect("write time");
+        rmp::encode::write_str(&mut payload, "lst").expect("write key");
+        write_msgp_time(&mut payload, last).expect("write time");
+        rmp::encode::write_str(&mut payload, "id").expect("write key");
+        rmp::encode::write_str(&mut payload, "resync-1").expect("write id");
+        rmp::encode::write_str(&mut payload, "rdt").expect("write key");
+        write_msgp_time(&mut payload, before).expect("write time");
+        rmp::encode::write_str(&mut payload, "rst").expect("write key");
+        rmp::encode::write_i32(&mut payload, 3).expect("write status");
+        rmp::encode::write_str(&mut payload, "fs").expect("write key");
+        rmp::encode::write_i64(&mut payload, 11).expect("write fs");
+        rmp::encode::write_str(&mut payload, "frc").expect("write key");
+        rmp::encode::write_i64(&mut payload, 2).expect("write frc");
+        rmp::encode::write_str(&mut payload, "rs").expect("write key");
+        rmp::encode::write_i64(&mut payload, 101).expect("write rs");
+        rmp::encode::write_str(&mut payload, "rrc").expect("write key");
+        rmp::encode::write_i64(&mut payload, 9).expect("write rrc");
+        rmp::encode::write_str(&mut payload, "bkt").expect("write key");
+        rmp::encode::write_str(&mut payload, "bucket-a").expect("write bucket");
+        rmp::encode::write_str(&mut payload, "obj").expect("write key");
+        rmp::encode::write_str(&mut payload, "object-a").expect("write obj");
+        rmp::encode::write_str(&mut payload, "id").expect("write key");
+        rmp::encode::write_i32(&mut payload, 42).expect("write id");
+        rmp::encode::write_str(&mut payload, "lu").expect("write key");
+        write_msgp_time(&mut payload, bucket_last).expect("write lu");
+
+        let got = BucketReplicationResyncStatus::unmarshal_msg(&payload).expect("decode");
+        assert_eq!(got.version, 1);
+        assert_eq!(got.id, 42);
+        assert_eq!(got.last_update, Some(bucket_last));
+        let tgt = got.targets_map.get("arn:replication::1:dest").expect("target exists");
+        assert_eq!(tgt.resync_id, "resync-1");
+        assert_eq!(tgt.resync_status, ResyncStatusType::ResyncStarted);
+        assert_eq!(tgt.bucket, "bucket-a");
+        assert_eq!(tgt.object, "object-a");
+        assert_eq!(tgt.start_time, Some(start));
+        assert_eq!(tgt.last_update, Some(last));
+        assert_eq!(tgt.resync_before_date, Some(before));
+    }
+
+    #[test]
+    fn test_unmarshal_legacy_resync_payload() {
+        let mut status = BucketReplicationResyncStatus::new();
+        status.id = 7;
+        status.version = 1;
+        status.last_update = Some(OffsetDateTime::from_unix_timestamp(1_700_222_222).expect("valid ts"));
+        status.targets_map = HashMap::from([(
+            "legacy-arn".to_string(),
+            TargetReplicationResyncStatus {
+                resync_id: "legacy-1".to_string(),
+                resync_status: ResyncStatusType::ResyncCompleted,
+                ..Default::default()
+            },
+        )]);
+
+        let old_payload = rmp_serde::to_vec(&status).expect("legacy encode");
+        let got = BucketReplicationResyncStatus::unmarshal_legacy_msg(&old_payload).expect("legacy decode");
+        assert_eq!(got.id, 7);
+        assert_eq!(got.version, 1);
+        assert_eq!(got.targets_map["legacy-arn"].resync_id, "legacy-1");
+        assert_eq!(got.targets_map["legacy-arn"].resync_status, ResyncStatusType::ResyncCompleted);
+    }
+
+    #[test]
+    fn test_resync_file_roundtrip_wire_format() {
+        let mut status = BucketReplicationResyncStatus::new();
+        status.id = 19;
+        status.last_update = Some(OffsetDateTime::from_unix_timestamp(1_700_333_333).expect("valid ts"));
+        status.targets_map = HashMap::from([(
+            "arn:replication::1:dest".to_string(),
+            TargetReplicationResyncStatus {
+                resync_id: "wire-1".to_string(),
+                resync_status: ResyncStatusType::ResyncStarted,
+                replicated_count: 5,
+                ..Default::default()
+            },
+        )]);
+
+        let bytes = encode_resync_file(&status).expect("encode file");
+        assert_eq!(&bytes[0..2], &RESYNC_META_FORMAT.to_le_bytes());
+        assert_eq!(&bytes[2..4], &RESYNC_META_VERSION.to_le_bytes());
+
+        let got = decode_resync_file(&bytes).expect("decode file");
+        assert_eq!(got.version, RESYNC_META_VERSION);
+        assert_eq!(got.id, 19);
+        assert_eq!(got.targets_map["arn:replication::1:dest"].resync_id, "wire-1");
+        assert_eq!(got.targets_map["arn:replication::1:dest"].replicated_count, 5);
+    }
+
+    #[test]
+    fn test_resync_file_decodes_legacy_payload() {
+        let mut status = BucketReplicationResyncStatus::new();
+        status.id = 7;
+        status.version = RESYNC_META_VERSION;
+        status.targets_map = HashMap::from([(
+            "legacy-arn".to_string(),
+            TargetReplicationResyncStatus {
+                resync_id: "legacy-v1".to_string(),
+                resync_status: ResyncStatusType::ResyncCompleted,
+                ..Default::default()
+            },
+        )]);
+
+        let legacy_payload = rmp_serde::to_vec(&status).expect("legacy encode");
+        let mut file_bytes = Vec::new();
+        file_bytes.extend_from_slice(&RESYNC_META_FORMAT.to_le_bytes());
+        file_bytes.extend_from_slice(&RESYNC_META_VERSION.to_le_bytes());
+        file_bytes.extend_from_slice(&legacy_payload);
+
+        let got = decode_resync_file(&file_bytes).expect("decode legacy");
+        assert_eq!(got.id, 7);
+        assert_eq!(got.targets_map["legacy-arn"].resync_id, "legacy-v1");
+        assert_eq!(got.targets_map["legacy-arn"].resync_status, ResyncStatusType::ResyncCompleted);
+    }
+
+    #[test]
+    fn test_resync_none_time_encodes_as_wire_zero_and_decodes_to_none() {
+        let wire_zero = OffsetDateTime::from_unix_timestamp(WIRE_ZERO_TIME_UNIX).expect("valid wire zero timestamp");
+
+        let mut with_none = BucketReplicationResyncStatus::new();
+        with_none.id = 77;
+        with_none.targets_map = HashMap::from([(
+            "arn:replication::1:dest".to_string(),
+            TargetReplicationResyncStatus {
+                resync_id: "wire-none".to_string(),
+                resync_status: ResyncStatusType::ResyncStarted,
+                replicated_count: 1,
+                ..Default::default()
+            },
+        )]);
+
+        let mut with_zero = with_none.clone();
+        with_zero.last_update = Some(wire_zero);
+        if let Some(target) = with_zero.targets_map.get_mut("arn:replication::1:dest") {
+            target.start_time = Some(wire_zero);
+            target.last_update = Some(wire_zero);
+            target.resync_before_date = Some(wire_zero);
+        }
+
+        let encoded_none = encode_resync_file(&with_none).expect("encode with none");
+        let encoded_zero = encode_resync_file(&with_zero).expect("encode with zero");
+        assert_eq!(encoded_none, encoded_zero);
+
+        let decoded = decode_resync_file(&encoded_none).expect("decode");
+        let target = decoded
+            .targets_map
+            .get("arn:replication::1:dest")
+            .expect("target should exist");
+        assert_eq!(decoded.last_update, None);
+        assert_eq!(target.start_time, None);
+        assert_eq!(target.last_update, None);
+        assert_eq!(target.resync_before_date, None);
+    }
+
     #[test]
     fn test_heal_should_use_check_replicate_delete_failed_non_delete_marker() {
         let oi = ObjectInfo {

crates/ecstore/src/bucket/utils.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::disk::RUSTFS_META_BUCKET;
+use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result, StorageError};
 use regex::Regex;
 use rustfs_utils::path::SLASH_SEPARATOR;
@@ -20,7 +20,7 @@ use s3s::xml;
 use tracing::instrument;
 
 pub fn is_meta_bucketname(name: &str) -> bool {
-    name.starts_with(RUSTFS_META_BUCKET)
+    name.starts_with(RUSTFS_META_BUCKET) || name.starts_with(MIGRATING_META_BUCKET)
 }
 
 lazy_static::lazy_static! {

crates/ecstore/src/client/api_get_options.rs

@@ -64,20 +64,32 @@ impl GetObjectOptions {
     pub fn header(&self) -> HeaderMap {
         let mut headers: HeaderMap = HeaderMap::with_capacity(self.headers.len());
         for (k, v) in &self.headers {
-            if let Ok(header_name) = HeaderName::from_bytes(k.as_bytes()) {
-                headers.insert(header_name, v.parse().expect("err"));
-            } else {
-                warn!("Invalid header name: {}", k);
+            match (HeaderName::from_bytes(k.as_bytes()), HeaderValue::from_str(v)) {
+                (Ok(header_name), Ok(header_value)) => {
+                    headers.insert(header_name, header_value);
+                }
+                (Err(_), _) => {
+                    warn!("Invalid header name: {}", k);
+                }
+                (_, Err(_)) => {
+                    warn!("Invalid header value for {}: {:?}", k, v);
+                }
             }
         }
         if self.checksum {
-            headers.insert("x-amz-checksum-mode", "ENABLED".parse().expect("err"));
+            headers.insert(HeaderName::from_static("x-amz-checksum-mode"), HeaderValue::from_static("ENABLED"));
         }
         headers
     }
 
-    pub fn set(&self, key: &str, value: &str) {
-        //self.headers[http.CanonicalHeaderKey(key)] = value;
+    pub fn set(&mut self, key: &str, value: &str) -> Result<(), std::io::Error> {
+        let header_name = HeaderName::from_bytes(key.as_bytes())
+            .map_err(|err| std::io::Error::other(err_invalid_argument(&format!("Invalid header name {key}: {err}"))))?;
+        HeaderValue::from_str(value)
+            .map_err(|err| std::io::Error::other(err_invalid_argument(&format!("Invalid header value for {key}: {err}"))))?;
+
+        self.headers.insert(header_name.as_str().to_string(), value.to_string());
+        Ok(())
     }
 
     pub fn set_req_param(&mut self, key: &str, value: &str) {
@@ -89,12 +101,12 @@ impl GetObjectOptions {
     }
 
     pub fn set_match_etag(&mut self, etag: &str) -> Result<(), std::io::Error> {
-        self.set("If-Match", &format!("\"{etag}\""));
+        self.set("If-Match", &format!("\"{etag}\""))?;
         Ok(())
     }
 
     pub fn set_match_etag_except(&mut self, etag: &str) -> Result<(), std::io::Error> {
-        self.set("If-None-Match", &format!("\"{etag}\""));
+        self.set("If-None-Match", &format!("\"{etag}\""))?;
         Ok(())
     }
 
@@ -102,7 +114,7 @@ impl GetObjectOptions {
         if mod_time.unix_timestamp() == 0 {
             return Err(std::io::Error::other(err_invalid_argument("Modified since cannot be empty.")));
         }
-        self.set("If-Unmodified-Since", &mod_time.to_string());
+        self.set("If-Unmodified-Since", &mod_time.to_string())?;
         Ok(())
     }
 
@@ -110,17 +122,17 @@ impl GetObjectOptions {
         if mod_time.unix_timestamp() == 0 {
             return Err(std::io::Error::other(err_invalid_argument("Modified since cannot be empty.")));
         }
-        self.set("If-Modified-Since", &mod_time.to_string());
+        self.set("If-Modified-Since", &mod_time.to_string())?;
         Ok(())
     }
 
     pub fn set_range(&mut self, start: i64, end: i64) -> Result<(), std::io::Error> {
         if start == 0 && end < 0 {
-            self.set("Range", &format!("bytes={}", end));
+            self.set("Range", &format!("bytes={}", end))?;
         } else if 0 < start && end == 0 {
-            self.set("Range", &format!("bytes={}-", start));
+            self.set("Range", &format!("bytes={}-", start))?;
         } else if 0 <= start && start <= end {
-            self.set("Range", &format!("bytes={}-{}", start, end));
+            self.set("Range", &format!("bytes={}-{}", start, end))?;
         } else {
             return Err(std::io::Error::other(err_invalid_argument(&format!(
                 "Invalid range specified: start={} end={}",
@@ -146,3 +158,40 @@ impl GetObjectOptions {
         url_values
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::GetObjectOptions;
+
+    #[test]
+    fn set_range_populates_range_header() {
+        let mut opts = GetObjectOptions::default();
+        opts.set_range(5, 9).expect("valid range should succeed");
+
+        let headers = opts.header();
+        let range = headers.get("range").expect("range header should be present");
+        assert_eq!(range.to_str().expect("range header must be valid ascii"), "bytes=5-9");
+    }
+
+    #[test]
+    fn set_rejects_invalid_header_value() {
+        let mut opts = GetObjectOptions::default();
+
+        let err = opts
+            .set("Range", "bytes=5-\n9")
+            .expect_err("invalid header value should fail");
+
+        assert!(err.to_string().contains("Invalid header value"));
+        assert!(opts.headers.is_empty(), "invalid headers must not be stored");
+    }
+
+    #[test]
+    fn header_skips_invalid_prepopulated_header_value() {
+        let mut opts = GetObjectOptions::default();
+        opts.headers.insert("Range".to_string(), "bytes=5-\n9".to_string());
+
+        let headers = opts.header();
+
+        assert!(headers.get("range").is_none(), "invalid stored header values should be ignored");
+    }
+}

crates/ecstore/src/client/transition_api.rs

@@ -375,7 +375,7 @@ impl TransitionClient {
         //debug!("http_resp_body: {}", String::from_utf8(b).unwrap());
 
         //if self.is_trace_enabled && !(self.trace_errors_only && resp.status() == StatusCode::OK) {
-        if resp.status() != StatusCode::OK {
+        if !resp.status().is_success() {
             //self.dump_http(&cloned_req, &resp)?;
             let mut body_vec = Vec::new();
             let mut body = resp.into_body();

crates/ecstore/src/config/com.rs

@@ -12,17 +12,20 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::config::{Config, GLOBAL_STORAGE_CLASS, storageclass};
-use crate::disk::RUSTFS_META_BUCKET;
+use crate::config::{Config, GLOBAL_STORAGE_CLASS, KVS, oidc, storageclass};
+use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result};
+use crate::global::is_first_cluster_node_local;
 use crate::store_api::{ObjectInfo, ObjectOptions, PutObjReader, StorageAPI};
 use http::HeaderMap;
-use rustfs_config::DEFAULT_DELIMITER;
+use rustfs_config::oidc::{IDENTITY_OPENID_KEYS, IDENTITY_OPENID_SUB_SYS, OIDC_REDIRECT_URI_DYNAMIC};
+use rustfs_config::{COMMENT_KEY, DEFAULT_DELIMITER, ENABLE_KEY, EnableState, RUSTFS_REGION};
 use rustfs_utils::path::SLASH_SEPARATOR;
-use std::collections::HashSet;
+use serde_json::{Map, Value};
+use std::collections::{HashMap, HashSet};
 use std::sync::Arc;
 use std::sync::LazyLock;
-use tracing::{error, instrument, warn};
+use tracing::{debug, error, info, instrument, warn};
 
 pub const CONFIG_PREFIX: &str = "config";
 const CONFIG_FILE: &str = "config.json";
@@ -43,6 +46,19 @@ pub async fn read_config<S: StorageAPI>(api: Arc<S>, file: &str) -> Result<Vec<u
     Ok(data)
 }
 
+pub async fn read_config_no_lock<S: StorageAPI>(api: Arc<S>, file: &str) -> Result<Vec<u8>> {
+    let (data, _obj) = read_config_with_metadata(
+        api,
+        file,
+        &ObjectOptions {
+            no_lock: true,
+            ..Default::default()
+        },
+    )
+    .await?;
+    Ok(data)
+}
+
 pub async fn read_config_with_metadata<S: StorageAPI>(
     api: Arc<S>,
     file: &str,
@@ -136,10 +152,446 @@ fn get_config_file() -> String {
     format!("{CONFIG_PREFIX}{SLASH_SEPARATOR}{CONFIG_FILE}")
 }
 
+fn storage_class_kvs_mut(cfg: &mut Config) -> &mut crate::config::KVS {
+    let sub_cfg = cfg.0.entry(STORAGE_CLASS_SUB_SYS.to_string()).or_insert_with(|| {
+        let mut section = HashMap::new();
+        section.insert(DEFAULT_DELIMITER.to_string(), storageclass::DEFAULT_KVS.clone());
+        section
+    });
+    sub_cfg
+        .entry(DEFAULT_DELIMITER.to_string())
+        .or_insert_with(|| storageclass::DEFAULT_KVS.clone())
+}
+
+fn parse_storage_class_value(value: &Value) -> Option<String> {
+    match value {
+        Value::String(v) => Some(v.trim().to_string()),
+        Value::Object(m) => m
+            .get("parity")
+            .and_then(Value::as_u64)
+            .map(|parity| if parity == 0 { String::new() } else { format!("EC:{parity}") }),
+        _ => None,
+    }
+}
+
+fn parse_inline_block_value(value: &Value) -> Option<String> {
+    match value {
+        Value::String(v) if !v.trim().is_empty() => Some(v.trim().to_string()),
+        Value::Number(v) => Some(v.to_string()),
+        _ => None,
+    }
+}
+
+fn parse_oidc_scalar_value(key: &str, value: &Value) -> Option<String> {
+    match value {
+        Value::String(v) => Some(v.trim().to_string()),
+        Value::Bool(v) if key == ENABLE_KEY || key == OIDC_REDIRECT_URI_DYNAMIC => Some(if *v {
+            EnableState::On.to_string()
+        } else {
+            EnableState::Off.to_string()
+        }),
+        Value::Bool(v) => Some(v.to_string()),
+        Value::Number(v) => Some(v.to_string()),
+        Value::Array(values) if key == rustfs_config::oidc::OIDC_SCOPES => {
+            let scopes = values
+                .iter()
+                .filter_map(Value::as_str)
+                .map(str::trim)
+                .filter(|scope| !scope.is_empty())
+                .collect::<Vec<_>>()
+                .join(",");
+            Some(scopes)
+        }
+        Value::Null => None,
+        _ => None,
+    }
+}
+
+fn decode_oidc_provider_object(provider: &Map<String, Value>) -> KVS {
+    let mut kvs = oidc::DEFAULT_IDENTITY_OPENID_KVS.clone();
+
+    for (key, value) in provider {
+        if !IDENTITY_OPENID_KEYS.contains(&key.as_str()) || key == COMMENT_KEY {
+            continue;
+        }
+
+        if let Some(parsed) = parse_oidc_scalar_value(key, value) {
+            kvs.insert(key.clone(), parsed);
+        }
+    }
+
+    kvs
+}
+
+fn apply_external_oidc_map(cfg: &mut Config, root: &Map<String, Value>) -> bool {
+    let oidc_root = root.get("openid").or_else(|| root.get(IDENTITY_OPENID_SUB_SYS));
+    let Some(Value::Object(oidc_obj)) = oidc_root else {
+        return false;
+    };
+
+    if oidc_obj.is_empty() {
+        return false;
+    }
+
+    let subsystem = cfg.0.entry(IDENTITY_OPENID_SUB_SYS.to_string()).or_default();
+    let mut applied = false;
+
+    for (raw_instance, provider) in oidc_obj {
+        let instance_key = if raw_instance == "default" {
+            DEFAULT_DELIMITER.to_string()
+        } else {
+            raw_instance.to_string()
+        };
+
+        match provider {
+            Value::Object(provider_obj) => {
+                subsystem.insert(instance_key, decode_oidc_provider_object(provider_obj));
+                applied = true;
+            }
+            Value::Array(_) => {
+                if let Ok(kvs) = serde_json::from_value::<KVS>(provider.clone()) {
+                    subsystem.insert(instance_key, kvs);
+                    applied = true;
+                }
+            }
+            _ => {}
+        }
+    }
+
+    applied
+}
+
+fn apply_external_storage_class_map(cfg: &mut Config, root: &Map<String, Value>) -> bool {
+    let sc = root.get("storageclass").or_else(|| root.get("storage_class"));
+    let Some(Value::Object(sc_obj)) = sc else {
+        return false;
+    };
+
+    let mut applied = false;
+    let kvs = storage_class_kvs_mut(cfg);
+
+    if let Some(v) = sc_obj.get("standard").and_then(parse_storage_class_value) {
+        kvs.insert(storageclass::CLASS_STANDARD.to_string(), v);
+        applied = true;
+    }
+    if let Some(v) = sc_obj.get("rrs").and_then(parse_storage_class_value) {
+        kvs.insert(storageclass::CLASS_RRS.to_string(), v);
+        applied = true;
+    }
+    if let Some(Value::String(v)) = sc_obj.get("optimize")
+        && !v.trim().is_empty()
+    {
+        kvs.insert(storageclass::OPTIMIZE.to_string(), v.clone());
+        applied = true;
+    }
+    if let Some(v) = sc_obj.get("inline_block").and_then(parse_inline_block_value) {
+        kvs.insert(storageclass::INLINE_BLOCK.to_string(), v);
+        applied = true;
+    }
+
+    applied
+}
+
+fn decode_server_config_blob(data: &[u8]) -> Result<Config> {
+    if let Ok(cfg) = Config::unmarshal(data) {
+        return Ok(cfg);
+    }
+
+    let value: Value = serde_json::from_slice(data)?;
+    let Value::Object(root) = value else {
+        return Err(Error::other("unrecognized external server config shape"));
+    };
+
+    let mut cfg = Config::new();
+    let has_storage = apply_external_storage_class_map(&mut cfg, &root);
+    let has_oidc = apply_external_oidc_map(&mut cfg, &root);
+    let has_header = root.contains_key("version") || root.contains_key("region") || root.contains_key("credential");
+    if !has_storage && !has_oidc && !has_header {
+        return Err(Error::other("unrecognized external server config shape"));
+    }
+    Ok(cfg)
+}
+
+fn parse_object_seed(data: &[u8]) -> Option<Map<String, Value>> {
+    let value: Value = serde_json::from_slice(data).ok()?;
+    value.as_object().cloned()
+}
+
+fn build_storageclass_object(cfg: &Config) -> Map<String, Value> {
+    let kvs = cfg.get_value(STORAGE_CLASS_SUB_SYS, DEFAULT_DELIMITER).unwrap_or_default();
+    let mut sc_obj = Map::new();
+    sc_obj.insert(
+        "standard".to_string(),
+        Value::String(kvs.lookup(storageclass::CLASS_STANDARD).unwrap_or_default()),
+    );
+    sc_obj.insert("rrs".to_string(), Value::String(kvs.lookup(storageclass::CLASS_RRS).unwrap_or_default()));
+    let optimize = kvs
+        .lookup(storageclass::OPTIMIZE)
+        .filter(|v| !v.trim().is_empty())
+        .unwrap_or_else(|| "availability".to_string());
+    sc_obj.insert("optimize".to_string(), Value::String(optimize));
+    if let Some(v) = kvs.lookup(storageclass::INLINE_BLOCK).filter(|v| !v.trim().is_empty()) {
+        sc_obj.insert("inline_block".to_string(), Value::String(v));
+    }
+    sc_obj
+}
+
+fn build_oidc_provider_object(kvs: &KVS) -> Map<String, Value> {
+    let mut provider = Map::new();
+
+    for kv in &kvs.0 {
+        if kv.key == COMMENT_KEY || (kv.hidden_if_empty && kv.value.trim().is_empty()) {
+            continue;
+        }
+
+        if kv.value.trim().is_empty() {
+            continue;
+        }
+
+        if kv.key == ENABLE_KEY || kv.key == OIDC_REDIRECT_URI_DYNAMIC {
+            let enabled = kv
+                .value
+                .parse::<EnableState>()
+                .map(|state| state.is_enabled())
+                .unwrap_or(false);
+            provider.insert(kv.key.clone(), Value::Bool(enabled));
+            continue;
+        }
+
+        if kv.key == rustfs_config::oidc::OIDC_SCOPES {
+            let scopes = kv
+                .value
+                .split(',')
+                .map(str::trim)
+                .filter(|scope| !scope.is_empty())
+                .map(|scope| Value::String(scope.to_string()))
+                .collect::<Vec<_>>();
+            provider.insert(kv.key.clone(), Value::Array(scopes));
+            continue;
+        }
+
+        provider.insert(kv.key.clone(), Value::String(kv.value.clone()));
+    }
+
+    provider
+}
+
+fn build_oidc_object(cfg: &Config) -> Map<String, Value> {
+    let Some(subsystem) = cfg.0.get(IDENTITY_OPENID_SUB_SYS) else {
+        return Map::new();
+    };
+
+    let mut providers = subsystem.iter().collect::<Vec<_>>();
+    providers.sort_by(|(lhs, _), (rhs, _)| lhs.cmp(rhs));
+
+    let mut oidc_obj = Map::new();
+    for (instance_key, kvs) in providers {
+        if kvs
+            .lookup(rustfs_config::oidc::OIDC_CONFIG_URL)
+            .unwrap_or_default()
+            .trim()
+            .is_empty()
+        {
+            continue;
+        }
+
+        let provider = build_oidc_provider_object(kvs);
+        if provider.is_empty() {
+            continue;
+        }
+
+        let external_key = if instance_key == DEFAULT_DELIMITER {
+            "default".to_string()
+        } else {
+            instance_key.clone()
+        };
+        oidc_obj.insert(external_key, Value::Object(provider));
+    }
+
+    oidc_obj
+}
+
+fn build_semantic_oidc_object(cfg: &Config) -> Map<String, Value> {
+    let Some(subsystem) = cfg.0.get(IDENTITY_OPENID_SUB_SYS) else {
+        return Map::new();
+    };
+
+    let mut providers = subsystem.iter().collect::<Vec<_>>();
+    providers.sort_by(|(lhs, _), (rhs, _)| lhs.cmp(rhs));
+
+    let mut oidc_obj = Map::new();
+    for (instance_key, kvs) in providers {
+        let mut normalized = oidc::DEFAULT_IDENTITY_OPENID_KVS.clone();
+        normalized.extend(kvs.clone());
+
+        if normalized
+            .lookup(rustfs_config::oidc::OIDC_CONFIG_URL)
+            .unwrap_or_default()
+            .trim()
+            .is_empty()
+        {
+            continue;
+        }
+
+        let provider = build_oidc_provider_object(&normalized);
+        if provider.is_empty() {
+            continue;
+        }
+
+        let external_key = if instance_key == DEFAULT_DELIMITER {
+            "default".to_string()
+        } else {
+            instance_key.clone()
+        };
+        oidc_obj.insert(external_key, Value::Object(provider));
+    }
+
+    oidc_obj
+}
+
+fn encode_server_config_blob(cfg: &Config, seed: Option<&[u8]>) -> Result<Vec<u8>> {
+    let mut root = seed.and_then(parse_object_seed).unwrap_or_default();
+
+    if !matches!(root.get("version"), Some(Value::String(v)) if !v.trim().is_empty()) {
+        root.insert("version".to_string(), Value::String("33".to_string()));
+    }
+    if !matches!(root.get("region"), Some(Value::String(v)) if !v.trim().is_empty()) {
+        root.insert("region".to_string(), Value::String(RUSTFS_REGION.to_string()));
+    }
+
+    let mut sc_obj = match root.remove("storageclass") {
+        Some(Value::Object(v)) => v,
+        _ => Map::new(),
+    };
+    for (k, v) in build_storageclass_object(cfg) {
+        sc_obj.insert(k, v);
+    }
+    root.insert("storageclass".to_string(), Value::Object(sc_obj));
+    root.remove("storage_class");
+
+    let oidc_obj = build_oidc_object(cfg);
+    if oidc_obj.is_empty() {
+        root.remove("openid");
+        root.remove(IDENTITY_OPENID_SUB_SYS);
+    } else {
+        root.insert("openid".to_string(), Value::Object(oidc_obj));
+        root.remove(IDENTITY_OPENID_SUB_SYS);
+    }
+
+    Ok(serde_json::to_vec(&Value::Object(root))?)
+}
+
+fn is_standard_object_server_config(data: &[u8]) -> bool {
+    let Ok(value) = serde_json::from_slice::<Value>(data) else {
+        return false;
+    };
+    let Value::Object(root) = value else {
+        return false;
+    };
+    matches!(root.get("version"), Some(Value::String(v)) if !v.trim().is_empty())
+        && matches!(root.get("storageclass"), Some(Value::Object(_)))
+        && !root.contains_key("storage_class")
+}
+
+fn configs_semantically_equal(lhs: &Config, rhs: &Config) -> bool {
+    build_storageclass_object(lhs) == build_storageclass_object(rhs)
+        && build_semantic_oidc_object(lhs) == build_semantic_oidc_object(rhs)
+}
+
+fn is_object_not_found(err: &Error) -> bool {
+    *err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _) | Error::BucketNotFound(_))
+}
+
+pub async fn try_migrate_server_config<S: StorageAPI>(api: Arc<S>) {
+    let config_file = get_config_file();
+    match api
+        .get_object_info(
+            RUSTFS_META_BUCKET,
+            &config_file,
+            &ObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+    {
+        Ok(_) => {
+            debug!("server config already exists in RustFS metadata bucket, skip migration");
+            return;
+        }
+        Err(err) if is_object_not_found(&err) => {}
+        Err(err) => {
+            warn!("check target server config failed, skip migration: {:?}", err);
+            return;
+        }
+    }
+
+    let opts = ObjectOptions {
+        max_parity: true,
+        no_lock: true,
+        ..Default::default()
+    };
+
+    let mut rd = match api
+        .get_object_reader(MIGRATING_META_BUCKET, &config_file, None, HeaderMap::new(), &opts)
+        .await
+    {
+        Ok(v) => v,
+        Err(err) => {
+            if !is_object_not_found(&err) {
+                warn!("read legacy server config failed: {:?}", err);
+            }
+            return;
+        }
+    };
+
+    let data = match rd.read_all().await {
+        Ok(v) if !v.is_empty() => v,
+        Ok(_) => {
+            debug!("legacy server config is empty, skip migration");
+            return;
+        }
+        Err(err) => {
+            warn!("read legacy server config body failed: {:?}", err);
+            return;
+        }
+    };
+
+    let cfg = match decode_server_config_blob(&data) {
+        Ok(v) => v,
+        Err(err) => {
+            warn!("legacy server config format is incompatible, skip migration: {:?}", err);
+            return;
+        }
+    };
+    let normalized = match encode_server_config_blob(&cfg, Some(&data)) {
+        Ok(v) => v,
+        Err(err) => {
+            warn!("serialize migrated server config failed, skip migration: {:?}", err);
+            return;
+        }
+    };
+
+    match save_config(api, &config_file, normalized).await {
+        Ok(()) => {
+            info!("Migrated compatible server config from legacy metadata bucket");
+        }
+        Err(err) => {
+            warn!("write migrated server config failed: {:?}", err);
+        }
+    }
+}
+
 /// Handle the situation where the configuration file does not exist, create and save a new configuration
 async fn handle_missing_config<S: StorageAPI>(api: Arc<S>, context: &str) -> Result<Config> {
     warn!("Configuration not found ({}): Start initializing new configuration", context);
-    let cfg = new_and_save_server_config(api).await?;
+    let cfg = if is_first_cluster_node_local().await {
+        new_and_save_server_config(api.clone()).await?
+    } else {
+        let mut cfg = new_server_config();
+        lookup_configs(&mut cfg, api).await;
+        cfg
+    };
     warn!("Configuration initialization complete ({})", context);
     Ok(cfg)
 }
@@ -154,7 +606,7 @@ pub async fn read_config_without_migrate<S: StorageAPI>(api: Arc<S>) -> Result<C
     let config_file = get_config_file();
 
     // Try to read the configuration file
-    match read_config(api.clone(), &config_file).await {
+    match read_config_no_lock(api.clone(), &config_file).await {
         Ok(data) => read_server_config(api, &data).await,
         Err(Error::ConfigNotFound) => handle_missing_config(api, "Read the main configuration").await,
         Err(err) => handle_config_read_error(err, &config_file),
@@ -168,10 +620,10 @@ async fn read_server_config<S: StorageAPI>(api: Arc<S>, data: &[u8]) -> Result<C
         warn!("Received empty configuration data, try to reread from '{}'", config_file);
 
         // Try to read the configuration again
-        match read_config(api.clone(), &config_file).await {
+        match read_config_no_lock(api.clone(), &config_file).await {
             Ok(cfg_data) => {
                 // TODO: decrypt
-                let cfg = Config::unmarshal(&cfg_data)?;
+                let cfg = decode_server_config_blob(&cfg_data)?;
                 return Ok(cfg.merge());
             }
             Err(Error::ConfigNotFound) => return handle_missing_config(api, "Read alternate configuration").await,
@@ -180,14 +632,35 @@ async fn read_server_config<S: StorageAPI>(api: Arc<S>, data: &[u8]) -> Result<C
     }
 
     // Process non-empty configuration data
-    let cfg = Config::unmarshal(data)?;
+    let cfg = decode_server_config_blob(data)?;
     Ok(cfg.merge())
 }
 
 pub async fn save_server_config<S: StorageAPI>(api: Arc<S>, cfg: &Config) -> Result<()> {
-    let data = cfg.marshal()?;
-
     let config_file = get_config_file();
+    let existing = match read_config(api.clone(), &config_file).await {
+        Ok(v) => Some(v),
+        Err(Error::ConfigNotFound) => None,
+        Err(err) => {
+            warn!("read existing server config before save failed, continue with clean output: {:?}", err);
+            None
+        }
+    };
+
+    if let Some(current) = existing.as_deref()
+        && is_standard_object_server_config(current)
+        && let Ok(decoded_current) = decode_server_config_blob(current)
+        && configs_semantically_equal(&decoded_current, cfg)
+    {
+        debug!("server config unchanged and already in standard object shape, skip write");
+        return Ok(());
+    }
+
+    let data = encode_server_config_blob(cfg, existing.as_deref())?;
+    if existing.as_deref().is_some_and(|current| current == data.as_slice()) {
+        debug!("server config bytes unchanged after encode, skip write");
+        return Ok(());
+    }
 
     save_config(api, &config_file, data).await
 }
@@ -232,3 +705,211 @@ async fn apply_dynamic_config_for_sub_sys<S: StorageAPI>(cfg: &mut Config, api:
 
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{
+        configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
+        storage_class_kvs_mut,
+    };
+    use crate::config::{Config, oidc};
+    use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
+    use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState};
+    use serde_json::Value;
+
+    #[test]
+    fn test_decode_server_config_accepts_legacy_hidden_if_empty_alias() {
+        let input = r#"{"storage_class":{"_":[{"key":"standard","value":"EC:2","hiddenIfEmpty":true}]}}"#;
+        let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
+        let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
+        assert!(kvs.0[0].hidden_if_empty);
+    }
+
+    #[test]
+    fn test_decode_server_config_accepts_missing_hidden_if_empty() {
+        let input = r#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"}]}}"#;
+        let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
+        let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
+        assert!(!kvs.0[0].hidden_if_empty);
+    }
+
+    #[test]
+    fn test_decode_server_config_accepts_v33_object_shape() {
+        let input = r#"{
+          "version":"33",
+          "credential":{"accessKey":"test","secretKey":"testtesttest"},
+          "region":"us-east-1",
+          "worm":"off",
+          "storageclass":{"standard":"EC:2","rrs":"EC:1"},
+          "notify":{},
+          "logger":{},
+          "compress":{"enabled":false},
+          "openid":{},
+          "policy":{"opa":{}},
+          "ldapserverconfig":{}
+        }"#;
+
+        let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
+        let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
+        assert_eq!(kvs.get("standard"), "EC:2");
+        assert_eq!(kvs.get("rrs"), "EC:1");
+        assert_eq!(kvs.get("optimize"), "availability");
+    }
+
+    #[test]
+    fn test_decode_server_config_reads_openid_providers() {
+        let input = r#"{
+          "version":"33",
+          "storageclass":{"standard":"EC:2","rrs":"EC:1"},
+          "openid":{
+            "default":{
+              "enable":true,
+              "config_url":"https://example.com/.well-known/openid-configuration",
+              "client_id":"console",
+              "client_secret":"secret-value",
+              "scopes":["openid","profile","email"],
+              "redirect_uri_dynamic":true,
+              "display_name":"Default Provider"
+            },
+            "smoke":{
+              "enable":false,
+              "config_url":"https://issuer.example.com/.well-known/openid-configuration",
+              "client_id":"smoke-client",
+              "scopes":["openid"],
+              "redirect_uri_dynamic":false
+            }
+          }
+        }"#;
+
+        let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
+
+        let default_kvs = cfg
+            .get_value(IDENTITY_OPENID_SUB_SYS, DEFAULT_DELIMITER)
+            .expect("default oidc provider should exist");
+        assert_eq!(
+            default_kvs.get(rustfs_config::oidc::OIDC_CONFIG_URL),
+            "https://example.com/.well-known/openid-configuration"
+        );
+        assert_eq!(default_kvs.get(rustfs_config::oidc::OIDC_CLIENT_ID), "console");
+        assert_eq!(default_kvs.get(rustfs_config::oidc::OIDC_SCOPES), "openid,profile,email");
+        assert_eq!(default_kvs.get(ENABLE_KEY), EnableState::On.to_string());
+
+        let smoke_kvs = cfg
+            .get_value(IDENTITY_OPENID_SUB_SYS, "smoke")
+            .expect("named oidc provider should exist");
+        assert_eq!(smoke_kvs.get(rustfs_config::oidc::OIDC_CLIENT_ID), "smoke-client");
+        assert_eq!(
+            smoke_kvs.get(rustfs_config::oidc::OIDC_REDIRECT_URI_DYNAMIC),
+            EnableState::Off.to_string()
+        );
+    }
+
+    #[test]
+    fn test_encode_server_config_writes_external_object_shape() {
+        let mut cfg = Config::new();
+        let kvs = storage_class_kvs_mut(&mut cfg);
+        kvs.insert("standard".to_string(), "EC:2".to_string());
+        kvs.insert("rrs".to_string(), "EC:1".to_string());
+
+        let out = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+        let v: Value = serde_json::from_slice(&out).expect("output should be json");
+        assert!(v.get("version").is_some(), "external object should have version");
+        assert!(v.get("storageclass").is_some(), "external object should have storageclass");
+        assert!(v.get("storage_class").is_none(), "should not write rustfs map shape");
+    }
+
+    #[test]
+    fn test_encode_server_config_writes_openid_object_shape() {
+        let mut cfg = Config::new();
+        let mut oidc_section = std::collections::HashMap::new();
+        let mut default_provider = oidc::DEFAULT_IDENTITY_OPENID_KVS.clone();
+        default_provider.insert(ENABLE_KEY.to_string(), EnableState::On.to_string());
+        default_provider.insert(
+            rustfs_config::oidc::OIDC_CONFIG_URL.to_string(),
+            "https://example.com/.well-known/openid-configuration".to_string(),
+        );
+        default_provider.insert(rustfs_config::oidc::OIDC_CLIENT_ID.to_string(), "console".to_string());
+        default_provider.insert(rustfs_config::oidc::OIDC_SCOPES.to_string(), "openid,profile,email".to_string());
+        oidc_section.insert(DEFAULT_DELIMITER.to_string(), default_provider);
+        cfg.0.insert(IDENTITY_OPENID_SUB_SYS.to_string(), oidc_section);
+
+        let out = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+        let v: Value = serde_json::from_slice(&out).expect("output should be json");
+        let openid = v
+            .get("openid")
+            .and_then(Value::as_object)
+            .expect("output should include openid object");
+        let default_provider = openid
+            .get("default")
+            .and_then(Value::as_object)
+            .expect("default provider should be encoded");
+
+        assert_eq!(
+            default_provider
+                .get(rustfs_config::oidc::OIDC_CLIENT_ID)
+                .and_then(Value::as_str),
+            Some("console")
+        );
+        assert_eq!(
+            default_provider
+                .get(rustfs_config::oidc::OIDC_SCOPES)
+                .and_then(Value::as_array)
+                .map(|values| values.iter().filter_map(Value::as_str).collect::<Vec<_>>()),
+            Some(vec!["openid", "profile", "email"])
+        );
+        assert_eq!(default_provider.get(ENABLE_KEY).and_then(Value::as_bool), Some(true));
+    }
+
+    #[test]
+    fn test_is_standard_object_server_config_detection() {
+        let external = br#"{"version":"33","storageclass":{"standard":"EC:2","rrs":"EC:1"}}"#;
+        assert!(is_standard_object_server_config(external));
+
+        let legacy = br#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"}]}}"#;
+        assert!(!is_standard_object_server_config(legacy));
+    }
+
+    #[test]
+    fn test_configs_semantically_equal_for_equivalent_shapes() {
+        let external = br#"{"version":"33","storageclass":{"standard":"EC:2","rrs":"EC:1","optimize":"availability"}}"#;
+        let legacy = br#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"},{"key":"rrs","value":"EC:1"},{"key":"optimize","value":"availability"}]}}"#;
+        let lhs = decode_server_config_blob(external).expect("decode external");
+        let rhs = decode_server_config_blob(legacy).expect("decode legacy");
+        assert!(configs_semantically_equal(&lhs, &rhs));
+    }
+
+    #[test]
+    fn test_configs_semantically_equal_accounts_for_openid() {
+        let external = br#"{
+          "version":"33",
+          "storageclass":{"standard":"EC:2","rrs":"EC:1","optimize":"availability"},
+          "openid":{
+            "default":{
+              "enable":true,
+              "config_url":"https://example.com/.well-known/openid-configuration",
+              "client_id":"console",
+              "scopes":["openid","profile","email"],
+              "redirect_uri_dynamic":true
+            }
+          }
+        }"#;
+        let legacy = br#"{
+          "storage_class":{"_":[
+            {"key":"standard","value":"EC:2"},
+            {"key":"rrs","value":"EC:1"},
+            {"key":"optimize","value":"availability"}
+          ]},
+          "identity_openid":{"_":[
+            {"key":"enable","value":"on"},
+            {"key":"config_url","value":"https://example.com/.well-known/openid-configuration"},
+            {"key":"client_id","value":"console"},
+            {"key":"scopes","value":"openid,profile,email"},
+            {"key":"redirect_uri_dynamic","value":"on"}
+          ]}
+        }"#;
+
+        let lhs = decode_server_config_blob(external).expect("decode external");
+        let rhs = decode_server_config_blob(legacy).expect("decode legacy");
+        assert!(configs_semantically_equal(&lhs, &rhs));
+    }
+}

crates/ecstore/src/config/mod.rs

@@ -75,14 +75,19 @@ pub async fn init_global_config_sys(api: Arc<ECStore>) -> Result<()> {
     GLOBAL_CONFIG_SYS.init(api).await
 }
 
-#[derive(Debug, Deserialize, Serialize, Clone)]
+pub async fn try_migrate_server_config(api: Arc<ECStore>) {
+    com::try_migrate_server_config(api).await
+}
+
+#[derive(Debug, Deserialize, Serialize, Clone, PartialEq, Eq)]
 pub struct KV {
     pub key: String,
     pub value: String,
+    #[serde(default, alias = "hiddenIfEmpty")]
     pub hidden_if_empty: bool,
 }
 
-#[derive(Debug, Deserialize, Serialize, Clone)]
+#[derive(Debug, Deserialize, Serialize, Clone, PartialEq, Eq)]
 pub struct KVS(pub Vec<KV>);
 
 impl Default for KVS {
@@ -158,7 +163,7 @@ impl KVS {
     }
 }
 
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, PartialEq, Eq)]
 pub struct Config(pub HashMap<String, HashMap<String, KVS>>);
 
 impl Default for Config {

crates/ecstore/src/disk/disk_store.rs

@@ -18,6 +18,7 @@ use crate::disk::{
     WalkDirOptions,
     local::{LocalDisk, ScanGuard},
 };
+use crate::global::GLOBAL_LOCAL_DISK_ID_MAP;
 use bytes::Bytes;
 use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
 use std::{
@@ -410,7 +411,19 @@ impl LocalDiskWrapper {
     /// Set the disk ID
     pub async fn set_disk_id_internal(&self, id: Option<Uuid>) -> Result<()> {
         let mut disk_id = self.disk_id.write().await;
+        let previous = *disk_id;
         *disk_id = id;
+        drop(disk_id);
+
+        if self.disk.is_local() {
+            let mut disk_id_map = GLOBAL_LOCAL_DISK_ID_MAP.write().await;
+            if let Some(previous_id) = previous {
+                disk_id_map.remove(&previous_id);
+            }
+            if let Some(current_id) = id {
+                disk_id_map.insert(current_id, self.disk.endpoint().to_string());
+            }
+        }
         Ok(())
     }
 

crates/ecstore/src/disk/local.rs

@@ -29,7 +29,6 @@ use crate::disk::{
     os::{check_path_length, is_empty_dir, is_root_disk, rename_all},
 };
 use crate::erasure_coding::bitrot_verify;
-use crate::file_cache::{get_global_file_cache, prefetch_metadata_patterns, read_metadata_cached};
 use crate::global::{GLOBAL_IsErasureSD, GLOBAL_RootDiskThreshold};
 use bytes::Bytes;
 use parking_lot::RwLock as ParkingLotRwLock;
@@ -492,56 +491,6 @@ impl LocalDisk {
         Ok(results.into_iter().map(|(_, path)| path).collect())
     }
 
-    // Optimized metadata reading with caching
-    async fn read_metadata_cached(&self, path: PathBuf) -> Result<Arc<FileMeta>> {
-        read_metadata_cached(path).await
-    }
-
-    // Smart prefetching for related files
-    async fn read_version_with_prefetch(
-        &self,
-        volume: &str,
-        path: &str,
-        version_id: &str,
-        opts: &ReadOptions,
-    ) -> Result<FileInfo> {
-        let file_path = self.get_object_path(volume, path)?;
-
-        // Async prefetch related files, don't block current read
-        if let Some(parent) = file_path.parent() {
-            prefetch_metadata_patterns(parent, &[STORAGE_FORMAT_FILE, "part.1", "part.2", "part.meta"]).await;
-        }
-
-        // Main read logic
-        let file_dir = self.get_bucket_path(volume)?;
-        let (data, _) = self.read_raw(volume, file_dir, file_path, opts.read_data).await?;
-
-        get_file_info(
-            &data,
-            volume,
-            path,
-            version_id,
-            FileInfoOpts {
-                data: opts.read_data,
-                include_free_versions: false,
-            },
-        )
-        .map_err(|_e| DiskError::Unexpected)
-    }
-
-    // Batch metadata reading for multiple objects
-    async fn read_metadata_batch(&self, requests: Vec<(String, String)>) -> Result<Vec<Option<Arc<FileMeta>>>> {
-        let paths: Vec<PathBuf> = requests
-            .iter()
-            .map(|(bucket, key)| self.get_object_path(bucket, &format!("{}/{}", key, STORAGE_FORMAT_FILE)))
-            .collect::<Result<Vec<_>>>()?;
-
-        let cache = get_global_file_cache();
-        let results = cache.get_metadata_batch(paths).await;
-
-        Ok(results.into_iter().map(|r| r.ok()).collect())
-    }
-
     // /// Write to the filesystem atomically.
     // /// This is done by first writing to a temporary location and then moving the file.
     // pub(crate) async fn prepare_file_write<'a>(&self, path: &'a PathBuf) -> Result<FileWriter<'a>> {
@@ -697,7 +646,6 @@ impl LocalDisk {
                 match self.read_metadata_with_dmtime(meta_path).await {
                     Ok(res) => Ok(res),
                     Err(err) => {
-                        warn!("read_raw: error: {:?}", err);
                         if err == Error::FileNotFound
                             && !skip_access_checks(volume_dir.as_ref().to_string_lossy().to_string().as_str())
                             && let Err(e) = access(volume_dir.as_ref()).await
@@ -866,8 +814,6 @@ impl LocalDisk {
 
         rename_all(tmp_file_path, &file_path, volume_dir).await?;
 
-        // Invalidate cache after successful write
-        get_global_file_cache().invalidate(&file_path).await;
         Ok(())
     }
 
@@ -892,7 +838,9 @@ impl LocalDisk {
         check_path_length(file_path.to_string_lossy().as_ref())?;
 
         self.write_all_internal(&file_path, InternalBuf::Owned(buf), sync, skip_parent)
-            .await
+            .await?;
+
+        Ok(())
     }
     // write_all_internal do write file
     async fn write_all_internal(&self, file_path: &Path, data: InternalBuf<'_>, sync: bool, skip_parent: &Path) -> Result<()> {
@@ -1493,6 +1441,12 @@ impl DiskAPI for LocalDisk {
         let erasure = &fi.erasure;
         for (i, part) in fi.parts.iter().enumerate() {
             let checksum_info = erasure.get_checksum_info(part.number);
+            let checksum_algo =
+                if fi.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
+                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
+                } else {
+                    checksum_info.algorithm
+                };
             let part_path = self.get_object_path(
                 volume,
                 path_join_buf(&[
@@ -1506,7 +1460,7 @@ impl DiskAPI for LocalDisk {
                 .bitrot_verify(
                     &part_path,
                     erasure.shard_file_size(part.size as i64) as usize,
-                    checksum_info.algorithm,
+                    checksum_algo,
                     &checksum_info.hash,
                     erasure.shard_size(),
                 )
@@ -2058,7 +2012,6 @@ impl DiskAPI for LocalDisk {
         let search_version_id = fi.version_id.or(Some(Uuid::nil()));
 
         // Check if there's an existing version with the same version_id that has a data_dir to clean up
-        // Note: For non-versioned buckets, fi.version_id is None, but in xl.meta it's stored as Some(Uuid::nil())
         let has_old_data_dir = {
             xlmeta.find_version(search_version_id).ok().and_then(|(_, ver)| {
                 // shard_count == 0 means no other version shares this data_dir
@@ -2474,7 +2427,7 @@ impl DiskAPI for LocalDisk {
                 file_path.as_path(),
                 Path::new(format!("{path}{SLASH_SEPARATOR}{STORAGE_FORMAT_FILE}").as_str()),
             ]);
-            return rename_all(src_path, dst_path, file_path).await;
+            return rename_all(&src_path, &dst_path, file_path).await;
         }
 
         self.delete_file(&volume_dir, &xl_path, true, false).await
@@ -2597,17 +2550,9 @@ impl DiskAPI for LocalDisk {
 
     #[tracing::instrument(skip(self))]
     async fn read_metadata(&self, volume: &str, path: &str) -> Result<Bytes> {
-        // Try to use cached file content reading for better performance, with safe fallback
         let file_path = self.get_object_path(volume, path)?;
-        // let file_path = file_path.join(Path::new(STORAGE_FORMAT_FILE));
-
-        // First, try the cache
-        if let Ok(bytes) = get_global_file_cache().get_file_content(file_path.clone()).await {
-            return Ok(bytes);
-        }
-
-        // Fallback to direct read if cache fails
-        let (data, _) = self.read_metadata_with_dmtime(&file_path).await?;
+        let volume_dir = self.get_bucket_path(volume)?;
+        let (data, _) = self.read_all_data_with_dmtime(volume, volume_dir, file_path).await?;
         Ok(data.into())
     }
 }

crates/ecstore/src/disk/mod.rs

@@ -23,6 +23,7 @@ pub mod local;
 pub mod os;
 
 pub const RUSTFS_META_BUCKET: &str = ".rustfs.sys";
+pub const MIGRATING_META_BUCKET: &str = ".minio.sys";
 pub const RUSTFS_META_MULTIPART_BUCKET: &str = ".rustfs.sys/multipart";
 pub const RUSTFS_META_TMP_BUCKET: &str = ".rustfs.sys/tmp";
 pub const RUSTFS_META_TMP_DELETED_BUCKET: &str = ".rustfs.sys/tmp/.trash";

crates/ecstore/src/endpoints.rs

@@ -791,14 +791,17 @@ mod test {
             panic!("No non-loop back IP address found for this host");
         }
         let non_loop_back_ip = non_loop_back_i_ps[0];
+        let remote_ip1 = "192.0.2.10";
+        let remote_ip2 = "192.0.2.11";
+        let remote_ip3 = "192.0.2.12";
 
         let case1_endpoint1 = format!("http://{non_loop_back_ip}/d1");
         let case1_endpoint2 = format!("http://{non_loop_back_ip}/d2");
         let args = vec![
             format!("http://{}:10000/d1", non_loop_back_ip),
             format!("http://{}:10000/d2", non_loop_back_ip),
-            "http://example.org:10000/d3".to_string(),
-            "http://example.com:10000/d4".to_string(),
+            format!("http://{remote_ip1}:10000/d3"),
+            format!("http://{remote_ip2}:10000/d4"),
         ];
         let (case1_ur_ls, case1_local_flags) = get_expected_endpoints(args, format!("http://{non_loop_back_ip}:10000/"));
 
@@ -807,26 +810,26 @@ mod test {
         let args = vec![
             format!("http://{}:10000/d1", non_loop_back_ip),
             format!("http://{}:9000/d2", non_loop_back_ip),
-            "http://example.org:10000/d3".to_string(),
-            "http://example.com:10000/d4".to_string(),
+            format!("http://{remote_ip1}:10000/d3"),
+            format!("http://{remote_ip2}:10000/d4"),
         ];
         let (case2_ur_ls, case2_local_flags) = get_expected_endpoints(args, format!("http://{non_loop_back_ip}:10000/"));
 
         let case3_endpoint1 = format!("http://{non_loop_back_ip}/d1");
         let args = vec![
             format!("http://{}:80/d1", non_loop_back_ip),
-            "http://example.org:9000/d2".to_string(),
-            "http://example.com:80/d3".to_string(),
-            "http://example.net:80/d4".to_string(),
+            format!("http://{remote_ip1}:9000/d2"),
+            format!("http://{remote_ip2}:80/d3"),
+            format!("http://{remote_ip3}:80/d4"),
         ];
         let (case3_ur_ls, case3_local_flags) = get_expected_endpoints(args, format!("http://{non_loop_back_ip}:80/"));
 
         let case4_endpoint1 = format!("http://{non_loop_back_ip}/d1");
         let args = vec![
             format!("http://{}:9000/d1", non_loop_back_ip),
-            "http://example.org:9000/d2".to_string(),
-            "http://example.com:9000/d3".to_string(),
-            "http://example.net:9000/d4".to_string(),
+            format!("http://{remote_ip1}:9000/d2"),
+            format!("http://{remote_ip2}:9000/d3"),
+            format!("http://{remote_ip3}:9000/d4"),
         ];
         let (case4_ur_ls, case4_local_flags) = get_expected_endpoints(args, format!("http://{non_loop_back_ip}:9000/"));
 
@@ -844,8 +847,8 @@ mod test {
 
         let case6_endpoint1 = format!("http://{non_loop_back_ip}:9003/d4");
         let args = vec![
-            "http://localhost:9000/d1".to_string(),
-            "http://localhost:9001/d2".to_string(),
+            "http://127.0.0.1:9000/d1".to_string(),
+            "http://127.0.0.1:9001/d2".to_string(),
             "http://127.0.0.1:9002/d3".to_string(),
             case6_endpoint1.clone(),
         ];
@@ -864,8 +867,8 @@ mod test {
             // Erasure Single Drive
             TestCase {
                 num: 2,
-                server_addr: "localhost:9000",
-                args: vec!["http://localhost/d1"],
+                server_addr: "127.0.0.1:9000",
+                args: vec!["http://127.0.0.1/d1"],
                 expected_err: Some(Error::other("use path style endpoint for single node setup")),
                 ..Default::default()
             },
@@ -885,7 +888,7 @@ mod test {
             },
             TestCase {
                 num: 4,
-                server_addr: "localhost:10000",
+                server_addr: "127.0.0.1:10000",
                 args: vec!["/d1"],
                 expected_endpoints: Some(Endpoints(vec![Endpoint {
                     url: must_file_path("/d1"),
@@ -899,12 +902,12 @@ mod test {
             },
             TestCase {
                 num: 5,
-                server_addr: "localhost:9000",
+                server_addr: "127.0.0.1:9000",
                 args: vec![
                     "https://127.0.0.1:9000/d1",
-                    "https://localhost:9001/d1",
-                    "https://example.com/d1",
-                    "https://example.com/d2",
+                    "https://127.0.0.1:9001/d1",
+                    "https://192.0.2.1/d1",
+                    "https://192.0.2.1/d2",
                 ],
                 expected_err: Some(Error::other("same path '/d1' can not be served by different port on same address")),
                 ..Default::default()
@@ -952,35 +955,35 @@ mod test {
                 num: 7,
                 server_addr: "0.0.0.0:9000",
                 args: vec![
-                    "http://localhost/d1",
-                    "http://localhost/d2",
-                    "http://localhost/d3",
-                    "http://localhost/d4",
+                    "http://127.0.0.1/d1",
+                    "http://127.0.0.1/d2",
+                    "http://127.0.0.1/d3",
+                    "http://127.0.0.1/d4",
                 ],
                 expected_endpoints: Some(Endpoints(vec![
                     Endpoint {
-                        url: must_url("http://localhost:9000/d1"),
+                        url: must_url("http://127.0.0.1:9000/d1"),
                         is_local: true,
                         pool_idx: 0,
                         set_idx: 0,
                         disk_idx: 0,
                     },
                     Endpoint {
-                        url: must_url("http://localhost:9000/d2"),
+                        url: must_url("http://127.0.0.1:9000/d2"),
                         is_local: true,
                         pool_idx: 0,
                         set_idx: 0,
                         disk_idx: 0,
                     },
                     Endpoint {
-                        url: must_url("http://localhost:9000/d3"),
+                        url: must_url("http://127.0.0.1:9000/d3"),
                         is_local: true,
                         pool_idx: 0,
                         set_idx: 0,
                         disk_idx: 0,
                     },
                     Endpoint {
-                        url: must_url("http://localhost:9000/d4"),
+                        url: must_url("http://127.0.0.1:9000/d4"),
                         is_local: true,
                         pool_idx: 0,
                         set_idx: 0,
@@ -995,8 +998,8 @@ mod test {
                 num: 8,
                 server_addr: "127.0.0.1:10000",
                 args: vec![
-                    "http://localhost/d1",
-                    "http://localhost/d2",
+                    "http://[::1]/d1",
+                    "http://[::1]/d2",
                     "http://127.0.0.1/d3",
                     "http://127.0.0.1/d4",
                 ],
@@ -1034,8 +1037,8 @@ mod test {
                 args: vec![
                     case1_endpoint1.as_str(),
                     case1_endpoint2.as_str(),
-                    "http://example.org/d3",
-                    "http://example.com/d4",
+                    "http://192.0.2.10/d3",
+                    "http://192.0.2.11/d4",
                 ],
                 expected_endpoints: Some(Endpoints(vec![
                     Endpoint {
@@ -1076,8 +1079,8 @@ mod test {
                 args: vec![
                     case2_endpoint1.as_str(),
                     case2_endpoint2.as_str(),
-                    "http://example.org/d3",
-                    "http://example.com/d4",
+                    "http://192.0.2.10/d3",
+                    "http://192.0.2.11/d4",
                 ],
                 expected_endpoints: Some(Endpoints(vec![
                     Endpoint {
@@ -1117,9 +1120,9 @@ mod test {
                 server_addr: "0.0.0.0:80",
                 args: vec![
                     case3_endpoint1.as_str(),
-                    "http://example.org:9000/d2",
-                    "http://example.com/d3",
-                    "http://example.net/d4",
+                    "http://192.0.2.10:9000/d2",
+                    "http://192.0.2.11/d3",
+                    "http://192.0.2.12/d4",
                 ],
                 expected_endpoints: Some(Endpoints(vec![
                     Endpoint {
@@ -1159,9 +1162,9 @@ mod test {
                 server_addr: "0.0.0.0:9000",
                 args: vec![
                     case4_endpoint1.as_str(),
-                    "http://example.org/d2",
-                    "http://example.com/d3",
-                    "http://example.net/d4",
+                    "http://192.0.2.10/d2",
+                    "http://192.0.2.11/d3",
+                    "http://192.0.2.12/d4",
                 ],
                 expected_endpoints: Some(Endpoints(vec![
                     Endpoint {
@@ -1242,8 +1245,8 @@ mod test {
                 num: 16,
                 server_addr: "0.0.0.0:9003",
                 args: vec![
-                    "http://localhost:9000/d1",
-                    "http://localhost:9001/d2",
+                    "http://127.0.0.1:9000/d1",
+                    "http://127.0.0.1:9001/d2",
                     "http://127.0.0.1:9002/d3",
                     case6_endpoint1.as_str(),
                 ],

crates/ecstore/src/erasure_coding/bitrot.rs

@@ -15,6 +15,7 @@
 use bytes::Bytes;
 use pin_project_lite::pin_project;
 use rustfs_utils::HashAlgorithm;
+use std::io::IoSlice;
 use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
 use tracing::error;
 use uuid::Uuid;
@@ -155,25 +156,51 @@ where
                 error!("bitrot writer write hash error: hash is empty");
                 return Err(std::io::Error::new(std::io::ErrorKind::InvalidInput, "hash is empty"));
             }
-            self.inner.write_all(hash.as_ref()).await?;
+            write_all_vectored(&mut self.inner, hash.as_ref(), buf).await?;
+        } else {
+            self.inner.write_all(buf).await?;
         }
 
-        self.inner.write_all(buf).await?;
-
-        self.inner.flush().await?;
-
         let n = buf.len();
 
         Ok(n)
     }
 
     pub async fn shutdown(&mut self) -> std::io::Result<()> {
+        self.inner.flush().await?;
         self.inner.shutdown().await
     }
 }
 
+async fn write_all_vectored<W>(writer: &mut W, hash: &[u8], data: &[u8]) -> std::io::Result<()>
+where
+    W: AsyncWrite + Unpin,
+{
+    let mut hash_offset = 0;
+    let mut data_offset = 0;
+
+    while hash_offset < hash.len() || data_offset < data.len() {
+        let slices = [IoSlice::new(&hash[hash_offset..]), IoSlice::new(&data[data_offset..])];
+        let written = writer.write_vectored(&slices).await?;
+        if written == 0 {
+            return Err(std::io::Error::new(std::io::ErrorKind::WriteZero, "failed to write hash and data"));
+        }
+
+        let hash_remaining = hash.len() - hash_offset;
+        if written < hash_remaining {
+            hash_offset += written;
+            continue;
+        }
+
+        hash_offset = hash.len();
+        data_offset += written - hash_remaining;
+    }
+
+    Ok(())
+}
+
 pub fn bitrot_shard_file_size(size: usize, shard_size: usize, algo: HashAlgorithm) -> usize {
-    if algo != HashAlgorithm::HighwayHash256S {
+    if algo != HashAlgorithm::HighwayHash256S && algo != HashAlgorithm::HighwayHash256SLegacy {
         return size;
     }
     size.div_ceil(shard_size) * algo.size() + size
@@ -292,6 +319,33 @@ impl AsyncWrite for CustomWriter {
             }
         }
     }
+
+    fn poll_write_vectored(
+        self: std::pin::Pin<&mut Self>,
+        cx: &mut std::task::Context<'_>,
+        bufs: &[IoSlice<'_>],
+    ) -> std::task::Poll<std::io::Result<usize>> {
+        match self.get_mut() {
+            Self::InlineBuffer(data) => {
+                let total = bufs.iter().map(|buf| buf.len()).sum::<usize>();
+                for buf in bufs {
+                    data.extend_from_slice(buf);
+                }
+                std::task::Poll::Ready(Ok(total))
+            }
+            Self::Other(writer) => {
+                let pinned_writer = std::pin::Pin::new(writer.as_mut());
+                pinned_writer.poll_write_vectored(cx, bufs)
+            }
+        }
+    }
+
+    fn is_write_vectored(&self) -> bool {
+        match self {
+            Self::InlineBuffer(_) => true,
+            Self::Other(writer) => writer.is_write_vectored(),
+        }
+    }
 }
 
 /// Wrapper around BitrotWriter that uses our custom writer
@@ -361,7 +415,74 @@ mod tests {
     use super::BitrotReader;
     use super::BitrotWriter;
     use rustfs_utils::HashAlgorithm;
-    use std::io::Cursor;
+    use std::io::{Cursor, IoSlice};
+    use std::sync::{
+        Arc,
+        atomic::{AtomicUsize, Ordering},
+    };
+    use std::task::{Context, Poll};
+    use tokio::io::AsyncWrite;
+
+    #[derive(Default)]
+    struct VectoredCountingWriter {
+        vectored_writes: Arc<AtomicUsize>,
+        writes: Vec<u8>,
+    }
+
+    impl AsyncWrite for VectoredCountingWriter {
+        fn poll_write(self: std::pin::Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<std::io::Result<usize>> {
+            Poll::Ready(Err(std::io::Error::other("poll_write should not be used")))
+        }
+
+        fn poll_flush(self: std::pin::Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: std::pin::Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_write_vectored(
+            mut self: std::pin::Pin<&mut Self>,
+            _cx: &mut Context<'_>,
+            bufs: &[IoSlice<'_>],
+        ) -> Poll<std::io::Result<usize>> {
+            self.vectored_writes.fetch_add(1, Ordering::SeqCst);
+            let total = bufs.iter().map(|buf| buf.len()).sum::<usize>();
+            for buf in bufs {
+                self.writes.extend_from_slice(buf);
+            }
+            Poll::Ready(Ok(total))
+        }
+
+        fn is_write_vectored(&self) -> bool {
+            true
+        }
+    }
+
+    #[derive(Default)]
+    struct CountingWriter {
+        flushes: Arc<AtomicUsize>,
+        shutdowns: Arc<AtomicUsize>,
+        writes: Vec<u8>,
+    }
+
+    impl AsyncWrite for CountingWriter {
+        fn poll_write(mut self: std::pin::Pin<&mut Self>, _cx: &mut Context<'_>, buf: &[u8]) -> Poll<std::io::Result<usize>> {
+            self.writes.extend_from_slice(buf);
+            Poll::Ready(Ok(buf.len()))
+        }
+
+        fn poll_flush(self: std::pin::Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            self.flushes.fetch_add(1, Ordering::SeqCst);
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: std::pin::Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            self.shutdowns.fetch_add(1, Ordering::SeqCst);
+            Poll::Ready(Ok(()))
+        }
+    }
 
     #[tokio::test]
     async fn test_bitrot_read_write_ok() {
@@ -471,4 +592,41 @@ mod tests {
         assert_eq!(n, data_size);
         assert_eq!(data, &out[..]);
     }
+
+    #[tokio::test]
+    async fn test_bitrot_writer_flushes_once_on_shutdown() {
+        let flushes = Arc::new(AtomicUsize::new(0));
+        let shutdowns = Arc::new(AtomicUsize::new(0));
+        let writer = CountingWriter {
+            flushes: flushes.clone(),
+            shutdowns: shutdowns.clone(),
+            writes: Vec::new(),
+        };
+        let mut bitrot_writer = BitrotWriter::new(writer, 8, HashAlgorithm::None);
+
+        bitrot_writer.write(b"12345678").await.unwrap();
+        bitrot_writer.write(b"abc").await.unwrap();
+
+        assert_eq!(flushes.load(Ordering::SeqCst), 0);
+        assert_eq!(shutdowns.load(Ordering::SeqCst), 0);
+
+        bitrot_writer.shutdown().await.unwrap();
+
+        assert_eq!(flushes.load(Ordering::SeqCst), 1);
+        assert_eq!(shutdowns.load(Ordering::SeqCst), 1);
+    }
+
+    #[tokio::test]
+    async fn test_bitrot_writer_uses_vectored_write_for_hash_and_data() {
+        let vectored_writes = Arc::new(AtomicUsize::new(0));
+        let writer = VectoredCountingWriter {
+            vectored_writes: vectored_writes.clone(),
+            writes: Vec::new(),
+        };
+        let mut bitrot_writer = BitrotWriter::new(writer, 8, HashAlgorithm::HighwayHash256);
+
+        bitrot_writer.write(b"payload").await.unwrap();
+
+        assert!(vectored_writes.load(Ordering::SeqCst) > 0);
+    }
 }

crates/ecstore/src/erasure_coding/encode.rs

@@ -112,11 +112,66 @@ impl<'a> MultiWriter<'a> {
         )))
     }
 
-    pub async fn _shutdown(&mut self) -> std::io::Result<()> {
-        for writer in self.writers.iter_mut().flatten() {
-            writer.shutdown().await?;
+    async fn shutdown_writer(writer_opt: &mut Option<BitrotWriterWrapper>, err: &mut Option<Error>) {
+        match writer_opt {
+            Some(writer) => match writer.shutdown().await {
+                Ok(()) => {
+                    *err = None;
+                }
+                Err(e) => {
+                    *err = Some(Error::from(e));
+                    *writer_opt = None;
+                }
+            },
+            None => {
+                *err = Some(Error::DiskNotFound);
+            }
         }
-        Ok(())
+    }
+
+    pub async fn shutdown(&mut self) -> std::io::Result<()> {
+        {
+            let mut futures = FuturesUnordered::new();
+            for (writer_opt, err) in self.writers.iter_mut().zip(self.errs.iter_mut()) {
+                if err.is_some() {
+                    continue;
+                }
+                futures.push(Self::shutdown_writer(writer_opt, err));
+            }
+            while let Some(()) = futures.next().await {}
+        }
+
+        let nil_count = self.errs.iter().filter(|&e| e.is_none()).count();
+        if nil_count >= self.write_quorum {
+            return Ok(());
+        }
+
+        if let Some(write_err) = reduce_write_quorum_errs(&self.errs, OBJECT_OP_IGNORED_ERRS, self.write_quorum) {
+            error!(
+                "reduce_write_quorum_errs during shutdown: {:?}, offline-disks={}/{}, errs={:?}",
+                write_err,
+                count_errs(&self.errs, &Error::DiskNotFound),
+                self.writers.len(),
+                self.errs
+            );
+            return Err(std::io::Error::other(format!(
+                "Failed to shutdown writers: {} (offline-disks={}/{})",
+                write_err,
+                count_errs(&self.errs, &Error::DiskNotFound),
+                self.writers.len()
+            )));
+        }
+
+        Err(std::io::Error::other(format!(
+            "Failed to shutdown writers: (offline-disks={}/{}): {}",
+            count_errs(&self.errs, &Error::DiskNotFound),
+            self.writers.len(),
+            self.errs
+                .iter()
+                .map(|e| e.as_ref().map_or("<nil>".to_string(), |e| e.to_string()))
+                .collect::<Vec<_>>()
+                .join(", ")
+        )))
     }
 }
 
@@ -176,7 +231,69 @@ impl Erasure {
         }
 
         let (reader, total) = task.await??;
-        // writers.shutdown().await?;
+        writers.shutdown().await?;
         Ok((reader, total))
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::erasure_coding::{BitrotWriterWrapper, CustomWriter};
+    use rustfs_utils::HashAlgorithm;
+    use std::pin::Pin;
+    use std::sync::{Arc, Mutex};
+    use std::task::{Context, Poll};
+    use tokio::io::AsyncWrite;
+
+    #[derive(Clone, Default)]
+    struct DeferredCommitWriter {
+        buffered: Vec<u8>,
+        committed: Arc<Mutex<Vec<u8>>>,
+    }
+
+    impl DeferredCommitWriter {
+        fn new(committed: Arc<Mutex<Vec<u8>>>) -> Self {
+            Self {
+                buffered: Vec::new(),
+                committed,
+            }
+        }
+    }
+
+    impl AsyncWrite for DeferredCommitWriter {
+        fn poll_write(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &[u8]) -> Poll<std::io::Result<usize>> {
+            self.buffered.extend_from_slice(buf);
+            Poll::Ready(Ok(buf.len()))
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(mut self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            let buffered = std::mem::take(&mut self.buffered);
+            let mut committed = self.committed.lock().unwrap();
+            committed.extend_from_slice(&buffered);
+            Poll::Ready(Ok(()))
+        }
+    }
+
+    #[tokio::test]
+    async fn encode_shutdowns_writers_after_small_shards() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed.clone());
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 16));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(b"small payload".to_vec()));
+        let (_reader, written) = erasure.encode(reader, &mut writers, 1).await.unwrap();
+
+        assert_eq!(written, b"small payload".len());
+        assert!(!committed.lock().unwrap().is_empty());
+    }
+}

crates/ecstore/src/erasure_coding/erasure.rs

@@ -12,31 +12,12 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Erasure coding implementation using Reed-Solomon SIMD backend.
+//! Erasure coding implementation using reed-solomon-erasure (GF(2^8)).
+//! Supports legacy (reed-solomon-simd) for reading/healing old-version files.
 //!
-//! This module provides erasure coding functionality with high-performance SIMD
-//! Reed-Solomon implementation:
-//!
-//! ## Reed-Solomon Implementation
-//!
-//! ### SIMD Mode (Only)
-//! - **Performance**: Uses SIMD optimization for high-performance encoding/decoding
-//! - **Compatibility**: Works with any shard size through SIMD implementation
-//! - **Reliability**: High-performance SIMD implementation for large data processing
-//! - **Use case**: Optimized for maximum performance in large data processing scenarios
-//!
-//! ## Example
-//!
-//! ```ignore
-//! use rustfs_ecstore::erasure_coding::Erasure;
-//!
-//! let erasure = Erasure::new(4, 2, 1024); // 4 data shards, 2 parity shards, 1KB block size
-//! let data = b"hello world";
-//! let shards = erasure.encode_data(data).unwrap();
-//! // Simulate loss and recovery...
-//! ```
 
 use bytes::{Bytes, BytesMut};
+use reed_solomon_erasure::galois_8::ReedSolomon;
 use reed_solomon_simd;
 use smallvec::SmallVec;
 use std::io;
@@ -44,132 +25,88 @@ use tokio::io::AsyncRead;
 use tracing::warn;
 use uuid::Uuid;
 
-/// Reed-Solomon encoder using SIMD implementation.
-pub struct ReedSolomonEncoder {
+/// Legacy calc_shard_size formula: (block_size.div_ceil(data_shards) + 1) & !1
+/// Matches main branch and filemeta::ErasureInfo for old-version files.
+pub fn calc_shard_size_legacy(block_size: usize, data_shards: usize) -> usize {
+    (block_size.div_ceil(data_shards) + 1) & !1
+}
+
+/// Reed-Solomon encoder for legacy (main branch) format using reed-solomon-simd.
+/// Used when decoding/encoding files with uses_legacy_checksum == true.
+struct LegacyReedSolomonEncoder {
     data_shards: usize,
     parity_shards: usize,
-    // Use RwLock to ensure thread safety, implementing Send + Sync
     encoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
     decoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
 }
 
-impl Clone for ReedSolomonEncoder {
+impl Clone for LegacyReedSolomonEncoder {
     fn clone(&self) -> Self {
         Self {
             data_shards: self.data_shards,
             parity_shards: self.parity_shards,
-            // Create an empty cache for the new instance instead of sharing one
             encoder_cache: std::sync::RwLock::new(None),
             decoder_cache: std::sync::RwLock::new(None),
         }
     }
 }
 
-impl ReedSolomonEncoder {
-    /// Create a new Reed-Solomon encoder with specified data and parity shards.
-    pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
-        Ok(ReedSolomonEncoder {
-            data_shards,
-            parity_shards,
+impl LegacyReedSolomonEncoder {
+    fn new(_data_shards: usize, _parity_shards: usize) -> io::Result<Self> {
+        Ok(Self {
+            data_shards: _data_shards,
+            parity_shards: _parity_shards,
             encoder_cache: std::sync::RwLock::new(None),
             decoder_cache: std::sync::RwLock::new(None),
         })
     }
 
-    /// Encode data shards with parity.
-    pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
+    fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
         let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
         if shards_vec.is_empty() {
             return Ok(());
         }
-
-        let simd_result = self.encode_with_simd(&mut shards_vec);
-
-        match simd_result {
-            Ok(()) => Ok(()),
-            Err(simd_error) => {
-                warn!("SIMD encoding failed: {}", simd_error);
-                Err(simd_error)
-            }
-        }
-    }
-
-    fn encode_with_simd(&self, shards_vec: &mut [&mut [u8]]) -> io::Result<()> {
         let shard_len = shards_vec[0].len();
-
-        // Get or create encoder
         let mut encoder = {
             let mut cache_guard = self
                 .encoder_cache
                 .write()
                 .map_err(|_| io::Error::other("Failed to acquire encoder cache lock"))?;
-
             match cache_guard.take() {
-                Some(mut cached_encoder) => {
-                    // Use reset method to reset existing encoder to adapt to new parameters
-                    if let Err(e) = cached_encoder.reset(self.data_shards, self.parity_shards, shard_len) {
-                        warn!("Failed to reset SIMD encoder: {:?}, creating new one", e);
-                        // If reset fails, create new encoder
+                Some(mut cached) => {
+                    if cached.reset(self.data_shards, self.parity_shards, shard_len).is_err() {
                         reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
                             .map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?
                     } else {
-                        cached_encoder
+                        cached
                     }
                 }
-                None => {
-                    // First use, create new encoder
-                    reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
-                        .map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?
-                }
+                None => reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
+                    .map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?,
             }
         };
-
-        // Add original shards
         for (i, shard) in shards_vec.iter().enumerate().take(self.data_shards) {
             encoder
                 .add_original_shard(shard)
                 .map_err(|e| io::Error::other(format!("Failed to add shard {i}: {e:?}")))?;
         }
-
-        // Encode and get recovery shards
         let result = encoder
             .encode()
             .map_err(|e| io::Error::other(format!("SIMD encoding failed: {e:?}")))?;
-
-        // Copy recovery shards to output buffer
         for (i, recovery_shard) in result.recovery_iter().enumerate() {
             if i + self.data_shards < shards_vec.len() {
                 shards_vec[i + self.data_shards].copy_from_slice(recovery_shard);
             }
         }
-
-        // Return encoder to cache (encoder is automatically reset after result is dropped, can be reused)
-        drop(result); // Explicitly drop result to ensure encoder is reset
-
+        drop(result);
         *self
             .encoder_cache
             .write()
             .map_err(|_| io::Error::other("Failed to return encoder to cache"))? = Some(encoder);
-
         Ok(())
     }
 
-    /// Reconstruct missing shards.
-    pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
-        // Use SIMD for reconstruction
-        let simd_result = self.reconstruct_with_simd(shards);
-
-        match simd_result {
-            Ok(()) => Ok(()),
-            Err(simd_error) => {
-                warn!("SIMD reconstruction failed: {}", simd_error);
-                Err(simd_error)
-            }
-        }
-    }
-
-    fn reconstruct_with_simd(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
-        // Find a valid shard to determine length
+    fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         let shard_len = shards
             .iter()
             .find_map(|s| s.as_ref().map(|v| v.len()))
@@ -185,7 +122,6 @@ impl ReedSolomonEncoder {
                 Some(mut cached_decoder) => {
                     if let Err(e) = cached_decoder.reset(self.data_shards, self.parity_shards, shard_len) {
                         warn!("Failed to reset SIMD decoder: {:?}, creating new one", e);
-
                         reed_solomon_simd::ReedSolomonDecoder::new(self.data_shards, self.parity_shards, shard_len)
                             .map_err(|e| io::Error::other(format!("Failed to create SIMD decoder: {e:?}")))?
                     } else {
@@ -197,7 +133,6 @@ impl ReedSolomonEncoder {
             }
         };
 
-        // Add available shards (both data and parity)
         for (i, shard_opt) in shards.iter().enumerate() {
             if let Some(shard) = shard_opt {
                 if i < self.data_shards {
@@ -217,7 +152,6 @@ impl ReedSolomonEncoder {
             .decode()
             .map_err(|e| io::Error::other(format!("SIMD decode error: {e:?}")))?;
 
-        // Fill in missing data shards from reconstruction result
         for (i, shard_opt) in shards.iter_mut().enumerate() {
             if shard_opt.is_none() && i < self.data_shards {
                 for (restored_index, restored_data) in result.restored_original_iter() {
@@ -240,6 +174,67 @@ impl ReedSolomonEncoder {
     }
 }
 
+/// Reed-Solomon encoder using reed-solomon-erasure
+pub struct ReedSolomonEncoder {
+    data_shards: usize,
+    parity_shards: usize,
+    encoder: Option<ReedSolomon>,
+}
+
+impl Clone for ReedSolomonEncoder {
+    fn clone(&self) -> Self {
+        Self {
+            data_shards: self.data_shards,
+            parity_shards: self.parity_shards,
+            encoder: self.encoder.clone(),
+        }
+    }
+}
+
+impl ReedSolomonEncoder {
+    /// Create a new Reed-Solomon encoder with specified data and parity shards.
+    pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
+        let encoder = if parity_shards > 0 {
+            ReedSolomon::new(data_shards, parity_shards)
+                .map_err(|e| io::Error::other(format!("Failed to create Reed-Solomon encoder: {e:?}")))
+                .map(Some)?
+        } else {
+            None
+        };
+
+        Ok(ReedSolomonEncoder {
+            data_shards,
+            parity_shards,
+            encoder,
+        })
+    }
+
+    /// Encode data shards with parity.
+    pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
+        let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
+        if shards_vec.is_empty() {
+            return Ok(());
+        }
+
+        if let Some(ref rs) = self.encoder {
+            rs.encode(&mut shards_vec)
+                .map_err(|e| io::Error::other(format!("Reed-Solomon encode failed: {e:?}")))
+        } else {
+            Ok(())
+        }
+    }
+
+    /// Reconstruct missing shards.
+    pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        if let Some(ref rs) = self.encoder {
+            rs.reconstruct_data(shards)
+                .map_err(|e| io::Error::other(format!("Reed-Solomon reconstruct failed: {e:?}")))
+        } else {
+            Ok(())
+        }
+    }
+}
+
 /// Erasure coding utility for data reliability using Reed-Solomon codes.
 ///
 /// This struct provides encoding and decoding of data into data and parity shards.
@@ -262,24 +257,41 @@ impl ReedSolomonEncoder {
 /// let shards = erasure.encode_data(data).unwrap();
 /// // Simulate loss and recovery...
 /// ```
-
-#[derive(Default)]
 pub struct Erasure {
     pub data_shards: usize,
     pub parity_shards: usize,
     encoder: Option<ReedSolomonEncoder>,
+    legacy_encoder: Option<LegacyReedSolomonEncoder>,
     pub block_size: usize,
+    uses_legacy: bool,
     _id: Uuid,
     _buf: Vec<u8>,
 }
 
+impl Default for Erasure {
+    fn default() -> Self {
+        Self {
+            data_shards: 0,
+            parity_shards: 0,
+            encoder: None,
+            legacy_encoder: None,
+            block_size: 0,
+            uses_legacy: false,
+            _id: Uuid::nil(),
+            _buf: vec![],
+        }
+    }
+}
+
 impl Clone for Erasure {
     fn clone(&self) -> Self {
         Self {
             data_shards: self.data_shards,
             parity_shards: self.parity_shards,
             encoder: self.encoder.clone(),
+            legacy_encoder: self.legacy_encoder.clone(),
             block_size: self.block_size,
+            uses_legacy: self.uses_legacy,
             _id: Uuid::new_v4(), // Generate new ID for clone
             _buf: vec![0u8; self.block_size],
         }
@@ -287,28 +299,44 @@ impl Clone for Erasure {
 }
 
 pub fn calc_shard_size(block_size: usize, data_shards: usize) -> usize {
-    (block_size.div_ceil(data_shards) + 1) & !1
+    block_size.div_ceil(data_shards)
 }
 
 impl Erasure {
-    /// Create a new Erasure instance.
+    /// Create a new Erasure instance
     ///
     /// # Arguments
     /// * `data_shards` - Number of data shards.
     /// * `parity_shards` - Number of parity shards.
     /// * `block_size` - Block size for each shard.
     pub fn new(data_shards: usize, parity_shards: usize, block_size: usize) -> Self {
-        let encoder = if parity_shards > 0 {
+        Self::new_with_options(data_shards, parity_shards, block_size, false)
+    }
+
+    /// Create a new Erasure instance with legacy format support.
+    ///
+    /// When `uses_legacy` is true, uses main-branch shard_size formula and reed-solomon-simd
+    /// for decode/reconstruct (for reading and healing old-version files).
+    pub fn new_with_options(data_shards: usize, parity_shards: usize, block_size: usize, uses_legacy: bool) -> Self {
+        let encoder = if !uses_legacy && parity_shards > 0 {
             Some(ReedSolomonEncoder::new(data_shards, parity_shards).unwrap())
         } else {
             None
         };
 
+        let legacy_encoder = if uses_legacy && parity_shards > 0 {
+            Some(LegacyReedSolomonEncoder::new(data_shards, parity_shards).unwrap())
+        } else {
+            None
+        };
+
         Erasure {
             data_shards,
             parity_shards,
             block_size,
             encoder,
+            legacy_encoder,
+            uses_legacy,
             _id: Uuid::new_v4(),
             _buf: vec![0u8; block_size],
         }
@@ -323,28 +351,29 @@ impl Erasure {
     /// A vector of encoded shards as `Bytes`.
     #[tracing::instrument(level = "debug", skip_all, fields(data_len=data.len()))]
     pub fn encode_data(&self, data: &[u8]) -> io::Result<Vec<Bytes>> {
-        // let shard_size = self.shard_size();
-        // let total_size = shard_size * self.total_shard_count();
-
-        // Data shard count
-        let per_shard_size = calc_shard_size(data.len(), self.data_shards);
-        // Total required size
+        let shard_size_fn = if self.uses_legacy {
+            calc_shard_size_legacy
+        } else {
+            calc_shard_size
+        };
+        let per_shard_size = shard_size_fn(data.len(), self.data_shards);
         let need_total_size = per_shard_size * self.total_shard_count();
 
-        // Create a new buffer with the required total length for all shards
         let mut data_buffer = BytesMut::with_capacity(need_total_size);
-
-        // Copy source data
         data_buffer.extend_from_slice(data);
         data_buffer.resize(need_total_size, 0u8);
 
         {
-            // EC encode, the result will be written into data_buffer
             let data_slices: SmallVec<[&mut [u8]; 16]> = data_buffer.chunks_exact_mut(per_shard_size).collect();
 
-            // Only do EC if parity_shards > 0
             if self.parity_shards > 0 {
-                if let Some(encoder) = self.encoder.as_ref() {
+                if self.uses_legacy {
+                    if let Some(encoder) = self.legacy_encoder.as_ref() {
+                        encoder.encode(data_slices)?;
+                    } else {
+                        warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
+                    }
+                } else if let Some(encoder) = self.encoder.as_ref() {
                     encoder.encode(data_slices)?;
                 } else {
                     warn!("parity_shards > 0, but encoder is None");
@@ -372,7 +401,13 @@ impl Erasure {
     /// Ok if reconstruction succeeds, error otherwise.
     pub fn decode_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         if self.parity_shards > 0 {
-            if let Some(encoder) = self.encoder.as_ref() {
+            if self.uses_legacy {
+                if let Some(encoder) = self.legacy_encoder.as_ref() {
+                    encoder.reconstruct(shards)?;
+                } else {
+                    warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
+                }
+            } else if let Some(encoder) = self.encoder.as_ref() {
                 encoder.reconstruct(shards)?;
             } else {
                 warn!("parity_shards > 0, but encoder is None");
@@ -395,7 +430,11 @@ impl Erasure {
 
     /// Calculate the size of each shard.
     pub fn shard_size(&self) -> usize {
-        calc_shard_size(self.block_size, self.data_shards)
+        if self.uses_legacy {
+            calc_shard_size_legacy(self.block_size, self.data_shards)
+        } else {
+            calc_shard_size(self.block_size, self.data_shards)
+        }
     }
     /// Calculate the total erasure file size for a given original size.
     // Returns the final erasure size from the original size
@@ -408,10 +447,15 @@ impl Erasure {
         }
 
         let total_length = total_length as usize;
+        let shard_size_fn = if self.uses_legacy {
+            calc_shard_size_legacy
+        } else {
+            calc_shard_size
+        };
 
         let num_shards = total_length / self.block_size;
         let last_block_size = total_length % self.block_size;
-        let last_shard_size = calc_shard_size(last_block_size, self.data_shards);
+        let last_shard_size = shard_size_fn(last_block_size, self.data_shards);
         (num_shards * self.shard_size() + last_shard_size) as i64
     }
 
@@ -494,8 +538,7 @@ mod tests {
     #[test]
     fn test_shard_file_size_cases2() {
         let erasure = Erasure::new(12, 4, 1024 * 1024);
-
-        assert_eq!(erasure.shard_file_size(1572864), 131074);
+        assert_eq!(erasure.shard_file_size(1572864), 131073);
     }
 
     #[test]
@@ -517,11 +560,14 @@ mod tests {
         // Case 5: total_length > block_size, aligned
         assert_eq!(erasure.shard_file_size(16), 4); // 16/8=2, last=0, 2*2+0=4
 
-        assert_eq!(erasure.shard_file_size(1248739), 312186); // 1248739/8=156092, last=3, 3 div_ceil 4=1, 156092*2+1=312185
+        // MinIO-compatible: 1248739/8=156092, last=3, ceil(3/4)=1, 156092*2+1=312185
+        assert_eq!(erasure.shard_file_size(1248739), 312185);
 
-        assert_eq!(erasure.shard_file_size(43), 12); // 43/8=5, last=3, 3 div_ceil 4=1, 5*2+1=11
+        // MinIO-compatible: 43/8=5, last=3, ceil(3/4)=1, 5*2+1=11
+        assert_eq!(erasure.shard_file_size(43), 11);
 
-        assert_eq!(erasure.shard_file_size(1572864), 393216); // 43/8=5, last=3, 3 div_ceil 4=1, 5*2+1=11
+        // 1572864 with block_size=8: 196608 full blocks, last=0, 196608*2+0=393216
+        assert_eq!(erasure.shard_file_size(1572864), 393216);
     }
 
     #[test]
@@ -601,10 +647,70 @@ mod tests {
     #[test]
     fn test_shard_size_and_file_size() {
         let erasure = Erasure::new(4, 2, 8);
+        assert_eq!(erasure.shard_file_size(33), 9);
+        assert_eq!(erasure.shard_file_size(0), 0);
+    }
+
+    #[test]
+    fn test_legacy_shard_size_and_file_size() {
+        let erasure = Erasure::new_with_options(4, 2, 8, true);
+        assert_eq!(erasure.shard_size(), 2);
+        assert_eq!(calc_shard_size_legacy(8, 4), 2);
+        assert_eq!(calc_shard_size_legacy(1, 4), 2);
         assert_eq!(erasure.shard_file_size(33), 10);
         assert_eq!(erasure.shard_file_size(0), 0);
     }
 
+    #[test]
+    fn test_legacy_encode_decode_roundtrip() {
+        let data_shards = 4;
+        let parity_shards = 2;
+        let block_size = 1024;
+        let erasure = Erasure::new_with_options(data_shards, parity_shards, block_size, true);
+
+        let data = b"Legacy encode/decode roundtrip test data with sufficient length.".repeat(20);
+        let encoded_shards = erasure.encode_data(&data).unwrap();
+        assert_eq!(encoded_shards.len(), data_shards + parity_shards);
+
+        let mut decode_input: Vec<Option<Vec<u8>>> = vec![None; data_shards + parity_shards];
+        for i in 0..data_shards {
+            decode_input[i] = Some(encoded_shards[i].to_vec());
+        }
+
+        erasure.decode_data(&mut decode_input).unwrap();
+
+        let mut recovered = Vec::new();
+        for shard in decode_input.iter().take(data_shards) {
+            recovered.extend_from_slice(shard.as_ref().unwrap());
+        }
+        recovered.truncate(data.len());
+        assert_eq!(&recovered, &data);
+    }
+
+    #[test]
+    fn test_legacy_decode_with_missing_shards() {
+        let data_shards = 4;
+        let parity_shards = 2;
+        let block_size = 256;
+        let erasure = Erasure::new_with_options(data_shards, parity_shards, block_size, true);
+
+        let data = b"Legacy decode with missing shards test.".repeat(10);
+        let encoded_shards = erasure.encode_data(&data).unwrap();
+
+        let mut shards_opt: Vec<Option<Vec<u8>>> = encoded_shards.iter().map(|s| Some(s.to_vec())).collect();
+        shards_opt[1] = None;
+        shards_opt[5] = None;
+
+        erasure.decode_data(&mut shards_opt).unwrap();
+
+        let mut recovered = Vec::new();
+        for shard in shards_opt.iter().take(data_shards) {
+            recovered.extend_from_slice(shard.as_ref().unwrap());
+        }
+        recovered.truncate(data.len());
+        assert_eq!(&recovered, &data);
+    }
+
     #[test]
     fn test_shard_file_offset() {
         let erasure = Erasure::new(8, 8, 1024 * 1024);
@@ -887,6 +993,57 @@ mod tests {
             assert_eq!(&recovered, &data);
         }
 
+        /// Generates 7557 bytes identical to MinIO generateCompatTestData.
+        fn generate_compat_test_data(size: usize) -> Vec<u8> {
+            (0..size).map(|i| ((i * 7 + 13) % 256) as u8).collect()
+        }
+
+        /// Verifies reed-solomon-simd produces same shards.
+        /// Data shards (0-3) must match for MinIO to read RustFS part files.
+        /// Parity shards (4-5) differ: reed-solomon-simd vs klauspost use different RS encoding.
+        /// Run: cargo test -p rustfs-ecstore test_reed_solomon_compat
+        #[test]
+        fn test_reed_solomon_compat() {
+            let data = generate_compat_test_data(7557);
+            let erasure = Erasure::new(4, 2, 7557);
+            let shards = erasure.encode_data(&data).unwrap();
+            assert_eq!(shards.len(), 6, "expected 6 shards (4 data + 2 parity)");
+
+            // Per-shard HighwayHash
+            let expected_hashes: [&str; 6] = [
+                "fb3db9338e610cec541504ddae4b0bfd54445bcbd45318cf21f35f024240914d", // data 0
+                "a545269a3196e18e77ef9f5ec6e735a4f4ebe82d342db666b11a5256eb305720", // data 1
+                "2adbf0058f36c4cbcb5c9c16c38a6530c54198dfe504179a6f92d2349f245318", // data 2
+                "898e6d060b0cb4f0e830add7e1f936bc8b78442bf582283ee244a3a058602db8", // data 3
+                "4a20460bca044b3a777b26f2b0bcd371e3eab2f156f84778be3ccd8edd521ef2", // parity 4
+                "eb8ba4c0db15ca910d58d031f74e4601ba2fed62ad03ec29cadde3367ab0d415", // parity 5
+            ];
+
+            let mut data_shards_match = true;
+            let mut parity_shards_match = true;
+            for (i, shard) in shards.iter().enumerate() {
+                let hash = rustfs_utils::HashAlgorithm::HighwayHash256S.hash_encode(shard);
+                let got = hex_simd::encode_to_string(hash.as_ref(), hex_simd::AsciiCase::Lower);
+                let matches = got == expected_hashes[i];
+                if i < 4 {
+                    data_shards_match &= matches;
+                } else {
+                    parity_shards_match &= matches;
+                }
+                if !matches {
+                    eprintln!(
+                        "Shard {} ({}): got {} want {}",
+                        i,
+                        if i < 4 { "data" } else { "parity" },
+                        got,
+                        expected_hashes[i]
+                    );
+                }
+            }
+            assert!(data_shards_match, "Data shards (0-3) must match");
+            assert!(parity_shards_match, "Parity shards (4-5): reed-solomon-simd differs");
+        }
+
         #[test]
         fn test_simd_small_data_handling() {
             let data_shards = 4;

crates/ecstore/src/erasure_coding/mod.rs

@@ -19,4 +19,4 @@ pub mod erasure;
 pub mod heal;
 pub use bitrot::*;
 
-pub use erasure::{Erasure, ReedSolomonEncoder, calc_shard_size};
+pub use erasure::{Erasure, ReedSolomonEncoder, calc_shard_size, calc_shard_size_legacy};

crates/ecstore/src/file_cache.rs

@@ -1,332 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! High-performance file content and metadata caching using moka
-//!
-//! This module provides optimized caching for file operations to reduce
-//! redundant I/O and improve overall system performance.
-
-use super::disk::error::{Error, Result};
-use bytes::Bytes;
-use moka::future::Cache;
-use rustfs_filemeta::FileMeta;
-use std::path::{Path, PathBuf};
-use std::sync::Arc;
-use std::time::Duration;
-
-pub struct OptimizedFileCache {
-    // Use moka as high-performance async cache
-    metadata_cache: Cache<PathBuf, Arc<FileMeta>>,
-    file_content_cache: Cache<PathBuf, Bytes>,
-    // Performance monitoring
-    cache_hits: std::sync::atomic::AtomicU64,
-    cache_misses: std::sync::atomic::AtomicU64,
-}
-
-impl OptimizedFileCache {
-    pub fn new() -> Self {
-        Self {
-            metadata_cache: Cache::builder()
-                .max_capacity(2048)
-                .time_to_live(Duration::from_secs(300)) // 5 minutes TTL
-                .time_to_idle(Duration::from_secs(60)) // 1 minute idle
-                .build(),
-
-            file_content_cache: Cache::builder()
-                .max_capacity(512) // Smaller file content cache
-                .time_to_live(Duration::from_secs(120))
-                .weigher(|_key: &PathBuf, value: &Bytes| value.len() as u32)
-                .build(),
-
-            cache_hits: std::sync::atomic::AtomicU64::new(0),
-            cache_misses: std::sync::atomic::AtomicU64::new(0),
-        }
-    }
-
-    pub async fn get_metadata(&self, path: PathBuf) -> Result<Arc<FileMeta>> {
-        if let Some(cached) = self.metadata_cache.get(&path).await {
-            self.cache_hits.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
-            return Ok(cached);
-        }
-
-        self.cache_misses.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
-
-        // Cache miss, read file
-        let data = tokio::fs::read(&path)
-            .await
-            .map_err(|e| Error::other(format!("Read metadata failed: {e}")))?;
-
-        let mut meta = FileMeta::default();
-        meta.unmarshal_msg(&data)?;
-
-        let arc_meta = Arc::new(meta);
-        self.metadata_cache.insert(path, arc_meta.clone()).await;
-
-        Ok(arc_meta)
-    }
-
-    pub async fn get_file_content(&self, path: PathBuf) -> Result<Bytes> {
-        if let Some(cached) = self.file_content_cache.get(&path).await {
-            self.cache_hits.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
-            return Ok(cached);
-        }
-
-        self.cache_misses.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
-
-        let data = tokio::fs::read(&path)
-            .await
-            .map_err(|e| Error::other(format!("Read file failed: {e}")))?;
-
-        let bytes = Bytes::from(data);
-        self.file_content_cache.insert(path, bytes.clone()).await;
-
-        Ok(bytes)
-    }
-
-    // Prefetch related files
-    pub async fn prefetch_related(&self, base_path: &Path, patterns: &[&str]) {
-        let mut prefetch_tasks = Vec::new();
-
-        for pattern in patterns {
-            let path = base_path.join(pattern);
-            if tokio::fs::metadata(&path).await.is_ok() {
-                let cache = self.clone();
-                let path_clone = path.clone();
-                prefetch_tasks.push(async move {
-                    let _ = cache.get_metadata(path_clone).await;
-                });
-            }
-        }
-
-        // Parallel prefetch, don't wait for completion
-        if !prefetch_tasks.is_empty() {
-            tokio::spawn(async move {
-                futures::future::join_all(prefetch_tasks).await;
-            });
-        }
-    }
-
-    // Batch metadata reading with deduplication
-    pub async fn get_metadata_batch(
-        &self,
-        paths: Vec<PathBuf>,
-    ) -> Vec<std::result::Result<Arc<FileMeta>, rustfs_filemeta::Error>> {
-        let mut results = Vec::with_capacity(paths.len());
-        let mut cache_futures = Vec::new();
-
-        // First, attempt to get from cache
-        for (i, path) in paths.iter().enumerate() {
-            if let Some(cached) = self.metadata_cache.get(path).await {
-                results.push((i, Ok(cached)));
-                self.cache_hits.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
-            } else {
-                cache_futures.push((i, path.clone()));
-            }
-        }
-
-        // For cache misses, read from filesystem
-        if !cache_futures.is_empty() {
-            let mut fs_results = Vec::new();
-
-            for (i, path) in cache_futures {
-                self.cache_misses.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
-
-                match tokio::fs::read(&path).await {
-                    Ok(data) => {
-                        let mut meta = FileMeta::default();
-                        match meta.unmarshal_msg(&data) {
-                            Ok(_) => {
-                                let arc_meta = Arc::new(meta);
-                                self.metadata_cache.insert(path, arc_meta.clone()).await;
-                                fs_results.push((i, Ok(arc_meta)));
-                            }
-                            Err(e) => {
-                                fs_results.push((i, Err(e)));
-                            }
-                        }
-                    }
-                    Err(_e) => {
-                        fs_results.push((i, Err(rustfs_filemeta::Error::Unexpected)));
-                    }
-                }
-            }
-
-            results.extend(fs_results);
-        }
-
-        // Sort results back to original order
-        results.sort_by_key(|(i, _)| *i);
-        results.into_iter().map(|(_, result)| result).collect()
-    }
-
-    // Invalidate cache entries for a path
-    pub async fn invalidate(&self, path: &Path) {
-        self.metadata_cache.remove(path).await;
-        self.file_content_cache.remove(path).await;
-    }
-
-    // Get cache statistics
-    pub fn get_stats(&self) -> FileCacheStats {
-        let hits = self.cache_hits.load(std::sync::atomic::Ordering::Relaxed);
-        let misses = self.cache_misses.load(std::sync::atomic::Ordering::Relaxed);
-        let hit_rate = if hits + misses > 0 {
-            (hits as f64 / (hits + misses) as f64) * 100.0
-        } else {
-            0.0
-        };
-
-        FileCacheStats {
-            metadata_cache_size: self.metadata_cache.entry_count(),
-            content_cache_size: self.file_content_cache.entry_count(),
-            cache_hits: hits,
-            cache_misses: misses,
-            hit_rate,
-            total_weight: 0, // Simplified for compatibility
-        }
-    }
-
-    // Clear all caches
-    pub async fn clear(&self) {
-        self.metadata_cache.invalidate_all();
-        self.file_content_cache.invalidate_all();
-
-        // Wait for invalidation to complete
-        self.metadata_cache.run_pending_tasks().await;
-        self.file_content_cache.run_pending_tasks().await;
-    }
-}
-
-impl Clone for OptimizedFileCache {
-    fn clone(&self) -> Self {
-        Self {
-            metadata_cache: self.metadata_cache.clone(),
-            file_content_cache: self.file_content_cache.clone(),
-            cache_hits: std::sync::atomic::AtomicU64::new(self.cache_hits.load(std::sync::atomic::Ordering::Relaxed)),
-            cache_misses: std::sync::atomic::AtomicU64::new(self.cache_misses.load(std::sync::atomic::Ordering::Relaxed)),
-        }
-    }
-}
-
-#[derive(Debug)]
-pub struct FileCacheStats {
-    pub metadata_cache_size: u64,
-    pub content_cache_size: u64,
-    pub cache_hits: u64,
-    pub cache_misses: u64,
-    pub hit_rate: f64,
-    pub total_weight: u64,
-}
-
-impl Default for OptimizedFileCache {
-    fn default() -> Self {
-        Self::new()
-    }
-}
-
-// Global cache instance
-use std::sync::OnceLock;
-
-static GLOBAL_FILE_CACHE: OnceLock<OptimizedFileCache> = OnceLock::new();
-
-pub fn get_global_file_cache() -> &'static OptimizedFileCache {
-    GLOBAL_FILE_CACHE.get_or_init(OptimizedFileCache::new)
-}
-
-// Utility functions for common operations
-pub async fn read_metadata_cached(path: PathBuf) -> Result<Arc<FileMeta>> {
-    get_global_file_cache().get_metadata(path).await
-}
-
-pub async fn read_file_content_cached(path: PathBuf) -> Result<Bytes> {
-    get_global_file_cache().get_file_content(path).await
-}
-
-pub async fn prefetch_metadata_patterns(base_path: &Path, patterns: &[&str]) {
-    get_global_file_cache().prefetch_related(base_path, patterns).await;
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::io::Write;
-    use tempfile::tempdir;
-
-    #[tokio::test]
-    async fn test_file_cache_basic() {
-        let cache = OptimizedFileCache::new();
-
-        // Create a temporary file
-        let dir = tempdir().unwrap();
-        let file_path = dir.path().join("test.txt");
-        let mut file = std::fs::File::create(&file_path).unwrap();
-        writeln!(file, "test content").unwrap();
-        drop(file);
-
-        // First read should be cache miss
-        let content1 = cache.get_file_content(file_path.clone()).await.unwrap();
-        assert_eq!(content1, Bytes::from("test content\n"));
-
-        // Second read should be cache hit
-        let content2 = cache.get_file_content(file_path.clone()).await.unwrap();
-        assert_eq!(content2, content1);
-
-        let stats = cache.get_stats();
-        assert!(stats.cache_hits > 0);
-        assert!(stats.cache_misses > 0);
-    }
-
-    #[tokio::test]
-    async fn test_metadata_batch_read() {
-        let cache = OptimizedFileCache::new();
-
-        // Create test files
-        let dir = tempdir().unwrap();
-        let mut paths = Vec::new();
-
-        for i in 0..5 {
-            let file_path = dir.path().join(format!("test_{i}.txt"));
-            let mut file = std::fs::File::create(&file_path).unwrap();
-            writeln!(file, "content {i}").unwrap();
-            paths.push(file_path);
-        }
-
-        // Note: This test would need actual FileMeta files to work properly
-        // For now, we just test that the function runs without errors
-        let results = cache.get_metadata_batch(paths).await;
-        assert_eq!(results.len(), 5);
-    }
-
-    #[tokio::test]
-    async fn test_cache_invalidation() {
-        let cache = OptimizedFileCache::new();
-
-        let dir = tempdir().unwrap();
-        let file_path = dir.path().join("test.txt");
-        let mut file = std::fs::File::create(&file_path).unwrap();
-        writeln!(file, "test content").unwrap();
-        drop(file);
-
-        // Read file to populate cache
-        let _ = cache.get_file_content(file_path.clone()).await.unwrap();
-
-        // Invalidate cache
-        cache.invalidate(&file_path).await;
-
-        // Next read should be cache miss again
-        let _ = cache.get_file_content(file_path.clone()).await.unwrap();
-
-        let stats = cache.get_stats();
-        assert!(stats.cache_misses >= 2);
-    }
-}

crates/ecstore/src/global.rs

@@ -46,6 +46,7 @@ lazy_static! {
     pub static ref GLOBAL_IsDistErasure: RwLock<bool> = RwLock::new(false);
     pub static ref GLOBAL_IsErasureSD: RwLock<bool> = RwLock::new(false);
     pub static ref GLOBAL_LOCAL_DISK_MAP: Arc<RwLock<HashMap<String, Option<DiskStore>>>> = Arc::new(RwLock::new(HashMap::new()));
+    pub static ref GLOBAL_LOCAL_DISK_ID_MAP: Arc<RwLock<HashMap<Uuid, String>>> = Arc::new(RwLock::new(HashMap::new()));
     pub static ref GLOBAL_LOCAL_DISK_SET_DRIVES: Arc<RwLock<TypeLocalDiskSetDrives>> = Arc::new(RwLock::new(Vec::new()));
     pub static ref GLOBAL_Endpoints: OnceLock<EndpointServerPools> = OnceLock::new();
     pub static ref GLOBAL_RootDiskThreshold: RwLock<u64> = RwLock::new(0);
@@ -153,6 +154,10 @@ pub fn get_global_endpoints_opt() -> Option<EndpointServerPools> {
     GLOBAL_Endpoints.get().cloned()
 }
 
+pub async fn is_first_cluster_node_local() -> bool {
+    get_global_endpoints().first_local()
+}
+
 pub fn get_global_tier_config_mgr() -> Arc<RwLock<TierConfigMgr>> {
     GLOBAL_TierConfigMgr.clone()
 }

crates/ecstore/src/lib.rs

@@ -28,7 +28,6 @@ pub mod disks_layout;
 pub mod endpoints;
 pub mod erasure_coding;
 pub mod error;
-pub mod file_cache;
 pub mod global;
 pub mod metrics_realtime;
 pub mod notification_sys;

crates/ecstore/src/metrics_realtime.rs

@@ -14,8 +14,11 @@
 
 use crate::{admin_server_info::get_local_server_property, new_object_layer_fn, store_api::StorageAPI};
 use chrono::Utc;
-use rustfs_common::{GLOBAL_LOCAL_NODE_NAME, GLOBAL_RUSTFS_ADDR, heal_channel::DriveState, metrics::global_metrics};
-use rustfs_madmin::metrics::{DiskIOStats, DiskMetric, RealtimeMetrics};
+use rustfs_common::{
+    GLOBAL_LOCAL_NODE_NAME, GLOBAL_RUSTFS_ADDR, heal_channel::DriveState, internode_metrics::global_internode_metrics,
+    metrics::global_metrics,
+};
+use rustfs_madmin::metrics::{DiskIOStats, DiskMetric, NetDevLine, NetMetrics, RPCMetrics, RealtimeMetrics};
 use rustfs_utils::os::get_drive_stats;
 use serde::{Deserialize, Serialize};
 use std::collections::{HashMap, HashSet};
@@ -120,13 +123,50 @@ pub async fn collect_local_metrics(types: MetricType, opts: &CollectMetricsOpts)
 
     // if types.contains(&MetricType::SITE_RESYNC) {}
 
-    // if types.contains(&MetricType::NET) {}
+    if types.contains(&MetricType::NET) {
+        let snapshot = global_internode_metrics().snapshot();
+        real_time_metrics.aggregated.net = Some(NetMetrics {
+            collected_at: Utc::now(),
+            interface_name: "internode".to_string(),
+            net_stats: NetDevLine {
+                name: "internode".to_string(),
+                rx_bytes: snapshot.recv_bytes_total,
+                tx_bytes: snapshot.sent_bytes_total,
+                ..Default::default()
+            },
+        });
+    }
 
     // if types.contains(&MetricType::MEM) {}
 
     // if types.contains(&MetricType::CPU) {}
 
-    // if types.contains(&MetricType::RPC) {}
+    if types.contains(&MetricType::RPC) {
+        let collected_at = Utc::now();
+        let snapshot = global_internode_metrics().snapshot();
+        let last_connect_time =
+            chrono::DateTime::<Utc>::from_timestamp_millis(snapshot.last_dial_unix_millis as i64).unwrap_or(collected_at);
+
+        real_time_metrics.aggregated.rpc = Some(RPCMetrics {
+            collected_at,
+            connected: i32::from(snapshot.last_dial_unix_millis > 0),
+            reconnect_count: snapshot.dial_errors_total.min(i32::MAX as u64) as i32,
+            disconnected: 0,
+            outgoing_streams: 0,
+            incoming_streams: 0,
+            outgoing_bytes: snapshot.sent_bytes_total.min(i64::MAX as u64) as i64,
+            incoming_bytes: snapshot.recv_bytes_total.min(i64::MAX as u64) as i64,
+            outgoing_messages: snapshot.outgoing_requests_total.min(i64::MAX as u64) as i64,
+            incoming_messages: snapshot.incoming_requests_total.min(i64::MAX as u64) as i64,
+            out_queue: 0,
+            last_pong_time: collected_at,
+            last_ping_ms: snapshot.dial_avg_time_nanos as f64 / 1_000_000.0,
+            max_ping_dur_ms: snapshot.dial_avg_time_nanos as f64 / 1_000_000.0,
+            last_connect_time,
+            by_destination: None,
+            by_caller: None,
+        });
+    }
 
     real_time_metrics
         .by_host
@@ -211,7 +251,9 @@ async fn collect_local_disks_metrics(disks: &HashSet<String>) -> HashMap<String,
 
 #[cfg(test)]
 mod test {
-    use super::MetricType;
+    use super::*;
+    use rustfs_common::internode_metrics::global_internode_metrics;
+    use std::time::Duration;
 
     #[test]
     fn tes_types() {
@@ -229,4 +271,30 @@ mod test {
         let disk = MetricType::new(1 << 1);
         assert!(disk.contains(&MetricType::DISK));
     }
+
+    #[tokio::test]
+    async fn collect_local_metrics_reports_internode_net_and_rpc() {
+        let metrics = global_internode_metrics();
+        metrics.reset_for_test();
+        metrics.record_sent_bytes(128);
+        metrics.record_recv_bytes(64);
+        metrics.record_outgoing_request();
+        metrics.record_incoming_request();
+        metrics.record_dial_result(Duration::from_millis(4), true);
+
+        let realtime = collect_local_metrics(MetricType::NET, &CollectMetricsOpts::default()).await;
+        let net = realtime.aggregated.net.expect("net metrics");
+        assert_eq!(net.net_stats.tx_bytes, 128);
+        assert_eq!(net.net_stats.rx_bytes, 64);
+
+        let realtime = collect_local_metrics(MetricType::RPC, &CollectMetricsOpts::default()).await;
+        let rpc = realtime.aggregated.rpc.expect("rpc metrics");
+        assert_eq!(rpc.outgoing_bytes, 128);
+        assert_eq!(rpc.incoming_bytes, 64);
+        assert_eq!(rpc.outgoing_messages, 1);
+        assert_eq!(rpc.incoming_messages, 1);
+        assert!(rpc.last_ping_ms > 0.0);
+
+        metrics.reset_for_test();
+    }
 }

crates/ecstore/src/rpc/remote_disk.rs

@@ -23,6 +23,7 @@ use crate::disk::{
 };
 use crate::disk::{disk_store::DiskHealthTracker, error::DiskError, local::ScanGuard};
 use crate::rpc::client::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
+use crate::set_disk::DEFAULT_READ_BUFFER_SIZE;
 use crate::{
     disk::error::{Error, Result},
     rpc::build_auth_headers,
@@ -40,7 +41,9 @@ use rustfs_protos::proto_gen::node_service::{
     node_service_client::NodeServiceClient,
 };
 use rustfs_rio::{HttpReader, HttpWriter};
+use serde::{Serialize, de::DeserializeOwned};
 use std::{
+    io::Cursor,
     path::PathBuf,
     sync::{
         Arc,
@@ -49,12 +52,36 @@ use std::{
     time::Duration,
 };
 use tokio::time;
-use tokio::{io::AsyncWrite, net::TcpStream, time::timeout};
+use tokio::{
+    io::{self, AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt},
+    net::TcpStream,
+    time::timeout,
+};
 use tokio_util::sync::CancellationToken;
 use tonic::{Request, service::interceptor::InterceptedService, transport::Channel};
 use tracing::{debug, info, warn};
 use uuid::Uuid;
 
+async fn copy_stream_with_buffer<R, W>(reader: &mut R, writer: &mut W, buffer_size: usize) -> io::Result<u64>
+where
+    R: AsyncRead + Unpin,
+    W: AsyncWrite + Unpin,
+{
+    let mut copied = 0_u64;
+    let mut buffer = vec![0_u8; buffer_size];
+
+    loop {
+        let bytes_read = reader.read(&mut buffer).await?;
+        if bytes_read == 0 {
+            writer.flush().await?;
+            return Ok(copied);
+        }
+
+        writer.write_all(&buffer[..bytes_read]).await?;
+        copied += bytes_read as u64;
+    }
+}
+
 #[derive(Debug)]
 pub struct RemoteDisk {
     pub id: Mutex<Option<Uuid>>,
@@ -259,6 +286,27 @@ impl RemoteDisk {
             .await
             .map_err(|err| Error::other(format!("can not get client, err: {err}")))
     }
+
+    async fn disk_ref(&self) -> String {
+        (*self.id.lock().await)
+            .map(|id| id.to_string())
+            .unwrap_or_else(|| self.endpoint.to_string())
+    }
+}
+
+fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
+    let mut serializer = rmp_serde::Serializer::new(Vec::new());
+    value.serialize(&mut serializer)?;
+    Ok(serializer.into_inner())
+}
+
+fn decode_msgpack_or_json<T: DeserializeOwned>(binary: &[u8], json: &str) -> Result<T> {
+    if !binary.is_empty() {
+        let mut deserializer = rmp_serde::Deserializer::new(Cursor::new(binary));
+        return T::deserialize(&mut deserializer).map_err(Error::from);
+    }
+
+    serde_json::from_str(json).map_err(Error::from)
 }
 
 // TODO: all api need to handle errors
@@ -707,18 +755,21 @@ impl DiskAPI for RemoteDisk {
     async fn write_metadata(&self, _org_volume: &str, volume: &str, path: &str, fi: FileInfo) -> Result<()> {
         info!("write_metadata {}/{}", volume, path);
         let file_info = serde_json::to_string(&fi)?;
+        let file_info_bin = encode_msgpack(&fi)?;
 
         self.execute_with_timeout(
             || async {
+                let disk = self.disk_ref().await;
                 let mut client = self
                     .get_client()
                     .await
                     .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
                 let request = Request::new(WriteMetadataRequest {
-                    disk: self.endpoint.to_string(),
+                    disk,
                     volume: volume.to_string(),
                     path: path.to_string(),
                     file_info: file_info.clone(),
+                    file_info_bin: file_info_bin.clone(),
                 });
 
                 let response = client.write_metadata(request).await?.into_inner();
@@ -735,6 +786,7 @@ impl DiskAPI for RemoteDisk {
     }
 
     async fn read_metadata(&self, volume: &str, path: &str) -> Result<Bytes> {
+        let disk = self.disk_ref().await;
         let mut client = self
             .get_client()
             .await
@@ -742,7 +794,7 @@ impl DiskAPI for RemoteDisk {
         let request = Request::new(ReadMetadataRequest {
             volume: volume.to_string(),
             path: path.to_string(),
-            disk: self.endpoint.to_string(),
+            disk,
         });
 
         let response = client.read_metadata(request).await?.into_inner();
@@ -759,19 +811,24 @@ impl DiskAPI for RemoteDisk {
         info!("update_metadata");
         let file_info = serde_json::to_string(&fi)?;
         let opts_str = serde_json::to_string(&opts)?;
+        let file_info_bin = encode_msgpack(&fi)?;
+        let opts_bin = encode_msgpack(opts)?;
 
         self.execute_with_timeout(
             || async {
+                let disk = self.disk_ref().await;
                 let mut client = self
                     .get_client()
                     .await
                     .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
                 let request = Request::new(UpdateMetadataRequest {
-                    disk: self.endpoint.to_string(),
+                    disk,
                     volume: volume.to_string(),
                     path: path.to_string(),
                     file_info: file_info.clone(),
                     opts: opts_str.clone(),
+                    file_info_bin: file_info_bin.clone(),
+                    opts_bin: opts_bin.clone(),
                 });
 
                 let response = client.update_metadata(request).await?.into_inner();
@@ -798,19 +855,22 @@ impl DiskAPI for RemoteDisk {
     ) -> Result<FileInfo> {
         info!("read_version");
         let opts_str = serde_json::to_string(opts)?;
+        let opts_bin = encode_msgpack(opts)?;
 
         self.execute_with_timeout(
             || async {
+                let disk = self.disk_ref().await;
                 let mut client = self
                     .get_client()
                     .await
                     .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
                 let request = Request::new(ReadVersionRequest {
-                    disk: self.endpoint.to_string(),
+                    disk,
                     volume: volume.to_string(),
                     path: path.to_string(),
                     version_id: version_id.to_string(),
                     opts: opts_str.clone(),
+                    opts_bin: opts_bin.clone(),
                 });
 
                 let response = client.read_version(request).await?.into_inner();
@@ -819,7 +879,7 @@ impl DiskAPI for RemoteDisk {
                     return Err(response.error.unwrap_or_default().into());
                 }
 
-                let file_info = serde_json::from_str::<FileInfo>(&response.file_info)?;
+                let file_info = decode_msgpack_or_json::<FileInfo>(&response.file_info_bin, &response.file_info)?;
 
                 Ok(file_info)
             },
@@ -834,12 +894,13 @@ impl DiskAPI for RemoteDisk {
 
         self.execute_with_timeout(
             || async {
+                let disk = self.disk_ref().await;
                 let mut client = self
                     .get_client()
                     .await
                     .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
                 let request = Request::new(ReadXlRequest {
-                    disk: self.endpoint.to_string(),
+                    disk,
                     volume: volume.to_string(),
                     path: path.to_string(),
                     read_data,
@@ -851,7 +912,7 @@ impl DiskAPI for RemoteDisk {
                     return Err(response.error.unwrap_or_default().into());
                 }
 
-                let raw_file_info = serde_json::from_str::<RawFileInfo>(&response.raw_file_info)?;
+                let raw_file_info = decode_msgpack_or_json::<RawFileInfo>(&response.raw_file_info_bin, &response.raw_file_info)?;
 
                 Ok(raw_file_info)
             },
@@ -909,13 +970,14 @@ impl DiskAPI for RemoteDisk {
         if self.health.is_faulty() {
             return Err(DiskError::FaultyDisk);
         }
+        let disk = self.disk_ref().await;
 
         let mut client = self
             .get_client()
             .await
             .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
         let request = Request::new(ListDirRequest {
-            disk: self.endpoint.to_string(),
+            disk,
             volume: volume.to_string(),
             dir_path: dir_path.to_string(),
             count,
@@ -937,12 +999,9 @@ impl DiskAPI for RemoteDisk {
         if self.health.is_faulty() {
             return Err(DiskError::FaultyDisk);
         }
+        let disk = self.disk_ref().await;
 
-        let url = format!(
-            "{}/rustfs/rpc/walk_dir?disk={}",
-            self.endpoint.grid_host(),
-            urlencoding::encode(self.endpoint.to_string().as_str()),
-        );
+        let url = format!("{}/rustfs/rpc/walk_dir?disk={}", self.endpoint.grid_host(), urlencoding::encode(&disk),);
 
         let opts = serde_json::to_vec(&opts)?;
 
@@ -952,33 +1011,14 @@ impl DiskAPI for RemoteDisk {
 
         let mut reader = HttpReader::new(url, Method::GET, headers, Some(opts)).await?;
 
-        tokio::io::copy(&mut reader, wr).await?;
+        copy_stream_with_buffer(&mut reader, wr, DEFAULT_READ_BUFFER_SIZE).await?;
 
         Ok(())
     }
 
     #[tracing::instrument(level = "debug", skip(self))]
     async fn read_file(&self, volume: &str, path: &str) -> Result<FileReader> {
-        info!("read_file {}/{}", volume, path);
-
-        if self.health.is_faulty() {
-            return Err(DiskError::FaultyDisk);
-        }
-
-        let url = format!(
-            "{}/rustfs/rpc/read_file_stream?disk={}&volume={}&path={}&offset={}&length={}",
-            self.endpoint.grid_host(),
-            urlencoding::encode(self.endpoint.to_string().as_str()),
-            urlencoding::encode(volume),
-            urlencoding::encode(path),
-            0,
-            0
-        );
-
-        let mut headers = HeaderMap::new();
-        headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
-        build_auth_headers(&url, &Method::GET, &mut headers);
-        Ok(Box::new(HttpReader::new(url, Method::GET, headers, None).await?))
+        self.read_file_stream(volume, path, 0, 0).await
     }
 
     #[tracing::instrument(level = "debug", skip(self))]
@@ -995,11 +1035,12 @@ impl DiskAPI for RemoteDisk {
         if self.health.is_faulty() {
             return Err(DiskError::FaultyDisk);
         }
+        let disk = self.disk_ref().await;
 
         let url = format!(
             "{}/rustfs/rpc/read_file_stream?disk={}&volume={}&path={}&offset={}&length={}",
             self.endpoint.grid_host(),
-            urlencoding::encode(self.endpoint.to_string().as_str()),
+            urlencoding::encode(&disk),
             urlencoding::encode(volume),
             urlencoding::encode(path),
             offset,
@@ -1019,11 +1060,12 @@ impl DiskAPI for RemoteDisk {
         if self.health.is_faulty() {
             return Err(DiskError::FaultyDisk);
         }
+        let disk = self.disk_ref().await;
 
         let url = format!(
             "{}/rustfs/rpc/put_file_stream?disk={}&volume={}&path={}&append={}&size={}",
             self.endpoint.grid_host(),
-            urlencoding::encode(self.endpoint.to_string().as_str()),
+            urlencoding::encode(&disk),
             urlencoding::encode(volume),
             urlencoding::encode(path),
             true,
@@ -1049,11 +1091,12 @@ impl DiskAPI for RemoteDisk {
         if self.health.is_faulty() {
             return Err(DiskError::FaultyDisk);
         }
+        let disk = self.disk_ref().await;
 
         let url = format!(
             "{}/rustfs/rpc/put_file_stream?disk={}&volume={}&path={}&append={}&size={}",
             self.endpoint.grid_host(),
-            urlencoding::encode(self.endpoint.to_string().as_str()),
+            urlencoding::encode(&disk),
             urlencoding::encode(volume),
             urlencoding::encode(path),
             false,
@@ -1261,13 +1304,16 @@ impl DiskAPI for RemoteDisk {
         self.execute_with_timeout(
             || async {
                 let read_multiple_req = serde_json::to_string(&req)?;
+                let read_multiple_req_bin = encode_msgpack(&req)?;
+                let disk = self.disk_ref().await;
                 let mut client = self
                     .get_client()
                     .await
                     .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
                 let request = Request::new(ReadMultipleRequest {
-                    disk: self.endpoint.to_string(),
+                    disk,
                     read_multiple_req,
+                    read_multiple_req_bin,
                 });
 
                 let response = client.read_multiple(request).await?.into_inner();
@@ -1276,11 +1322,19 @@ impl DiskAPI for RemoteDisk {
                     return Err(response.error.unwrap_or_default().into());
                 }
 
-                let read_multiple_resps = response
-                    .read_multiple_resps
-                    .into_iter()
-                    .filter_map(|json_str| serde_json::from_str::<ReadMultipleResp>(&json_str).ok())
-                    .collect();
+                let read_multiple_resps = if !response.read_multiple_resps_bin.is_empty() {
+                    response
+                        .read_multiple_resps_bin
+                        .into_iter()
+                        .filter_map(|buf| decode_msgpack_or_json::<ReadMultipleResp>(&buf, "").ok())
+                        .collect()
+                } else {
+                    response
+                        .read_multiple_resps
+                        .into_iter()
+                        .filter_map(|json_str| serde_json::from_str::<ReadMultipleResp>(&json_str).ok())
+                        .collect()
+                };
 
                 Ok(read_multiple_resps)
             },
@@ -1295,12 +1349,13 @@ impl DiskAPI for RemoteDisk {
 
         self.execute_with_timeout(
             || async {
+                let disk = self.disk_ref().await;
                 let mut client = self
                     .get_client()
                     .await
                     .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
                 let request = Request::new(WriteAllRequest {
-                    disk: self.endpoint.to_string(),
+                    disk,
                     volume: volume.to_string(),
                     path: path.to_string(),
                     data,
@@ -1325,12 +1380,13 @@ impl DiskAPI for RemoteDisk {
 
         self.execute_with_timeout(
             || async {
+                let disk = self.disk_ref().await;
                 let mut client = self
                     .get_client()
                     .await
                     .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
                 let request = Request::new(ReadAllRequest {
-                    disk: self.endpoint.to_string(),
+                    disk,
                     volume: volume.to_string(),
                     path: path.to_string(),
                 });
@@ -1386,6 +1442,7 @@ impl DiskAPI for RemoteDisk {
 mod tests {
     use super::*;
     use std::sync::Once;
+    use tokio::io::duplex;
     use tokio::net::TcpListener;
     use tracing::Level;
     use uuid::Uuid;
@@ -1543,6 +1600,24 @@ mod tests {
         assert!(!remote_disk.is_online().await);
     }
 
+    #[tokio::test]
+    async fn test_copy_stream_with_buffer_copies_full_payload() {
+        let payload = b"walk-dir-stream".repeat(1024);
+        let expected = payload.clone();
+        let (mut write_half, mut read_half) = duplex(128);
+
+        let copy_task = tokio::spawn(async move {
+            let mut cursor = Cursor::new(payload);
+            copy_stream_with_buffer(&mut cursor, &mut write_half, 4 * 1024).await.unwrap();
+        });
+
+        let mut copied = Vec::new();
+        read_half.read_to_end(&mut copied).await.unwrap();
+        copy_task.await.unwrap();
+
+        assert_eq!(copied, expected);
+    }
+
     #[tokio::test]
     async fn test_remote_disk_disk_id() {
         let url = url::Url::parse("http://remote-server:9000").unwrap();
@@ -1579,6 +1654,30 @@ mod tests {
         assert!(cleared_id.is_none());
     }
 
+    #[tokio::test]
+    async fn test_remote_disk_ref_prefers_disk_id() {
+        let url = url::Url::parse("http://remote-server:9000").unwrap();
+        let endpoint = Endpoint {
+            url,
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+        let disk_option = DiskOption {
+            cleanup: false,
+            health_check: false,
+        };
+
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        assert_eq!(remote_disk.disk_ref().await, endpoint.to_string());
+
+        let disk_id = Uuid::new_v4();
+        remote_disk.set_disk_id(Some(disk_id)).await.unwrap();
+
+        assert_eq!(remote_disk.disk_ref().await, disk_id.to_string());
+    }
+
     #[tokio::test]
     async fn test_remote_disk_endpoints_with_different_schemes() {
         let test_cases = vec![

crates/ecstore/src/rpc/remote_locker.rs

@@ -52,6 +52,7 @@ impl RemoteClient {
             metadata: LockMetadata::default(),
             priority: LockPriority::Normal,
             deadlock_detection: false,
+            suppress_contention_logs: false,
         }
     }
 

crates/ecstore/src/set_disk.rs

@@ -80,9 +80,12 @@ use rustfs_lock::{FastLockGuard, NamespaceLock, NamespaceLockGuard, NamespaceLoc
 use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem};
 use rustfs_rio::{EtagResolvable, HashReader, HashReaderMut, TryGetIndex as _, WarpReader};
 use rustfs_s3_common::EventName;
-use rustfs_utils::http::RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM;
+use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
 use rustfs_utils::http::headers::AMZ_STORAGE_CLASS;
-use rustfs_utils::http::headers::{AMZ_OBJECT_TAGGING, RESERVED_METADATA_PREFIX, RESERVED_METADATA_PREFIX_LOWER};
+use rustfs_utils::http::{
+    SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
+    contains_key_str, get_header_map, get_str, insert_str, remove_header_map,
+};
 use rustfs_utils::{
     HashAlgorithm,
     crypto::hex,
@@ -118,6 +121,21 @@ use uuid::Uuid;
 
 pub const DEFAULT_READ_BUFFER_SIZE: usize = MI_B; // 1 MiB = 1024 * 1024;
 pub const MAX_PARTS_COUNT: usize = 10000;
+
+/// Get the duplex buffer size from environment variable or use default.
+///
+/// This function reads `RUSTFS_DUPLEX_BUFFER_SIZE` environment variable
+/// to allow runtime configuration of the duplex pipe buffer size.
+/// A larger buffer (e.g., 4MB) helps prevent backpressure-related hangs
+/// when reading large objects (20-26MB) under high concurrency.
+///
+/// Default: 4MB (4 * 1024 * 1024 bytes)
+pub fn get_duplex_buffer_size() -> usize {
+    rustfs_utils::get_env_usize(
+        rustfs_config::ENV_OBJECT_DUPLEX_BUFFER_SIZE,
+        rustfs_config::DEFAULT_OBJECT_DUPLEX_BUFFER_SIZE,
+    )
+}
 const DISK_ONLINE_TIMEOUT: Duration = Duration::from_secs(1);
 const DISK_HEALTH_CACHE_TTL: Duration = Duration::from_millis(750);
 
@@ -133,7 +151,96 @@ mod write;
 /// Get lock acquire timeout from environment variable RUSTFS_LOCK_ACQUIRE_TIMEOUT (in seconds)
 /// Defaults to 30 seconds if not set or invalid
 pub fn get_lock_acquire_timeout() -> Duration {
-    Duration::from_secs(rustfs_utils::get_env_u64("RUSTFS_LOCK_ACQUIRE_TIMEOUT", 5))
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT,
+        rustfs_config::DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT,
+    ))
+}
+
+/// Check if lock optimization is enabled.
+/// When enabled, read locks are released after metadata read instead of
+/// being held for the entire data transfer duration.
+pub fn is_lock_optimization_enabled() -> bool {
+    rustfs_utils::get_env_bool(
+        rustfs_config::ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE,
+        rustfs_config::DEFAULT_OBJECT_LOCK_OPTIMIZATION_ENABLE,
+    )
+}
+
+/// Check if deadlock detection is enabled.
+/// When enabled, lock operations are recorded for deadlock analysis.
+pub fn is_deadlock_detection_enabled() -> bool {
+    rustfs_utils::get_env_bool(
+        rustfs_config::ENV_OBJECT_DEADLOCK_DETECTION_ENABLE,
+        rustfs_config::DEFAULT_OBJECT_DEADLOCK_DETECTION_ENABLE,
+    )
+}
+
+/// Record a lock acquisition for deadlock detection.
+/// This records detailed lock information for deadlock analysis.
+/// Returns the lock_id for later release tracking.
+#[inline]
+fn record_lock_acquire(bucket: &str, object: &str, lock_type: &str) -> String {
+    let lock_id = format!("{}:{}", bucket, object);
+
+    if !is_deadlock_detection_enabled() {
+        return lock_id;
+    }
+
+    let request_id = format!("get-{}-{}", bucket, object);
+    let resource = format!("{}/{}", bucket, object);
+
+    // Log with structured fields for analysis
+    debug!(
+        request_id = %request_id,
+        lock_id = %lock_id,
+        lock_type = %lock_type,
+        resource = %resource,
+        "Lock acquired for deadlock tracking"
+    );
+
+    lock_id
+}
+
+/// Record a lock release for deadlock detection.
+#[inline]
+fn record_lock_release(bucket: &str, object: &str, lock_id: &str, lock_type: &str) {
+    if !is_deadlock_detection_enabled() {
+        return;
+    }
+
+    let request_id = format!("get-{}-{}", bucket, object);
+
+    debug!(
+        request_id = %request_id,
+        lock_id = %lock_id,
+        lock_type = %lock_type,
+        "Lock released for deadlock tracking"
+    );
+}
+
+fn build_tiered_decommission_file_info(
+    bucket: &str,
+    object: &str,
+    fi: &FileInfo,
+    disk_count: usize,
+    default_parity_count: usize,
+    storage_class: Option<&str>,
+) -> (FileInfo, usize) {
+    let parity_drives = GLOBAL_STORAGE_CLASS
+        .get()
+        .and_then(|sc| sc.get_parity_for_sc(storage_class.unwrap_or_default()))
+        .unwrap_or(default_parity_count);
+    let data_drives = disk_count - parity_drives;
+    let mut write_quorum = data_drives;
+    if data_drives == parity_drives {
+        write_quorum += 1;
+    }
+
+    let mut updated = fi.clone();
+    updated.erasure = FileInfo::new([bucket, object].join("/").as_str(), data_drives, parity_drives).erasure;
+
+    (updated, write_quorum)
 }
 
 #[derive(Clone, Debug)]
@@ -424,20 +531,44 @@ impl ObjectIO for SetDisks {
         h: HeaderMap,
         opts: &ObjectOptions,
     ) -> Result<GetObjectReader> {
+        // Check if lock optimization is enabled
+        // When enabled, read locks are released after metadata read
+        let lock_optimization_enabled = is_lock_optimization_enabled();
+
         // Acquire a shared read-lock early to protect read consistency
         let read_lock_guard = if !opts.no_lock {
-            Some(
-                self.new_ns_lock(bucket, object)
-                    .await?
-                    .get_read_lock(get_lock_acquire_timeout())
-                    .await
-                    .map_err(|e| {
-                        Error::other(format!(
-                            "Failed to acquire read lock: {}",
-                            self.format_lock_error_from_error(bucket, object, "read", &e)
-                        ))
-                    })?,
-            )
+            let acquire_start = Instant::now();
+
+            // Record lock wait for deadlock detection
+            if is_deadlock_detection_enabled() {
+                debug!(
+                    lock_id = format!("{}:{}", bucket, object),
+                    lock_type = "read",
+                    resource = format!("{}/{}", bucket, object),
+                    "Waiting for read lock"
+                );
+            }
+
+            let guard = self
+                .new_ns_lock(bucket, object)
+                .await?
+                .get_read_lock(get_lock_acquire_timeout())
+                .await
+                .map_err(|e| {
+                    Error::other(format!(
+                        "Failed to acquire read lock: {}",
+                        self.format_lock_error_from_error(bucket, object, "read", &e)
+                    ))
+                })?;
+
+            // Record lock acquisition for deadlock detection
+            let _lock_id = record_lock_acquire(bucket, object, "read");
+
+            // Record lock statistics
+            metrics::counter!("rustfs.lock.acquire.total", "type" => "read").increment(1);
+            metrics::histogram!("rustfs.lock.acquire.duration.seconds").record(acquire_start.elapsed().as_secs_f64());
+
+            Some(guard)
         } else {
             None
         };
@@ -485,7 +616,28 @@ impl ObjectIO for SetDisks {
             return Ok(gr);
         }
 
-        let (rd, wd) = tokio::io::duplex(DEFAULT_READ_BUFFER_SIZE);
+        // Lock optimization: release read lock after metadata read if enabled
+        // This reduces lock contention by not holding the lock during data transfer
+        let read_lock_guard = if lock_optimization_enabled {
+            // Record lock release for deadlock detection
+            if read_lock_guard.is_some() {
+                let lock_id = format!("{}:{}", bucket, object);
+                record_lock_release(bucket, object, &lock_id, "read");
+
+                // Record early lock release statistics
+                metrics::counter!("rustfs.lock.release.early.total", "type" => "read").increment(1);
+            }
+            // Explicitly drop the lock guard to release the lock early
+            drop(read_lock_guard);
+            debug!(bucket, object, "Lock optimization: released read lock after metadata read");
+            None
+        } else {
+            read_lock_guard
+        };
+
+        let duplex_buffer_size = get_duplex_buffer_size();
+        let (rd, wd) = tokio::io::duplex(duplex_buffer_size);
+        debug!(bucket, object, duplex_buffer_size, "Created duplex pipe for object data transfer");
 
         let (reader, offset, length) = GetObjectReader::new(Box::new(rd), range, &object_info, opts, &h)?;
 
@@ -496,9 +648,10 @@ impl ObjectIO for SetDisks {
         let pool_index = self.pool_index;
         let skip_verify = opts.skip_verify_bitrot;
         // Move the read-lock guard into the task so it lives for the duration of the read
+        // Note: when lock optimization is enabled, read_lock_guard is None
         // let _guard_to_hold = _read_lock_guard; // moved into closure below
         tokio::spawn(async move {
-            let _guard = read_lock_guard; // keep guard alive until task ends
+            let _guard = read_lock_guard; // keep guard alive until task ends (None if optimization enabled)
             let mut writer = wd;
             if let Err(e) = Self::get_object_with_fileinfo(
                 &bucket,
@@ -620,7 +773,7 @@ impl ObjectIO for SetDisks {
                     &tmp_object,
                     erasure.shard_file_size(data.size()),
                     erasure.shard_size(),
-                    HashAlgorithm::HighwayHash256,
+                    HashAlgorithm::HighwayHash256S,
                 )
                 .await
                 {
@@ -678,8 +831,8 @@ impl ObjectIO for SetDisks {
             )));
         }
 
-        if user_defined.contains_key(&format!("{RESERVED_METADATA_PREFIX_LOWER}compression")) {
-            user_defined.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression-size"), w_size.to_string());
+        if contains_key_str(&user_defined, SUFFIX_COMPRESSION) {
+            insert_str(&mut user_defined, SUFFIX_COMPRESSION_SIZE, w_size.to_string());
         }
 
         let index_op = data.stream.try_get_index().map(|v| v.clone().into_vec());
@@ -724,7 +877,7 @@ impl ObjectIO for SetDisks {
             pfi.metadata = user_defined.clone();
             if is_inline_buffer {
                 if let Some(writer) = writers[i].take() {
-                    pfi.data = Some(writer.into_inline_data().map(bytes::Bytes::from).unwrap_or_default());
+                    pfi.data = Some(writer.into_inline_data().map(Bytes::from).unwrap_or_default());
                 }
 
                 pfi.set_inline_data();
@@ -765,7 +918,7 @@ impl ObjectIO for SetDisks {
         .await?;
 
         if let Some(old_dir) = op_old_dir {
-            self.commit_rename_data_dir(&shuffle_disks, bucket, object, &old_dir.to_string(), write_quorum)
+            self.commit_rename_data_dir(&online_disks, bucket, object, &old_dir.to_string(), write_quorum)
                 .await?;
         }
 
@@ -1066,7 +1219,7 @@ impl ObjectOperations for SetDisks {
         let mut unique_objects: HashSet<String> = HashSet::new();
         for dobj in &objects {
             if unique_objects.insert(dobj.object_name.clone()) {
-                batch = batch.add_write_lock(rustfs_lock::ObjectKey::new(bucket, dobj.object_name.clone()));
+                batch = batch.add_write_lock(ObjectKey::new(bucket, dobj.object_name.clone()));
             }
         }
 
@@ -1336,6 +1489,12 @@ impl ObjectOperations for SetDisks {
         let mut delete_marker = opts.versioned;
 
         if opts.version_id.is_some() {
+            // Decommission/rebalance may recreate a delete marker on a new pool before that
+            // exact version exists there, so we must still treat it as a mark-delete write.
+            if opts.data_movement && opts.delete_marker && !version_found {
+                mark_delete = true;
+            }
+
             if version_found && opts.delete_marker_replication_status() == ReplicationStatusType::Replica {
                 mark_delete = false;
             }
@@ -1784,19 +1943,54 @@ impl ObjectOperations for SetDisks {
         //}
 
         let mut uploaded_parts: Vec<CompletePart> = vec![];
-        let rs: Option<HTTPRangeSpec> = None;
-        let gr = get_transitioned_object_reader(bucket, object, &rs, &HeaderMap::new(), &oi, opts).await;
-        if let Err(err) = gr {
-            return set_restore_header_fn(&mut oi, Some(StorageError::Io(err))).await;
-        }
-        let gr = gr.unwrap();
-
-        for part_info in &oi.parts {
-            let reader = BufReader::new(Cursor::new(vec![] /*gr.stream*/));
+        let parts = oi.parts.clone();
+        let mut part_offset: i64 = 0;
+        for part_info in &parts {
+            let mut part_opts = opts.clone();
+            part_opts.part_number = Some(part_info.number);
+            if part_info.actual_size <= 0 {
+                return set_restore_header_fn(
+                    &mut oi,
+                    Some(Error::other(format!("invalid multipart restore part size {}", part_info.actual_size))),
+                )
+                .await;
+            }
+            let part_end = match part_offset.checked_add(part_info.actual_size - 1) {
+                Some(end) => end,
+                None => {
+                    return set_restore_header_fn(
+                        &mut oi,
+                        Some(Error::other("multipart restore part range overflow".to_string())),
+                    )
+                    .await;
+                }
+            };
+            let rs = Some(HTTPRangeSpec {
+                is_suffix_length: false,
+                start: part_offset,
+                end: part_end,
+            });
+            part_offset = match part_end.checked_add(1) {
+                Some(next) => next,
+                None => {
+                    return set_restore_header_fn(
+                        &mut oi,
+                        Some(Error::other("multipart restore part offset overflow".to_string())),
+                    )
+                    .await;
+                }
+            };
+            let gr = match get_transitioned_object_reader(bucket, object, &rs, &HeaderMap::new(), &oi, &part_opts).await {
+                Ok(reader) => reader,
+                Err(err) => {
+                    return set_restore_header_fn(&mut oi, Some(StorageError::Io(err))).await;
+                }
+            };
+            let reader = BufReader::new(gr.stream);
             let hash_reader = HashReader::new(
                 Box::new(WarpReader::new(reader)),
-                part_info.size as i64,
-                part_info.size as i64,
+                part_info.actual_size,
+                part_info.actual_size,
                 None,
                 None,
                 false,
@@ -1809,7 +2003,7 @@ impl ObjectOperations for SetDisks {
             //if let Err(err) = p_info {
             //    return set_restore_header_fn(&mut oi, err).await;
             //}
-            if p_info.size != part_info.size {
+            if p_info.size as i64 != part_info.actual_size {
                 return set_restore_header_fn(
                     &mut oi,
                     Some(Error::other(ObjectApiError::InvalidObjectState(GenericError {
@@ -1889,6 +2083,68 @@ impl ObjectOperations for SetDisks {
     }
 }
 
+impl SetDisks {
+    #[tracing::instrument(skip(self, fi, opts))]
+    pub(crate) async fn decommission_tiered_object(
+        &self,
+        bucket: &str,
+        object: &str,
+        fi: &FileInfo,
+        opts: &ObjectOptions,
+    ) -> Result<()> {
+        let _lock_guard = if !opts.no_lock {
+            Some(
+                self.new_ns_lock(bucket, object)
+                    .await?
+                    .get_write_lock(get_lock_acquire_timeout())
+                    .await
+                    .map_err(|e| {
+                        Error::other(format!(
+                            "Failed to acquire write lock: {}",
+                            self.format_lock_error_from_error(bucket, object, "write", &e)
+                        ))
+                    })?,
+            )
+        } else {
+            None
+        };
+
+        let disks = self.disks.read().await.clone();
+        let storage_class = opts.user_defined.get(AMZ_STORAGE_CLASS).map(String::as_str);
+        let (fi, write_quorum) =
+            build_tiered_decommission_file_info(bucket, object, fi, disks.len(), self.default_parity_count, storage_class);
+        let parts_metadata = vec![fi.clone(); disks.len()];
+        let (shuffle_disks, parts_metadata) = Self::shuffle_disks_and_parts_metadata(&disks, &parts_metadata, &fi);
+
+        let mut errs = Vec::with_capacity(shuffle_disks.len());
+        let mut futures = Vec::with_capacity(shuffle_disks.len());
+        for (index, disk) in shuffle_disks.iter().enumerate() {
+            let mut file_info = parts_metadata[index].clone();
+            file_info.erasure.index = index + 1;
+            futures.push(async move {
+                if let Some(disk) = disk {
+                    disk.write_metadata("", bucket, object, file_info).await
+                } else {
+                    Err(DiskError::DiskNotFound)
+                }
+            });
+        }
+
+        for result in join_all(futures).await {
+            match result {
+                Ok(_) => errs.push(None),
+                Err(err) => errs.push(Some(err)),
+            }
+        }
+
+        if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, write_quorum) {
+            return Err(to_object_err(err.into(), vec![bucket, object]));
+        }
+
+        Ok(())
+    }
+}
+
 #[async_trait::async_trait]
 impl ListOperations for SetDisks {
     #[tracing::instrument(skip(self))]
@@ -2007,7 +2263,7 @@ impl MultipartOperations for SetDisks {
                     &tmp_part_path,
                     erasure.shard_file_size(data.size()),
                     erasure.shard_size(),
-                    HashAlgorithm::HighwayHash256,
+                    HashAlgorithm::HighwayHash256S,
                 )
                 .await
                 {
@@ -2454,11 +2710,11 @@ impl MultipartOperations for SetDisks {
 
         fi.data_dir = Some(Uuid::new_v4());
 
-        if let Some(cssum) = user_defined.get(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM)
+        if let Some(cssum) = get_header_map(&user_defined, SUFFIX_REPLICATION_SSEC_CRC)
             && !cssum.is_empty()
         {
-            fi.checksum = base64_simd::STANDARD.decode_to_vec(cssum).ok().map(Bytes::from);
-            user_defined.remove(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM);
+            fi.checksum = base64_simd::STANDARD.decode_to_vec(&cssum).ok().map(Bytes::from);
+            remove_header_map(&mut user_defined, SUFFIX_REPLICATION_SSEC_CRC);
         }
 
         let parts_metadata = vec![fi.clone(); disks.len()];
@@ -2662,8 +2918,7 @@ impl MultipartOperations for SetDisks {
         // Build a lookup map for O(1) part resolution instead of O(n) find() in the loop
         // This optimizes from O(n^2) to O(n) when processing many parts
         use std::collections::HashMap;
-        let part_lookup: HashMap<usize, &rustfs_filemeta::ObjectPartInfo> =
-            curr_fi.parts.iter().map(|part| (part.number, part)).collect();
+        let part_lookup: HashMap<usize, &ObjectPartInfo> = curr_fi.parts.iter().map(|part| (part.number, part)).collect();
 
         for (i, p) in uploaded_parts.iter().enumerate() {
             let Some(ext_part) = part_lookup.get(&p.part_num) else {
@@ -2809,8 +3064,8 @@ impl MultipartOperations for SetDisks {
             }
         }
 
-        if let Some(rc_crc) = opts.user_defined.get(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM) {
-            if let Ok(rc_crc_bytes) = base64_simd::STANDARD.decode_to_vec(rc_crc) {
+        if let Some(rc_crc) = get_header_map(&opts.user_defined, SUFFIX_REPLICATION_SSEC_CRC) {
+            if let Ok(rc_crc_bytes) = base64_simd::STANDARD.decode_to_vec(&rc_crc) {
                 fi.checksum = Some(Bytes::from(rc_crc_bytes));
             } else {
                 error!("complete_multipart_upload decode rc_crc failed rc_crc={}", rc_crc);
@@ -2849,25 +3104,19 @@ impl MultipartOperations for SetDisks {
         fi.metadata.insert("etag".to_owned(), etag);
 
         if opts.replication_request {
-            if let Some(actual_size) = opts
-                .user_defined
-                .get(format!("{RESERVED_METADATA_PREFIX_LOWER}Actual-Object-Size").as_str())
-            {
+            if let Some(actual_size) = get_str(&opts.user_defined, SUFFIX_ACTUAL_OBJECT_SIZE_CAP) {
+                insert_str(&mut fi.metadata, SUFFIX_ACTUAL_SIZE, actual_size.clone());
                 fi.metadata
-                    .insert(format!("{RESERVED_METADATA_PREFIX}actual-size"), actual_size.clone());
-                fi.metadata
-                    .insert("x-rustfs-encryption-original-size".to_string(), actual_size.to_string());
+                    .insert("x-rustfs-encryption-original-size".to_string(), actual_size);
             }
         } else {
-            fi.metadata
-                .insert(format!("{RESERVED_METADATA_PREFIX}actual-size"), object_actual_size.to_string());
+            insert_str(&mut fi.metadata, SUFFIX_ACTUAL_SIZE, object_actual_size.to_string());
             fi.metadata
                 .insert("x-rustfs-encryption-original-size".to_string(), object_actual_size.to_string());
         }
 
         if fi.is_compressed() {
-            fi.metadata
-                .insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression-size"), object_size.to_string());
+            insert_str(&mut fi.metadata, SUFFIX_COMPRESSION_SIZE, object_size.to_string());
         }
 
         if opts.data_movement {
@@ -2927,7 +3176,7 @@ impl MultipartOperations for SetDisks {
         .await?;
 
         if let Some(old_dir) = op_old_dir {
-            self.commit_rename_data_dir(&shuffle_disks, bucket, object, &old_dir.to_string(), write_quorum)
+            self.commit_rename_data_dir(&online_disks, bucket, object, &old_dir.to_string(), write_quorum)
                 .await?;
         }
 
@@ -3356,12 +3605,17 @@ async fn disks_with_all_parts(
         if (meta.data.is_some() || meta.size == 0) && !meta.parts.is_empty() {
             if let Some(data) = &meta.data {
                 let checksum_info = meta.erasure.get_checksum_info(meta.parts[0].number);
+                let checksum_algo = if meta.uses_legacy_checksum && checksum_info.algorithm == HashAlgorithm::HighwayHash256S {
+                    HashAlgorithm::HighwayHash256SLegacy
+                } else {
+                    checksum_info.algorithm
+                };
                 let data_len = data.len();
                 let verify_err = bitrot_verify(
                     Box::new(Cursor::new(data.clone())),
                     data_len,
                     meta.erasure.shard_file_size(meta.size) as usize,
-                    checksum_info.algorithm,
+                    checksum_algo,
                     checksum_info.hash,
                     meta.erasure.shard_size(),
                 )
@@ -4161,6 +4415,33 @@ mod tests {
         assert!(e_tag_matches("\"abc\"", "*"));
     }
 
+    #[test]
+    fn test_build_tiered_decommission_file_info_preserves_transition_metadata() {
+        let version_id = Uuid::new_v4();
+        let transition_version_id = Uuid::new_v4();
+        let original = FileInfo {
+            version_id: Some(version_id),
+            transition_status: TRANSITION_COMPLETE.to_string(),
+            transitioned_objname: "remote/object".to_string(),
+            transition_tier: "WARM-TIER".to_string(),
+            transition_version_id: Some(transition_version_id),
+            erasure: FileInfo::new("old-bucket/old-object", 8, 8).erasure,
+            ..Default::default()
+        };
+
+        let (updated, write_quorum) = build_tiered_decommission_file_info("bucket", "object", &original, 16, 4, None);
+
+        assert_eq!(updated.version_id, original.version_id);
+        assert_eq!(updated.transition_status, original.transition_status);
+        assert_eq!(updated.transitioned_objname, original.transitioned_objname);
+        assert_eq!(updated.transition_tier, original.transition_tier);
+        assert_eq!(updated.transition_version_id, original.transition_version_id);
+        assert_eq!(updated.erasure.data_blocks, 12);
+        assert_eq!(updated.erasure.parity_blocks, 4);
+        assert_eq!(write_quorum, 12);
+        assert_ne!(updated.erasure.distribution, original.erasure.distribution);
+    }
+
     #[test]
     fn test_should_prevent_write() {
         let oi = ObjectInfo {

crates/ecstore/src/set_disk/heal.rs

@@ -124,11 +124,12 @@ impl SetDisks {
                         );
 
                         let erasure = if !latest_meta.deleted && !latest_meta.is_remote() {
-                            // Initialize erasure coding
-                            erasure_coding::Erasure::new(
+                            // Initialize erasure coding; use legacy mode for old-version files
+                            erasure_coding::Erasure::new_with_options(
                                 latest_meta.erasure.data_blocks,
                                 latest_meta.erasure.parity_blocks,
                                 latest_meta.erasure.block_size,
+                                latest_meta.uses_legacy_checksum,
                             )
                         } else {
                             erasure_coding::Erasure::default()
@@ -347,7 +348,14 @@ impl SetDisks {
 
                             for (part_index, part) in latest_meta.parts.iter().enumerate() {
                                 let till_offset = erasure.shard_file_offset(0, part.size, part.size);
-                                let checksum_algo = erasure_info.get_checksum_info(part.number).algorithm;
+                                let checksum_info = erasure_info.get_checksum_info(part.number);
+                                let checksum_algo = if latest_meta.uses_legacy_checksum
+                                    && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S
+                                {
+                                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
+                                } else {
+                                    checksum_info.algorithm
+                                };
                                 let mut readers = Vec::with_capacity(latest_disks.len());
                                 let mut writers = Vec::with_capacity(out_dated_disks.len());
                                 // let mut errors = Vec::with_capacity(out_dated_disks.len());
@@ -420,7 +428,7 @@ impl SetDisks {
                                             ]),
                                             erasure.shard_file_size(part.size as i64),
                                             erasure.shard_size(),
-                                            HashAlgorithm::HighwayHash256,
+                                            HashAlgorithm::HighwayHash256S,
                                         )
                                         .await
                                         {

crates/ecstore/src/set_disk/read.rs

@@ -603,7 +603,12 @@ impl SetDisks {
             object, offset, length, end_offset, part_index, last_part_index, last_part_relative_offset, "Multipart read bounds"
         );
 
-        let erasure = erasure_coding::Erasure::new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size);
+        let erasure = erasure_coding::Erasure::new_with_options(
+            fi.erasure.data_blocks,
+            fi.erasure.parity_blocks,
+            fi.erasure.block_size,
+            fi.uses_legacy_checksum,
+        );
 
         let part_indices: Vec<usize> = (part_index..=last_part_index).collect();
         debug!(bucket, object, ?part_indices, "Multipart part indices to stream");
@@ -648,6 +653,14 @@ impl SetDisks {
                 "Streaming multipart part"
             );
 
+            let checksum_info = fi.erasure.get_checksum_info(part_number);
+            let checksum_algo =
+                if fi.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
+                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
+                } else {
+                    checksum_info.algorithm
+                };
+
             let mut readers = Vec::with_capacity(disks.len());
             let mut errors = Vec::with_capacity(disks.len());
             for (idx, disk_op) in disks.iter().enumerate() {
@@ -659,7 +672,7 @@ impl SetDisks {
                     read_offset,
                     till_offset,
                     erasure.shard_size(),
-                    HashAlgorithm::HighwayHash256,
+                    checksum_algo.clone(),
                     skip_verify_bitrot,
                 )
                 .await

crates/ecstore/src/store.rs

@@ -27,7 +27,7 @@ use crate::bucket::utils::check_new_multipart_args;
 use crate::bucket::utils::check_object_args;
 use crate::bucket::utils::check_put_object_args;
 use crate::bucket::utils::check_put_object_part_args;
-use crate::bucket::utils::{check_valid_bucket_name, check_valid_bucket_name_strict, is_meta_bucketname};
+use crate::bucket::utils::{check_valid_bucket_name, check_valid_bucket_name_strict};
 use crate::config::GLOBAL_STORAGE_CLASS;
 use crate::config::storageclass;
 use crate::disk::endpoint::{Endpoint, EndpointType};
@@ -142,8 +142,8 @@ mod rebalance;
 
 use peer::init_local_peer;
 pub use peer::{
-    all_local_disk, all_local_disk_path, find_local_disk, get_disk_infos, get_disk_via_endpoint, has_space_for, init_local_disks,
-    init_lock_clients,
+    all_local_disk, all_local_disk_path, find_local_disk, find_local_disk_by_ref, get_disk_infos, get_disk_via_endpoint,
+    has_space_for, init_local_disks, init_lock_clients,
 };
 
 #[derive(Debug)]
@@ -156,7 +156,7 @@ pub struct ECStore {
     // pub local_disks: Vec<DiskStore>,
     pub pool_meta: RwLock<PoolMeta>,
     pub rebalance_meta: RwLock<Option<RebalanceMeta>>,
-    pub decommission_cancelers: Vec<Option<usize>>,
+    pub decommission_cancelers: RwLock<Vec<Option<CancellationToken>>>,
 }
 
 // impl Clone for ECStore {

crates/ecstore/src/store/bucket.rs

@@ -13,6 +13,11 @@
 // limitations under the License.
 
 use super::*;
+use crate::bucket::utils::is_meta_bucketname;
+
+fn should_override_created_from_metadata(created: OffsetDateTime) -> bool {
+    created != OffsetDateTime::UNIX_EPOCH
+}
 
 impl ECStore {
     #[instrument(skip(self))]
@@ -68,7 +73,9 @@ impl ECStore {
         let mut info = self.peer_sys.get_bucket_info(bucket, opts).await?;
 
         if let Ok(sys) = metadata_sys::get(bucket).await {
-            info.created = Some(sys.created);
+            if should_override_created_from_metadata(sys.created) {
+                info.created = Some(sys.created);
+            }
             info.versioning = sys.versioning();
             info.object_locking = sys.object_locking();
         }
@@ -84,7 +91,9 @@ impl ECStore {
 
         if !opts.no_metadata {
             for bucket in buckets.iter_mut() {
-                if let Ok(created) = metadata_sys::created_at(&bucket.name).await {
+                if let Ok(created) = metadata_sys::created_at(&bucket.name).await
+                    && should_override_created_from_metadata(created)
+                {
                     bucket.created = Some(created);
                 }
             }
@@ -148,3 +157,20 @@ impl ECStore {
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::should_override_created_from_metadata;
+    use time::OffsetDateTime;
+
+    #[test]
+    fn should_not_override_when_metadata_created_is_unix_epoch() {
+        assert!(!should_override_created_from_metadata(OffsetDateTime::UNIX_EPOCH));
+    }
+
+    #[test]
+    fn should_override_when_metadata_created_is_valid_time() {
+        let created = OffsetDateTime::from_unix_timestamp(1704067200).expect("valid timestamp");
+        assert!(should_override_created_from_metadata(created));
+    }
+}

crates/ecstore/src/store/init.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use super::*;
+use crate::global::is_first_cluster_node_local;
 
 impl ECStore {
     #[allow(clippy::new_ret_no_self)]
@@ -149,7 +150,7 @@ impl ECStore {
         let mut pool_meta = PoolMeta::new(&pools, &PoolMeta::default());
         pool_meta.dont_save = true;
 
-        let decommission_cancelers = vec![None; pools.len()];
+        let decommission_cancelers = RwLock::new(vec![None; pools.len()]);
         let ec = Arc::new(ECStore {
             id: deployment_id.unwrap(),
             disk_map,
@@ -203,6 +204,7 @@ impl ECStore {
         let mut meta = PoolMeta::default();
         meta.load(self.pools[0].clone(), self.pools.clone()).await?;
         let update = meta.validate(self.pools.clone())?;
+        let should_persist_pool_meta = is_first_cluster_node_local().await;
 
         if !update {
             {
@@ -211,7 +213,9 @@ impl ECStore {
             }
         } else {
             let new_meta = PoolMeta::new(&self.pools, &meta);
-            new_meta.save(self.pools.clone()).await?;
+            if should_persist_pool_meta {
+                new_meta.save(self.pools.clone()).await?;
+            }
             {
                 let mut pool_meta = self.pool_meta.write().await;
                 *pool_meta = new_meta;
@@ -265,6 +269,7 @@ impl ECStore {
         init_background_expiry(self.clone()).await;
 
         TransitionState::init(self.clone()).await;
+        crate::tier::tier::try_migrate_tiering_config(self.clone()).await;
 
         if let Err(err) = GLOBAL_TierConfigMgr.write().await.init(self.clone()).await {
             info!("TierConfigMgr init error: {}", err);

crates/ecstore/src/store/object.rs

@@ -14,7 +14,64 @@
 
 use super::*;
 
+fn select_data_movement_target_pool(
+    existing_pool_idx: Result<usize>,
+    src_pool_idx: usize,
+    delete_marker: bool,
+) -> Result<Option<usize>> {
+    match existing_pool_idx {
+        Ok(pool_idx) => {
+            if delete_marker && pool_idx == src_pool_idx {
+                Ok(None)
+            } else {
+                Ok(Some(pool_idx))
+            }
+        }
+        Err(err) => {
+            if is_err_read_quorum(&err) {
+                return Err(StorageError::ErasureWriteQuorum);
+            }
+            if delete_marker && (is_err_object_not_found(&err) || is_err_version_not_found(&err)) {
+                Ok(None)
+            } else {
+                Err(err)
+            }
+        }
+    }
+}
+
 impl ECStore {
+    #[instrument(skip(self, fi, opts))]
+    pub(crate) async fn decommission_tiered_object(
+        &self,
+        bucket: &str,
+        object: &str,
+        fi: &rustfs_filemeta::FileInfo,
+        opts: &ObjectOptions,
+    ) -> Result<()> {
+        check_put_object_args(bucket, object)?;
+
+        let object = encode_dir_object(object);
+
+        if self.single_pool() {
+            return Err(Error::other(format!("error decommissioning {bucket}/{object}")));
+        }
+
+        let idx = self.get_pool_idx_no_lock(bucket, &object, fi.size).await?;
+        if opts.data_movement && idx == opts.src_pool_idx {
+            return Err(StorageError::DataMovementOverwriteErr(
+                bucket.to_owned(),
+                object.to_owned(),
+                opts.version_id.clone().unwrap_or_default(),
+            ));
+        }
+
+        self.pools[idx]
+            .get_disks_by_key(&object)
+            .decommission_tiered_object(bucket, &object, fi, opts)
+            .await
+    }
+
     #[instrument(level = "debug", skip(self))]
     pub(super) async fn handle_get_object_reader(
         &self,
@@ -179,6 +236,30 @@ impl ECStore {
         let mut gopts = opts.clone();
         gopts.no_lock = true;
 
+        if opts.data_movement {
+            let existing_pool_idx = self
+                .get_pool_info_existing_with_opts(bucket, object, &gopts)
+                .await
+                .map(|(pinfo, _)| pinfo.index);
+            let target_pool_idx =
+                match select_data_movement_target_pool(existing_pool_idx, opts.src_pool_idx, opts.delete_marker)? {
+                    Some(pool_idx) => pool_idx,
+                    None => self.get_pool_idx_no_lock(bucket, object, 0).await?,
+                };
+
+            if opts.src_pool_idx == target_pool_idx {
+                return Err(StorageError::DataMovementOverwriteErr(
+                    bucket.to_owned(),
+                    object.to_owned(),
+                    opts.version_id.unwrap_or_default(),
+                ));
+            }
+
+            let mut obj = self.pools[target_pool_idx].delete_object(bucket, object, opts).await?;
+            obj.name = decode_dir_object(obj.name.as_str());
+            return Ok(obj);
+        }
+
         // Determine which pool contains it
         let (mut pinfo, errs) = self
             .get_pool_info_existing_with_opts(bucket, object, &gopts)
@@ -204,12 +285,6 @@ impl ECStore {
             ));
         }
 
-        if opts.data_movement {
-            let mut obj = self.pools[pinfo.index].delete_object(bucket, object, opts).await?;
-            obj.name = decode_dir_object(obj.name.as_str());
-            return Ok(obj);
-        }
-
         if !errs.is_empty() && !opts.versioned && !opts.version_suspended {
             return self.delete_object_from_all_pools(bucket, object, &opts, errs).await;
         }
@@ -565,3 +640,27 @@ impl ECStore {
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn delete_marker_data_movement_falls_back_when_only_source_pool_has_object() {
+        let target = select_data_movement_target_pool(Ok(1), 1, true).unwrap();
+        assert_eq!(target, None);
+    }
+
+    #[test]
+    fn delete_marker_data_movement_falls_back_when_version_does_not_exist_yet() {
+        let err = StorageError::ObjectNotFound("bucket".to_string(), "object".to_string());
+        let target = select_data_movement_target_pool(Err(err), 1, true).unwrap();
+        assert_eq!(target, None);
+    }
+
+    #[test]
+    fn non_delete_marker_data_movement_keeps_existing_pool() {
+        let target = select_data_movement_target_pool(Ok(0), 1, false).unwrap();
+        assert_eq!(target, Some(0));
+    }
+}

crates/ecstore/src/store/peer.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use super::*;
+use crate::global::GLOBAL_LOCAL_DISK_ID_MAP;
 
 pub async fn find_local_disk(disk_path: &String) -> Option<DiskStore> {
     let disk_map = GLOBAL_LOCAL_DISK_MAP.read().await;
@@ -24,6 +25,19 @@ pub async fn find_local_disk(disk_path: &String) -> Option<DiskStore> {
     }
 }
 
+pub async fn find_local_disk_by_ref(disk_ref: &str) -> Option<DiskStore> {
+    if let Some(disk) = find_local_disk(&disk_ref.to_string()).await {
+        return Some(disk);
+    }
+
+    let Ok(disk_id) = Uuid::parse_str(disk_ref) else {
+        return None;
+    };
+
+    let disk_path = GLOBAL_LOCAL_DISK_ID_MAP.read().await.get(&disk_id).cloned()?;
+    find_local_disk(&disk_path).await
+}
+
 pub async fn get_disk_via_endpoint(endpoint: &Endpoint) -> Option<DiskStore> {
     let global_set_drives = GLOBAL_LOCAL_DISK_SET_DRIVES.read().await;
     if global_set_drives.is_empty() {

crates/ecstore/src/store/rebalance.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use super::*;
+use crate::bucket::utils::is_meta_bucketname;
 
 impl ECStore {
     #[instrument(level = "debug", skip(self))]

crates/ecstore/src/store_api.rs

@@ -27,8 +27,8 @@ use bytes::Bytes;
 use http::{HeaderMap, HeaderValue};
 use rustfs_common::heal_channel::HealOpts;
 use rustfs_filemeta::{
-    FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, REPLICATION_RESET, REPLICATION_STATUS, ReplicateDecision, ReplicationState,
-    ReplicationStatusType, RestoreStatusOps as _, VersionPurgeStatusType, parse_restore_obj_status, replication_statuses_map,
+    FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, ReplicateDecision, ReplicationState, ReplicationStatusType,
+    RestoreStatusOps as _, VersionPurgeStatusType, parse_restore_obj_status, replication_statuses_map,
     version_purge_statuses_map,
 };
 use rustfs_lock::NamespaceLockWrapper;
@@ -36,7 +36,7 @@ use rustfs_madmin::heal_commands::HealResultItem;
 use rustfs_rio::Checksum;
 use rustfs_rio::{DecompressReader, HashReader, LimitReader, WarpReader};
 use rustfs_utils::CompressionAlgorithm;
-use rustfs_utils::http::headers::{AMZ_OBJECT_TAGGING, RESERVED_METADATA_PREFIX_LOWER};
+use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
 use rustfs_utils::http::{AMZ_BUCKET_REPLICATION_STATUS, AMZ_RESTORE, AMZ_STORAGE_CLASS};
 use rustfs_utils::path::decode_dir_object;
 use serde::{Deserialize, Serialize};