docs: remove agent-generated planning docs and forbid committing them (#4771)

docs: remove agent-generated planning docs, forbid committing them Delete one-shot planning/progress artifacts that were checked into the tree: the 14 superpowers plan/tracker docs under docs/superpowers/plans/, plus issue-scoped implementation plans, optimization conclusions, and dated benchmark-result snapshots under docs/ (issue-4003 ListObjectsV2 plans, get-small-file conclusion, issue824/issue829 benchmark results, issue-713 >1GiB GET baseline summary and ops guide). Codify the rule so they do not come back: - .gitignore drops the docs/superpowers whitelist, so anything new under docs/ stays ignored unless force-added. - AGENTS.md gains an explicit 'do not commit planning-type documents' rule scoping version control to the durable architecture/operations/testing sets. - docs/architecture/README.md, overview.md, arch-checks SKILL.md, and check_doc_paths.sh drop their references to the removed archive.
2026-07-26 08:18:18 +00:00 · 2026-07-12 14:14:15 +08:00
parent b235762fdb
commit c4c198670d
27 changed files with 21 additions and 16103 deletions
@@ -47,5 +47,4 @@ consider adding it to the script's `checked_files` list.
 Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
 `docs/architecture/*.md`) must not reference repo file paths that no longer
 exist. If your refactor moved code, update the docs that point at it — the
-error message lists `doc -> stale-path` pairs. Historical plans under
+error message lists `doc -> stale-path` pairs.
 `docs/superpowers/plans/` are exempt.
@@ -57,10 +57,6 @@ docs/*
 !docs/operations/**
 !docs/testing/
 !docs/testing/**
 !docs/superpowers/
 !docs/superpowers/**
 !docs/testing/
 !docs/testing/**
 docs/heal-scanner-logging-governance.md
 docs/benchmark/rustfs-target-bench/
 docs/benchmark/*.md
@@ -59,7 +59,6 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - High-level architecture and crate map: `ARCHITECTURE.md`
 - Migration guardrails, readiness contracts, support matrices:
  `docs/architecture/README.md` (routes by audience)
 - Historical implementation plans and trackers: `docs/superpowers/plans/`
 - Shared agent skills (all tools): `.agents/skills/` — each `SKILL.md` carries
  a frontmatter `description` stating when it applies. Scan the descriptions
  before starting a task and follow any skill that matches, even if your tool
@@ -71,6 +70,17 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 Avoid duplicating long crate lists or command matrices in instruction files.
 Reference the source files above instead.
 Do not commit planning-type documents — one-shot implementation/optimization
 plans, task trackers, migration-progress ledgers, phase/PR templates,
 issue-scoped benchmark-result snapshots or optimization conclusions, or
 agent-generated working notes (e.g. anything a `superpowers`/scratch workflow
 produces). Keep that work in the issue tracker or your local worktree, not in
 the repository. Only durable reference — the architecture set under
 `docs/architecture/`, repeatable operational runbooks under `docs/operations/`,
 and the test-suite references under `docs/testing/` — belongs in version
 control; `.gitignore` ignores everything else under `docs/` by default, so a new
 plan file will not be tracked unless someone force-adds it — don't.
 ## Verification Before PR
 Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
@@ -6,8 +6,9 @@ contracts, boundary rules, and support matrices.
 Two rules keep this directory healthy:
 1. **Durable reference only.** One-shot implementation plans, task trackers,
-   and PR templates live in [`docs/superpowers/plans/`](../superpowers/plans/)
+   and PR templates do not belong in the repository — keep them in the issue
-   and are archived there when their work closes.
+   tracker or your local worktree. When their work closes, delete them rather
   than archiving them here.
 2. **No copies of other sources of truth.** Crate lists come from
   `Cargo.toml`, CI steps from `.github/workflows/ci.yml`, code structure from
   the code. `scripts/check_doc_paths.sh` fails the pre-commit gate when a
@@ -54,9 +55,8 @@ Two rules keep this directory healthy:
 - [compat-cleanup-register.md](compat-cleanup-register.md)
 Historical plans and trackers (rebalance/decommission phases,
-migration-progress ledger) were moved to
+migration-progress ledger, and the one-shot migration snapshots that fed it —
-[`docs/superpowers/plans/`](../superpowers/plans/) in 2026-07. One-shot
+startup timeline, scheduler baseline, profiling/NUMA capability inventory, KMS
-migration snapshots whose only consumer was that ledger (startup timeline,
+development defaults inventory) were retired in 2026-07 once the
-scheduler baseline, profiling/NUMA capability inventory, KMS development
+architecture-review ledger they served closed out (backlog#660/#665). Planning
-defaults inventory) followed once the architecture-review ledger they fed
+documents are no longer kept in the repository.
 closed out (backlog#660/#665).
@@ -50,8 +50,6 @@ hot-path behavior must not drift during this migration.
 - [`config-model-boundary-adr.md`](config-model-boundary-adr.md): target crate,
  module path, dependency rules, and verification gates for moving the pure
  server-config model.
 - [`migration-progress.md`](../superpowers/plans/migration-progress.md): historical
  task ledger for the closed migration (execution tracking moved to issue #665).
 - [`compat-cleanup-register.md`](compat-cleanup-register.md): temporary
  compatibility code that must be removed later.
@@ -1,86 +0,0 @@
 # Issue 824 GET Pressure Validation Results
 Date: 2026-07-05
 ## Scope
 This run validates the current branch `houseme/get-object-performance-hardening`
 against the closest local historical GET metrics gate baseline.
 Historical baseline:
 - Path: `docs/benchmark/rustfs-target-bench/get-metrics-gate-baseline/warp/median_summary.csv`
 - Git head: `92c50156a3cbbdfa454be1d14062098708851f51`
 - Parameters: `10MiB`, warp, concurrency `32`, duration `10s`, rounds `3`
 - Median throughput: `5734882344.960000` B/s
 - Median req/s: `546.920000`
 - Median latency: `48.700000` ms
 Current run:
 - Path: `target/bench/issue824-get-metrics-gate-current-20260705`
 - Git head recorded by script: `55ad8df1c2f574178669052093aa185fae8185ca`
 - Branch/worktree contained staged and unstaged issue-824 changes.
 - Parameters: `10MiB`, warp, concurrency `32`, duration `10s`, rounds `3`
 - RustFS binary: `target/release/rustfs`
 - Observability export: disabled
 - Scanner: disabled
 - Command:
 ```bash
 scripts/run_get_metrics_gate_smoke.sh \
  --skip-build \
  --rustfs-bin target/release/rustfs \
  --size 10MiB \
  --concurrency 32 \
  --duration 10s \
  --rounds 3 \
  --round-cooldown-secs 1 \
  --baseline-csv docs/benchmark/rustfs-target-bench/get-metrics-gate-baseline/warp/median_summary.csv \
  --out-dir target/bench/issue824-get-metrics-gate-current-20260705 \
  --data-root /private/tmp/issue824-get-metrics-gate-current-20260705
 ```
 ## Raw Rounds
 | round | throughput | req/s | avg latency | p90 | p99 |
 | --- | ---: | ---: | ---: | ---: | ---: |
 | 1 | 6505.75 MiB/s | 650.57 | 47.9 ms | 73.0 ms | 103.9 ms |
 | 2 | 5631.35 MiB/s | 563.13 | 52.1 ms | 80.1 ms | 106.3 ms |
 | 3 | 5830.30 MiB/s | 583.03 | 52.9 ms | 72.2 ms | 92.7 ms |
 Corrected median from raw round data:
 - Median throughput: `6113512652.800000` B/s (`5830.30 MiB/s`)
 - Median req/s: `583.030000`
 - Median latency: `52.100000` ms
 - Median p90: `73.000000` ms
 - Median p99: `103.900000` ms
 ## Baseline Comparison
 | metric | baseline | current | delta |
 | --- | ---: | ---: | ---: |
 | throughput B/s | 5734882344.960000 | 6113512652.800000 | +6.60% |
 | req/s | 546.920000 | 583.030000 | +6.60% |
 | avg latency ms | 48.700000 | 52.100000 | +6.98% |
 Interpretation:
 - No throughput regression was observed in this local release-binary smoke.
 - Latency moved higher by 6.98% against the selected baseline. This is not a
  blocking regression by itself because host state, cooldown, and branch state
  differ, but it should be watched in the PR benchmark matrix.
 - The run validates that the current changes do not obviously collapse the
  default 10MiB/C32 GET hot path.
 ## Artifact Caveat
 The script completed with exit code `0`, but its generated
 `warp/median_summary.csv` and `warp/baseline_compare.csv` contain `N/A` values.
 The terminal summary and `warp/round_results.csv` preserved the actual round
 values. The corrected medians above are computed from those raw round logs.
 Follow-up: fix `scripts/run_object_batch_bench_enhanced.sh` CSV parsing/report
 generation so the persisted median and baseline comparison match the terminal
 summary.
@@ -1,283 +0,0 @@
 # Issue 829 Default GET Hot Path Benchmark and Attribution Runbook
 Date: 2026-07-05
 ## Scope
 This runbook defines the evidence package for proving the current default GET
 hot path. It intentionally separates throughput evidence from path attribution:
 - Metrics-off runs are throughput evidence.
 - Metrics-on diagnostic runs are path attribution evidence.
 - Codec and direct-memory experiments are A/B evidence, not default-path proof
  unless their enabling environment is recorded.
 The current review baseline is:
 - Ordinary non-inline GETs are expected to use ECStore metadata/cache lookup and
  the legacy duplex reader path.
 - Metadata early-stop is not expected for normal `read_data` GETs.
 - Direct-memory, codec streaming, and metadata early-stop are opt-in or gated.
 - Diagnostic metrics add overhead and must not be used as pure throughput proof.
 - Metadata cache is topology-sensitive: distributed erasure currently bypasses
  the metadata GET cache, so distributed baselines should assume full metadata
  fanout unless a later change records otherwise.
 - Rollout percentage is not the primary safety gate for codec streaming or
  metadata early-stop in the current defaults; record the base enable flag and
  compatibility switches before interpreting any percentage value.
 - Codec-streaming cost accounting must include both shard-to-output-buffer and
  output-buffer-to-client copies, plus the `+ Sync` reader wrapper lock cost.
 ## Artifact Contract
 Every run directory must keep enough information to reproduce or reject the
 result:
 - `git_head` or `git rev-parse HEAD`
 - branch name and dirty-tree status
 - full script command and arguments
 - RustFS binary path and build mode
 - object size, object class, bucket, concurrency, duration, rounds
 - metric mode: metrics-off throughput or metrics-on attribution
 - environment affecting GET path selection
 - `manifest.env` and/or `environment.txt`
 - `warp/median_summary.csv` and `warp/round_results.csv` for throughput runs
 - `service_metrics_summary.csv`, `service_metrics_stage_distribution.csv`, or
  raw before/after Prometheus snapshots for attribution runs
 Do not compare a metrics-on run against a metrics-off run as a throughput win or
 loss. The only valid cross-mode claim is attribution.
 ## Baseline Dimensions
 Object sizes:
 - `1KiB`
 - `4KiB`
 - `10KiB`
 - `64KiB`
 - `100KiB`
 - `128KiB`
 - `1MiB`
 - `10MiB`
 - `128MiB`
 Object classes:
 - default inline candidate
 - forced non-inline candidate
 - plain single-part
 - multipart
 - range request
 - versioned bucket
 - checksum-mode request
 - CORS request
 Run the small default-path proof first. Expand into large, range, multipart, and
 versioned dimensions only after the small matrix has stable artifacts.
 ## Step 0: Script Syntax Gate
 ```bash
 for s in \
  scripts/run_get_codec_streaming_smoke.sh \
  scripts/run_get_metrics_gate_smoke.sh \
  scripts/run_object_batch_bench_enhanced.sh \
  scripts/run_object_data_cache_bench.sh
 do
  bash -n "$s"
 done
 ```
 ## Step 1: Metrics-Off Throughput Smoke
 Use this run only for throughput and latency. Observability export and detailed
 GET stage attribution are disabled by the script.
 ```bash
 scripts/run_get_metrics_gate_smoke.sh \
  --skip-build \
  --size 1KiB \
  --concurrency 32 \
  --duration 10s \
  --rounds 3 \
  --out-dir target/bench/get-default-metrics-off-1kib-c32
 ```
 Required checks:
 - `target/bench/get-default-metrics-off-1kib-c32/manifest.env` records
  observability export as disabled.
 - `warp/median_summary.csv` and `warp/round_results.csv` exist.
 - No statement about reader path is made from this run alone.
 ## Step 2: Metrics-On Default Path Attribution
 Use this run to prove the default reader path. It may be slower than the
 metrics-off smoke and that is expected.
 ```bash
 scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 1KiB,4KiB,10MiB \
  --concurrency 32 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._](reader_path|stage_duration|reader_setup|disk_permit|request|reader)' \
  --out-dir target/bench/get-default-attribution-legacy
 ```
 Required checks:
 - `manifest.env` records `mode=legacy` and diagnostic metrics enabled.
 - `service_metrics_summary.csv` exists.
 - The default path proof reports the delta for
  `rustfs_io_get_object_reader_path_total{path="legacy_duplex"}`.
 - If `legacy_duplex` is not positive, report the observed path instead of
  forcing the expected conclusion.
 ## Step 3: Small-Object Matrix
 Run the small sizes with metrics-off first:
 ```bash
 for size in 1KiB 4KiB 10KiB 64KiB 100KiB 128KiB 1MiB; do
  scripts/run_get_metrics_gate_smoke.sh \
    --skip-build \
    --size "$size" \
    --concurrency 32 \
    --duration 10s \
    --rounds 3 \
    --out-dir "target/bench/get-default-metrics-off-${size}-c32"
 done
 ```
 Then run attribution only for representative sizes:
 ```bash
 scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 1KiB,10KiB,128KiB,1MiB \
  --concurrency 32 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._](reader_path|stage_duration|reader_setup|disk_permit|request|reader)' \
  --out-dir target/bench/get-default-small-attribution
 ```
 ## Step 4: Large Sequential and Range Matrix
 Use the dedicated GT1G helper for very large sequential/ranged GETs when the
 target machine has enough local disk and time:
 ```bash
 scripts/run_gt1g_get_http_matrix.sh \
  --skip-build \
  --out-dir target/bench/get-default-large-http-matrix
 ```
 For a shorter release-gate matrix:
 ```bash
 scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 10MiB,128MiB \
  --concurrency 16 \
  --duration 20s \
  --rounds 3 \
  --out-dir target/bench/get-default-large-metrics-off
 ```
 Add `--diagnostic-metrics` only in a separate attribution run.
 ## Step 5: Codec Streaming A/B Matrix
 Codec streaming is not default-path proof. Use it only after the default legacy
 path has been proven.
 ```bash
 scripts/run_get_codec_streaming_smoke.sh \
  --mode both \
  --profile-order reverse \
  --codec-engine rustfs \
  --sizes 1MiB,10MiB,128MiB \
  --concurrency 16 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --out-dir target/bench/get-codec-ab-attribution
 ```
 Required checks:
 - Legacy profile has a positive `legacy_duplex` path delta.
 - Codec profile has a positive `codec_streaming` path delta only when codec
  settings are enabled.
 - Compatibility summaries pass before any performance conclusion is accepted.
 ## Step 6: Admission and Permit Fallback Stress
 Use this to validate workload admission visibility under pressure:
 ```bash
 RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=1 \
 RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=1 \
 scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 10MiB \
  --concurrency 64 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._]disk_permit[._]bypass|rustfs_io_disk_permit|rustfs_io_queue_|rustfs_io_get_object_' \
  --out-dir target/bench/get-permit-admission
 ```
 Required checks:
 - `rustfs.get_object.disk_permit.bypass.total` or its exported equivalent is
  captured when fallback happens.
 - Queue/admission metrics are present in the attribution snapshot.
 - Slow-client behavior does not deadlock the benchmark.
 ## Reporting Template
 Use this structure in the final issue or PR comment:
 ```text
 Commit:
 Branch:
 Dirty tree:
 Host:
 RustFS binary:
 Metric mode:
 Command:
 Out dir:
 Throughput evidence:
 - median req/s:
 - median latency:
 - p90:
 - p99:
 Attribution evidence:
 - reader_path legacy_duplex delta:
 - reader_path codec_streaming delta:
 - top stage durations:
 - disk permit fallback delta:
 Conclusion:
 - Default path:
 - Compatibility:
 - Follow-up:
 ```
 ## Acceptance Checklist
 - Metrics-off throughput and metrics-on attribution are separate directories.
 - Every artifact directory records commit, env, object layout, and arguments.
 - Default path hit rate is explicitly reported from diagnostic metrics.
 - Any default-path claim cites the metric delta used to prove it.
 - Codec/direct-memory/metadata-early-stop claims include their enabling env.
@@ -1,323 +0,0 @@
 # GET 中小文件性能优化结论
 ## 1. 项目概述
 ### 1.1 目标
 针对 RustFS GET 中小文件（1KiB-1MiB）性能落后 MinIO 5-15 倍的问题，进行系统性优化。
 ### 1.2 背景
 基于 2026-06-28 的标准化测试数据：
 | 对象大小 | RustFS | MinIO | 差距 |
 |----------|--------|-------|------|
 | 1KiB | 1.36 MiB/s | 21.15 MiB/s | 15.5x |
 | 4KiB | 4.88 MiB/s | 51.19 MiB/s | 10.5x |
 | 10KiB | 12.98 MiB/s | 201.23 MiB/s | 15.5x |
 | 100KiB | 117.42 MiB/s | 1142.89 MiB/s | 9.7x |
 | 1MiB | 1328.75 MiB/s | 7264.10 MiB/s | 5.5x |
 ## 2. 五位专家分析结论
 ### 2.1 根因分析
 | 瓶颈 | 影响 | 专家来源 |
 |------|------|----------|
 | `get_bucket_info()` 每次 GET 调用 | 500-2000us | 系统性能专家 |
 | Metadata fanout 等待所有盘 | 500-2000us | 分布式系统专家 |
 | Inline 数据走 duplex pipe | 300-600us | Rust 异步运行时专家 |
 | `tokio::spawn` 不必要 | 16-32us | S3 协议专家 |
 | Lifecycle 每次检查 | 50-100us | 可观测性专家 |
 ### 2.2 关键发现
 1. **Bucket 验证开销**：每次 GET 都调用 `get_bucket_info()`，对所有盘执行 `stat_volume()`
 2. **Inline 数据未优化**：数据已在内存中，但仍走完整 duplex pipe + EC decode 路径
 3. **Metadata fanout 过重**：默认等待所有盘响应，而非 quorum 早停
 4. **不必要的 tokio::spawn**：metadata fanout 中的 tokio::spawn 增加额外开销
 5. **条件化检查缺失**：lifecycle 和 metrics 检查未条件化
 ## 3. 已完成的优化
 ### 3.1 任务清单
 | Issue | 任务 | Commit | 状态 |
 |-------|------|--------|------|
 | #766 | SF01: Bucket 验证缓存 | 46e28cfac | ✅ 完成 |
 | #767 | SF02: Inline 数据快速路径 | 096fadcb3 | ✅ 完成 |
 | #768 | SF03: Metadata Cache TTL | 295178df8 | ✅ 完成 |
 | #769 | SF04: 移除 tokio::spawn | c2acb9aeb | ✅ 完成 |
 | #770 | SF05: 跳过 IO Planning | a26c241b7 | ✅ 完成 |
 | #771 | SF06: 条件化 Lifecycle | ce43a2189 | ✅ 完成 |
 | #772 | SF07: 条件化 Metrics | 0bf32ab11 | ✅ 完成 |
 ### 3.2 代码变更摘要
 #### SF01: Bucket 验证缓存
 **文件**：`rustfs/src/storage/ecfs_extend.rs`
 使用 moka 缓存 bucket 验证结果，TTL 5 秒。避免每次 GET 都执行 `stat_volume()`。
 ```rust
 static BUCKET_VALIDATED_CACHE: OnceLock<moka::sync::Cache<String, ()>> = OnceLock::new();
 const BUCKET_VALIDATION_TTL: Duration = Duration::from_secs(5);
 ```
 **预期提升**：3-5x (小文件)
 #### SF02: Inline 数据快速路径
 **文件**：`crates/ecstore/src/set_disk/mod.rs`
 对 inline 数据的小对象（<= 128KB），跳过 duplex pipe + tokio::spawn + bitrot reader 创建，直接在内存中解码。
 **条件**：
 - 单 part 对象
 - Inline 数据可用
 - 大小 <= 128KB
 - 非加密/压缩/远程
 - 无 range 请求
 **预期提升**：2-3x (小文件)
 #### SF03: Metadata Cache TTL 增加
 **文件**：`crates/ecstore/src/set_disk/mod.rs`
 ```rust
 // 修改前
 const GET_OBJECT_METADATA_CACHE_TTL: Duration = Duration::from_millis(250);
 const GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: usize = 1024;
 // 修改后
 const GET_OBJECT_METADATA_CACHE_TTL: Duration = Duration::from_secs(2);
 const GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: usize = 4096;
 ```
 **预期提升**：10-50x (热点对象)
 #### SF04: 移除 tokio::spawn
 **文件**：`crates/ecstore/src/set_disk/read.rs`
 移除 `read_all_fileinfo_full_wait` 中不必要的 `tokio::spawn`，直接使用 async future。
 **预期提升**：16-32us/请求
 #### SF06: 条件化 Lifecycle 检查
 **文件**：`rustfs/src/app/object_usecase.rs`
 仅在对象有 `x-amz-expiration` metadata 时才调用 `resolve_put_object_expiration`。
 **预期提升**：50-100us/请求
 #### SF07: 条件化 Metrics 记录
 **文件**：`rustfs/src/app/object_usecase.rs`
 所有 hot path metrics 都被 `get_stage_metrics_enabled()` 保护。
 **预期提升**：20-50us/请求
 ## 4. 实际压测效果 (2026-06-28)
 ### 4.1 吞吐量对比 (MiB/s)
 | 对象大小 | MinIO | RustFS main | SF01-07 | SF01-07+SF05 | **重构后** | vs main | vs MinIO |
 |----------|------:|------------:|--------:|-------------:|-----------:|--------:|---------:|
 | 1KiB | 21.15 | 2.88 | 2.43 | 2.52 | **2.52** | -12.5% | -88.1% |
 | 4KiB | 51.19 | 11.30 | 9.73 | 10.12 | **10.11** | -10.5% | -80.2% |
 | 10KiB | 201.23 | 28.56 | 19.41 | 25.33 | **24.88** | -12.9% | -87.6% |
 | 100KiB | 1142.89 | 277.49 | 212.04 | 243.95 | **247.23** | -10.9% | -78.3% |
 | 1MiB | 7264.10 | 2270.27 | 1818.96 | 2070.06 | **2051.34** | -9.6% | -71.8% |
 ### 4.2 SF05 增量效果
 | 对象大小 | SF01-07 | SF01-07+SF05 | 重构后 | SF05 提升 |
 |----------|--------:|-------------:|-------:|----------:|
 | 1KiB | 2.43 | 2.52 | 2.52 | +3.7% |
 | 4KiB | 9.73 | 10.12 | 10.11 | +3.9% |
 | 10KiB | 19.41 | 25.33 | 24.88 | +28.2% |
 | 100KiB | 212.04 | 243.95 | 247.23 | +16.6% |
 | 1MiB | 1818.96 | 2070.06 | 2051.34 | +12.8% |
 ### 4.3 与 MinIO 差距
 | 对象大小 | 差距倍数 |
 |----------|----------|
 | 1KiB | 8.4x |
 | 4KiB | 5.1x |
 | 10KiB | 8.1x |
 | 100KiB | 4.6x |
 | 1MiB | 3.5x |
 ## 5. SF05 实现详情
 ### 5.1 修改内容
 **文件**：`rustfs/src/app/object_usecase.rs`
 1. `GetObjectIoPlanning._disk_permit`: `SemaphorePermit<'a>` → `Option<SemaphorePermit<'a>>`
 2. `GetObjectReadSetup`: 新增 `is_inline_fast_path: bool`
 3. `prepare_get_object_read`: 新增 inline 检测逻辑
 4. `prepare_get_object_read_execution`: 重排执行顺序，先读再决定是否获取 semaphore
 ### 5.2 执行流程变化
 **修改前**：
 ```
 prepare_get_object_read_execution
  → acquire_get_object_io_planning   [BLOCKS on semaphore]
  → prepare_get_object_read          [discovers inline here]
 ```
 **修改后**：
 ```
 prepare_get_object_read_execution
  → prepare_get_object_read          [no semaphore held]
  → if inline fast path:
      → skip semaphore               [saves 100-200us]
  → else:
      → acquire_get_object_io_planning
 ```
 ### 5.3 SF05 增量效果
 | 对象大小 | SF01-07 | SF01-07+SF05 | 提升 |
 |----------|--------:|-------------:|-----:|
 | 1KiB | 2.43 | 2.52 | +3.7% |
 | 4KiB | 9.73 | 10.12 | +4.0% |
 | 10KiB | 19.41 | 25.33 | **+30.5%** |
 | 100KiB | 212.04 | 243.95 | **+15.1%** |
 | 1MiB | 1818.96 | 2070.06 | **+13.8%** |
 ## 6. 未完成的任务
 ### 6.1 SF08: 标准化压测验证
 **状态**：已完成初步压测，详见 `docs/benchmark/sf05-optimization-results-2026-06-28.md`。
 **结论**：SF01-07+SF05 相比 main 分支差距缩小到 9-12%，相比历史 v2-refactor 分支提升 85-108%。
 ## 6. 兼容性保证
 ### 6.1 S3 协议兼容性
 所有优化都保持 S3 协议完全兼容：
 - ETag、Content-Length、Content-Type 等 header 不变
 - Range 请求正确处理
 - 加密/压缩对象走原路径
 - Versioning 语义不变
 ### 6.2 数据正确性
 - Inline 数据在写入时已通过 bitrot 校验
 - Metadata quorum 保证数据一致性
 - 写操作正确失效缓存
 ### 6.3 回滚机制
 所有优化都可通过环境变量或代码回滚：
 - SF01: 删除缓存代码
 - SF02: 删除快速路径代码
 - SF03: 恢复原 TTL 值
 - SF04: 恢复 tokio::spawn
 - SF05: 恢复 semaphore 必选
 - SF06: 恢复无条件 lifecycle 检查
 - SF07: 恢复无条件 metrics 记录
 ## 7. 测试验证
 ### 7.1 功能测试
 - [x] 所有现有测试通过
 - [x] Inline 数据快速路径正确性验证
 - [x] Metadata cache 失效机制验证
 - [x] S3 协议兼容性验证
 - [x] SF05 inline IO planning 跳过验证
 ### 7.2 性能测试
 - [x] SF01-07+SF05 标准化压测完成
 - 详见 `docs/benchmark/sf05-optimization-results-2026-06-28.md`
 ## 8. 分支与提交
 ### 8.1 分支信息
 ```
 分支: houseme/get-small-file-optimization
 基础: origin/main
 Commit 数: 9+
 ```
 ### 8.2 提交历史
 ```
 a26c241b7 feat(get): SF05 - skip IO planning for inline data
 2c493fb5f refactor: translate Chinese comments to English
 096fadcb3 feat(get): SF02 - inline data fast path
 46e28cfac refactor(get): SF01 - use moka instead of dashmap for bucket cache
 0bf32ab11 feat(get): SF07 - conditional metrics recording
 ce43a2189 feat(get): SF06 - conditional lifecycle check
 c2acb9aeb feat(get): SF04 - remove unnecessary tokio::spawn in metadata fanout
 295178df8 feat(get): SF03 - metadata cache TTL increase
 618377541 feat(get): SF01 - bucket validation cache
 ```
 ## 9. GitHub Issues
 | Issue | 标题 | 状态 |
 |-------|------|------|
 | [#765](https://github.com/rustfs/backlog/issues/765) | 主 Issue: 中小文件性能优化 | 进行中 |
 | [#766](https://github.com/rustfs/backlog/issues/766) | SF01: Bucket 验证缓存 | ✅ 完成 |
 | [#767](https://github.com/rustfs/backlog/issues/767) | SF02: Inline 数据快速路径 | ✅ 完成 |
 | [#768](https://github.com/rustfs/backlog/issues/768) | SF03: Metadata Cache TTL | ✅ 完成 |
 | [#769](https://github.com/rustfs/backlog/issues/769) | SF04: 移除 tokio::spawn | ✅ 完成 |
 | [#770](https://github.com/rustfs/backlog/issues/770) | SF05: 跳过 IO Planning | ✅ 完成 |
 | [#771](https://github.com/rustfs/backlog/issues/771) | SF06: 条件化 Lifecycle | ✅ 完成 |
 | [#772](https://github.com/rustfs/backlog/issues/772) | SF07: 条件化 Metrics | ✅ 完成 |
 | [#773](https://github.com/rustfs/backlog/issues/773) | SF08: 标准化压测验证 | ✅ 完成 |
 ## 10. 总结
 ### 10.1 成果
 1. **完成 7 项优化**：SF01, SF02, SF03, SF04, SF05, SF06, SF07
 2. **重构消除技术债**：统一 inline 检测逻辑，修复缓存失效、容错降级等问题
 3. **实际性能提升**：10KiB +28.2%, 100KiB +16.6%, 1MiB +12.8% (vs SF01-07)
 4. **与历史 v2-refactor 对比**：提升 54-110%
 5. **保持兼容性**：S3 协议完全兼容，数据正确性保证
 6. **代码质量**：消除重复逻辑，恢复容错能力
 ### 10.2 关键优化点
 1. **Bucket 验证缓存** (SF01)：消除每次 GET 的 stat_volume 调用
 2. **Inline 快速路径** (SF02)：跳过 duplex pipe 和 EC decode
 3. **跳过 IO Planning** (SF05)：inline 数据跳过 semaphore 获取
 4. **统一 inline 检测** (重构)：`ObjectInfo::is_inline_fast_path_eligible()` 共享方法
 5. **恢复容错** (重构)：`tokio::spawn` + `JoinError` 处理
 ### 10.3 后续建议
 1. **Profiling**：使用 flamegraph 找到真正瓶颈
 2. **Inline 验证**：确认 warp 上传的对象是否走 inline 路径
 3. **对比 MinIO**：分析 MinIO 在小文件上的优势来源
 4. **简化优化**：考虑移除效果不明显的 SF03/SF06/SF07
 ## 11. 相关文档
 - [GET 优化性能对比分析](get-optimization-performance-comparison.md)
 - [GET 优化复测分析](get-optimization-retest-analysis.md)
 - [GET 优化标准化测试结果](get-optimization-standardized-test-results.md)
 - [中小文件优化指南](get-optimization-small-medium-file-guide.md)
 - [子任务索引](tasks/get-small-file-optimization/README.md)
 - [SF05 压测结果](../benchmark/sf05-optimization-results-2026-06-28.md)
 - [重构后压测结果](../benchmark/refactored-results-2026-06-28.md)
 - [V2 优化压测结果](../benchmark/v2-optimization-results-2026-06-28.md)
 - [自适应缓存压测结果](../benchmark/adaptive-cache-results-2026-06-28.md)
 - [性能差距根因分析](../benchmark/performance-gap-analysis-2026-06-28.md)
 - [基准测试存档](../benchmark/issue714-local-single-machine-multidisk-get-2026-06-26.md)
@@ -1,304 +0,0 @@
 # Issue #4003 Pragmatic Execution Plan for ListObjectsV2 Performance
 Date: 2026-06-29
 Reference:
 - rustfs issue: https://github.com/rustfs/rustfs/issues/4003
 - backlog parent issue: https://github.com/rustfs/backlog/issues/788
 - subtask: #785 #786 #787
 ## 1) Problem statement and observed failure mode
 `ListObjectsV2` becomes very slow on very large buckets (~745k objects) under recursive listing workloads (`rclone size`, `rclone lsf --recursive`).
 Current implementation characteristics observed in code path:
 - List operations still inherit a conservative default quorum in many paths.
 - Listing for each page can trigger repeated full/near-full scan + merge behavior.
 - Continuation semantics are marker-based and can create high per-page amplification.
 ## 2) Root-cause hypothesis (five-perspective review)
 ### 2.1 Infrastructure / SRE view
 - Per-page scan work is multiplicative for deep namespaces.
 - No explicit low-overhead listing profile mode exists for listing-only workloads.
 - Operational rollback is possible, but currently not documented as a one-liner.
 ### 2.2 Storage engine view
 - Quorum configuration currently defaults to strict behavior (`RUSTFS_API_LIST_QUORUM=strict`) which is safe but can be expensive under read-heavy list workloads.
 - Merge and resolution path is correct but not optimized for pagination throughput.
 ### 2.3 API contract view
 - List API correctness requirements (no duplicates / no skips / stable order behavior) must stay unchanged.
 - Any cursor optimization must preserve existing token compatibility first.
 ### 2.4 Performance view
 - Listing is dominated by scan and merge fan-out before response assembly under large directories.
 - Early-stop / reduced ask set tuning can yield immediate wins without touching on-wire protocol.
 ### 2.5 Release risk view
 - Quick changes should be isolated behind an environment switch and default to a safer path in case of behavior concern.
 - Test/verification surface should track both correctness and latency delta.
 ## 3) Fixes completed in current patch (Phase-1 scope)
 ### 3.1 Introduce ListObjects-only quorum switch
 File: `crates/ecstore/src/store/list_objects.rs`
 - Added constants:
  - `RUSTFS_LIST_OBJECTS_QUORUM`
  - default `optimal`
 - Added helper:
  - `list_objects_quorum_from_env()`
 - Switched ask disks for listing operations to use this helper in 6 call sites:
  - `ECStore::list_objects`
  - `ECStore::list_object_versions`
  - `Sets::list_objects`
  - `Sets::list_object_versions`
  - `SetDisks::list_objects`
  - `SetDisks::list_object_versions`
 ### 3.2 Unit tests added
 In `#[cfg(test)]` for `list_objects.rs`:
 - `list_objects_quorum_from_env_defaults_to_optimal`
 - `list_objects_quorum_from_env_honors_supported_value`
 These verify default and explicit env behavior.
 ### 3.3 Why this is safe
 - API behavior does not change directly; only quorum selection for listing I/O path changes.
 - Existing `RUSTFS_API_LIST_QUORUM` remains untouched for non-listing paths.
 - Invalid value behavior remains deterministic via existing `normalize_list_quorum()` logic.
 ## 4) Phase plan (practical and staged)
 ### Phase 1 (Now): Observable list performance tuning without protocol change
 Scope:
 - Ship listing-specific quorum knob and default.
 - Add metrics/logging for per-page scan count and early-stop ratio.
 - Run baseline and post-change micro-bench on recursive list.
 Acceptance:
 - No correctness regression in list responses.
 - Reduction in list CPU + elapsed time under large recursive reads.
 - Rollback confirmed by setting `RUSTFS_LIST_OBJECTS_QUORUM=strict`.
 ### Phase 2 (Next): Resume semantics refactor
 Scope:
 - Introduce versioned continuation token for ListObjectsV2.
 - Keep legacy token compatibility in parsing path.
 - Add validation tests for deterministic pagination (no duplicate/missing entries).
 Acceptance:
 - Idempotent pagination under repeated same token requests.
 - Zero regression in API contract and clients.
 - Graceful fallback for unknown token format.
 ### Phase 3 (Long): Index-backed listing path
 Scope:
 - Evaluate/introduce index assist path for walker-heavy environments.
 - Fallback to walker on index inconsistency.
 - Define rebuild + repair strategy and runbooks.
 Acceptance:
 - Sustained throughput gains on very large buckets.
 - Operational risk under consistency faults bounded.
 - Rollback and recovery steps documented and tested.
 ## 5) Validation checklist (before merge)
 - `cargo test -p rustfs-ecstore list_objects -- --nocapture` (focused on list code)
 - `cargo test -p rustfs-ecstore list_objects_quorum_from_env` (if supported selector)
 - `scripts/validate_issue_787_list_quorum.sh` (offline) and `scripts/validate_issue_787_list_quorum.sh --full` (live, if environment ready)
 - Add/verify benchmark baseline:
  - recursive `rclone lsf --recursive` on staging bucket
  - `rclone size` style pass
 - Compare:
  - p99 listing latency
  - request count per page
  - duplicated/missing keys between consecutive pages
  - CPU and disk read amplification
 ## 6) #787 live benchmark and rollout acceptance (in progress)
 ### 6.1 live benchmark plan (execution contract)
 - Workload baseline:
  - Bucket: pre-populated production-equivalent bucket (>= 700k objects, mixed prefixes, deep tree)
  - Tool: `rclone lsf --recursive` + `rclone size`
  - Endpoint: `RUSTFS_ADDRESS=...` on fresh RustFS node with `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY`
  - Pages: `--max-keys` at 1000 / 5000 to measure p95/p99 page behavior
 - Comparative configurations (at least two runs):
  - Baseline: `RUSTFS_LIST_OBJECTS_QUORUM=strict`
  - Candidate: `RUSTFS_LIST_OBJECTS_QUORUM=optimal` (or `reduced` if strict is unavailable in workload)
 - Required outputs for each run:
  - p95/p99 page latency
  - wall-clock time for complete recursive listing pass
  - duplicates/missing count by token replay check
  - per-page key count and continuation token progression
 - `scripts/validate_issue_787_list_quorum.sh --full`
  - set `MODE_LIST` to the candidate matrix, for example:
    - `MODE_LIST=strict,optimal,reduced`
  - export `SERVER_RESTART_CMD` and `SERVER_WAIT_SECONDS=20` to produce mode-scoped logs under one run output directory.
 - Acceptance thresholds (proposal):
  - Candidate p99 page latency delta ≤ +10% and ≥ +10% wall-clock gain against baseline on same bucket class
  - Zero duplicates/missing objects across page transitions
  - No listing correctness regressions
 ### 6.2 completed local benchmark package (2026-06-30)
 Benchmark environment used to close the missing evidence gap:
 - single-node local RustFS with the same data directory reused across mode restarts
 - bucket: `issue787-bench`
 - object layout: 6,000 zero-byte objects across mixed recursive prefixes:
  - `alpha/obj-*`
  - `beta/deep/obj-*`
  - `gamma/nested/path/obj-*`
 - request shape: full bucket recursive `ListObjectsV2`
 - pagination: `max-keys=1000`
 - sample size: 15 full passes per mode, 6 pages per pass, 90 page samples per mode
 Results summary:
 | Mode | Page p95 (ms) | Page p99 (ms) | Wall avg (ms) | p99 delta vs strict | Wall avg delta vs strict | Continuity |
 | --- | ---: | ---: | ---: | ---: | ---: | --- |
 | `strict` | 255.413 | 275.438 | 1469.938 | baseline | baseline | pass |
 | `optimal` | 259.146 | 267.342 | 1485.912 | -2.939% | +1.087% | pass |
 | `reduced` | 255.937 | 272.557 | 1477.955 | -1.046% | +0.545% | pass |
 Decision from the benchmark package:
 - Both candidate modes stayed within the strict-baseline p99 guardrail in this local harness.
 - Neither candidate achieved the required `>= 10%` page-latency improvement or wall-clock gain on the same bucket/workload.
 - Therefore the current evidence does **not** justify quorum-only tuning as a production-safe standalone mitigation for issue `#4003`.
 - Recommendation: keep quorum tuning as an operator-controlled diagnostic/experimental gate and treat index-backed listing as the real Phase-3 mitigation candidate.
 ### 6.3 execution attempt log
 - Attempt start/run status:
  - `scripts/restart_local_single_node_multidisk_rustfs.sh` and direct `target/debug/rustfs server` launch path
  - Endpoint bind error observed: `Operation not permitted (os error 1)` on both 9000 and 19000
  - Artifact directory:
    - `target/issue-787-live-benchmark-20260630-031521/`
 - Current status:
  - Live benchmark for #787 is **blocked by environment bind permission** in this runner.
  - Offline/contract checks for #785/Phase-1 behavior remain passed and can be attached in `#787` as pre-condition.
 ### 6.4 rollout documentation package (required to close #787)
 - Add rollout control:
  - Config default: `RUSTFS_LIST_OBJECTS_QUORUM=optimal` (current patch)
  - Safety fallback: `RUSTFS_LIST_OBJECTS_QUORUM=strict`
  - Error budget guardrails for rollout (phase plan):
    - p99 latency +20% ceiling
    - error-rate +10% ceiling
    - continuity test zero duplicate/missing
 - Release sequencing:
  - Stage A: shadow traffic for large-recursive listing buckets (same object layout + namespace)
  - Stage B: partial bucket/range enablement with continuity probes
  - Stage C: scale to wider prefix families only after two consecutive acceptance windows pass
 ### 6.5 index-backed engineering scaffold (next milestone)
 - Scope: introduce an optional index-assisted listing mode with walk-based fallback.
 - Minimal state machine design:
  1. read-path planner reads request metadata from marker/continuation context and picks one of two strategies: `walk` or `index`
  2. if `index` is selected, enforce version+layout checksum check before scan
  3. when index cursor miss/validate failure, fallback immediately to `walk` and emit a `marker_fallback` metric
  4. emit cursor and source tags in `NextContinuationToken` only when index strategy was used
 - Acceptance for first pass:
  - zero functional regressions (`duplication=0`, `missing=0`) in offline smoke and token replay tests
  - fallback event rate below threshold in steady-state run (target: < 1%)
  - rollback via existing `RUSTFS_LIST_OBJECTS_QUORUM` control remains unchanged
 - Close-out action: keep this section as the live action list in the follow-up issue/subtask and bind each item to a dedicated PR before closure.
 ### 6.6 explicit index-backed design artifact
 Continuation-token compatibility matrix:
 | Token family | Current/Planned producer | Parser action | Preferred read path | Fallback behavior |
 | --- | --- | --- | --- | --- |
 | legacy marker-only | legacy clients / pre-v2 servers | parse as legacy continuation | walker | fail closed only on malformed XML/query |
 | v2 cache-tag token | current Phase-2 implementation | parse marker + cache tag, preserve replay semantics | walker with metacache resume | force cache refresh on corrupt/replay mismatch |
 | proposed v3 index-cursor token | future index-assisted planner | parse marker + index cursor metadata + manifest epoch | index | immediate downgrade to walker on token parse / epoch / checksum mismatch |
 Proposed index schema (design only, not implemented in this PR):
 - `bucket`
 - `prefix_scope`
 - `object_key`
 - `version_id` (optional for versioned buckets)
 - `delete_marker`
 - `etag`
 - `size`
 - `last_modified`
 - `cursor_seq`
 - `manifest_epoch`
 - `record_checksum`
 Proposed write-path hooks:
 - `PutObject`, `CopyObject`, `CompleteMultipartUpload`: emit index upsert records
 - `DeleteObject`, `DeleteObjectVersion`, lifecycle expiry: emit tombstone records
 - replication/heal/rebuild workflows: emit replayable repair events into the same mutation stream
 Consistency, backfill, and rebuild plan:
 - initial build trigger: bucket/range enrollment into index-assisted listing
 - backfill model: prefix-ordered walker snapshot with persisted watermark per bucket/range
 - drift detection: periodic sampled compare between index page output and walker page output using the same continuation boundary
 - rebuild trigger:
  - manifest epoch mismatch
  - record checksum mismatch
  - fallback rate above threshold
  - operator-requested rebuild after repair/heal incidents
 - proposed operator tooling surface (design-only names):
  - `list-index build`
  - `list-index verify`
  - `list-index rebuild`
  - `list-index repair`
 Fallback state machine requirements:
 - token parse failure -> walker
 - manifest epoch mismatch -> walker + `marker_fallback` metric
 - sampled drift mismatch -> walker + alert
 - high fallback/error budget burn -> disable index path for affected bucket/range
 ### 6.7 rollout / rollback closure package
 Rollout phases:
 | Phase | Scope | Success condition | Abort condition |
 | --- | --- | --- | --- |
 | 0 | local/offline validation | continuity pass, unit/static pass | duplicate/missing in local smoke |
 | A | shadow benchmark on production-shaped bucket | two consecutive windows with continuity pass and stable latency | any continuity failure |
 | B | canary bucket or prefix range | error budget within guardrail, fallback rate below threshold | p99 +20%, error +10%, fallback spike |
 | C | broadened prefix families | same as canary for two windows | any regression repeats in two windows |
 | D | wider rollout | observability remains stable | operator rollback trigger |
 Explicit rollback conditions:
 - duplicate objects across pages
 - missing objects after truncation / continuation transition
 - p99 latency regression `> +20%`
 - error-rate regression `> +10%`
 - fallback rate above the planned threshold
 - sampled index drift mismatch
 Rollback actions:
 - quorum-only experiment: set `RUSTFS_LIST_OBJECTS_QUORUM=strict`
 - future index-assisted planner: disable index path and force walker
 - token-family regression: stop emitting proposed v3 index tokens and continue accepting legacy / v2 parser branches only
 Closure interpretation for `#787`:
 - the benchmark evidence is now attached and complete;
 - the outcome is negative for quorum-only rollout;
 - the design/rollout package now supports moving the real mitigation focus to index-backed listing.
 ## 7) Risk register
 - Wrong env value fallback behavior: normalized by existing helper to safe default.
 - Quorum drift across environments: explicit rollout control via env variable in deployment chart.
 - Unexpected compatibility concerns: no client-facing protocol change in this phase.
 ## 8) Rollback plan
 - Set `RUSTFS_LIST_OBJECTS_QUORUM=strict` at runtime/restart.
 - If needed, revert this commit only.
 ## 9) Deliverables expected
 - Code: this Phase-1 patch in worktree branch `houseme/issue-4003-listobjects-v2-performance`.
 - Docs: English and Chinese documents (this plan and Chinese counterpart).
 - Tracking: backlog issues `#785`, `#786`, `#787` under parent `#788`.
@@ -1,281 +0,0 @@
 # Issue #4003：ListObjectsV2 大规模列表性能实战优化方案
 日期：2026-06-29
 关联：
 - rustfs 问题： https://github.com/rustfs/rustfs/issues/4003
 - Backlog 主 Issue： https://github.com/rustfs/backlog/issues/788
 - 子任务：#785 #786 #787
 ## 1) 问题定义与现象
 在超大桶（约 74.5 万对象）上，`ListObjectsV2` 在递归列表场景（`rclone size`、`rclone lsf --recursive`）表现非常慢。
 当前路径的核心问题：
 - 列表场景默认仍偏向保守的一致性配置；
 - 每页列表可能触发重复扫描与归并放大，导致分页成本远超期望；
 - 续传标记偏 marker 化，没形成可优化的“低开销续跑”语义。
 ## 2) 五位专家视角下的根因拆解
 ### 2.1 存储引擎视角
 - 列表时 ask-disks quorum 决策成本高，且“严格模式”对大目录带来更高开销。
 - merge/walk 逻辑正确但未对分页放大做特别优化。
 ### 2.2 API 兼容性视角
 - List API 的行为正确性（无重复、无漏）。
 - 后续 cursor 化不能破坏旧 token 兼容。
 ### 2.3 性能视角
 - 页面级放大是主要瓶颈，尤其在多磁盘元数据合并阶段。
 - 优化点应先聚焦“减少每页扫描成本”和“低风险可回退配置”。
 ### 2.4 运营风险视角
 - 优先以可运行时切换的参数落地，而非一次性重构。
 - 回退路径必须清晰且成本低。
 ### 2.5 SRE 观测视角
 - 需要把每页扫描量、归并次数、token 回退频率纳入指标，否则优化效果难量化。
 ## 3) 本次已完成变更（Phase-1 第一刀）
 ### 3.1 引入列表专用 quorum 配置
 文件：`crates/ecstore/src/store/list_objects.rs`
 - 新增环境变量：`RUSTFS_LIST_OBJECTS_QUORUM`
 - 默认值：`optimal`
 - 新增方法：`list_objects_quorum_from_env()`
 - 将列表相关 6 个入口改为该参数：
  - `ECStore::list_objects`
  - `ECStore::list_object_versions`
  - `Sets::list_objects`
  - `Sets::list_object_versions`
  - `SetDisks::list_objects`
  - `SetDisks::list_object_versions`
 ### 3.2 单测补齐
 `crates/ecstore/src/store/list_objects.rs` 测试模块新增：
 - `list_objects_quorum_from_env_defaults_to_optimal`
 - `list_objects_quorum_from_env_honors_supported_value`
 ## 4) 变更分阶段执行计划
 ### 阶段 1（当前）：列表性能基线优化，不改协议
 目标：只调优查询侧行为，保持外部语义不变。
 - 列表专用 quorum 与默认值上线；
 - 增强列表指标与日志（扫描/归并/提前停止）；
 - 建立对比基线，确认 `rclone` 场景改善。
 验收：
 - 响应正确性不变；
 - 大列表场景延时和 CPU 明显下降；
 - `RUSTFS_LIST_OBJECTS_QUORUM=strict` 可秒级回退。
 ### 阶段 2：分页续传语义工程
 目标：实现可版本化 cursor。
 - 引入版本化 continuation token；
 - 保持 legacy token 向后兼容；
 - 增补重复/跳页防护测试。
 验收：
 - 同输入同 token 重放结果稳定；
 - 不丢/不重；
 - 兼容旧客户端。
 ### 阶段 3：指数级场景结构优化
 目标：引入索引辅助列表主路径。
 - 评估并实现索引路径；
 - walker 保持 fallback；
 - 指定回放/重建策略。
 验收：
 - 超大桶 list 延时和放大系数持续下降；
 - 一致性问题有明确应急与恢复动作。
 ## 5) 验证清单
 - `cargo test -p rustfs-ecstore list_objects -- --nocapture`
 - `cargo test -p rustfs-ecstore list_objects_quorum_from_env`
 - `scripts/validate_issue_787_list_quorum.sh`（离线）以及 `scripts/validate_issue_787_list_quorum.sh --full`（如环境可连则做 live）
 - 压测对比：
  - `rclone lsf --recursive`
  - `rclone size`
 - 对照指标：
  - p99 延时
  - 每页扫描对象数/归并时间
  - 是否出现重复、跳过对象
  - 资源使用（CPU、磁盘读）
 ## 6) #787 验收与上线方案（进行中）
 ### 6.1 live benchmark 验证方案
 - 压测基线：
  - 目标桶：预先准备的生产同规模桶（建议 ≥ 70万 对象，含深层前缀）
  - 工具：`rclone lsf --recursive` + `rclone size`
  - 端点：本地可用的 RustFS 节点，配置 `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY`
  - 分页：`--max-keys` 取 1000 / 5000 两档，比较单页 p95/p99
 - 对比配置（至少两组）：
  - 基线：`RUSTFS_LIST_OBJECTS_QUORUM=strict`
  - 候选：`RUSTFS_LIST_OBJECTS_QUORUM=optimal`（或 `reduced`，如 strict 对某些环境不可接受）
 - 每次压测必交付：
  - p95/p99 分页延时
  - 全量递归列表完成耗时
  - 跨页重放校验下的重复/漏对象数
  - 每页 key 数及 continuation token 递进情况
 - `scripts/validate_issue_787_list_quorum.sh --full`
  - 用 `MODE_LIST` 表示要对比的模式，例如：
    - `MODE_LIST=strict,optimal,reduced`
  - 建议设置 `SERVER_RESTART_CMD` 与 `SERVER_WAIT_SECONDS=20`，在同一日志目录下产出生效模式分片。
 - 建议验收阈值（草案）：
  - 候选方案 p99 分页延时较 baseline 下降 ≥ 10%，回退不超过 +10%
  - 连续分页零重复/零漏
  - API 列表语义不变
 ### 6.2 已完成的本地 benchmark 证据包（2026-06-30）
 本次用于补齐证据缺口的本地 benchmark 环境：
 - 单节点本地 RustFS，同一数据目录在三模式间复用
 - 桶：`issue787-bench`
 - 对象布局：6,000 个零字节对象，混合递归前缀：
  - `alpha/obj-*`
  - `beta/deep/obj-*`
  - `gamma/nested/path/obj-*`
 - 请求形态：整桶递归 `ListObjectsV2`
 - 分页：`max-keys=1000`
 - 样本规模：每模式 15 次 full pass、每次 6 页、每模式共 90 个分页样本
 结果汇总：
 | 模式 | Page p95 (ms) | Page p99 (ms) | Wall avg (ms) | 相对 strict 的 p99 变化 | 相对 strict 的 wall avg 变化 | 连续性 |
 | --- | ---: | ---: | ---: | ---: | ---: | --- |
 | `strict` | 255.413 | 275.438 | 1469.938 | baseline | baseline | pass |
 | `optimal` | 259.146 | 267.342 | 1485.912 | -2.939% | +1.087% | pass |
 | `reduced` | 255.937 | 272.557 | 1477.955 | -1.046% | +0.545% | pass |
 基于 benchmark 的结论：
 - 两个候选模式在本地样本中都没有突破 strict 基线的 p99 安全护栏；
 - 但两者都没有达到 `>= 10%` 的分页延时改善或 wall-clock 改善要求；
 - 因此，当前证据**不足以支持**把 quorum-only 调优作为 `#4003` 的生产级独立缓解方案；
 - 建议：把 quorum 调优保留为可运营控制的实验/诊断开关，把真正的 Phase-3 缓解重心转到 index-backed listing。
 ### 6.3 现场执行记录（本次）
 - 当前在本地执行记录：
  - 直接执行 `scripts/restart_local_single_node_multidisk_rustfs.sh` 与 `target/debug/rustfs server` 启动链路
  - 9000 与 19000 端口均出现启动失败：`Operation not permitted (os error 1)`（HTTP bind 失败）
  - 归档日志目录：
    - `target/issue-787-live-benchmark-20260630-031521/`
 - 当前状态：
  - #787 live benchmark 当前 **受执行环境限制阻塞**，未能形成完整线上对比结果
  - #785 的离线验收已闭环，可作为 #787 后续评估前置条件
 ### 6.4 上线回滚文档要求
 - 回滚参数：
  - 当前默认值：`RUSTFS_LIST_OBJECTS_QUORUM=optimal`
  - 安全回退：`RUSTFS_LIST_OBJECTS_QUORUM=strict`
 - 上线阈值：
  - p99 延时失效阈值 +20%
  - 错误率失效阈值 +10%
  - 连续性测试重复/漏数异常立即回滚
 - 上线顺序：
  - 第 1 阶段：影子流量验证（与现网同构 prefix）
  - 第 2 阶段：小范围桶/前缀灰度
  - 第 3 阶段：连续两次验收窗口均通过后再扩大范围
 ### 6.5 index-backed 工程化路线（下一阶段）
 - 目标：引入可选 index 辅助列表路径，并保留 walk 回退。
 - 最小状态机（初版）：
  1. 读取 marker/continuation 的上下文后进行策略选择：`walk` 或 `index`
  2. 走 index 时先做版本与布局校验，不通过则立即回退到 walk 并记录 `marker_fallback`
  3. 当 fallback 触发时记录 metric 和日志，避免“悄悄切换”
  4. 仅 index 成功路径返回的 token 中携带 index 源标记，避免兼容歧义
 - 下一里程碑验收指标：
  - 跨页 smoke 保持零重复/零漏
  - index 回退率在稳定阶段可控（目标 < 1%）
  - 回退开关与 `RUSTFS_LIST_OBJECTS_QUORUM` 的现有保护链路保持独立互不影响
 ### 6.6 显式 index-backed 设计产物
 Continuation token 兼容矩阵：
 | Token 家族 | 当前/规划的产生方 | 解析动作 | 首选读路径 | 回退行为 |
 | --- | --- | --- | --- | --- |
 | legacy marker-only | 旧客户端 / pre-v2 服务端 | 按 legacy continuation 解析 | walker | 仅在 XML / query 自身损坏时失败 |
 | v2 cache-tag token | 当前 Phase-2 实现 | 解析 marker + cache tag，保持 replay 语义 | walker + metacache resume | cache tag 损坏或 replay 不一致时强制 refresh |
 | 规划中的 v3 index-cursor token | 未来 index-assisted planner | 解析 marker + index cursor 元数据 + manifest epoch | index | token 解析 / epoch / checksum 失败时立即降级 walker |
 规划中的索引 schema（仅设计，不代表当前 PR 已实现）：
 - `bucket`
 - `prefix_scope`
 - `object_key`
 - `version_id`（版本桶可选）
 - `delete_marker`
 - `etag`
 - `size`
 - `last_modified`
 - `cursor_seq`
 - `manifest_epoch`
 - `record_checksum`
 规划中的写路径 hook：
 - `PutObject` / `CopyObject` / `CompleteMultipartUpload`：写入 upsert 记录
 - `DeleteObject` / `DeleteObjectVersion` / 生命周期过期：写入 tombstone
 - replication / heal / rebuild：向同一 mutation stream 发送可重放 repair event
 一致性、回填、重建方案：
 - 初始构建触发：bucket/range 被纳入 index-assisted listing
 - 回填模型：按 prefix 顺序 walker snapshot，并为每个 bucket/range 持久化 watermark
 - 漂移检测：周期性对同一 continuation 边界下的 index page 与 walker page 做抽样比对
 - 重建触发：
  - manifest epoch 不一致
  - record checksum 不一致
  - fallback rate 超阈值
  - operator 在 repair/heal 之后手动触发
 - 规划中的运维工具接口（设计名）：
  - `list-index build`
  - `list-index verify`
  - `list-index rebuild`
  - `list-index repair`
 Fallback 状态机要求：
 - token 解析失败 -> walker
 - manifest epoch 不一致 -> walker + `marker_fallback` metric
 - 抽样漂移不一致 -> walker + alert
 - fallback/error budget burn 过快 -> 对受影响 bucket/range 关闭 index path
 ### 6.7 rollout / rollback 闭环产物
 上线阶段矩阵：
 | 阶段 | 范围 | 成功条件 | 中止条件 |
 | --- | --- | --- | --- |
 | 0 | 本地/离线验证 | continuity pass，unit/static pass | 本地 smoke 出现重复/漏数 |
 | A | 影子 benchmark（生产形态桶） | 连续两个窗口 continuity pass 且延时稳定 | 任意 continuity failure |
 | B | canary bucket / prefix range | error budget 在阈值内，fallback rate 受控 | p99 +20%、error +10%、fallback 激增 |
 | C | 扩大到更多 prefix family | 连续两个窗口保持 canary 水平 | 同类回归重复出现 |
 | D | 大范围启用 | 观测稳定 | operator 触发回滚 |
 显式回滚条件：
 - 跨页重复对象
 - truncation / continuation 切换后漏对象
 - p99 延时回归 `> +20%`
 - 错误率回归 `> +10%`
 - fallback rate 高于计划阈值
 - index 抽样漂移不一致
 回滚动作：
 - quorum-only 实验：设置 `RUSTFS_LIST_OBJECTS_QUORUM=strict`
 - 未来 index-assisted planner：关闭 index path，强制走 walker
 - token 家族回归：停止发放规划中的 v3 index token，仅保留 legacy / v2 解析兼容
 对 `#787` 的闭环解释：
 - benchmark 证据现在已补齐；
 - 结论是 quorum-only rollout 证据不足；
 - design / rollout 产物已经就位，后续真实缓解方向应转向 index-backed listing。
 ## 7) 风险与回滚
 - 风险：默认 `optimal` 在某些奇异部署下可能导致一致性感知差异。
 - 低风险处置：设置 `RUSTFS_LIST_OBJECTS_QUORUM=strict` 回退。
 - 任何问题：回退本次提交即可恢复原行为。
@@ -1,87 +0,0 @@
 # Issue #713 `>1GiB GET` baseline v1 小结
 ## 1. 执行范围
 本轮是 `#713` 的第一轮 plain baseline，先不碰 encrypted / compressed，只看：
 1. `1GiB`
 2. `2GiB`
 3. `sequential`
 4. `ranged_parallel`
 5. `concurrency=1,4,8`
 6. `range_workers=4`
 7. `rounds=1`
 结果目录：
 1. `rustfs/target/bench/issue713-gt1g-get-baseline-v1-20260625T105149Z`
 ## 2. 结果总表
 ### 2.1 `1GiB`
 `sequential`
 1. `c1`: `99.36 MiB/s`, `10807.0ms`
 2. `c4`: `518.53 MiB/s`, `8267.5ms`
 3. `c8`: `708.04 MiB/s`, `11362.4ms`
 `ranged_parallel`
 1. `c1`: `481.71 MiB/s`, `2229.0ms`
 2. `c4`: `1536.66 MiB/s`, `2728.5ms`
 3. `c8`: `1548.29 MiB/s`, `5010.9ms`
 ### 2.2 `2GiB`
 `sequential`
 1. `c1`: `116.43 MiB/s`, `18444.0ms`
 2. `c4`: `522.06 MiB/s`, `16442.5ms`
 3. `c8`: `712.24 MiB/s`, `21842.1ms`
 `ranged_parallel`
 1. `c1`: `503.28 MiB/s`, `4267.0ms`
 2. `c4`: `1817.59 MiB/s`, `4707.3ms`
 3. `c8`: `1681.50 MiB/s`, `9502.1ms`
 ## 3. 当前第一结论
 本轮 plain baseline 可以先收敛出三点：
 1. `ranged_parallel` 在当前单机多盘场景下，明显优于单流 `sequential`
 2. 这种优势在 `1GiB` 和 `2GiB` 两个点上都成立
 3. 即使 `sequential` 已经过了当前这轮 `>1GiB GET` buffer / readahead 优化，当前仍然没有追平 `4-way ranged_parallel`
 换句话说：
 1. 这轮服务端优化是必要的
 2. 但对当前这组 plain `>1GiB GET` workload 来说，客户端并行 range 下载仍然是更强的模式
 ## 4. 当前推荐
 在这轮 baseline 之后，最自然的下一步是：
 1. 先补一轮更稳妥的确认性复测，只保留重点点位：
   - `1GiB c1`
   - `1GiB c4`
   - `2GiB c1`
   - `2GiB c4`
 2. 如果结论仍稳定，再继续补：
   - larger `range_workers` 对比
   - encrypted `>1GiB GET`
   - compressed `>1GiB GET`
 ## 5. 当前边界说明
 这轮还不能直接下最终结论的地方：
 1. 只有 `1 round`
 2. `ranged_parallel` 当前只测了 `range_workers=4`
 3. 还没有拆 plain / encrypted / compressed
 因此当前更准确的表述是：
 1. plain `>1GiB GET` 的第一轮 baseline 已显示 `ranged_parallel` 明显领先
 2. 但还需要更小面复测来确认这个结论的稳定性
@@ -1,258 +0,0 @@
 # Issue #713 `>1GiB GET` 优化与验证手册
 ## 1. 目标
 本文用于收口 `backlog#713` 当前这一轮 `>1GiB GET` 优化的实现点与验证方式。
 当前这轮改动先解决三件事：
 1. `sequential hint` 真正参与 GET I/O 策略计算
 2. storage media buffer cap 不再被统一的 `1MiB` hard clamp 二次截断
 3. `>1GiB`、非 range、允许 readahead 的 streaming GET，会走更大的 stream buffer 策略
 ## 2. 当前代码行为变化
 ### 2.1 多因素调度
 涉及：
 1. `rustfs/src/storage/concurrency/io_schedule.rs`
 当前行为：
 1. `IoSchedulingContext.is_sequential_hint=true` 时，会优先按 `Sequential` 模式参与 buffer / readahead 决策
 2. storage profile 提供的 media cap 现在可以真实生效
 3. 例如默认常量下：
   - `NVMe`: `2MiB`
   - `SSD`: `1MiB`
   - `HDD`: `512KiB`
 ### 2.2 GET body stream 策略
 涉及：
 1. `rustfs/src/app/object_usecase.rs`
 当前行为：
 1. `optimal_buffer_size` 不再被额外 `min(base_buffer_size)` 压回去
 2. `>1GiB`、非 range、`enable_readahead=true` 的 streaming GET，会命中：
   - `large_sequential_readahead`
 3. 该路径当前把 `ReaderStream` buffer 扩到：
   - `min(optimal_buffer_size * 2, 4MiB)`
 ### 2.3 GET observability
 涉及：
 1. `crates/io-metrics/src/lib.rs`
 新增：
 1. `rustfs_io_get_object_stream_strategy_total{strategy=...}`
 2. `rustfs_io_get_object_stream_buffer_size_bytes{strategy=...}`
 3. `rustfs_io_get_object_stream_response_size_bytes{strategy=...}`
 当前策略标签：
 1. `standard`
 2. `large_sequential_readahead`
 ## 3. 推荐验证顺序
 按下面顺序做最稳妥：
 1. 先准备 plain `1GiB` / `2GiB` 对象
 2. 先跑 `sequential` baseline
 3. 再跑 `ranged_parallel`
 4. 最后再决定是否继续补 encrypted / compressed
 ## 4. 准备测试对象
 直接使用：
 1. `scripts/prepare_gt1g_get_test_objects.sh`
 示例：
 ```bash
 bash scripts/prepare_gt1g_get_test_objects.sh \
  --endpoint http://127.0.0.1:9000 \
  --access-key rustfsadmin \
  --secret-key rustfsadmin \
  --bucket rustfs-bench \
  --prefix bench/issue713/plain
 ```
 默认会准备：
 1. `1GiB=plain-1g.bin`
 2. `2GiB=plain-2g.bin`
 默认最终对象路径：
 1. `s3://rustfs-bench/bench/issue713/plain/plain-1g.bin`
 2. `s3://rustfs-bench/bench/issue713/plain/plain-2g.bin`
 ## 5. 执行 GET 矩阵
 直接使用：
 1. `scripts/run_gt1g_get_http_matrix.sh`
 这个脚本特点：
 1. 不把响应体落到本地磁盘
 2. 直接把 body 丢到 `/dev/null`
 3. 用 `curl` 的 `size_download` 做字节校验和吞吐计算
 4. 同时支持：
   - `sequential`
   - `ranged_parallel`
 补充说明：
 1. 如果本机没有 `mc`，脚本会自动 fallback 到仓库内置的 `rustfs/tests/gt1g_get_benchmark_tool.rs`
 2. fallback 模式下，结果目录会落在：
   - `rustfs/target/bench/...`
 3. 非 fallback 模式下，结果目录仍按脚本参数落在：
   - `target/bench/...`
 ### 5.1 baseline：sequential + ranged_parallel
 ```bash
 bash scripts/run_gt1g_get_http_matrix.sh \
  --endpoint http://127.0.0.1:9000 \
  --access-key rustfsadmin \
  --secret-key rustfsadmin \
  --bucket rustfs-bench \
  --objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
  --modes sequential,ranged_parallel \
  --concurrencies 1,4,8 \
  --range-workers 4 \
  --rounds 3 \
  --cooldown-secs 20
 ```
 ### 5.2 只跑 sequential
 ```bash
 bash scripts/run_gt1g_get_http_matrix.sh \
  --endpoint http://127.0.0.1:9000 \
  --access-key rustfsadmin \
  --secret-key rustfsadmin \
  --bucket rustfs-bench \
  --objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
  --modes sequential \
  --concurrencies 1,4,8 \
  --rounds 3 \
  --cooldown-secs 20
 ```
 ### 5.3 只跑 ranged_parallel
 ```bash
 bash scripts/run_gt1g_get_http_matrix.sh \
  --endpoint http://127.0.0.1:9000 \
  --access-key rustfsadmin \
  --secret-key rustfsadmin \
  --bucket rustfs-bench \
  --objects '1GiB=bench/issue713/plain/plain-1g.bin,2GiB=bench/issue713/plain/plain-2g.bin' \
  --modes ranged_parallel \
  --concurrencies 1,4,8 \
  --range-workers 4,8 \
  --rounds 3 \
  --cooldown-secs 20
 ```
 ## 6. 输出目录
 默认输出：
 ```text
 target/bench/gt1g-get-http-<timestamp>/
 ```
 重点看：
 1. `round_results.csv`
 2. `median_summary.csv`
 3. `logs/`
 `round_results.csv` 字段：
 1. `object_label`
 2. `object_key`
 3. `mode`
 4. `range_workers`
 5. `concurrency`
 6. `round`
 7. `status`
 8. `total_bytes`
 9. `elapsed_ms`
 10. `throughput_bps`
 11. `avg_client_latency_ms`
 ## 7. 当前推荐观察点
 ### 7.1 sequential 是否命中大对象策略
 查询：
 ```promql
 sum by (strategy) (
  increase(rustfs_io_get_object_stream_strategy_total[5m])
 )
 ```
 预期：
 1. `>1GiB` 顺序 GET 应出现 `large_sequential_readahead`
 ### 7.2 GET stream buffer 分布
 ```promql
 histogram_quantile(
  0.95,
  sum by (strategy, le) (
    rate(rustfs_io_get_object_stream_buffer_size_bytes_bucket[5m])
  )
 )
 ```
 ### 7.3 GET 端到端完成时长
 ```promql
 histogram_quantile(
  0.95,
  sum by (le) (
    rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])
  )
 )
 ```
 ## 8. 当前判断建议
 如果 sequential 在 `1GiB/2GiB`、`c1` 下仍然没有明显优于旧结果，优先检查：
 1. `large_sequential_readahead` 是否真的命中
 2. buffer size 是否仍停在 `1MiB`
 3. 请求是否实际是 range / encrypted / compressed，从而没走当前优化路径
 如果 ranged_parallel 明显优于 sequential，再继续补：
 1. 客户端并行分片数建议
 2. plain 与 encrypted/compressed 的拆分对比
 ## 9. 当前边界
 这轮实现还没有直接覆盖：
 1. encrypted `>1GiB GET` 的单独 read-ahead 策略
 2. compressed `>1GiB GET` 的专门 index / range 成本优化
 3. server-side 主动预取缓存
 因此当前更适合先做：
 1. plain sequential vs ranged_parallel 基线
 2. 命中策略与吞吐结果收敛
 3. 再决定下一刀是否继续进 encrypted / compressed
@@ -1,328 +0,0 @@
 # Site Replication Hardening Implementation Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Harden RustFS site replication against credential exposure, unsafe diagnostic authorization, incomplete remove cleanup, and weak operational visibility while preserving MinIO-compatible API behavior.
 **Architecture:** Start with low-risk guardrails that reduce blast radius, then add deterministic cleanup and status surfaces that support later durable replay work. Keep changes local to existing admin, madmin, and bucket replication modules before introducing larger state-machine refactors.
 **Tech Stack:** Rust, Axum admin handlers, serde madmin contracts, ecstore bucket replication metadata, RustFS route policy tests, focused cargo tests.
 ---
 ## File Map
 - `docs/architecture/site-replication-hardening-pr.md`: PR body draft and implementation log.
 - `crates/ecstore/src/bucket/target/bucket_target.rs`: credential redaction helpers and safe `Debug`.
 - `rustfs/src/admin/handlers/replication.rs`: remote target response/log redaction.
 - `rustfs/src/admin/handlers/bucket_meta.rs`: bucket metadata export redaction.
 - `rustfs/src/admin/route_policy.rs`: site replication diagnostic route action mapping.
 - `rustfs/src/admin/handlers/site_replication.rs`: diagnostic authorization, remove cleanup, status detail, and focused tests.
 - `crates/madmin/src/site_replication.rs`: response DTO fields for peer errors and pending operation visibility when needed.
 ## Task 1: Documentation Baseline
 **Files:**
 - Create: `docs/superpowers/plans/2026-06-25-site-replication-hardening.md`
 - Create: `docs/architecture/site-replication-hardening-pr.md`
 - [ ] **Step 1: Write the implementation plan and PR draft**
 Record the phased plan, implementation log, and PR template sections.
 - [ ] **Step 2: Commit**
 ```bash
 git add docs/superpowers/plans/2026-06-25-site-replication-hardening.md docs/architecture/site-replication-hardening-pr.md
 git commit -m "docs: plan site replication hardening"
 ```
 ## Task 2: Redact Site Replication Credentials
 **Files:**
 - Modify: `crates/ecstore/src/bucket/target/bucket_target.rs`
 - Modify: `rustfs/src/admin/handlers/replication.rs`
 - Modify: `rustfs/src/admin/handlers/bucket_meta.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [ ] **Step 1: Add safe credential formatting**
 Implement custom `Debug` for credential-bearing bucket target structures and expose redacted response/export helpers where existing APIs serialize targets.
 - [ ] **Step 2: Replace unsafe logs and admin responses**
 Stop returning or logging `secret_key` in remote target listing and bucket metadata export unless an existing privileged import path requires the raw stored form.
 - [ ] **Step 3: Add focused tests**
 Add assertions that serialized admin/export output and `Debug` output do not contain the configured secret value.
 - [ ] **Step 4: Run focused tests**
 ```bash
 cargo test -p rustfs replication --lib
 cargo test -p rustfs bucket_meta --lib
 cargo test -p rustfs site_replication --lib
 ```
 - [ ] **Step 5: Commit**
 ```bash
 git add crates/ecstore/src/bucket/target/bucket_target.rs rustfs/src/admin/handlers/replication.rs rustfs/src/admin/handlers/bucket_meta.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "fix: redact site replication target secrets"
 ```
 ## Task 3: Tighten Diagnostic Route Authorization
 **Files:**
 - Modify: `rustfs/src/admin/route_policy.rs`
 - Modify: `rustfs/src/admin/handlers/site_replication.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [ ] **Step 1: Move devnull and netperf off read-only info authorization**
 Map diagnostic POST endpoints to `SiteReplicationOperationAction` or a stricter existing action and update handlers to validate the same action.
 - [ ] **Step 2: Add request bounds**
 Ensure diagnostic body or duration inputs cannot be used as unbounded work by a low-privilege caller.
 - [ ] **Step 3: Add route policy tests**
 Assert `devnull` and `netperf` require operation-level permission.
 - [ ] **Step 4: Run focused tests**
 ```bash
 cargo test -p rustfs route_policy --lib
 cargo test -p rustfs site_replication --lib
 ```
 - [ ] **Step 5: Commit**
 ```bash
 git add rustfs/src/admin/route_policy.rs rustfs/src/admin/handlers/site_replication.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "fix: require operation access for replication diagnostics"
 ```
 ## Task 4: Clean Replication Targets on Site Remove
 **Files:**
 - Modify: `rustfs/src/admin/handlers/site_replication.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [ ] **Step 1: Add removed-peer target cleanup**
 When a site is removed, traverse buckets and remove `site-repl-{deployment_id}` rules and matching bucket targets before final state is saved.
 - [ ] **Step 2: Preserve runtime cache consistency**
 After changing `bucket-targets.json`, call existing target refresh paths so replication clients stop using removed peers.
 - [ ] **Step 3: Add focused tests**
 Create a bucket target/rule fixture and assert remove cleanup prunes only the removed deployment.
 - [ ] **Step 4: Run focused tests**
 ```bash
 cargo test -p rustfs site_replication --lib
 ```
 - [ ] **Step 5: Commit**
 ```bash
 git add rustfs/src/admin/handlers/site_replication.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "fix: clean site replication targets on remove"
 ```
 ## Task 5: Improve Status Diagnostics
 **Files:**
 - Modify: `crates/madmin/src/site_replication.rs`
 - Modify: `rustfs/src/admin/handlers/site_replication.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [ ] **Step 1: Expose peer fetch errors**
 Add machine-readable peer error fields to status responses without failing the entire status call.
 - [ ] **Step 2: Expose pending operations**
 Return pending remove or credential rotation state with acked and pending peers.
 - [ ] **Step 3: Add focused tests**
 Assert status includes peer fetch error details and pending operation summaries.
 - [ ] **Step 4: Run focused tests**
 ```bash
 cargo test -p rustfs site_replication --lib
 ```
 - [ ] **Step 5: Commit**
 ```bash
 git add crates/madmin/src/site_replication.rs rustfs/src/admin/handlers/site_replication.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "feat: expose site replication status diagnostics"
 ```
 ## Task 6: Compatibility and Release Verification
 **Files:**
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [ ] **Step 1: Run focused verification**
 ```bash
 cargo fmt --all --check
 cargo test -p rustfs route_policy --lib
 cargo test -p rustfs route_registration_test --lib
 cargo test -p rustfs site_replication --lib
 ```
 - [ ] **Step 2: Run broader gate if feasible**
 ```bash
 make pre-commit
 ```
 - [ ] **Step 3: Record verification and residual risk in PR draft**
 Update the PR draft with commands run, failures, skips, and remaining follow-up items.
 - [ ] **Step 4: Commit**
 ```bash
 git add docs/architecture/site-replication-hardening-pr.md
 git commit -m "docs: finalize site replication hardening notes"
 ```
 ## Task 7: Extend Scope for Full Single-PR Completion
 **Files:**
 - Modify: `docs/superpowers/plans/2026-06-25-site-replication-hardening.md`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [ ] **Step 1: Record the expanded single-PR scope**
 Add follow-up tasks for MinIO wire compatibility, add preflight validation, bootstrap sync, lifecycle compatibility, durable retry, and repair.
 - [ ] **Step 2: Commit**
 ```bash
 git add docs/superpowers/plans/2026-06-25-site-replication-hardening.md docs/architecture/site-replication-hardening-pr.md
 git commit -m "docs: expand site replication hardening scope"
 ```
 ## Task 8: MinIO-Compatible Peer Transport Contract
 **Files:**
 - Modify: `rustfs/src/admin/handlers/site_replication.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [ ] **Step 1: Add peer admin path mapping**
 Route outbound peer requests through MinIO-compatible `/minio/admin/v3/site-replication/...` paths when talking to a MinIO-compatible peer, while preserving RustFS compatibility through the existing alias handling.
 - [ ] **Step 2: Add focused tests**
 Assert peer path mapping preserves the request path used for signing and URL construction.
 - [ ] **Step 3: Commit**
 ```bash
 git add rustfs/src/admin/handlers/site_replication.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "fix: align site replication peer paths with minio"
 ```
 ## Task 9: Add-Time Topology Preflight
 **Files:**
 - Modify: `rustfs/src/admin/handlers/site_replication.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [x] **Step 1: Validate self, deployment identity, IDP, and initial data shape**
 Before persisting add state, fetch remote metainfo and IDP settings, reject duplicate deployment IDs, missing local site, existing site-replication topology gaps, IDP mismatch, and multiple non-empty initial sites.
 - [x] **Step 2: Add focused tests**
 Cover pure topology validation for duplicate deployments, missing self, IDP mismatch, and more than one non-empty site.
 - [x] **Step 3: Commit**
 ```bash
 git add rustfs/src/admin/handlers/site_replication.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "fix: validate site replication add topology"
 ```
 ## Task 10: Full Bootstrap Sync
 **Files:**
 - Modify: `rustfs/src/admin/handlers/site_replication.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [x] **Step 1: Add snapshot bootstrap after add/join**
 Use `build_sr_info` as the canonical local snapshot and sync IAM and bucket metadata to peers after add, before object resync.
 - [x] **Step 2: Add focused tests**
 Cover bootstrap task planning order and item counts without requiring live peers.
 - [x] **Step 3: Commit**
 ```bash
 git add rustfs/src/admin/handlers/site_replication.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "feat: bootstrap site replication metadata on add"
 ```
 ## Task 11: Lifecycle Compatibility
 **Files:**
 - Modify: `rustfs/src/app/bucket_usecase.rs`
 - Modify: `rustfs/src/admin/handlers/site_replication.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [x] **Step 1: Stop default full lifecycle replication**
 Only replicate lifecycle expiry metadata when `replicate_ilm_expiry` is enabled for site replication peers.
 - [x] **Step 2: Add focused tests**
 Assert lifecycle metadata is skipped by default and included only for expiry-enabled peers.
 - [x] **Step 3: Commit**
 ```bash
 git add rustfs/src/app/bucket_usecase.rs rustfs/src/admin/handlers/site_replication.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "fix: align lifecycle replication with minio semantics"
 ```
 ## Task 12: Durable Retry and Repair MVP
 **Files:**
 - Modify: `crates/madmin/src/site_replication.rs`
 - Modify: `rustfs/src/admin/handlers/site_replication.rs`
 - Modify: `docs/architecture/site-replication-hardening-pr.md`
 - [x] **Step 1: Add minimal persistent retry queue**
 Record failed bucket/IAM/state replication events with peer, path, payload, retry count, and last error; expose pending/failed counts in status.
 - [x] **Step 2: Add repair once operation**
 Provide an operation-level admin path that compares local snapshot to peer metainfo and replays missing metadata.
 - [x] **Step 3: Add focused tests**
 Cover queue serialization, retry status summaries, and repair task generation.
 - [x] **Step 4: Commit**
 ```bash
 git add crates/madmin/src/site_replication.rs rustfs/src/admin/handlers/site_replication.rs docs/architecture/site-replication-hardening-pr.md
 git commit -m "feat: add site replication retry and repair MVP"
 ```
@@ -1,105 +0,0 @@
 > **Archived migration snapshot** — moved from `docs/architecture/` (2026-07)
 > when the architecture-review ledger it fed closed out. Kept for history; not
 > maintained.
 # KMS Development Defaults Inventory
 This inventory tracks `KMSD-001` for
 [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660). It records
 current KMS development defaults before any production default hardening.
 ## Scope
 - Source files reviewed: `crates/kms/src/config.rs` and
  `crates/kms/src/api_types.rs`.
 - This is a `docs-only` task.
 - No runtime behavior, config serialization, admin authorization, startup order,
  global state, storage path, or crate boundary changes are included.
 - Follow-up hardening must be done in separate `security-change` PRs with focused
  tests.
 ## Current Defaults
 | Source | Default | Current behavior | Classification |
 |---|---|---|---|
 | `KmsConfig::default()` | Local backend | Uses `LocalConfig::default()` and validates successfully. | dev-only |
 | `LocalConfig::default().key_dir` | OS temp dir plus `rustfs_kms_keys` | Keys are stored under the process temp directory. | dev-only |
 | `LocalConfig::default().master_key` | `None` | Local key files are stored in plaintext when no master key is configured. | invalid for production |
 | `LocalConfig::default().file_permissions` | `0o600` | Owner read/write only for key files. | production-safe as a permission default, but not sufficient without encrypted key storage |
 | `KmsConfig::local(key_dir)` | caller-provided key dir, default local fields | Keeps `master_key = None` unless the caller supplies one later. | dev-only unless explicit encryption material is configured |
 | `KmsConfig::from_env()` local key dir | `./kms_keys` | The env loader builds a relative path, then existing validation rejects it because local key dirs must be absolute. | invalid as a standalone default |
 | `KmsConfig::from_env()` local master key | absent | Leaves `master_key = None`. | invalid for production |
 | `VaultConfig::default().address` | `http://localhost:8200` | HTTP is accepted by validation. | dev-only |
 | `VaultTransitConfig::default().address` | `http://localhost:8200` | HTTP is accepted by validation. | dev-only |
 | `VaultConfig::default().auth_method` | token dev-token | The default token is accepted if used as-is. | invalid for production |
 | `VaultTransitConfig::default().auth_method` | token dev-token | The default token is accepted if used as-is. | invalid for production |
 | `VaultConfig::default().tls` | `None` | No custom TLS settings. HTTPS without custom TLS relies on system CA when no skip flag is set. | production-safe only when HTTPS and system trust are intended |
 | `VaultTransitConfig::default().tls` | `None` | Same TLS behavior as Vault KV2. | production-safe only when HTTPS and system trust are intended |
 | `ConfigureVaultKmsRequest.skip_tls_verify` | omitted means false | `to_kms_config()` leaves TLS config as `None` unless the request explicitly sets true. | production-safe when omitted |
 | `ConfigureVaultTransitKmsRequest.skip_tls_verify` | omitted means false | Same behavior as Vault KV2 configure requests. | production-safe when omitted |
 | `skip_tls_verify = true` in configure requests | explicit insecure opt-in | Creates a TLS config with `skip_verify = true`. | invalid for production |
 ## Existing Validation Boundary
 - Local key directories must be absolute.
 - Timeout and retry attempts must be greater than zero.
 - Vault addresses must use HTTP or HTTPS.
 - Vault mount paths must be non-empty.
 - HTTPS with custom TLS config and verification enabled warns when relying on
  system CA instead of custom CA/client certificates.
 - Existing validation does not fail closed for HTTP Vault addresses, dev-token,
  missing local master key, temp key dirs, or explicit `skip_tls_verify = true`.
 ## Hardening Behavior
 `KMSD-002` makes Local KMS unsafe defaults explicit development opt-ins or
 production failures:
 - no local master key fails validation unless
  `allow_insecure_dev_defaults = true`;
 - local key directories under the process temp directory fail validation unless
  `allow_insecure_dev_defaults = true`;
 - `RUSTFS_KMS_LOCAL_MASTER_KEY` is the production-safe local CLI/env path for
  encrypted local key files;
 - `RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true` is the development-only escape
  hatch for local plaintext or temp-dir setups.
 `KMSD-003` makes Vault unsafe defaults explicit development opt-ins or
 production failures:
 - HTTP Vault addresses fail validation unless explicit development opt-in is set;
 - default `dev-token` credentials fail validation unless explicit development
  opt-in is set;
 - explicit `skip_tls_verify = true` fails validation unless explicit development
  opt-in is set;
 - `RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true` applies to `KmsConfig::from_env`;
 - admin configure requests can set `allow_insecure_dev_defaults = true` for the
  same development-only behavior.
 These checks run in `KmsConfig::validate()` so CLI startup, persisted dynamic
 configuration, service-manager start/reconfigure, and direct backend
 construction use the same fail-closed behavior.
 ## Compatibility Notes
 - Production Local KMS deployments should configure an absolute key directory
  outside the process temp directory and set `RUSTFS_KMS_LOCAL_MASTER_KEY`.
 - Local development setups that intentionally store plaintext key files or use
  temp directories must set `RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true` or the
  admin request field `allow_insecure_dev_defaults = true`.
 - Production Vault deployments should use HTTPS, non-default credentials, and
  TLS verification.
 - Local Vault development setups that intentionally use HTTP, `dev-token`, or
  skip TLS verification must set the same explicit development opt-in.
 - Legacy persisted KMS config JSON remains deserializable; old unsafe persisted
  values default to production mode and fail validation until secured or
  explicitly marked development-only.
 ## Test Coverage
 - Local production fail-closed and development opt-in validation.
 - Vault HTTP, default token, and skip-TLS fail-closed validation plus explicit
  development opt-in.
 - `KmsConfig::from_env()` development default rejection and opt-in behavior.
 - Admin configure request conversion to the same validation behavior.
 - KMS service manager rejects unsafe configs before moving to `Configured`.
@@ -1,99 +0,0 @@
 > **Archived migration snapshot** — moved from `docs/architecture/` (2026-07)
 > when the architecture-review ledger it fed closed out. Kept for history; not
 > maintained.
 # Profiling And NUMA Capability Inventory
 This inventory covers `G-013` for `rustfs/backlog#667`. It records the current
 profiling, memory sampling, allocator, and NUMA baseline before optional runtime
 sidecars are designed.
 ## Platform Support Matrix
 | Capability | Current support | Current owner | Baseline recommendation |
 |---|---|---|---|
 | CPU pprof dump | Linux and macOS builds use `pprof`; other targets return an unsupported-platform error. | `rustfs/src/profiling.rs` | Keep CPU profiling opt-in through existing env flags and cancellation token. |
 | Continuous CPU profiling | Linux and macOS builds can hold a continuous `ProfilerGuard` when enabled. | `rustfs/src/profiling.rs` | Preserve single-guard ownership and avoid starting multiple continuous guards. |
 | Periodic CPU profiling | Linux and macOS builds can spawn a periodic sampling loop. | `rustfs/src/profiling.rs` | Keep the loop cancellation-driven and non-fatal. |
 | Jemalloc memory pprof | Only `linux` + `gnu` + `x86_64` exposes jemalloc pprof dumping. Other supported builds return an unsupported-target error. | `rustfs/src/profiling.rs` | Treat memory pprof as optional and target-gated. |
 | Periodic memory pprof | Only runs where jemalloc profiling control is available and active. | `rustfs/src/profiling.rs` | Keep inactive jemalloc as a skipped dump, not a startup failure. |
 | Process/system memory sampling | Uses `rustfs_io_metrics::snapshot_process_resource_and_system` plus `sysinfo` total memory. | `rustfs/src/memory_observability.rs` | Keep sampling portable and metric-gated. |
 | cgroup memory sampling | Reads Linux cgroup v2 or v1 memory files when present. Missing files produce no cgroup split. | `rustfs/src/memory_observability.rs` | Keep cgroup data opportunistic and absent-safe. |
 | Allocator reclaim | Uses jemalloc backend on `linux` + `gnu` + `x86_64`; otherwise mimalloc variants. | `rustfs/src/allocator_reclaim.rs` | Keep backend detection read-only and preserve effective-force behavior. |
 | eBPF | No runtime eBPF sidecar is currently wired into startup. | N/A | Treat eBPF as future optional Linux-only inventory, never as a required baseline. |
 | NUMA | No NUMA placement or topology controller is currently wired into startup. | N/A | Treat NUMA as future optional capability with no-op fallback. |
 ## Cross-Platform Baseline
 The current safe baseline is:
 - Profiling is opt-in through env flags and must not make startup fatal.
 - Startup and shutdown call profiling through `startup_profiling` lifecycle
  hooks; `profiling.rs` remains the CPU/memory profiling implementation and
  admin dump API owner.
 - Unsupported profiling targets return structured unsupported errors or skip
  startup tasks.
 - Memory observability records process/system metrics and adds cgroup split
  only when cgroup files exist.
 - Allocator reclaim observes active HTTP, delete-tail, scanner, heal, erasure,
  and GET-buffer activity before reclaiming.
 - Runtime thread sizing remains owned by the Tokio runtime builder and sysinfo
  core detection, not NUMA topology.
 ## Optional Sidecar Invariants
 Future sidecars for profiling, eBPF, or NUMA must preserve these invariants:
 - Sidecars must be disabled by default or target-gated until explicitly enabled.
 - Unsupported targets must degrade to no-op status, not panic or fail startup.
 - Sidecars must use the runtime cancellation token or an equivalent explicit
  shutdown handle.
 - Sidecars must not mutate Tokio worker counts after runtime creation.
 - Profiling output directory fallback must stay local to profiling and must not
  affect object storage paths.
 - NUMA fallback must preserve current runtime thread defaults, storage set
  placement, and request admission behavior.
 ## First Implementation Candidates
 `API-013`:
 - Define a read-only capability contract for profiling, cgroup memory, eBPF,
  allocator backend, and NUMA availability.
 - Keep the contract in a low-dependency crate and report unsupported states
  explicitly.
 `R-016`:
 - Wire storage runtime startup to consume capability snapshots read-only.
 - Do not start sidecars or mutate runtime worker ownership in the same PR.
 `X-012`:
 - Define the `ops.profiler.v1` extension schema for profiling capability
  reporting, backend status, redaction requirements, and provenance.
 - Keep the schema capability-only; it must not request profiler execution,
  start sidecars, or change profile export behavior.
 - Keep unsupported targets, disabled sidecars, and unknown future backends
  representable as no-op capability states.
 `X-013`:
 - Add the extension capability snapshot contract for disabled, unsupported, and
  enabled profiler backends.
 - Verify optional profiler sidecar and Wasm runtimes stay disabled by default
  and cannot declare a startup fatal boundary.
 `R-021`:
 - If a runtime service sidecar is added later, enter it through the optional
  runtime boundary with explicit shutdown ownership.
 - Preserve current service order, KMS/audit/notification fatal boundaries, and
  scanner/heal startup semantics.
 `R-022`:
 - Keep optional runtime startup handoff in `startup_optional_runtimes` while
  leaving concrete protocol adapters in `startup_protocols`.
 - Preserve KMS-before-protocol startup ordering and disabled protocol no-op
  behavior.
@@ -1,81 +0,0 @@
 > **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
 > Kept for history; not maintained. File paths inside may reflect the pre-#3929
 > module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
 > `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
 # Rebalance and Decommission Implementation Plan Index
 > This index is based on `docs/architecture/rebalance-decommission-remediation-plan.md`. The remediation scope is intentionally split into smaller implementation plans because the fixes touch independent risk areas: object-version safety, distributed operation semantics, data movement internals, and operational hardening.
 ## Why Split the Work
 The remediation backlog has fourteen fix blocks. Implementing them in one PR would make review risky and would mix unrelated failure modes. The safer path is:
 1. Fix the data semantics that can lose or corrupt object-version meaning.
 2. Fix distributed start/stop/recovery semantics so operators can trust cluster state.
 3. Fix resource and metadata correctness in shared data movement.
 4. Add observability and compatibility hardening.
 Each plan below should be reviewed and executed independently unless the plan explicitly says two fixes must share the same implementation.
 ## Plan Set
 | Plan | Fixes | Status | Purpose |
 | --- | --- | --- | --- |
 | `rebalance-decommission-phase1-safety-plan.md` | F01, F02, F03, F04, F05 | Drafted | Protect object-version semantics and cluster operation safety |
 | `rebalance-decommission-phase2-data-movement-plan.md` | F06, F07, F08, F09, F10 | Drafted | Stream multipart migration, preserve metadata, and improve convergence |
 | `rebalance-decommission-phase3-hardening-plan.md` | F11, F12, F13, F14 | Drafted | Improve cleanup reporting, auditability, metadata decoding, and threshold docs |
 | `rebalance-decommission-followup-review-plan.md` | R01-R16 | Reviewed | Close post-implementation review gaps found after F01-F14 |
 ## Execution Recommendation
 Start with `rebalance-decommission-phase1-safety-plan.md`. Phase 1 contains the highest-risk issues and defines semantics that later fixes depend on.
 Within Phase 1:
 1. F01 should be analyzed first because rebalance delete marker and remote tiered behavior determines whether RustFS should move or skip these versions.
 2. F02 can be implemented independently once the expected decommission overwrite semantics are confirmed.
 3. F03, F04, and F05 can be designed together but should remain separate PRs unless a shared peer-failure helper is introduced.
 For a long-running implementation task, use the phase plans as checkpoints:
 1. Implement one fix block at a time.
 2. Run the focused tests listed in that block.
 3. Review the diff before moving to the next fix block.
 4. Run the phase-level test matrix before considering the phase complete.
 After the initial F01-F14 implementation pass, continue with `rebalance-decommission-followup-review-plan.md`. That follow-up plan is ordered by remaining risk and should be executed before treating the remediation as complete.
 ## Upstream Change Impact Notes
 ### 2026-06-17: `ed55857b refactor: move bucket operations contract (#3507)`
 This upstream change moved `BucketOperations` from `crates/ecstore/src/store_api/traits.rs` into `crates/storage-api/src/bucket.rs` and re-exported it from `rustfs_storage_api`.
 Impact on this remediation plan:
 - No F01-F14 risk item is fixed by this upstream change.
 - No planned fix block needs priority changes because of this upstream change.
 - Implementation code that imports or bounds `BucketOperations` must now use `rustfs_storage_api::BucketOperations`.
 - Generic implementations that need the ECStore error type should use an explicit associated error bound such as `BucketOperations<Error = crate::error::Error>`.
 - The only touched planning-relevant files are import/boundary changes in paths such as `crates/ecstore/src/pools.rs`, `crates/ecstore/src/store.rs`, `crates/ecstore/src/set_disk.rs`, and `rustfs/src/admin/handlers/rebalance.rs`; the rebalance/decommission safety logic remains unchanged.
 ## Review Gates Before Implementation
 Before any code patch starts for a fix block:
 - Confirm the selected behavior in the corresponding plan.
 - Identify the minimal set of files for that fix.
 - Write or update failing tests first.
 - Keep unrelated refactors out of scope.
 - Run focused tests for the touched crate before broader checks.
 ## Verification Baseline
 For any code PR generated from these plans:
 - Run focused `cargo test` commands for touched crates and modules.
 - Run `cargo fmt --all`.
 - Run `cargo fmt --all --check`.
 - Run `make pre-commit` before opening a PR when the implementation is ready.
 - Clean generated build artifacts after build-based verification.
@@ -1,381 +0,0 @@
 > **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
 > Kept for history; not maintained. File paths inside may reflect the pre-#3929
 > module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
 > `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
 # Rebalance and Decommission Phase 1 Safety Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Fix the highest-risk rebalance/decommission safety defects so object-version semantics and distributed operation state cannot silently diverge.
 **Architecture:** Phase 1 keeps the existing ECStore/Rebalance/PoolMeta architecture and introduces narrowly scoped behavior changes. The plan favors fail-closed semantics for unsafe migration, peer propagation, and startup conflicts. Each fix block should be implemented as a separate PR unless a shared helper is explicitly called out.
 **Tech Stack:** Rust, Tokio, ECStore, Axum admin handlers, tonic peer RPC, MessagePack metadata, existing RustFS test modules.
 ---
 ## Scope
 This plan covers:
 - F01: rebalance delete marker and remote tiered version migration semantics.
 - F02: decommission `DataMovementOverwriteErr` cleanup safety.
 - F03: decommission `pool_meta` reload as a start barrier.
 - F04: store startup recovery ordering for pool meta and rebalance.
 - F05: rebalance distributed start/stop semantics and `stopping` visibility.
 This plan does not cover multipart streaming, checksum preservation, lifecycle-expired cleanup, overwrite convergence, cleanup warning UX, audit fields, or metadata decoding hardening. Those belong to later phase plans.
 ## Shared Principles
 - Treat unknown target equivalence as unsafe.
 - Do not clean a source entry unless the target state is proven complete or the version is intentionally skipped by a documented policy.
 - Do not report distributed admin success when required peers failed.
 - Do not auto-run rebalance and decommission together after restart.
 - Prefer small local helpers near existing logic over broad refactors.
 ## F01: Rebalance Delete Marker and Remote Tiered Version Safety
 ### Decision Needed
 Remote tiered versions need a product decision before coding:
 - **Recommended short-term behavior:** Match MinIO and skip remote tiered versions during rebalance. This avoids changing remote-tier metadata during a balancing operation and removes the immediate source-cleanup data-loss risk.
 - **Later behavior:** Implement explicit cross-pool target metadata movement for remote tiered versions only after tests prove remote references, lifecycle state, and cleanup semantics.
 Delete markers should not be skipped. They preserve versioned delete semantics and must be copied to the selected target pool before source cleanup.
 ### Files
 - Modify: `crates/ecstore/src/rebalance.rs`
 - Possibly modify: `crates/ecstore/src/data_movement.rs`
 - Avoid unless necessary: `crates/ecstore/src/set_disk.rs`
 - Test: existing rebalance unit tests in `crates/ecstore/src/rebalance.rs`
 ### Design
 1. Add an ECStore-level helper for rebalance delete marker movement.
   - Input: source pool index, bucket, object name, source `FileInfo`, version ID.
   - Behavior: choose a target pool using the same placement rules as normal data movement, excluding source/rebalancing/decommissioned pools.
   - Write the delete marker metadata to the target pool with the original version ID, mod time, delete marker flag, and replication state.
   - Return success only when the target write succeeds or an equivalent target delete marker is confirmed.
 2. Change the delete marker branch in `migrate_entry_version_with_retry_wait`.
   - Do not call `SetDisks::delete_object_for_migration` for rebalance delete markers.
   - Call the ECStore-level helper via the `transfer`/backend abstraction or split delete marker handling out of the `SetDisks` backend path.
   - Do not set `moved = true` unless the target helper succeeds.
 3. Change remote tiered branch.
   - Short term: return `ignored = true`, `cleanup_ignored = false`, `moved = false` for remote tiered versions so they do not contribute to full source cleanup.
   - If skipping a remote tiered version prevents source cleanup, record a clear status reason so operators understand why the entry remains.
 4. Source cleanup must remain blocked if any version was skipped without cleanup permission.
 ### Implementation Tasks
 - [ ] Write a failing test proving a rebalance delete marker currently does not require target-pool metadata before counting complete.
 - [ ] Write a failing test proving remote tiered versions do not allow source cleanup under the short-term skip policy.
 - [ ] Implement the delete marker target write helper.
 - [ ] Replace the rebalance delete marker source-set branch with the target write helper.
 - [ ] Replace the remote tiered branch with the skip-without-cleanup policy.
 - [ ] Run focused rebalance tests.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore rebalance_delete_marker --lib
 cargo test -p rustfs-ecstore remote_tier --lib
 cargo test -p rustfs-ecstore rebalance_entry --lib
 ```
 ### Acceptance Criteria
 - Rebalance does not clean a source entry when a delete marker target write fails.
 - Rebalance preserves versioned delete behavior after source cleanup.
 - Remote tiered versions are either safely skipped without source cleanup or explicitly moved with verified target metadata.
 - Existing non-delete-marker rebalance behavior remains unchanged.
 ### Risks
 - The existing `MigrationBackend` trait is source-set oriented. If adapting it becomes invasive, prefer an ECStore-specific path in `rebalance_entry` rather than widening the trait for one branch.
 - Skipping remote tiered versions may leave more source entries behind. That is safer than deleting unverified metadata and can be improved in a later PR.
 ---
 ## F02: Decommission `DataMovementOverwriteErr` Cleanup Safety
 ### Decision
 `DataMovementOverwriteErr` must not be cleanup-safe by default. It can count as complete only after explicit target equivalence verification.
 ### Files
 - Modify: `crates/ecstore/src/pools.rs`
 - Possibly modify: `crates/ecstore/src/data_movement.rs`
 - Test: decommission tests in `crates/ecstore/src/pools.rs`
 ### Design
 1. Update delete marker and remote tiered decommission error handling.
   - Keep object-not-found and version-not-found handling as cleanup-safe only when existing semantics justify it.
   - Remove `is_err_data_movement_overwrite(&err)` from branches that set `cleanup_ignored = true`.
 2. Add a narrow equivalence helper only if an existing target version can be inspected cheaply.
   - Compare bucket, object, version ID, delete marker state, ETag where applicable, mod time, and key metadata.
   - If comparison cannot be done reliably in the current code path, treat overwrite as failure for Phase 1.
 3. Preserve source entry on unsafe overwrite.
   - Set `failure = true`.
   - Include a clear error stage in logs/status.
   - Do not increment the decommissioned count.
 ### Implementation Tasks
 - [ ] Write a failing unit test for delete marker decommission where `DataMovementOverwriteErr` must not count complete.
 - [ ] Write a failing unit test for remote tiered decommission where `DataMovementOverwriteErr` must not count complete.
 - [ ] Remove `is_err_data_movement_overwrite` from cleanup-safe conditions in those branches.
 - [ ] Add an explicit helper or comment documenting why overwrite is unsafe without equivalence.
 - [ ] Run focused decommission tests.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore decommission --lib
 cargo test -p rustfs-ecstore DataMovementOverwriteErr --lib
 ```
 ### Acceptance Criteria
 - `DataMovementOverwriteErr` does not set `cleanup_ignored = true` by default.
 - Source cleanup does not run when overwrite equivalence is unknown.
 - Logs make unsafe overwrite distinguishable from not-found cleanup-safe cases.
 ### Risks
 - Tightening this behavior may cause more decommission retries/failures in clusters that previously masked unsafe state. That is intended; status must make the failure actionable.
 ---
 ## F03: Decommission Pool Meta Reload Barrier
 ### Decision
 A successful decommission start must mean all required peers have acknowledged the updated pool meta or the API must return an explicit failure/degraded result.
 ### Files
 - Modify: `crates/ecstore/src/pools.rs`
 - Modify: `crates/ecstore/src/notification_sys.rs`
 - Modify: `rustfs/src/admin/handlers/pools.rs`
 - Possibly modify: `crates/ecstore/src/rpc/peer_rest_client.rs`
 - Possibly modify: `rustfs/src/storage/rpc/node_service.rs`
 ### Design
 1. Change `start_decommission` propagation behavior.
   - After `pool_meta.save`, call `reload_pool_meta`.
   - If peer reload returns an aggregate error, return that error to the admin handler.
   - Do not silently proceed.
 2. Decide rollback behavior.
   - Preferred minimal Phase 1: fail the API and leave persisted `pool_meta` in decommission state only if rollback is unsafe or unavailable; status must show reload failure.
   - Stronger option: persist a reverted pool meta before returning failure. This needs careful lock and persistence review and should not be attempted without tests.
 3. Ensure decommission workers do not spawn after reload failure.
   - If the caller starts workers after `start_decommission` returns, returning `Err` is sufficient.
   - If any background path can resume from persisted metadata after a failed reload, status must clearly show pending/degraded state.
 ### Implementation Tasks
 - [ ] Write a failing test for `start_decommission` with peer reload failure.
 - [ ] Change reload failure from `warn` to returned error.
 - [ ] Update admin handler error mapping so clients do not receive success.
 - [ ] Add status/log context for peer reload failure.
 - [ ] Run focused pool/decommission tests.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore start_decommission --lib
 cargo test -p rustfs-ecstore reload_pool_meta --lib
 cargo test -p rustfs decommission --lib
 ```
 ### Acceptance Criteria
 - Admin start decommission does not return success when peer reload fails.
 - Decommission workers do not start after reload failure through the admin path.
 - Failure output includes enough peer context for operators.
 ### Risks
 - Persisted pool meta may already be updated before reload failure is detected. Avoid adding rollback until the exact persistence safety is reviewed. Fail-closed reporting is the minimum safe fix.
 ---
 ## F04: Store Init Recovery Ordering
 ### Decision
 Store init must load and install pool meta before deciding whether rebalance can auto-start.
 ### Files
 - Modify: `crates/ecstore/src/store/init.rs`
 - Possibly modify: `crates/ecstore/src/rebalance.rs`
 - Possibly modify: `crates/ecstore/src/pools.rs`
 ### Design
 1. Reorder init sequence.
   - Initialize boot time.
   - Load pool meta.
   - Validate and install pool meta.
   - Resolve resumable decommission pools.
   - Load rebalance meta.
   - Start rebalance only if no decommission is active or resumable.
 2. Define conflict behavior.
   - If active decommission and started rebalance metadata coexist, decommission wins for Phase 1.
   - Rebalance should not start.
   - Record or expose a clear deferred/conflict reason if existing status structures can carry it without broad schema changes.
 3. Preserve existing happy path.
   - Clusters with only rebalance metadata should still auto-start rebalance.
   - Clusters with only decommission metadata should still resume decommission after the existing delay.
 ### Implementation Tasks
 - [ ] Write a failing init test for active decommission plus rebalance metadata.
 - [ ] Move pool meta load/validate/install before `load_rebalance_meta` and `start_rebalance`.
 - [ ] Add a helper such as `should_auto_start_rebalance_after_init`.
 - [ ] Ensure decommission resume still uses the installed pool meta.
 - [ ] Run focused init tests.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore init --lib
 cargo test -p rustfs-ecstore rebalance_meta --lib
 cargo test -p rustfs-ecstore decommission --lib
 ```
 ### Acceptance Criteria
 - Rebalance does not start when active decommission is loaded from pool meta.
 - Rebalance still starts when no decommission is active.
 - Decommission resume behavior is unchanged except that it no longer races with an already-started rebalance.
 ### Risks
 - Init ordering can affect existing startup failure behavior. Keep error handling and stage names explicit so operators can identify which stage failed.
 ---
 ## F05: Rebalance Distributed Start/Stop Semantics
 ### Decision
 Admin start/stop must not report plain success for partial cluster state. If stop remains asynchronous, status must expose that distinction.
 ### Files
 - Modify: `rustfs/src/admin/handlers/rebalance.rs`
 - Modify: `rustfs/src/storage/rpc/node_service.rs`
 - Modify: `crates/ecstore/src/notification_sys.rs`
 - Modify: `crates/ecstore/src/rpc/peer_rest_client.rs`
 - Possibly modify: `crates/ecstore/src/rebalance.rs`
 ### Design
 1. Fix peer RPC stop first.
   - `node_service::stop_rebalance` must return `success=false` or gRPC error when `store.stop_rebalance().await` fails.
   - `notification_sys.stop_rebalance` must propagate aggregate peer failure instead of warning and returning `Ok`.
 2. Fix start propagation.
   - Avoid returning success from peer `load_rebalance_meta(start=true)` before local start validation has completed.
   - If fully synchronous worker startup is too invasive, split semantics:
     - metadata loaded;
     - start accepted;
     - worker running observable through status.
   - Admin start should fail when any required peer fails load/start acceptance.
 3. Define rollback/degraded semantics.
   - Preferred Phase 1: return failure on propagation failure and attempt local stop if local rebalance was already started.
   - If rollback fails, return a degraded error and expose status.
 4. Add status language for stop.
   - If cancellation is requested but workers may still be running, status should not claim fully stopped.
   - Minimal approach: add a `stopping` or equivalent field if existing response types allow it.
 ### Implementation Tasks
 - [ ] Write a failing test for peer stop returning success despite local stop error.
 - [ ] Update `node_service::stop_rebalance` to return failure details.
 - [ ] Write a failing test for `notification_sys.stop_rebalance` aggregate peer failure.
 - [ ] Update `notification_sys.stop_rebalance` to return aggregate error.
 - [ ] Write a failing admin start propagation test.
 - [ ] Update admin start to fail or rollback on propagation failure.
 - [ ] Add status distinction for requested stop versus completed stop if response schema permits.
 - [ ] Run focused admin/RPC/rebalance tests.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore stop_rebalance --lib
 cargo test -p rustfs-ecstore load_rebalance_meta --lib
 cargo test -p rustfs rebalance --lib
 ```
 ### Acceptance Criteria
 - Peer stop failure is visible to callers.
 - Admin stop fails or reports degraded state on peer failure.
 - Admin start fails or reports degraded state on peer load/start failure.
 - Stop status does not mislead operators while workers are still winding down.
 ### Risks
 - API response schema changes may affect clients. Prefer backward-compatible additions where possible, and keep error responses compatible with existing admin error handling.
 ---
 ## Phase 1 Test Matrix
 Run these once the individual fix tests pass:
 ```bash
 cargo test -p rustfs-ecstore rebalance --lib
 cargo test -p rustfs-ecstore decommission --lib
 cargo test -p rustfs-ecstore pools --lib
 cargo test -p rustfs rebalance --lib
 cargo test -p rustfs pools --lib
 cargo fmt --all --check
 ```
 Before opening a PR with code changes:
 ```bash
 cargo fmt --all
 cargo fmt --all --check
 make pre-commit
 ```
 After build-based verification, clean generated build artifacts to avoid unnecessary disk usage.
 ## Suggested PR Split
 1. PR 1: F02 only. Smallest fail-closed fix.
 2. PR 2: F04 only. Startup ordering and conflict tests.
 3. PR 3: F03 only. Decommission peer reload barrier.
 4. PR 4: F05 stop semantics first, then start semantics if review scope remains manageable.
 5. PR 5: F01 delete marker fix and remote tiered skip policy.
 F01 is highest risk, but it may need the most design review. F02 and F04 are good first implementation candidates because they reduce safety risk with narrower code changes.
 ## Open Questions Before Coding
 1. For F01 remote tiered versions, should Phase 1 match MinIO and skip them during rebalance, or should RustFS implement cross-pool remote metadata movement now?
 2. For F03 reload failure, should the implementation attempt to rollback persisted pool meta, or fail closed and expose degraded metadata state?
 3. For F05 start propagation, is a backward-compatible degraded error response acceptable, or must the API stay strictly compatible with current success/error shapes?
@@ -1,368 +0,0 @@
 > **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
 > Kept for history; not maintained. File paths inside may reflect the pre-#3929
 > module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
 > `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
 # Rebalance and Decommission Phase 2 Data Movement Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Make shared data movement safe for large objects, metadata-preserving, and resilient to overwrite and cleanup races.
 **Architecture:** Phase 2 keeps `crates/ecstore/src/data_movement.rs` as the shared migration core, but tightens its streaming, metadata, and equivalence behavior. Rebalance and decommission should share small helpers where their safety checks are identical, while keeping operation-specific policy decisions in `rebalance.rs` and `pools.rs`.
 **Tech Stack:** Rust, Tokio async readers, ECStore object APIs, multipart APIs, lifecycle evaluator, filemeta metadata types, existing crate-local unit tests.
 ---
 ## Scope
 This plan covers:
 - F06: stream multipart data movement instead of buffering whole parts.
 - F07: preserve and verify checksum, replication, version purge, and object-lock metadata.
 - F08: align or explicitly define decommission lifecycle-expired cleanup behavior.
 - F09: handle rebalance overwrite races with explicit target equivalence checks.
 - F10: add final source cleanup verification before prefix deletion.
 This plan assumes Phase 1 either has landed or its semantics are stable enough to build on.
 ## Shared Principles
 - Streaming must keep memory bounded by buffer size, not part size.
 - Metadata equivalence should be explicit and testable.
 - A version is complete only when copied, intentionally skipped by policy, or proven equivalent on target.
 - Source cleanup must be the last step and must verify the source version set has not changed.
 - Fail closed when equivalence cannot be proven.
 ## F06: Stream Multipart Data Movement
 ### Decision
 Multipart migration must not allocate a buffer equal to `part.size`. It should stream each part into `put_object_part`.
 ### Files
 - Modify: `crates/ecstore/src/data_movement.rs`
 - Possibly modify: `crates/ecstore/src/store/multipart.rs`
 - Possibly modify: `crates/ecstore/src/store_api/readers.rs`
 - Test: data movement tests in `crates/ecstore/src/data_movement.rs`
 ### Design
 1. Replace per-part `Vec<u8>` allocation.
   - Current behavior reads each part with `read_exact` into `chunk`.
   - New behavior should create a bounded reader over the shared object stream for exactly `part.size` bytes.
 2. Preserve part boundaries.
   - Each `put_object_part` call must receive a reader that ends exactly at the part boundary.
   - If a part stream ends early, return a staged `read_part` error and abort the multipart upload.
 3. Preserve part index behavior.
   - Continue decoding `part.index`.
   - Preserve indexed reader behavior so erasure/indexed metadata remains compatible.
 4. Preserve abort behavior.
   - `abort_multipart_upload` must still run if any part upload or complete call fails before completion is marked.
 ### Implementation Tasks
 - [ ] Add a failing data movement test with a fake large multipart stream that panics or fails if code tries to allocate/read the full part into memory.
 - [ ] Add a test proving part boundary reads consume exactly the configured part sizes.
 - [ ] Introduce a bounded async reader helper for a single multipart part.
 - [ ] Replace `vec![0u8; part.size]` and `read_exact` in multipart migration with the bounded reader helper.
 - [ ] Ensure part upload still wraps data in `PutObjReader` with correct size, actual size, and index.
 - [ ] Keep existing abort-on-error behavior and add a focused test for failure during streamed part upload.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore data_movement --lib
 cargo test -p rustfs-ecstore multipart --lib
 ```
 ### Acceptance Criteria
 - Data movement memory is bounded and not proportional to multipart part size.
 - Multipart part boundaries, sizes, ETags, and indexes remain correct.
 - Failed streamed part upload aborts the temporary multipart upload.
 ### Risks
 - Async reader composition can accidentally over-read into the next part. The part-boundary test is mandatory.
 - If `put_object_part` requires a concrete reader type, introduce the smallest adapter needed rather than rewriting multipart internals.
 ---
 ## F07: Preserve and Verify Full Data Movement Metadata
 ### Decision
 Checksum preservation is a confirmed fix. Replication, version purge, and object-lock behavior must be tested first; production changes should follow only where tests reveal loss or mismatch.
 ### Files
 - Modify: `crates/ecstore/src/data_movement.rs`
 - Possibly modify: `crates/ecstore/src/store_api/types.rs`
 - Possibly modify: `crates/ecstore/src/set_disk.rs`
 - Possibly modify: `crates/filemeta/src/fileinfo.rs`
 - Possibly modify: `crates/filemeta/src/replication.rs`
 - Test: data movement tests in `crates/ecstore/src/data_movement.rs`
 ### Design
 1. Define metadata equivalence for data movement.
   - Required fields: version ID, ETag, size, actual size, mod time, user metadata, storage class, checksum, multipart checksum, replication state, version purge state, object-lock mode/date, legal hold.
   - Use a helper local to data movement tests first. Promote to production only if F09/F10 need it.
 2. Preserve multipart checksum.
   - When source part checksum exists, populate the corresponding `CompletePart` checksum fields.
   - Preserve object-level multipart checksum metadata when complete metadata is written.
 3. Validate replication state preservation.
   - Current code copies `user_defined` and `set_disk` reconstructs `replication_state_internal` from metadata.
   - Add tests to prove this works for replication status and version purge status.
   - If tests fail, extend `ObjectOptions` or write path metadata handling narrowly.
 4. Validate object lock preservation.
   - Current code copies `user_defined`, which should carry object-lock headers.
   - Add tests for retention mode/date and legal hold.
   - If tests fail, fix the metadata copy path without changing normal user copy semantics.
 ### Implementation Tasks
 - [ ] Add a metadata equivalence assertion helper in data movement tests.
 - [ ] Add a failing multipart checksum preservation test.
 - [ ] Populate `CompletePart` checksum fields from source part metadata.
 - [ ] Add single-part checksum preservation test and fix object-level checksum metadata if needed.
 - [ ] Add replication state and version purge state preservation tests.
 - [ ] Add object-lock retention and legal hold preservation tests.
 - [ ] Apply the minimal production fixes required by those tests.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore data_movement --lib
 cargo test -p rustfs-ecstore checksum --lib
 cargo test -p rustfs-ecstore replication --lib
 cargo test -p rustfs-ecstore object_lock --lib
 ```
 ### Acceptance Criteria
 - Migrated single-part and multipart objects preserve checksum behavior.
 - Migrated multipart parts preserve per-part checksum metadata where RustFS stores it.
 - Replication state and version purge state are equivalent after migration.
 - Object-lock retention and legal hold remain unchanged after migration.
 ### Risks
 - Some metadata may be derived rather than stored directly. Tests should compare externally observable object info, not only internal fields.
 - Avoid broad `ObjectOptions` expansion unless current metadata copy cannot preserve a required field.
 ---
 ## F08: Lifecycle-Expired Version Cleanup Semantics
 ### Decision
 RustFS should either align with MinIO by counting lifecycle-expired versions as decommission-complete, or explicitly document and expose retained expired source entries. The recommended behavior is to align with MinIO when lifecycle/object-lock/replication checks say the version is safe to expire.
 ### Files
 - Modify: `crates/ecstore/src/pools.rs`
 - Possibly modify: lifecycle evaluator under `crates/ecstore/src/bucket/lifecycle/`
 - Test: decommission tests in `crates/ecstore/src/pools.rs`
 ### Design
 1. Preserve existing lifecycle safety checks.
   - Do not bypass object-lock or replication constraints.
   - Keep `should_skip_lifecycle_for_data_movement` or equivalent evaluator as the source of truth.
 2. Count safe expired versions as complete for source cleanup.
   - Replace `expired == 0 && decommissioned == total_versions` with semantics that allow `decommissioned + expired == total_versions` when expired versions are safe.
 3. Keep status honest.
   - If a version is retained because expiration is not safe, status should show it as remaining, not completed.
 ### Implementation Tasks
 - [ ] Write a failing test where one migrated version plus one lifecycle-expired version should allow source cleanup.
 - [ ] Write a test where object-lock protected expired-looking version must not allow cleanup.
 - [ ] Write a test where replication-pending version must not allow cleanup.
 - [ ] Update source cleanup predicate to count safe expired versions.
 - [ ] Update counters/status if existing fields distinguish expired from decommissioned.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore decommission --lib
 cargo test -p rustfs-ecstore lifecycle --lib
 cargo test -p rustfs-ecstore object_lock --lib
 ```
 ### Acceptance Criteria
 - Source entry is cleaned when every version is migrated or safely lifecycle-expired.
 - Protected versions still block cleanup.
 - Decommission status remains understandable for migrated, expired, and blocked versions.
 ### Risks
 - Counting expired versions as complete can hide lifecycle evaluator bugs. Tests must include protected negative cases.
 ---
 ## F09: Rebalance Overwrite Race Equivalence
 ### Decision
 Unsafe `DataMovementOverwriteErr` remains a failure. Equivalent target overwrite can be counted complete only after explicit comparison.
 ### Files
 - Modify: `crates/ecstore/src/data_movement.rs`
 - Modify: `crates/ecstore/src/rebalance.rs`
 - Possibly modify: `crates/ecstore/src/store/object.rs`
 - Test: data movement and rebalance tests
 ### Design
 1. Define target equivalence helper.
   - Compare version ID, ETag, size, actual size, mod time, checksum, delete marker state, and selected user metadata.
   - Reuse the test helper from F07 if appropriate, but production helper must be limited to fields needed for safety.
 2. Apply helper only to overwrite race cases.
   - If `DataMovementOverwriteErr` occurs and target pool differs from source pool, inspect target object/version.
   - If equivalent, count migration as complete.
   - If not equivalent or cannot inspect target, return failure.
 3. Keep source-equals-target unsafe.
   - Do not convert source-equals-target into success without independent target proof.
 ### Implementation Tasks
 - [ ] Add a failing rebalance test where equivalent target version exists and overwrite should converge.
 - [ ] Add a failing rebalance test where target version differs and overwrite must fail.
 - [ ] Implement a minimal target equivalence helper.
 - [ ] Wire the helper into data movement overwrite handling for rebalance.
 - [ ] Keep decommission behavior aligned with F02.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore DataMovementOverwriteErr --lib
 cargo test -p rustfs-ecstore rebalance --lib
 cargo test -p rustfs-ecstore data_movement --lib
 ```
 ### Acceptance Criteria
 - Equivalent overwrite does not block rebalance convergence.
 - Non-equivalent overwrite does not clean source.
 - Logs/status distinguish equivalent overwrite from unsafe overwrite.
 ### Risks
 - Comparing too few fields can mark a corrupt target complete. Prefer strict comparison in Phase 2.
 - Comparing too many volatile fields can prevent convergence. Use tests to calibrate.
 ---
 ## F10: Source Cleanup Preflight Verification
 ### Decision
 Before deleting a source prefix, rebalance and decommission should verify the source version set still matches the migrated or intentionally skipped set.
 ### Files
 - Modify: `crates/ecstore/src/rebalance.rs`
 - Modify: `crates/ecstore/src/pools.rs`
 - Possibly modify: `crates/ecstore/src/set_disk.rs`
 - Test: rebalance and decommission cleanup tests
 ### Design
 1. Capture source version identity at scan time.
   - Version identity should include object name, version ID, delete marker flag, and enough metadata to detect a new or changed version.
 2. Re-read before cleanup.
   - Immediately before `delete_prefix`, read current source metadata.
   - Compare current identity set to the expected cleanup set.
 3. Defer on mismatch.
   - Rebalance should defer/retry the entry with a clear last error.
   - Decommission should fail the entry or retry according to existing decommission retry policy.
 4. Keep not-found idempotent.
   - If source entry is already gone, cleanup remains successful.
 ### Implementation Tasks
 - [ ] Add helper to build a stable version identity set from `FileInfoVersions`.
 - [ ] Add rebalance test where source metadata changes between migration and cleanup.
 - [ ] Add decommission test where source metadata changes between migration and cleanup.
 - [ ] Add cleanup preflight before rebalance source `delete_prefix`.
 - [ ] Add cleanup preflight before decommission source `delete_prefix`.
 - [ ] Return deferred/failure status with a clear reason on mismatch.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore rebalance_entry --lib
 cargo test -p rustfs-ecstore decommission_entry --lib
 cargo test -p rustfs-ecstore cleanup --lib
 ```
 ### Acceptance Criteria
 - Cleanup does not delete a source entry if the version set changed after migration started.
 - Already-deleted source remains idempotent.
 - Mismatch errors are visible in status or last-error fields.
 ### Risks
 - Extra metadata reads can add I/O to hot migration paths. Keep this as a cleanup-only check.
 ---
 ## Phase 2 Test Matrix
 Run after individual fix tests pass:
 ```bash
 cargo test -p rustfs-ecstore data_movement --lib
 cargo test -p rustfs-ecstore multipart --lib
 cargo test -p rustfs-ecstore rebalance --lib
 cargo test -p rustfs-ecstore decommission --lib
 cargo test -p rustfs-ecstore lifecycle --lib
 cargo fmt --all --check
 ```
 Before PR:
 ```bash
 cargo fmt --all
 cargo fmt --all --check
 make pre-commit
 ```
 Clean generated build artifacts after build-based verification.
 ## Suggested PR Split
 1. PR 1: F06 streaming multipart migration.
 2. PR 2: F07 checksum and metadata preservation tests/fixes.
 3. PR 3: F08 lifecycle-expired cleanup semantics.
 4. PR 4: F09 overwrite equivalence.
 5. PR 5: F10 cleanup preflight verification.
 F06 and F07 may combine only if checksum preservation requires the new streaming reader shape. F09 should wait for F07 if it depends on metadata equivalence helpers.
 ## Open Questions Before Coding
 1. Which metadata fields are considered mandatory for equivalence in Phase 2: strict internal fields or externally observable fields only?
 2. Should F08 fully match MinIO cleanup behavior, or should RustFS keep expired source entries but expose them as residual state?
 3. Should F10 apply to both rebalance and decommission in the same PR, or should rebalance land first as the higher-risk cleanup path?
@@ -1,325 +0,0 @@
 > **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
 > Kept for history; not maintained. File paths inside may reflect the pre-#3929
 > module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
 > `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
 # Rebalance and Decommission Phase 3 Hardening Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Improve operator visibility, auditability, metadata robustness, and documented compatibility after the Phase 1 and Phase 2 safety fixes.
 **Architecture:** Phase 3 should not change core data movement semantics unless a hardening test exposes a safety bug. It adds bounded status data, structured logs, metadata validation tests, and an explicit decision around rebalance completion tolerance.
 **Tech Stack:** Rust, tracing structured logs, admin status DTOs, serde/rmp metadata decoding, existing guardrail scripts, crate-local unit tests.
 ---
 ## Scope
 This plan covers:
 - F11: improve rebalance cleanup failure reporting.
 - F12: add structured audit fields for rebalance/decommission operations.
 - F13: harden persisted rebalance and pool metadata decoding.
 - F14: decide and document rebalance completion threshold behavior.
 This plan assumes the safety semantics from Phase 1 and the data movement behavior from Phase 2 are stable.
 ## Shared Principles
 - Hardening must not hide safety failures.
 - Status fields should be bounded to avoid unbounded metadata growth.
 - Logs must be structured, searchable, and free of secrets.
 - Metadata compatibility must be explicit: strict where possible, legacy-compatible where necessary.
 ## F11: Rebalance Cleanup Failure Reporting
 ### Decision
 Rebalance cleanup failures should not be invisible completion details. Admin status should expose whether completion had cleanup warnings and provide bounded object-level detail.
 ### Files
 - Modify: `crates/ecstore/src/rebalance.rs`
 - Modify: `rustfs/src/admin/handlers/rebalance.rs`
 - Possibly modify: admin response DTOs in the same module or related madmin types
 - Test: rebalance status and metadata tests
 ### Design
 1. Extend cleanup warning metadata.
   - Keep existing count.
   - Add a bounded list of recent cleanup failures, for example last 10 entries.
   - Each entry should include bucket, object, message, and timestamp.
 2. Keep metadata bounded.
   - Do not store every cleanup failure unboundedly in `rebalance.bin`.
   - Use a ring-buffer style helper or truncate older entries.
 3. Make terminal state visible.
   - If rebalance completes with cleanup warnings, status should show warning count and details.
   - Avoid reporting a clean completion when source cleanup failed.
 ### Implementation Tasks
 - [ ] Add a cleanup warning entry struct with bounded retention.
 - [ ] Add metadata compatibility defaults for older `rebalance.bin` without the new list.
 - [ ] Update `record_rebalance_cleanup_warning_in_meta` to append bounded entries and update count/last fields.
 - [ ] Update admin status serialization to expose warning count and entries.
 - [ ] Add tests for one warning, multiple warnings, and bounded truncation.
 - [ ] Add legacy metadata decode test.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore cleanup_warning --lib
 cargo test -p rustfs-ecstore rebalance_meta --lib
 cargo test -p rustfs rebalance --lib
 ```
 ### Acceptance Criteria
 - Multiple cleanup failures are visible through count and bounded details.
 - Legacy rebalance metadata still decodes.
 - Status distinguishes clean completion from completion with cleanup warnings.
 ### Risks
 - Adding status fields may affect clients if response schemas are strict. Prefer additive fields with defaults.
 ---
 ## F12: Structured Audit Fields
 ### Decision
 Rebalance and decommission admin operations should have consistent structured logs for success, rejection, propagation failure, and partial/degraded state.
 ### Files
 - Modify: `rustfs/src/admin/handlers/rebalance.rs`
 - Modify: `rustfs/src/admin/handlers/pools.rs`
 - Modify: `crates/ecstore/src/notification_sys.rs`
 - Modify: `crates/ecstore/src/rpc/peer_rest_client.rs`
 - Possibly modify: `scripts/check_logging_guardrails.sh` or related guardrails if logging rules require updates
 ### Design
 1. Define standard fields.
   - `event`
   - `component`
   - `subsystem`
   - `action`
   - `result`
   - `request_id` when available
   - `actor` or masked access key when available
   - `remote_addr` when available
   - `pool_indices` or `rebalance_id`
   - `peer`
   - `error`
 2. Apply to admin handlers.
   - Start, stop, cancel, status-affecting operations.
   - Authorization failures should remain handled by existing auth paths but should have enough context if already logged.
 3. Apply to peer propagation.
   - Peer success and failure should use stable event names.
   - Partial failure must be warn/error, not info-only.
 4. Avoid secret leakage.
   - Never log raw credentials, signatures, tokens, or full auth headers.
 ### Implementation Tasks
 - [ ] Inventory existing rebalance/decommission log events and field names.
 - [ ] Define a small local helper or convention for masked actor/request fields if one already exists.
 - [ ] Update rebalance admin start/stop/status logs.
 - [ ] Update decommission start/cancel/status logs.
 - [ ] Update notification peer propagation failure logs.
 - [ ] Add log-capture tests for success, partial failure, and rejected operation where feasible.
 - [ ] Run logging guardrail script if the repository has one.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs rebalance --lib
 cargo test -p rustfs pools --lib
 cargo test -p rustfs-ecstore notification --lib
 scripts/check_logging_guardrails.sh
 ```
 ### Acceptance Criteria
 - Operators can answer who requested an operation, what resource it targeted, and which peers failed.
 - Partial failures are searchable through stable structured fields.
 - Logs contain no secrets.
 ### Risks
 - Logging changes can be noisy. Keep high-volume per-object logs out of admin audit events.
 ---
 ## F13: Persisted Metadata Decode Hardening
 ### Decision
 Persisted rebalance and pool metadata should reject or surface unknown/corrupt fields where compatibility allows. Legacy compatibility must be tested explicitly.
 ### Files
 - Modify: `crates/ecstore/src/rebalance.rs`
 - Modify: `crates/ecstore/src/pools.rs`
 - Possibly add fixtures under an existing test fixture directory if one exists
 - Test: metadata decode tests in `rebalance.rs` and `pools.rs`
 ### Design
 1. Identify persisted structs.
   - Rebalance metadata and stats structs.
   - Pool metadata and decommission info structs.
 2. Choose strictness per struct.
   - Use strict unknown-field rejection where metadata is not expected to carry forward-compatible fields.
   - For legacy-compatible structs, keep decode lenient but log or validate unknown/unsupported state where possible.
 3. Add state validation.
   - Reject conflicting terminal states such as complete plus canceled plus running.
   - Reject impossible counters or invalid pool indices where existing validation can catch them.
 4. Preserve legacy fixtures.
   - Existing old metadata must still decode if compatibility is required.
 ### Implementation Tasks
 - [ ] List all rebalance and pool metadata structs currently deserialized from persisted bytes.
 - [ ] Add fixture tests for unknown fields, missing critical fields, and conflicting terminal states.
 - [ ] Add strict serde attributes only where fixtures show compatibility is safe.
 - [ ] Add post-decode validation helpers for state conflicts.
 - [ ] Update load paths to surface validation errors with actionable messages.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore rebalance_meta --lib
 cargo test -p rustfs-ecstore pool_meta --lib
 cargo test -p rustfs-ecstore metadata --lib
 ```
 ### Acceptance Criteria
 - Unknown/corrupt metadata does not silently become an apparently valid operation state.
 - Legacy metadata fixtures continue to decode through documented paths.
 - Conflicting terminal state is rejected or quarantined.
 ### Risks
 - Strict decode can block startup if old metadata contains benign extra fields. Start with tests and post-decode validation before broad strict attributes.
 ---
 ## F14: Rebalance Completion Threshold Decision
 ### Decision
 RustFS must explicitly choose whether to match MinIO's tolerance-based rebalance completion or keep strict completion. The recommended default is to match MinIO unless RustFS has a documented reason to move more data.
 ### Files
 - Modify: `crates/ecstore/src/rebalance.rs`
 - Possibly modify: admin docs if present
 - Test: rebalance goal/completion tests in `rebalance.rs`
 ### Design Options
 1. **Match MinIO tolerance.**
   - Complete when pool free-space ratio is within the MinIO-like tolerance of the goal.
   - Reduces movement and operational time.
   - Best for compatibility.
 2. **Keep strict behavior and document it.**
   - No code behavior change.
   - Add tests proving strict behavior is intentional.
   - Status/docs should avoid implying MinIO-compatible tolerance.
 3. **Configurable tolerance.**
   - Most flexible but not recommended unless operators need it.
   - Adds configuration and support burden.
 ### Recommended Plan
 Start with option 1 if compatibility is the goal. If maintainers prefer strictness, implement option 2 with explicit tests and documentation.
 ### Implementation Tasks
 - [ ] Add tests that capture current strict behavior around the completion goal.
 - [ ] Add tests for MinIO-like tolerance behavior.
 - [ ] Confirm maintainers choose MinIO-compatible or strict behavior.
 - [ ] Implement the selected threshold behavior.
 - [ ] Update status/docs comments so completion semantics are clear.
 ### Focused Test Commands
 ```bash
 cargo test -p rustfs-ecstore rebalance_goal --lib
 cargo test -p rustfs-ecstore check_if_rebalance_done --lib
 cargo test -p rustfs-ecstore rebalance --lib
 ```
 ### Acceptance Criteria
 - Completion threshold behavior is covered by deterministic tests.
 - Operators can understand whether RustFS matches MinIO tolerance.
 - Empty-queue completion path remains valid.
 ### Risks
 - Changing threshold may alter operational expectations for existing users. Document the change if behavior changes.
 ---
 ## Phase 3 Test Matrix
 Run after individual fix tests pass:
 ```bash
 cargo test -p rustfs-ecstore rebalance_meta --lib
 cargo test -p rustfs-ecstore pool_meta --lib
 cargo test -p rustfs-ecstore rebalance --lib
 cargo test -p rustfs rebalance --lib
 cargo test -p rustfs pools --lib
 cargo fmt --all --check
 ```
 If logging guardrails are relevant:
 ```bash
 scripts/check_logging_guardrails.sh
 ```
 Before PR:
 ```bash
 cargo fmt --all
 cargo fmt --all --check
 make pre-commit
 ```
 Clean generated build artifacts after build-based verification.
 ## Suggested PR Split
 1. PR 1: F11 cleanup warning status.
 2. PR 2: F12 structured audit fields.
 3. PR 3: F13 metadata decode hardening.
 4. PR 4: F14 completion threshold decision and tests.
 F11 may depend on Phase 2 cleanup semantics if F10 changes retry behavior. F12 can be done anytime after Phase 1 defines distributed failure semantics.
 ## Open Questions Before Coding
 1. What is the maximum number of cleanup warning entries to keep in persisted rebalance metadata?
 2. Which actor identity is safe and useful to log for admin requests?
 3. Should metadata unknown fields fail startup, warn and continue, or quarantine only the affected operation?
 4. Should RustFS match MinIO's rebalance completion tolerance by default?
@@ -1,137 +0,0 @@
 > **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
 > Kept for history; not maintained. File paths inside may reflect the pre-#3929
 > module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
 > `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
 # Rebalance and Decommission Post-Remediation Review
 > **Status:** Updated for branch `cxymds/rebalance-decommission-remediation`.
 >
 > **Scope:** Rebalance, decommission, shared data-movement helpers, source cleanup,
 > delete-marker handling, lifecycle expiry, restart/cancel recovery, and CI coverage.
 >
 > **Conclusion:** The main code-level P1/P2 remediation set has been implemented on
 > this branch. The remaining release gate is verification, not another known
 > correctness rewrite: full CI must pass, `make pre-commit` must pass before the
 > final PR state, and the data-movement e2e proof added to CI must complete in the
 > GitHub runner environment.
 ## Review Corrections
 The previous report needed two important corrections.
 First, the R28 decommission permit issue is real but was overstated. The worker
 permit was released before entry migration work, so `Workers` did not account for
 entry lifetime and `wk.wait()` could not prove entry work had drained. However,
 the current listing path awaits entry callbacks inline and bucket processing is
 bounded, so the old "unbounded object movement" wording was not supported by the
 code. The fix remains correct because permit lifetime now matches entry lifetime
 and prevents future spawn-based regressions.
 Second, SSE-C migration is related to raw data movement but has a distinct
 failure mode. Before the raw migration path, SSE-C objects did not silently
 corrupt through an empty `HeaderMap`; migration reads failed while resolving
 SSE-C material because the customer SSE-C headers were missing. The raw internal
 read path avoids requiring request customer headers for data movement.
 ## Implemented Remediation
 | Area | Current Branch State | Representative Guards |
 | --- | --- | --- |
 | Raw migration reads | Rebalance and decommission use raw data-movement read options instead of normal transformed GET semantics. | `test_rebalance_object_migration_read_opts_are_raw_data_movement`, `test_decommission_object_migration_read_opts_are_raw_data_movement` |
 | Raw write invariants | Data-movement writes preserve source version, ETag, tags, expires, part metadata, and checksums consistently. Single-part raw ETag validation no longer rejects preserved source ETags. | `test_data_movement_put_object_opts_preserves_version_and_etag`, `test_data_movement_opts_preserve_tags_and_expires`, `test_data_movement_single_part_raw_reader_does_not_validate_source_etag` |
 | Terminal decommission recovery | Cancel/failed/complete paths cancel in-memory workers before terminal persistence, and terminal-save failure no longer leaves ghost cancelers. | decommission canceler unit tests in `crates/ecstore/src/pools.rs` |
 | Rebalance operation identity | Rebalance merge/save/stop paths are id-aware and do not let stale operation state clobber a newer operation. | rebalance id-gate and merge tests |
 | Start coordination | Rebalance and decommission starts cross-check persisted peer operation state before starting conflicting work. | start cross-check tests |
 | Decommission worker accounting | Entry permits are held through entry completion. R28 is no longer a pending worker-accounting bug. | decommission worker permit tests |
 | Rebalance rollback | Failed start rollback finalizes metadata instead of leaving `Started` plus `stopping=true` without a worker. | rollback tests |
 | Decommission progress save | Periodic progress-save errors are best-effort; terminal saves remain strict. | progress-save tests |
 | Queue semantics | Multi-pool decommission is queued, local-leader prefix scheduling is supported, completed restart is rejected, completed queue prefixes can advance, failed/canceled entries block automatic promotion, failed/canceled retry preserves bucket progress, and promoted queued pools are canceled if cancellation arrives before work starts. | queue/promotion/retry tests in `pools.rs` |
 | Admin status and query hardening | Pool admin status exposes queued/progress state, dangerous pool/rebalance mutation queries reject unknown or ambiguous parameters, and pool status `by-id` parsing no longer falls back to pool 0. | `admin_pool_list_item`, `pools_handler_tests`, `rebalance_handler_tests` |
 | Cancel/clear operations | Non-leader cancel intent is accepted and failed/canceled terminal decommission can be explicitly cleared. | remote cancel and clear tests |
 | Resume equivalence | Delete-marker replication state, tiered-object metadata, multipart part numbers, tags, expires, and version counts are covered more strictly. | data-movement equivalence tests |
 | Lifecycle expiry | Data movement no longer treats failed lifecycle expiry application as a successful skip. | `resolve_data_movement_lifecycle_expiry_result_rejects_apply_failure` |
 | CI coverage | Existing RustFS e2e delete-marker migration proof is wired into the `e2e-tests` CI job with `--test-threads=1`, reusing the downloaded debug binary. | `.github/workflows/ci.yml` |
 ## Compatibility Decisions
 The current RustFS decommission contract is no longer single-pool only. Multi-pool
 requests are supported as queued operations on multi-pool deployments. The
 current behavior is documented in
 `docs/architecture/decommission-compatibility.md`.
 Delete-marker behavior is intentionally characterized rather than guessed:
 - a lone delete marker without replication is cleanup-only metadata and can be
  skipped;
 - delete markers with replication metadata remain eligible for movement;
 - rebalance and decommission share the same predicate.
 Lifecycle-expired versions are also explicit:
 - decommission can count safely expired versions toward source cleanup only when
  lifecycle expiry application succeeds;
 - rebalance remains stricter and requires actual data-movement completion for
  cleanup.
 ## Remaining Risks and Gates
 There are no remaining known P1/P2 code changes in this remediation set, but the
 branch is not final-release-ready until verification completes:
 1. `make pre-commit` must pass before the PR is marked ready.
 2. GitHub CI must pass, including the newly wired delete-marker migration e2e
   proof.
 3. Local full e2e verification was attempted but the machine ran out of disk
   space while building `rustfs`; this is an environment blocker, not a test
   assertion failure. The `target/` directory was cleaned afterward.
 4. If product requires stronger end-to-end proof for encrypted and compressed
   migration, add those scenarios to `crates/e2e_test` before release. Current
   branch coverage is strongest at the internal option/invariant layer, with CI
   e2e coverage focused on versioning/delete-marker semantics.
 ## Focused Verification Run During Implementation
 Focused checks were run per task, including:
 ```bash
 cargo test -p rustfs-ecstore data_movement_single_part_raw_reader --lib
 cargo test -p rustfs-ecstore test_merge_rebalance_meta_preserves_stopping_stop_snapshot --lib
 cargo test -p rustfs-ecstore test_merge_rebalance_pool_stats_clears_stopping_for_terminal_status --lib
 cargo test -p rustfs-ecstore first_resumable_decommission_queue_indices --lib
 cargo test -p rustfs-ecstore test_return_resumable_pools_skips_failed_decommission --lib
 cargo test -p rustfs-ecstore resolve_data_movement_lifecycle_expiry_result --lib
 cargo test -p rustfs-ecstore lifecycle_action_removes_data_movement_version --lib
 cargo test -p rustfs-ecstore test_pool_meta_promoted_queued_decommission_can_be_canceled --lib
 cargo test -p rustfs admin_pool_list_item --lib
 cargo test -p rustfs pools_handler_tests --lib
 cargo test -p rustfs rebalance_handler_tests --lib
 cargo test -p rustfs admin_query_pool_status_by_id --lib
 cargo fmt --all --check
 ```
 The attempted local e2e command was:
 ```bash
 cargo test -p e2e_test delete_marker_migration_semantics -- --nocapture
 ```
 It failed because the local filesystem reached `No space left on device` while
 building the RustFS debug binary. CI now runs the proof in the e2e job after the
 debug binary artifact has been downloaded, and uses `--test-threads=1` to avoid
 parallel test-server startup for that proof.
 ## Final PR Gate
 Before marking the draft PR ready:
 ```bash
 cargo fmt --all
 cargo fmt --all --check
 make pre-commit
 ```
 If CI reports failures after the branch is pushed, inspect the failing job logs
 instead of weakening the e2e gate. The delete-marker migration proof was added
 because skipping it would leave the exact versioning regression class invisible
 to CI.
@@ -1,551 +0,0 @@
 > **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
 > Kept for history; not maintained. File paths inside may reflect the pre-#3929
 > module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
 > `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
 # Rebalance and Decommission Remediation Plan
 > This document turns `docs/architecture/expert-review-analysis.md` into an actionable remediation backlog. It is intentionally split into independent fix blocks so each item can be analyzed, assigned, implemented, and verified separately.
 ## Scope
 This plan covers the confirmed and calibrated issues from the rebalance/decommission expert review:
 - Critical and high-risk data integrity issues.
 - Distributed state propagation and recovery semantics.
 - Data movement resource usage and metadata preservation.
 - Operational visibility, auditability, and hardening gaps.
 This file is not a code patch. Each block below should be expanded into a focused implementation plan before code changes begin.
 ## Priority Order
 | Order | Fix ID | Source | Priority | Main Risk |
 | --- | --- | --- | --- | --- |
 | 1 | F01 | P1.1 | Critical | Rebalance may drop delete marker or remote tiered metadata |
 | 2 | F02 | P1.5 | High | Decommission may treat unsafe overwrite as cleanup-safe |
 | 3 | F03 | P1.4 | High | Retiring pool may remain writable on stale peers |
 | 4 | F04 | P1.2 | High | Rebalance and decommission may both resume after restart |
 | 5 | F05 | P1.3, A4 | High | Admin start/stop may report success for partial cluster state |
 | 6 | F06 | P2.1 | High | Multipart migration can allocate whole parts in memory |
 | 7 | F07 | P2.2, A3 | Medium | Migrated objects may lose checksum or other metadata semantics |
 | 8 | F08 | P2.4 | Medium | Decommission leaves lifecycle-expired source entries behind |
 | 9 | F09 | P2.3 | Medium | Rebalance overwrite races may fail instead of converging |
 | 10 | F10 | A2 | Medium | Source cleanup lacks a final version-set guard |
 | 11 | F11 | A1 | Medium | Rebalance cleanup failures are completion warnings only |
 | 12 | F12 | P3.2 | Low | Admin operations are not sufficiently auditable |
 | 13 | F13 | P3.3 | Low | Persisted metadata accepts unknown or inconsistent fields |
 | 14 | F14 | P3.4 | Low | Rebalance completion threshold differs from MinIO |
 ## Phase 1: Data Integrity and Safety
 ### F01: Fix rebalance delete marker and remote tiered version migration
 **Source:** P1.1
 **Decision:** Confirmed fix required.
 **Target priority:** Critical.
 **Problem:** `rebalance_entry` marks delete markers and remote tiered versions as moved, but those branches operate through the source `SetDisks` path rather than a confirmed cross-pool target write. Source cleanup can then remove the only metadata that preserves tombstone or tiered-object semantics.
 **Primary files:**
 - `crates/ecstore/src/rebalance.rs`
 - `crates/ecstore/src/data_movement.rs`
 - `crates/ecstore/src/set_disk.rs`
 - Tests near existing rebalance migration tests in `crates/ecstore/src/rebalance.rs`
 **Preferred fix direction:**
 - Route rebalance delete marker movement through an `ECStore`-level data movement path that chooses a non-source target pool and writes the exact version metadata to that target.
 - For remote tiered versions, either:
  - align with MinIO and skip tiered versions during rebalance, or
  - implement a target-pool metadata move that preserves the remote tier pointer and lifecycle state.
 - Do not count the version as rebalanced until the target metadata is confirmed.
 **Acceptance criteria:**
 - A versioned object with a delete marker remains deleted after rebalance source cleanup.
 - A remote tiered version remains readable or correctly listed after rebalance source cleanup.
 - Rebalance does not mark delete marker or tiered versions as complete when target metadata was not written.
 **Required tests:**
 - Multi-pool rebalance test for a versioned object with a delete marker.
 - Multi-pool rebalance test for a remote tiered version.
 - Failure test where target metadata write fails and source cleanup must not happen.
 **Dependencies:** None. This is the first safety fix.
 ---
 ### F02: Stop treating `DataMovementOverwriteErr` as cleanup-safe in decommission
 **Source:** P1.5
 **Decision:** Confirmed fix required.
 **Target priority:** High.
 **Problem:** Decommission treats `DataMovementOverwriteErr` like object-not-found/version-not-found and sets `cleanup_ignored = true`. That error only means source and destination pool are the same; it does not prove an equivalent target version exists.
 **Primary files:**
 - `crates/ecstore/src/pools.rs`
 - `crates/ecstore/src/data_movement.rs`
 - `crates/ecstore/src/store/object.rs`
 **Preferred fix direction:**
 - Remove `DataMovementOverwriteErr` from cleanup-safe branches for delete marker and remote tiered decommission paths.
 - If overwrite occurs because an equivalent target version already exists, perform an explicit equivalence check before counting the version complete.
 - Otherwise return a migration failure and keep the source entry.
 **Acceptance criteria:**
 - `DataMovementOverwriteErr` does not increment the decommissioned count unless equivalence is proven.
 - Source cleanup is blocked when target equivalence is unknown.
 - Logs/status clearly state whether overwrite was equivalent-complete or unsafe.
 **Required tests:**
 - Delete marker decommission returns `DataMovementOverwriteErr` and does not clean source.
 - Remote tiered decommission returns `DataMovementOverwriteErr` and does not clean source.
 - Equivalent target version exists; overwrite can be counted complete only after metadata equality check passes.
 **Dependencies:** F01 clarifies remote/tombstone movement semantics, but F02 can be implemented independently for decommission.
 ---
 ### F03: Make decommission `pool_meta` reload a cluster-wide start barrier
 **Source:** P1.4
 **Decision:** Confirmed fix required.
 **Target priority:** High.
 **Problem:** `start_decommission` saves pool meta and then calls peer reload. Reload failure is logged but the operation still returns success, leaving stale peers able to write into the retiring pool.
 **Primary files:**
 - `crates/ecstore/src/pools.rs`
 - `crates/ecstore/src/notification_sys.rs`
 - `crates/ecstore/src/rpc/peer_rest_client.rs`
 - `rustfs/src/admin/handlers/pools.rs`
 - `rustfs/src/storage/rpc/node_service.rs`
 **Preferred fix direction:**
 - Treat reload failure as a failed start unless the system has an explicit partial/degraded operation state.
 - Return peer failure information to the admin handler.
 - Do not spawn or resume decommission workers until required peers acknowledge the updated pool meta.
 **Acceptance criteria:**
 - Admin decommission start fails or returns a clearly degraded response when peer reload fails.
 - No peer can select the retiring pool for new writes after decommission start is reported successful.
 - Status exposes peer reload failure details.
 **Required tests:**
 - Unit test for `start_decommission` with mocked `reload_pool_meta` failure.
 - RPC/admin test showing start does not return plain success on peer failure.
 - Write-placement test verifying retiring pools are skipped only after successful cluster reload.
 **Dependencies:** None.
 ---
 ### F04: Reorder store init recovery to restore pool meta before rebalance
 **Source:** P1.2
 **Decision:** Confirmed fix required.
 **Target priority:** High.
 **Problem:** Store init loads and starts rebalance before loading persisted pool meta. If unfinished decommission exists on disk, `start_rebalance` can miss it and both processes may resume.
 **Primary files:**
 - `crates/ecstore/src/store/init.rs`
 - `crates/ecstore/src/rebalance.rs`
 - `crates/ecstore/src/pools.rs`
 **Preferred fix direction:**
 - Load and install `PoolMeta` before attempting to start rebalance.
 - After pool meta is loaded, explicitly decide:
  - if decommission is active, do not auto-start rebalance;
  - if rebalance metadata exists but conflicts with decommission, mark rebalance as blocked/deferred and expose it in status.
 - Keep existing decommission resume delay, but ensure rebalance is not already running when decommission resumes.
 **Acceptance criteria:**
 - Restart with active decommission metadata and rebalance metadata does not run both workers.
 - Startup status explains which operation was deferred and why.
 - Existing startup without decommission still resumes rebalance normally.
 **Required tests:**
 - Init test with persisted active decommission plus started rebalance metadata.
 - Init test with only rebalance metadata still resumes rebalance.
 - Init test with corrupt or missing pool meta follows existing error policy.
 **Dependencies:** None, but coordinate with F05 status semantics.
 ---
 ### F05: Make rebalance start/stop distributed semantics explicit and fail-safe
 **Source:** P1.3, A4
 **Decision:** Confirmed fix required.
 **Target priority:** High.
 **Problem:** Rebalance start can return success even when peer propagation fails or peer background start later fails. Stop can return success while peer stop failed or local stop errors were ignored. Stop is also asynchronous, but admin semantics do not clearly expose `stopping` versus `stopped`.
 **Primary files:**
 - `rustfs/src/admin/handlers/rebalance.rs`
 - `rustfs/src/storage/rpc/node_service.rs`
 - `crates/ecstore/src/notification_sys.rs`
 - `crates/ecstore/src/rpc/peer_rest_client.rs`
 - `crates/ecstore/src/rebalance.rs`
 **Preferred fix direction:**
 - For start:
  - propagate rebalance metadata to peers before returning success;
  - have peer RPC synchronously validate/start or return a structured failure;
  - on partial failure, rollback local start or report a degraded state.
 - For stop:
  - never ignore `store.stop_rebalance()` errors;
  - aggregate peer stop failures and return them to admin;
  - expose `stop_requested`, `stopping`, and `stopped` in status if worker shutdown remains asynchronous.
 **Acceptance criteria:**
 - Admin start does not return plain success when any required peer fails to load/start.
 - Admin stop does not return plain success when any required peer fails to stop.
 - Status can distinguish a requested stop from all workers fully stopped.
 **Required tests:**
 - Peer `load_rebalance_meta(true)` failure.
 - Peer background `start_rebalance` failure.
 - Peer `stop_rebalance` local save failure.
 - Stop requested while a worker is in a long migration operation.
 **Dependencies:** Coordinate with F04 so startup and admin semantics use the same conflict model.
 ---
 ## Phase 2: Data Movement Correctness and Resource Control
 ### F06: Stream multipart data movement instead of buffering whole parts
 **Source:** P2.1
 **Decision:** Confirmed fix required.
 **Target priority:** High.
 **Problem:** Multipart migration allocates `Vec<u8>` for each part and reads the whole part into memory. Large parts plus concurrent workers can OOM the process.
 **Primary files:**
 - `crates/ecstore/src/data_movement.rs`
 - `crates/ecstore/src/store/multipart.rs`
 - `crates/ecstore/src/store/object.rs`
 **Preferred fix direction:**
 - Replace whole-part buffering with bounded streaming.
 - Preserve the current part index and checksum/etag behavior through the streaming reader.
 - Ensure multipart abort still runs on any failed part or complete call.
 **Acceptance criteria:**
 - Migrating a large multipart object does not allocate memory proportional to part size.
 - Part ETag, size, actual size, and index handling remain correct.
 - Failed migration aborts the temporary multipart upload.
 **Required tests:**
 - Fake large multipart reader proving memory is bounded.
 - Multipart migration preserves part order and ETags.
 - Failure during part upload aborts the multipart upload.
 **Dependencies:** Coordinate with F07 for checksum preservation.
 ---
 ### F07: Preserve and verify full data movement metadata
 **Source:** P2.2, A3
 **Decision:** Confirmed fix required for checksum; additional metadata requires verification and likely fixes.
 **Target priority:** Medium.
 **Problem:** Data movement preserves basic metadata but does not fully prove checksum, multipart checksum, replication state, version purge status, or object-lock metadata equivalence after migration.
 **Primary files:**
 - `crates/ecstore/src/data_movement.rs`
 - `crates/ecstore/src/store_api/types.rs`
 - `crates/ecstore/src/set_disk.rs`
 - `crates/filemeta/src/fileinfo.rs`
 - `crates/filemeta/src/replication.rs`
 **Preferred fix direction:**
 - For multipart migration, populate `CompletePart` checksum fields when source part checksum exists.
 - Preserve or reconstruct object-level checksum metadata.
 - Add migration equivalence checks for:
  - ETag;
  - checksum and multipart checksum;
  - replication status and version purge status;
  - object lock retention mode/date and legal hold;
  - version ID and mod time.
 - Only change production metadata handling after tests show a real loss path.
 **Acceptance criteria:**
 - Migrated object metadata matches source for the fields above.
 - `GetObjectAttributes` checksum behavior remains correct after migration.
 - Object lock retention/legal hold remains unchanged across migration.
 **Required tests:**
 - Single-part object with checksum metadata.
 - Multipart object with per-part checksum metadata.
 - Object with replication state and version purge state.
 - Object with governance/compliance retention and legal hold.
 **Dependencies:** F06 should land first or in the same PR if checksum streaming needs shared reader changes.
 ---
 ### F08: Align decommission lifecycle-expired version cleanup semantics
 **Source:** P2.4
 **Decision:** Confirmed fix required or explicit design decision required.
 **Target priority:** Medium.
 **Problem:** Decommission skips lifecycle-expired versions but does not clean the source entry if any expired version exists. MinIO counts expired versions as completed and can clean the source entry.
 **Primary files:**
 - `crates/ecstore/src/pools.rs`
 - Lifecycle evaluator code under `crates/ecstore/src/bucket/lifecycle/`
 **Preferred fix direction:**
 - Treat lifecycle-expired versions as completed for decommission source cleanup when they are not required to remain accessible.
 - Keep lifecycle/object-lock/replication checks in place before treating a version as expired.
 - If RustFS intentionally keeps expired source versions, expose that state in status and exclude the pool from being considered fully clean.
 **Acceptance criteria:**
 - Decommission can clean a source entry when all remaining versions are either migrated or lifecycle-expired.
 - Object-lock or replication-protected versions are not wrongly treated as cleanup-safe.
 - Status accurately reflects residual expired-source entries if they are intentionally retained.
 **Required tests:**
 - Bucket with one migrated version plus one lifecycle-expired version.
 - Expired version protected by object lock must not be cleaned.
 - Replication-pending version must not be treated as cleanup-safe.
 **Dependencies:** Coordinate with F07 metadata tests for object lock and replication.
 ---
 ### F09: Handle rebalance overwrite races using explicit equivalence checks
 **Source:** P2.3
 **Decision:** Fix likely required; first confirm exact race cases with tests.
 **Target priority:** Medium.
 **Problem:** RustFS treats `DataMovementOverwriteErr` as non-transient during rebalance, while MinIO ignores acceptable overwrite races. A strict failure can prevent rebalance convergence when the target already has an equivalent version.
 **Primary files:**
 - `crates/ecstore/src/rebalance.rs`
 - `crates/ecstore/src/data_movement.rs`
 - `crates/ecstore/src/store/object.rs`
 **Preferred fix direction:**
 - Keep unsafe source-equals-target cases as failures.
 - If the target version already exists, compare version ID, ETag, size, mod time, checksum, and key metadata.
 - Count as complete only when equivalence passes.
 - Add last-error/status text when overwrite is unsafe.
 **Acceptance criteria:**
 - Equivalent target version allows rebalance to continue.
 - Non-equivalent target version fails and does not clean source.
 - Behavior is documented as compatible with MinIO's acceptable overwrite race handling.
 **Required tests:**
 - Rebalance overwrite with equivalent target version.
 - Rebalance overwrite with mismatched checksum or metadata.
 - Rebalance overwrite where source and target pool are the same and no equivalent target exists.
 **Dependencies:** F07 defines the metadata equivalence fields.
 ---
 ### F10: Add source cleanup preflight verification before deleting migrated entries
 **Source:** A2
 **Decision:** Fix recommended as defense-in-depth.
 **Target priority:** Medium.
 **Problem:** After versions are migrated, source cleanup deletes the source prefix based on the version list read before migration. A final verification would protect against state propagation failures, recovery races, or repair processes changing metadata before cleanup.
 **Primary files:**
 - `crates/ecstore/src/rebalance.rs`
 - `crates/ecstore/src/pools.rs`
 - `crates/ecstore/src/set_disk.rs`
 **Preferred fix direction:**
 - Before source cleanup, re-read source metadata or a generation marker.
 - Verify the version set scheduled for deletion equals the version set that was migrated or intentionally ignored.
 - If the version set changed, defer/retry the entry instead of deleting the prefix.
 **Acceptance criteria:**
 - Cleanup does not delete versions created or discovered after the migration scan.
 - Changed source metadata causes a retry/defer state with a clear status error.
 - The guard applies to rebalance and decommission where source prefix deletion is used.
 **Required tests:**
 - Simulate source metadata changing between migration and cleanup.
 - Simulate peer state mismatch where a write lands on a rebalancing/decommissioning pool.
 - Confirm unchanged metadata still allows cleanup.
 **Dependencies:** Works best after F03 and F05 reduce state propagation failures.
 ---
 ## Phase 3: Cleanup Semantics, Observability, and Hardening
 ### F11: Improve rebalance cleanup failure handling and reporting
 **Source:** A1
 **Decision:** Fix recommended.
 **Target priority:** Medium.
 **Problem:** Rebalance source cleanup failure is converted to a warning and the task can still complete. Only a count and last warning are preserved.
 **Primary files:**
 - `crates/ecstore/src/rebalance.rs`
 - `rustfs/src/admin/handlers/rebalance.rs`
 - Rebalance status DTOs and serialization paths
 **Preferred fix direction:**
 - Preserve a bounded list of cleanup failures, not only the last one.
 - Add a clear terminal or partial terminal state when cleanup warnings exist.
 - Decide whether cleanup failure should:
  - defer the entry for retry, or
  - allow completion but mark the pool as `completed_with_cleanup_warnings`.
 **Acceptance criteria:**
 - Admin status reports cleanup warning count and representative object list.
 - Operators can identify which objects need manual cleanup or retry.
 - Rebalance terminal state is not misleading when source data remains.
 **Required tests:**
 - Multiple cleanup failures preserve count and bounded object details.
 - Status response includes cleanup warning data.
 - Completion state reflects cleanup warnings.
 **Dependencies:** Coordinate with F10 if cleanup failures become retryable.
 ---
 ### F12: Add structured audit fields for rebalance and decommission operations
 **Source:** P3.2
 **Decision:** Fix recommended.
 **Target priority:** Low.
 **Problem:** Critical admin operations do not consistently log actor, remote address, request ID, pool indices, peer host, result, and partial failure state.
 **Primary files:**
 - `rustfs/src/admin/handlers/rebalance.rs`
 - `rustfs/src/admin/handlers/pools.rs`
 - `crates/ecstore/src/notification_sys.rs`
 - `crates/ecstore/src/rpc/peer_rest_client.rs`
 - Logging guardrail scripts if applicable
 **Preferred fix direction:**
 - Standardize structured fields for start, stop, cancel, status-affecting propagation, and peer RPC failures.
 - Mask sensitive actor material.
 - Use warn/error for partial failure rather than info-only logs.
 **Acceptance criteria:**
 - Start/stop/cancel logs include actor, request ID, pool/rebalance ID, peer, result, and error when present.
 - Logs do not expose secrets.
 - Partial failures are searchable by stable event fields.
 **Required tests:**
 - Log-capture tests for success, rejection, and partial failure.
 - Guardrail check if logging conventions are enforced by script.
 **Dependencies:** F03 and F05 define partial-failure semantics.
 ---
 ### F13: Harden persisted rebalance and pool metadata decoding
 **Source:** P3.3
 **Decision:** Fix recommended after compatibility review.
 **Target priority:** Low.
 **Problem:** Persisted rebalance and pool metadata can accept unknown fields silently. This may be acceptable for compatibility, but it weakens corruption and typo detection.
 **Primary files:**
 - `crates/ecstore/src/rebalance.rs`
 - `crates/ecstore/src/pools.rs`
 - Metadata fixture tests near existing metadata tests
 **Preferred fix direction:**
 - For formats that can be strict, add `deny_unknown_fields`.
 - Where backward compatibility requires leniency, detect and warn on unknown fields during legacy decode.
 - Add explicit validation for conflicting terminal states.
 **Acceptance criteria:**
 - Unknown fields either fail decode or produce a warning through a documented compatibility path.
 - Missing critical fields fail safely.
 - Conflicting states such as completed plus running are rejected.
 **Required tests:**
 - Unknown field fixture.
 - Missing critical field fixture.
 - Conflicting terminal state fixture.
 - Legacy fixture proving backward compatibility.
 **Dependencies:** None.
 ---
 ### F14: Decide and document rebalance completion threshold behavior
 **Source:** P3.4
 **Decision:** Design decision required.
 **Target priority:** Low.
 **Problem:** RustFS uses a stricter rebalance completion goal than MinIO's tolerance-based behavior. This may increase movement work and operational time.
 **Primary files:**
 - `crates/ecstore/src/rebalance.rs`
 - Admin status documentation if present
 - Tests around rebalance goal calculation
 **Preferred fix direction:**
 - Decide whether RustFS should:
  - match MinIO tolerance;
  - keep strict behavior and document it;
  - make tolerance configurable with a conservative default.
 - Avoid configurability unless a real operational requirement exists.
 **Acceptance criteria:**
 - Completion behavior is intentional and covered by tests.
 - Status/ETA reporting does not imply MinIO-compatible tolerance if strict behavior remains.
 - Large-pool behavior is covered by a deterministic unit test.
 **Required tests:**
 - Rebalance goal just below target.
 - Rebalance goal within MinIO-like tolerance.
 - Empty queue completion path remains valid.
 **Dependencies:** None.
 ---
 ## Cross-Cutting Test Matrix
 Each fix should add focused tests, but the following end-to-end scenarios should also be covered before enabling production use:
 1. Rebalance a versioned object with normal versions, delete marker, and source cleanup.
 2. Rebalance a remote tiered version or verify it is intentionally skipped.
 3. Decommission a pool while peer reload fails.
 4. Restart with both active decommission metadata and rebalance metadata.
 5. Stop rebalance while a large multipart migration is in progress.
 6. Migrate a large multipart object without memory proportional to part size.
 7. Migrate objects with checksum, replication state, retention, and legal hold metadata.
 8. Trigger source cleanup failure and verify admin status exposes it.
 9. Trigger overwrite race with equivalent and non-equivalent target versions.
 10. Decommission lifecycle-expired versions with object-lock and replication constraints.
 ## Suggested Execution Strategy
 1. Start with F01 and F02 because they protect object version semantics.
 2. Implement F03, F04, and F05 before broader rollout because they define distributed operation safety.
 3. Implement F06 before running large-scale data movement tests.
 4. Implement F07 through F11 as compatibility, convergence, and observability hardening.
 5. Keep F12 through F14 as final hardening unless release criteria require them earlier.
 Each fix should be developed in a separate PR unless two adjacent fixes share the same core implementation. In particular:
 - F01 and F09 should not be merged together unless the equivalence helper is clearly isolated.
 - F06 and F07 may be combined only if streaming part migration and checksum preservation share the same reader changes.
 - F03 and F05 may share peer failure aggregation utilities, but their admin behavior should be tested separately.
@@ -1,81 +0,0 @@
 > **Archived migration snapshot** — moved from `docs/architecture/` (2026-07)
 > when the architecture-review ledger it fed closed out. Kept for history; not
 > maintained.
 # Scheduler Baseline Inventory
 This inventory covers `G-011` for `rustfs/backlog#675`. It is a docs-only
 snapshot of the current scheduling, backpressure, worker, scanner, heal, and
 runtime-builder ownership. It does not define new behavior.
 ## Current Owners
 | Surface | Current owner | Current responsibility | Migration boundary |
 |---|---|---|---|
 | `ConcurrencyManager` | `rustfs/src/storage/concurrency/manager.rs` | Owns the RustFS S3 read-path disk-read semaphore, I/O metrics, priority queue, storage media detection, access-pattern detection, and buffer strategy. | Keep request admission and I/O metrics behavior stable until a controller can consume the same state explicitly. |
 | I/O scheduler core | `crates/io-core/src/scheduler.rs` | Owns the reusable buffer-size and priority algorithms consumed by the RustFS S3 read path (`rustfs/src/storage/concurrency/io_schedule.rs`). The former `SchedulerManager` facade in `rustfs-concurrency` was removed as zero-caller dead code (backlog#1025). | Treat `rustfs-io-core` as the reusable algorithm surface; the RustFS S3 read path owns its own scheduling wiring. |
 | RustFS backpressure monitor | `rustfs/src/storage/backpressure.rs` | Tracks object-pipe watermark state used by RustFS storage backpressure tests and helpers, using the shared `PipeBackpressurePolicy` from `crates/concurrency/src/backpressure.rs`. The former `BackpressureManager`/`BackpressurePipe` facade in `rustfs-concurrency` was removed as zero-caller dead code (backlog#1025). | Preserve current state labels and watermark semantics; keep pipe sizing and watermark policy separate from object-read disk semaphore admission. |
 | `Workers` | `crates/concurrency/src/workers.rs` | Provides cooperative worker-slot admission with `take`, `give`, and `wait`; current background workflows use it for bounded set workers. | Preserve blocking/wakeup semantics and over-release clamping. |
 | Scanner cycle budget | `crates/scanner/src/scanner_budget.rs` | Cancels a child token when runtime, object-count, or directory-count budget is reached. | Preserve partial-cycle reason mapping and checkpoint accounting. |
 | Heal admission | `crates/heal/src/heal/manager.rs`, `crates/heal/src/heal/channel.rs`, `rustfs_common::heal_channel` | Owns priority queue admission, duplicate merge/drop/full results, active-task tracking, retry admission, and channel responses. | Preserve low-priority scanner behavior and high-priority escalation gates. |
 | Tokio runtime builder | `rustfs/src/server/runtime.rs` | Builds the multi-thread runtime from env/defaults, sets thread counts, stack, queue/event intervals, I/O event cap, thread name, and optional dial9 tracing. | Keep runtime defaults and env names stable when later startup phases move ownership. |
 ## Current Flow
 ```mermaid
 flowchart TD
    http["HTTP/S3 request"] --> app["app object usecase"]
    app --> guard["ConcurrencyManager::track_request"]
    app --> permit["disk-read semaphore permit"]
    permit --> strategy["I/O queue status and buffer strategy"]
    strategy --> ecstore["ECStore object/read path"]
    ecstore --> setdisks["hashed set disks"]
    scanner["scanner cycle"] --> budget["ScannerCycleBudget"]
    budget --> folder["folder/object scan"]
    folder --> healreq["heal channel request"]
    healreq --> admission["HealManager admission queue"]
    admission --> healworker["heal workers and retries"]
    startup["startup entrypoint"] --> runtime["Tokio runtime builder"]
    runtime --> services["background services"]
 ```
 ## Missing State For Later Work
 `R-015` storage foundation:
 - Needs a stable inventory of endpoint publication, local disk prewarm, lock
  client setup, and per-set readiness state before any scheduler/controller
  consumes storage topology.
 - Must not infer set availability only from request-path I/O metrics.
 `E-011` extension/runtime consumers:
 - Need explicit ownership for runtime admission snapshots before extensions can
  observe scheduler or backpressure state.
 - Must not receive mutable handles to `ConcurrencyManager`, heal queues, or
  scanner budget tokens.
 `C-011` controller work:
 - Needs desired/current/status snapshots for request admission, scanner budget,
  and heal queue pressure before any controller can reconcile them.
 - Must keep worker mutation explicit. Read-only status should report `None` or
  no-op mutation until a reviewed worker lifecycle PR exists.
 ## Preservation Invariants
 - Request reads must keep the same disk-read semaphore admission and active GET
  accounting.
 - I/O queue status and congestion metrics must remain derived from the same
  permit counts.
 - Scanner budget cancellation must keep its reason as runtime, objects, or
  directories.
 - Scanner inline-heal compatibility must continue to use asynchronous heal
  admission.
 - Heal duplicate admission must prefer merge semantics before full-queue
  rejection.
 - High-priority heal admission must still be able to displace lower-priority
  queued work where the current manager allows it.
 - Tokio runtime env names and fallback defaults must remain unchanged.
@@ -1,105 +0,0 @@
 > **Archived implementation plan/tracker** — moved from `docs/architecture/` (2026-07).
 > Kept for history; not maintained. File paths inside may reflect the pre-#3929
 > module layout (e.g. `crates/ecstore/src/rebalance.rs` is now
 > `crates/ecstore/src/store/rebalance.rs`; `set_disk.rs` is now `set_disk/`).
 ## Related Issues
 N/A
 ## Summary of Changes
 This PR hardens the site replication control plane in small, reviewable commits. The work starts with guardrails identified during the MinIO comparison and multi-review pass: credential redaction, stricter diagnostic authorization, remove cleanup, and clearer status diagnostics.
 ### Implementation Log
 1. `docs: plan site replication hardening`
   - Purpose: establish the execution plan and PR draft before code changes.
   - Reason: the site replication follow-up touches security, compatibility, operations, and tests, so reviewers need a durable record of each step and why it exists.
 2. `fix: redact site replication target secrets`
   - Purpose: prevent bucket replication target credentials from leaking through admin listing, bucket metadata export, and debug formatting.
   - Reason: site replication stores the `site-replicator-0` secret in bucket targets today; keeping the internal storage format intact while redacting external surfaces reduces immediate blast radius without changing replication target loading.
 3. `fix: require operation access for replication diagnostics`
   - Purpose: require operation-level admin authorization for site replication diagnostic POST endpoints and clamp netperf duration.
   - Reason: `devnull` and `netperf` read request bodies and exercise diagnostic work, so granting them through read-only site replication info permission was broader than intended.
 4. `fix: clean site replication targets on remove`
   - Purpose: remove bucket targets and `site-repl-*` replication rules that point at a removed deployment before completing pending remove.
   - Reason: removing a site only from site replication state leaves bucket-level replication configuration behind, so object replication can continue attempting to reach a removed peer.
 5. `feat: expose site replication status diagnostics`
   - Purpose: add machine-readable peer fetch errors and pending operation progress to site replication status.
   - Reason: operators previously saw only `Unknown` sync state or logs when peer metainfo fetches failed or remove/rotation was pending, which made automation and troubleshooting unnecessarily opaque.
 6. `docs: expand site replication hardening scope`
   - Purpose: keep the single-PR plan aligned with the expanded request to include MinIO compatibility, add preflight, bootstrap, lifecycle compatibility, retry, and repair work.
   - Reason: the remaining work is larger than the first hardening pass, so reviewers need to see the intended sequence before more behavior changes land.
 7. `fix: align site replication peer paths with minio`
   - Purpose: send peer site-replication calls over MinIO-compatible admin paths and accept the MinIO-style peer join route.
   - Reason: RustFS already accepts `/minio/admin` as an alias, so using MinIO wire paths improves mixed-cluster compatibility without removing RustFS route support.
 8. `fix: validate site replication add topology`
   - Purpose: preflight add requests with remote metainfo and IDP settings before creating service accounts, joining peers, or persisting state.
   - Reason: MinIO rejects unsafe add topologies up front; RustFS now rejects duplicate deployment IDs, missing local deployment, IDP mismatch, multiple non-empty initial sites, and peers already configured with a different site-replication set.
 9. `feat: bootstrap site replication metadata on add`
   - Purpose: replay the local site replication snapshot to joined peers during add, before object backfill starts.
   - Reason: existing hooks only replicate changes after site replication is enabled; existing policies, users, groups, bucket metadata, and buckets must be bootstrapped so a newly joined site does not start with an incomplete control-plane snapshot.
 10. `fix: align lifecycle replication with minio semantics`
   - Purpose: replicate bucket lifecycle metadata only when site replication has `replicate-ilm-expiry` enabled.
   - Reason: MinIO keeps lifecycle expiry replication opt-in; RustFS should not replicate lifecycle metadata by default during live bucket metadata hooks or add-time bootstrap.
 11. `feat: add site replication retry and repair MVP`
   - Purpose: persist failed peer replication attempts as retry metadata, expose retry counts in status, and add an operation-level repair endpoint that replays the current local snapshot.
   - Reason: transient peer failures were previously only visible through logs and required manual reconstruction; a durable, secret-safe queue plus repair replay gives operators a concrete recovery path without persisting sensitive request payloads.
 12. `fix: satisfy site replication pre-pr checks`
   - Purpose: remove a redundant MinIO admin-path branch and construct retry test fixtures directly.
   - Reason: the full pre-PR gate enforces clippy warnings as errors; this keeps the final branch green without changing site replication behavior.
 ## Verification
 Baseline started from `origin/main` at `758677da`, then was rebased onto the latest `origin/main` at `1d6a8259`.
 - Passed: `cargo fmt --all --check`
 - Passed: `cargo test -p rustfs-ecstore bucket_target --lib`
 - Passed: `cargo test -p rustfs-madmin site_replication --lib`
 - Passed: `cargo test -p rustfs route_policy --lib`
 - Passed: `cargo test -p rustfs site_replication --lib`
 - Passed: `cargo test -p rustfs route_policy --lib`
 - Passed: `cargo test -p rustfs site_replication --lib`
 - Passed: `cargo fmt --all`
 - Passed: `cargo fmt --all --check`
 - Passed: `cargo test -p rustfs site_replication --lib`
 - Passed: `cargo fmt --all --check`
 - Passed: `cargo test -p rustfs-madmin site_replication --lib`
 - Passed: `cargo test -p rustfs route_policy --lib`
 - Passed: `cargo test -p rustfs route_registration_test --lib`
 - Passed: `cargo test -p rustfs site_replication --lib`
 - Passed: `cargo fmt --all --check`
 - Passed: `cargo test -p rustfs site_replication --lib`
 - Passed: `cargo fmt --all --check`
 - Passed: `cargo test -p rustfs site_replication --lib`
 - Passed before the final main rebase: `make pre-pr`
 - Not completed after rebasing onto `1d6a8259`: `make pre-pr` was interrupted during clippy at request time; formatting, unsafe-code, architecture, and logging guardrails had already passed in that run.
 ## Impact
 The planned implementation is intended to reduce credential exposure, require stronger permission for diagnostic write-like endpoints, clean removed site replication targets, and make replication status more actionable. Compatibility-sensitive behavior is called out per commit in the implementation log.
 The credential redaction step changes admin/export visibility of bucket target secrets. Stored target configuration is not migrated or reformatted by this PR step.
 The diagnostic authorization step intentionally changes required permission for `POST /site-replication/devnull` and `POST /site-replication/netperf` from `SiteReplicationInfoAction` to `SiteReplicationOperationAction`.
 The remove cleanup step preserves user-managed replication rules and non-replication targets, and only prunes targets/rules associated with removed site replication deployment IDs.
 The status diagnostics step adds optional `PeerErrors` and `PendingOperation` fields to the status response. Existing clients can ignore them, while automation can use them to distinguish peer reachability/auth failures from content mismatch.
 ## Additional Notes
 This draft PR includes bootstrap, durable retry, and repair work in one reviewable branch. It should remain draft until CI or a full local pre-PR run confirms the latest default branch baseline.
 The expanded single-PR scope also includes MinIO wire-contract bootstrap validation, durable site-replication retry, full add-time IAM/bootstrap sync, lifecycle compatibility, and site-level repair.
 The peer path compatibility step maps outbound peer requests to `/minio/admin/v3/site-replication/...`. Peer join uses the encrypted MinIO payload contract, while internal peer metadata/IAM/remove/edit requests remain plain JSON to match MinIO handlers.
 The add preflight step performs remote reads before state mutation, so add fails earlier when a peer cannot report metainfo or IDP settings.
 The bootstrap step replays IAM policies, built-in users with stored secret material, group membership/status, policy mappings, bucket creation, and bucket metadata through the existing peer replication handlers. It does not synthesize ordinary service-account secrets when the snapshot does not contain them; the site-replicator service account remains distributed through the join flow.
 Lifecycle metadata is now skipped by default in site replication hooks and bootstrap snapshots unless the site replication peer state enables `replicate-ilm-expiry`.
 The retry MVP intentionally stores peer/path/error/count metadata only. Repair regenerates payloads from the current local snapshot, avoiding persistent storage of user secrets, service-account secrets, or bucket target credentials in the retry queue.
@@ -1,90 +0,0 @@
 > **Archived migration snapshot** — moved from `docs/architecture/` (2026-07)
 > when the architecture-review ledger it fed closed out. Kept for history; not
 > maintained.
 # Startup Timeline Baseline
 This document records the current binary startup order before runtime/lifecycle
 migration work. It is a behavior-preservation baseline only; it does not define
 new startup semantics.
 ## Scope
 - Baseline commit: `ae9d25879d72bc8977f08e61062c022e2142483b`
 - Entry points covered: `rustfs/src/main.rs::main`,
  `rustfs/src/startup_entrypoint.rs::{run_process, async_main, run}`, and
  startup lifecycle helpers
 - Related migration task: `G-007`
 - Out of scope for this baseline: embedded startup, admin route-action matrix,
  and any runtime/lifecycle code movement
 ## Startup Stages
 | Step | Source | Current action | Side effects | Fatal boundary | Ready stage |
 |---|---|---|---|---|---|
 | `BOOT-001` | `rustfs/src/startup_entrypoint.rs` | Apply external-prefix environment compatibility during async startup before command parsing. | Copies supported external env aliases into canonical `RUSTFS_*` process env keys and prints warnings or info to stderr. | Non-fatal; failure is logged to stderr and startup continues. | None |
 | `BOOT-002` | `rustfs/src/main.rs` and `rustfs/src/startup_entrypoint.rs` | Call the startup entrypoint and build the Tokio runtime. | Installs runtime configuration and any runtime telemetry guard created by the runtime builder. | Fatal through `expect`; process exits if the runtime cannot be built. | None |
 | `BOOT-003` | `rustfs/src/startup_entrypoint.rs` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None |
 | `BOOT-004` | `rustfs/src/startup_preflight.rs` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None |
 | `BOOT-005` | `rustfs/src/startup_preflight.rs` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None |
 | `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, `rustfs/src/startup_tls_material.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None |
 | `RUN-001` | `rustfs/src/startup_server.rs` | Enter startup run orchestration and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready |
 | `RUN-002` | `rustfs/src/startup_server.rs` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None |
 | `RUN-003` | `rustfs/src/startup_server.rs` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None |
 | `RUN-004` | `rustfs/src/startup_server.rs` | Initialize global action credentials. | Publishes root/action credentials used by auth paths. | Fatal if global credentials cannot be initialized. | None |
 | `RUN-005` | `rustfs/src/startup_server.rs` | Publish server port and address. | Updates global RustFS port and global address. | Non-fatal in this path. | None |
 | `RUN-006` | `rustfs/src/startup_storage.rs` | Build endpoint pools and enforce unsupported filesystem policy. | Derives pool/set/disk layout from configured volumes and validates unsupported filesystem policy. | Fatal on endpoint build or unsupported filesystem policy error. | None |
 | `RUN-007` | `rustfs/src/startup_storage.rs` | Publish endpoints and erasure type. | Updates global endpoints and erasure type. | Non-fatal in this path. | None |
 | `RUN-008` | `rustfs/src/startup_storage.rs` | Initialize local disks, prewarm local disk id map, and initialize lock clients. | Opens local disk state, primes disk id lookup, and creates global lock clients. | Local disk init is fatal; prewarm and lock-client setup are non-fatal in this path. | None |
 | `RUN-009` | `rustfs/src/startup_server.rs` | Initialize capacity management and service state manager. | Starts capacity management and moves service state to `Starting`. | Non-fatal in this path. | None |
 | `RUN-010` | `rustfs/src/startup_server.rs` | Start S3 HTTP listener and optional console listener before storage is ready. | Starts HTTP servers with readiness gates; console listener starts only when enabled and configured. | Fatal if a configured listener cannot start. | Requests remain gated until full readiness except probe/admin/console/rpc/tonic/table-catalog exempt paths; see [`readiness-matrix.md`](../../architecture/readiness-matrix.md) |
 | `RUN-011` | `rustfs/src/startup_storage.rs` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None |
 | `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, lock quorum, and gated peer health before `FullReady` |
 | `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady`; KMS compatibility readiness remains feature-gated health behavior |
 | `RUN-014` | `rustfs/src/startup_optional_runtime_sidecars.rs` and `rustfs/src/startup_protocols.rs` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None |
 | `RUN-015` | `rustfs/src/startup_services.rs`, `rustfs/src/startup_audit.rs`, and `rustfs/src/startup_deadlock.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
 | `RUN-016` | `rustfs/src/startup_bucket_metadata.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |
 | `RUN-017` | `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task |
 | `RUN-018` | `rustfs/src/startup_auth.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None |
 | `RUN-019` | `rustfs/src/startup_notification.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None |
 | `RUN-020` | `rustfs/src/startup_background.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None |
 | `RUN-021` | `rustfs/src/startup_observability.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None |
 | `RUN-022` | `rustfs/src/startup_lifecycle.rs` and `rustfs/src/startup_iam.rs` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here |
 | `RUN-023` | `rustfs/src/startup_lifecycle.rs` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery |
 | `RUN-024` | `rustfs/src/startup_lifecycle.rs` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state |
 ## Deferred IAM Readiness
 | Step | Source | Current action | Side effects | Fatal boundary | Ready stage |
 |---|---|---|---|---|---|
 | `IAM-001` | `rustfs/src/startup_iam.rs:256` | Attempt `init_iam_sys` during bootstrap. | Initializes IAM against the ECStore object layer when possible. | Recoverable failures can enter deferred mode; unrecoverable errors propagate from bootstrap. | None |
 | `IAM-002` | `rustfs/src/startup_iam.rs:73` | Spawn IAM recovery loop when bootstrap is deferred. | Retries IAM initialization with exponential backoff until shutdown or success. | Retry failures are logged; the service remains degraded. | None |
 | `IAM-003` | `rustfs/src/startup_iam.rs:52` | Finalize IAM recovery after init succeeds. | Initializes `AppContext` if needed, marks `IamReady`, and calls runtime readiness publication. | Finalize failures are retried by the recovery loop. | Marks `IamReady`, then `FullReady` when runtime readiness succeeds |
 ## Readiness Gate
 | Step | Source | Current action | Side effects | Fatal boundary | Ready stage |
 |---|---|---|---|---|---|
 | `READY-001` | `rustfs/src/server/readiness.rs:130` | Treat exact probe paths and admin/console/rpc/tonic/table-catalog prefixes as readiness-gate bypass paths. | Bypass paths continue to the inner service while the global readiness gate is not ready. | Non-fatal. | Does not change readiness stages |
 | `READY-002` | `rustfs/src/server/readiness.rs:171` | Reject non-probe requests while `GlobalReadiness` is not ready. | Returns `503 Service Unavailable`, `Retry-After: 5`, `Content-Type: text/plain; charset=utf-8`, and `Cache-Control: no-store`. | Non-fatal. | Does not change readiness stages |
 | `READY-003` | `rustfs/src/server/readiness.rs:202` | Wait for runtime storage, IAM, lock quorum, and gated peer-health readiness before publishing ready state. | Marks `FullReady` and updates `ServiceState` to `Ready` only when a state manager is provided. | Returns an error on timeout; inline startup treats that as fatal, while deferred IAM recovery retries finalization. | Marks `FullReady` |
 ## Shutdown Order
 | Step | Source | Current action | Side effects | Fatal boundary | Ready stage |
 |---|---|---|---|---|---|
 | `STOP-001` | `rustfs/src/startup_shutdown.rs` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here |
 | `STOP-002` | `rustfs/src/startup_shutdown.rs` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change |
 | `STOP-003` | `rustfs/src/startup_optional_runtime_sidecars.rs` | Plan optional runtime shutdown and log stopping state for FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown handles. | Non-fatal in this path. | No readiness-stage change |
 | `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change |
 | `STOP-005` | `rustfs/src/startup_shutdown.rs` and `rustfs/src/startup_optional_runtime_sidecars.rs` | Stop S3 and console HTTP servers, signal and wait for optional protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here |
 ## Migration Rules
 - Runtime/lifecycle PRs must map each moved startup line back to one of the
  `BOOT-*`, `RUN-*`, `IAM-*`, `READY-*`, or `STOP-*` rows.
 - A `pure-move` PR must keep the fatal boundary and ready-stage column unchanged.
 - Any intentional change to this table is a separate `behavior-change` PR with
  focused negative tests.
 - Do not use this document to justify changing readiness, IAM recovery, HTTP
  listener timing, lock quorum, or shutdown order in a docs-only PR.
@@ -5,8 +5,7 @@
 # ARCHITECTURE.md / docs/architecture honest after refactors move code.
 #
 # Checked files: all tracked AGENTS.md, CLAUDE.md, ARCHITECTURE.md, and
-# docs/architecture/*.md (archived plans under docs/superpowers/plans are
+# docs/architecture/*.md.
 # intentionally NOT checked — they are historical).
 #
 # A reference is any token starting with crates/, rustfs/, scripts/, docs/,
 # .github/ or .config/ that ends in a known file extension. Tokens containing