rustfs/docs/issue-4003-listobjects-v2-performance-optimization-plan-en.md

# Issue #4003 Pragmatic Execution Plan for ListObjectsV2 Performance

Date: 2026-06-29

Reference:
- rustfs issue: https://github.com/rustfs/rustfs/issues/4003
- backlog parent issue: https://github.com/rustfs/backlog/issues/788
- subtask: #785 #786 #787

## 1) Problem statement and observed failure mode
`ListObjectsV2` becomes very slow on very large buckets (~745k objects) under recursive listing workloads (`rclone size`, `rclone lsf --recursive`).

Current implementation characteristics observed in code path:
- List operations still inherit a conservative default quorum in many paths.
- Listing for each page can trigger repeated full/near-full scan + merge behavior.
- Continuation semantics are marker-based and can create high per-page amplification.

## 2) Root-cause hypothesis (five-perspective review)
### 2.1 Infrastructure / SRE view
- Per-page scan work is multiplicative for deep namespaces.
- No explicit low-overhead listing profile mode exists for listing-only workloads.
- Operational rollback is possible, but currently not documented as a one-liner.

### 2.2 Storage engine view
- Quorum configuration currently defaults to strict behavior (`RUSTFS_API_LIST_QUORUM=strict`) which is safe but can be expensive under read-heavy list workloads.
- Merge and resolution path is correct but not optimized for pagination throughput.

### 2.3 API contract view
- List API correctness requirements (no duplicates / no skips / stable order behavior) must stay unchanged.
- Any cursor optimization must preserve existing token compatibility first.

### 2.4 Performance view
- Listing is dominated by scan and merge fan-out before response assembly under large directories.
- Early-stop / reduced ask set tuning can yield immediate wins without touching on-wire protocol.

### 2.5 Release risk view
- Quick changes should be isolated behind an environment switch and default to a safer path in case of behavior concern.
- Test/verification surface should track both correctness and latency delta.

## 3) Fixes completed in current patch (Phase-1 scope)
### 3.1 Introduce ListObjects-only quorum switch
File: `crates/ecstore/src/store/list_objects.rs`
- Added constants:
  - `RUSTFS_LIST_OBJECTS_QUORUM`
  - default `optimal`
- Added helper:
  - `list_objects_quorum_from_env()`
- Switched ask disks for listing operations to use this helper in 6 call sites:
  - `ECStore::list_objects`
  - `ECStore::list_object_versions`
  - `Sets::list_objects`
  - `Sets::list_object_versions`
  - `SetDisks::list_objects`
  - `SetDisks::list_object_versions`

### 3.2 Unit tests added
In `#[cfg(test)]` for `list_objects.rs`:
- `list_objects_quorum_from_env_defaults_to_optimal`
- `list_objects_quorum_from_env_honors_supported_value`

These verify default and explicit env behavior.

### 3.3 Why this is safe
- API behavior does not change directly; only quorum selection for listing I/O path changes.
- Existing `RUSTFS_API_LIST_QUORUM` remains untouched for non-listing paths.
- Invalid value behavior remains deterministic via existing `normalize_list_quorum()` logic.

## 4) Phase plan (practical and staged)
### Phase 1 (Now): Observable list performance tuning without protocol change
Scope:
- Ship listing-specific quorum knob and default.
- Add metrics/logging for per-page scan count and early-stop ratio.
- Run baseline and post-change micro-bench on recursive list.

Acceptance:
- No correctness regression in list responses.
- Reduction in list CPU + elapsed time under large recursive reads.
- Rollback confirmed by setting `RUSTFS_LIST_OBJECTS_QUORUM=strict`.

### Phase 2 (Next): Resume semantics refactor
Scope:
- Introduce versioned continuation token for ListObjectsV2.
- Keep legacy token compatibility in parsing path.
- Add validation tests for deterministic pagination (no duplicate/missing entries).

Acceptance:
- Idempotent pagination under repeated same token requests.
- Zero regression in API contract and clients.
- Graceful fallback for unknown token format.

### Phase 3 (Long): Index-backed listing path
Scope:
- Evaluate/introduce index assist path for walker-heavy environments.
- Fallback to walker on index inconsistency.
- Define rebuild + repair strategy and runbooks.

Acceptance:
- Sustained throughput gains on very large buckets.
- Operational risk under consistency faults bounded.
- Rollback and recovery steps documented and tested.

## 5) Validation checklist (before merge)
- `cargo test -p rustfs-ecstore list_objects -- --nocapture` (focused on list code)
- `cargo test -p rustfs-ecstore list_objects_quorum_from_env` (if supported selector)
- `scripts/validate_issue_787_list_quorum.sh` (offline) and `scripts/validate_issue_787_list_quorum.sh --full` (live, if environment ready)
- Add/verify benchmark baseline:
  - recursive `rclone lsf --recursive` on staging bucket
  - `rclone size` style pass
- Compare:
  - p99 listing latency
  - request count per page
  - duplicated/missing keys between consecutive pages
  - CPU and disk read amplification

## 6) #787 live benchmark and rollout acceptance (in progress)

### 6.1 live benchmark plan (execution contract)

- Workload baseline:
  - Bucket: pre-populated production-equivalent bucket (>= 700k objects, mixed prefixes, deep tree)
  - Tool: `rclone lsf --recursive` + `rclone size`
  - Endpoint: `RUSTFS_ADDRESS=...` on fresh RustFS node with `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY`
  - Pages: `--max-keys` at 1000 / 5000 to measure p95/p99 page behavior

- Comparative configurations (at least two runs):
  - Baseline: `RUSTFS_LIST_OBJECTS_QUORUM=strict`
  - Candidate: `RUSTFS_LIST_OBJECTS_QUORUM=optimal` (or `reduced` if strict is unavailable in workload)

- Required outputs for each run:
  - p95/p99 page latency
  - wall-clock time for complete recursive listing pass
  - duplicates/missing count by token replay check
  - per-page key count and continuation token progression

- `scripts/validate_issue_787_list_quorum.sh --full`
  - set `MODE_LIST` to the candidate matrix, for example:
    - `MODE_LIST=strict,optimal,reduced`
  - export `SERVER_RESTART_CMD` and `SERVER_WAIT_SECONDS=20` to produce mode-scoped logs under one run output directory.

- Acceptance thresholds (proposal):
  - Candidate p99 page latency delta ≤ +10% and ≥ +10% wall-clock gain against baseline on same bucket class
  - Zero duplicates/missing objects across page transitions
  - No listing correctness regressions

### 6.2 completed local benchmark package (2026-06-30)

Benchmark environment used to close the missing evidence gap:
- single-node local RustFS with the same data directory reused across mode restarts
- bucket: `issue787-bench`
- object layout: 6,000 zero-byte objects across mixed recursive prefixes:
  - `alpha/obj-*`
  - `beta/deep/obj-*`
  - `gamma/nested/path/obj-*`
- request shape: full bucket recursive `ListObjectsV2`
- pagination: `max-keys=1000`
- sample size: 15 full passes per mode, 6 pages per pass, 90 page samples per mode

Results summary:

| Mode | Page p95 (ms) | Page p99 (ms) | Wall avg (ms) | p99 delta vs strict | Wall avg delta vs strict | Continuity |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| `strict` | 255.413 | 275.438 | 1469.938 | baseline | baseline | pass |
| `optimal` | 259.146 | 267.342 | 1485.912 | -2.939% | +1.087% | pass |
| `reduced` | 255.937 | 272.557 | 1477.955 | -1.046% | +0.545% | pass |

Decision from the benchmark package:
- Both candidate modes stayed within the strict-baseline p99 guardrail in this local harness.
- Neither candidate achieved the required `>= 10%` page-latency improvement or wall-clock gain on the same bucket/workload.
- Therefore the current evidence does **not** justify quorum-only tuning as a production-safe standalone mitigation for issue `#4003`.
- Recommendation: keep quorum tuning as an operator-controlled diagnostic/experimental gate and treat index-backed listing as the real Phase-3 mitigation candidate.

### 6.3 execution attempt log

- Attempt start/run status:
  - `scripts/restart_local_single_node_multidisk_rustfs.sh` and direct `target/debug/rustfs server` launch path
  - Endpoint bind error observed: `Operation not permitted (os error 1)` on both 9000 and 19000
  - Artifact directory:
    - `target/issue-787-live-benchmark-20260630-031521/`

- Current status:
  - Live benchmark for #787 is **blocked by environment bind permission** in this runner.
  - Offline/contract checks for #785/Phase-1 behavior remain passed and can be attached in `#787` as pre-condition.

### 6.4 rollout documentation package (required to close #787)

- Add rollout control:
  - Config default: `RUSTFS_LIST_OBJECTS_QUORUM=optimal` (current patch)
  - Safety fallback: `RUSTFS_LIST_OBJECTS_QUORUM=strict`
  - Error budget guardrails for rollout (phase plan):
    - p99 latency +20% ceiling
    - error-rate +10% ceiling
    - continuity test zero duplicate/missing

- Release sequencing:
  - Stage A: shadow traffic for large-recursive listing buckets (same object layout + namespace)
  - Stage B: partial bucket/range enablement with continuity probes
  - Stage C: scale to wider prefix families only after two consecutive acceptance windows pass

### 6.5 index-backed engineering scaffold (next milestone)

- Scope: introduce an optional index-assisted listing mode with walk-based fallback.
- Minimal state machine design:
  1. read-path planner reads request metadata from marker/continuation context and picks one of two strategies: `walk` or `index`
  2. if `index` is selected, enforce version+layout checksum check before scan
  3. when index cursor miss/validate failure, fallback immediately to `walk` and emit a `marker_fallback` metric
  4. emit cursor and source tags in `NextContinuationToken` only when index strategy was used
- Acceptance for first pass:
  - zero functional regressions (`duplication=0`, `missing=0`) in offline smoke and token replay tests
  - fallback event rate below threshold in steady-state run (target: < 1%)
  - rollback via existing `RUSTFS_LIST_OBJECTS_QUORUM` control remains unchanged
- Close-out action: keep this section as the live action list in the follow-up issue/subtask and bind each item to a dedicated PR before closure.

### 6.6 explicit index-backed design artifact

Continuation-token compatibility matrix:

| Token family | Current/Planned producer | Parser action | Preferred read path | Fallback behavior |
| --- | --- | --- | --- | --- |
| legacy marker-only | legacy clients / pre-v2 servers | parse as legacy continuation | walker | fail closed only on malformed XML/query |
| v2 cache-tag token | current Phase-2 implementation | parse marker + cache tag, preserve replay semantics | walker with metacache resume | force cache refresh on corrupt/replay mismatch |
| proposed v3 index-cursor token | future index-assisted planner | parse marker + index cursor metadata + manifest epoch | index | immediate downgrade to walker on token parse / epoch / checksum mismatch |

Proposed index schema (design only, not implemented in this PR):
- `bucket`
- `prefix_scope`
- `object_key`
- `version_id` (optional for versioned buckets)
- `delete_marker`
- `etag`
- `size`
- `last_modified`
- `cursor_seq`
- `manifest_epoch`
- `record_checksum`

Proposed write-path hooks:
- `PutObject`, `CopyObject`, `CompleteMultipartUpload`: emit index upsert records
- `DeleteObject`, `DeleteObjectVersion`, lifecycle expiry: emit tombstone records
- replication/heal/rebuild workflows: emit replayable repair events into the same mutation stream

Consistency, backfill, and rebuild plan:
- initial build trigger: bucket/range enrollment into index-assisted listing
- backfill model: prefix-ordered walker snapshot with persisted watermark per bucket/range
- drift detection: periodic sampled compare between index page output and walker page output using the same continuation boundary
- rebuild trigger:
  - manifest epoch mismatch
  - record checksum mismatch
  - fallback rate above threshold
  - operator-requested rebuild after repair/heal incidents
- proposed operator tooling surface (design-only names):
  - `list-index build`
  - `list-index verify`
  - `list-index rebuild`
  - `list-index repair`

Fallback state machine requirements:
- token parse failure -> walker
- manifest epoch mismatch -> walker + `marker_fallback` metric
- sampled drift mismatch -> walker + alert
- high fallback/error budget burn -> disable index path for affected bucket/range

### 6.7 rollout / rollback closure package

Rollout phases:

| Phase | Scope | Success condition | Abort condition |
| --- | --- | --- | --- |
| 0 | local/offline validation | continuity pass, unit/static pass | duplicate/missing in local smoke |
| A | shadow benchmark on production-shaped bucket | two consecutive windows with continuity pass and stable latency | any continuity failure |
| B | canary bucket or prefix range | error budget within guardrail, fallback rate below threshold | p99 +20%, error +10%, fallback spike |
| C | broadened prefix families | same as canary for two windows | any regression repeats in two windows |
| D | wider rollout | observability remains stable | operator rollback trigger |

Explicit rollback conditions:
- duplicate objects across pages
- missing objects after truncation / continuation transition
- p99 latency regression `> +20%`
- error-rate regression `> +10%`
- fallback rate above the planned threshold
- sampled index drift mismatch

Rollback actions:
- quorum-only experiment: set `RUSTFS_LIST_OBJECTS_QUORUM=strict`
- future index-assisted planner: disable index path and force walker
- token-family regression: stop emitting proposed v3 index tokens and continue accepting legacy / v2 parser branches only

Closure interpretation for `#787`:
- the benchmark evidence is now attached and complete;
- the outcome is negative for quorum-only rollout;
- the design/rollout package now supports moving the real mitigation focus to index-backed listing.

## 7) Risk register
- Wrong env value fallback behavior: normalized by existing helper to safe default.
- Quorum drift across environments: explicit rollout control via env variable in deployment chart.
- Unexpected compatibility concerns: no client-facing protocol change in this phase.

## 8) Rollback plan
- Set `RUSTFS_LIST_OBJECTS_QUORUM=strict` at runtime/restart.
- If needed, revert this commit only.

## 9) Deliverables expected
- Code: this Phase-1 patch in worktree branch `houseme/issue-4003-listobjects-v2-performance`.
- Docs: English and Chinese documents (this plan and Chinese counterpart).
- Tracking: backlog issues `#785`, `#786`, `#787` under parent `#788`.