rustfs/docs/benchmark/issue829-default-get-hot-path-runbook-2026-07-05.md

# Issue 829 Default GET Hot Path Benchmark and Attribution Runbook

Date: 2026-07-05

## Scope

This runbook defines the evidence package for proving the current default GET
hot path. It intentionally separates throughput evidence from path attribution:

- Metrics-off runs are throughput evidence.
- Metrics-on diagnostic runs are path attribution evidence.
- Codec and direct-memory experiments are A/B evidence, not default-path proof
  unless their enabling environment is recorded.

The current review baseline is:

- Ordinary non-inline GETs are expected to use ECStore metadata/cache lookup and
  the legacy duplex reader path.
- Metadata early-stop is not expected for normal `read_data` GETs.
- Direct-memory, codec streaming, and metadata early-stop are opt-in or gated.
- Diagnostic metrics add overhead and must not be used as pure throughput proof.
- Metadata cache is topology-sensitive: distributed erasure currently bypasses
  the metadata GET cache, so distributed baselines should assume full metadata
  fanout unless a later change records otherwise.
- Rollout percentage is not the primary safety gate for codec streaming or
  metadata early-stop in the current defaults; record the base enable flag and
  compatibility switches before interpreting any percentage value.
- Codec-streaming cost accounting must include both shard-to-output-buffer and
  output-buffer-to-client copies, plus the `+ Sync` reader wrapper lock cost.

## Artifact Contract

Every run directory must keep enough information to reproduce or reject the
result:

- `git_head` or `git rev-parse HEAD`
- branch name and dirty-tree status
- full script command and arguments
- RustFS binary path and build mode
- object size, object class, bucket, concurrency, duration, rounds
- metric mode: metrics-off throughput or metrics-on attribution
- environment affecting GET path selection
- `manifest.env` and/or `environment.txt`
- `warp/median_summary.csv` and `warp/round_results.csv` for throughput runs
- `service_metrics_summary.csv`, `service_metrics_stage_distribution.csv`, or
  raw before/after Prometheus snapshots for attribution runs

Do not compare a metrics-on run against a metrics-off run as a throughput win or
loss. The only valid cross-mode claim is attribution.

## Baseline Dimensions

Object sizes:

- `1KiB`
- `4KiB`
- `10KiB`
- `64KiB`
- `100KiB`
- `128KiB`
- `1MiB`
- `10MiB`
- `128MiB`

Object classes:

- default inline candidate
- forced non-inline candidate
- plain single-part
- multipart
- range request
- versioned bucket
- checksum-mode request
- CORS request

Run the small default-path proof first. Expand into large, range, multipart, and
versioned dimensions only after the small matrix has stable artifacts.

## Step 0: Script Syntax Gate

```bash
for s in \
  scripts/run_get_codec_streaming_smoke.sh \
  scripts/run_get_metrics_gate_smoke.sh \
  scripts/run_object_batch_bench_enhanced.sh \
  scripts/run_object_data_cache_bench.sh
do
  bash -n "$s"
done
```

## Step 1: Metrics-Off Throughput Smoke

Use this run only for throughput and latency. Observability export and detailed
GET stage attribution are disabled by the script.

```bash
scripts/run_get_metrics_gate_smoke.sh \
  --skip-build \
  --size 1KiB \
  --concurrency 32 \
  --duration 10s \
  --rounds 3 \
  --out-dir target/bench/get-default-metrics-off-1kib-c32
```

Required checks:

- `target/bench/get-default-metrics-off-1kib-c32/manifest.env` records
  observability export as disabled.
- `warp/median_summary.csv` and `warp/round_results.csv` exist.
- No statement about reader path is made from this run alone.

## Step 2: Metrics-On Default Path Attribution

Use this run to prove the default reader path. It may be slower than the
metrics-off smoke and that is expected.

```bash
scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 1KiB,4KiB,10MiB \
  --concurrency 32 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._](reader_path|stage_duration|reader_setup|disk_permit|request|reader)' \
  --out-dir target/bench/get-default-attribution-legacy
```

Required checks:

- `manifest.env` records `mode=legacy` and diagnostic metrics enabled.
- `service_metrics_summary.csv` exists.
- The default path proof reports the delta for
  `rustfs_io_get_object_reader_path_total{path="legacy_duplex"}`.
- If `legacy_duplex` is not positive, report the observed path instead of
  forcing the expected conclusion.

## Step 3: Small-Object Matrix

Run the small sizes with metrics-off first:

```bash
for size in 1KiB 4KiB 10KiB 64KiB 100KiB 128KiB 1MiB; do
  scripts/run_get_metrics_gate_smoke.sh \
    --skip-build \
    --size "$size" \
    --concurrency 32 \
    --duration 10s \
    --rounds 3 \
    --out-dir "target/bench/get-default-metrics-off-${size}-c32"
done
```

Then run attribution only for representative sizes:

```bash
scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 1KiB,10KiB,128KiB,1MiB \
  --concurrency 32 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._](reader_path|stage_duration|reader_setup|disk_permit|request|reader)' \
  --out-dir target/bench/get-default-small-attribution
```

## Step 4: Large Sequential and Range Matrix

Use the dedicated GT1G helper for very large sequential/ranged GETs when the
target machine has enough local disk and time:

```bash
scripts/run_gt1g_get_http_matrix.sh \
  --skip-build \
  --out-dir target/bench/get-default-large-http-matrix
```

For a shorter release-gate matrix:

```bash
scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 10MiB,128MiB \
  --concurrency 16 \
  --duration 20s \
  --rounds 3 \
  --out-dir target/bench/get-default-large-metrics-off
```

Add `--diagnostic-metrics` only in a separate attribution run.

## Step 5: Codec Streaming A/B Matrix

Codec streaming is not default-path proof. Use it only after the default legacy
path has been proven.

```bash
scripts/run_get_codec_streaming_smoke.sh \
  --mode both \
  --profile-order reverse \
  --codec-engine rustfs \
  --sizes 1MiB,10MiB,128MiB \
  --concurrency 16 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --out-dir target/bench/get-codec-ab-attribution
```

Required checks:

- Legacy profile has a positive `legacy_duplex` path delta.
- Codec profile has a positive `codec_streaming` path delta only when codec
  settings are enabled.
- Compatibility summaries pass before any performance conclusion is accepted.

## Step 6: Admission and Permit Fallback Stress

Use this to validate workload admission visibility under pressure:

```bash
RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=1 \
RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=1 \
scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 10MiB \
  --concurrency 64 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._]disk_permit[._]bypass|rustfs_io_disk_permit|rustfs_io_queue_|rustfs_io_get_object_' \
  --out-dir target/bench/get-permit-admission
```

Required checks:

- `rustfs.get_object.disk_permit.bypass.total` or its exported equivalent is
  captured when fallback happens.
- Queue/admission metrics are present in the attribution snapshot.
- Slow-client behavior does not deadlock the benchmark.

## Reporting Template

Use this structure in the final issue or PR comment:

```text
Commit:
Branch:
Dirty tree:
Host:
RustFS binary:
Metric mode:
Command:
Out dir:

Throughput evidence:
- median req/s:
- median latency:
- p90:
- p99:

Attribution evidence:
- reader_path legacy_duplex delta:
- reader_path codec_streaming delta:
- top stage durations:
- disk permit fallback delta:

Conclusion:
- Default path:
- Compatibility:
- Follow-up:
```

## Acceptance Checklist

- Metrics-off throughput and metrics-on attribution are separate directories.
- Every artifact directory records commit, env, object layout, and arguments.
- Default path hit rate is explicitly reported from diagnostic metrics.
- Any default-path claim cites the metric delta used to prove it.
- Codec/direct-memory/metadata-early-stop claims include their enabling env.