rustfs/docs/benchmark/issue829-default-get-hot-path-runbook-2026-07-05.md

Issue 829 Default GET Hot Path Benchmark and Attribution Runbook

Date: 2026-07-05

Scope

This runbook defines the evidence package for proving the current default GET hot path. It intentionally separates throughput evidence from path attribution:

• Metrics-off runs are throughput evidence.
• Metrics-on diagnostic runs are path attribution evidence.
• Codec and direct-memory experiments are A/B evidence, not default-path proof unless their enabling environment is recorded.

The current review baseline is:

• Ordinary non-inline GETs are expected to use ECStore metadata/cache lookup and the legacy duplex reader path.
• Metadata early-stop is not expected for normal read_data GETs.
• Direct-memory, codec streaming, and metadata early-stop are opt-in or gated.
• Diagnostic metrics add overhead and must not be used as pure throughput proof.
• Metadata cache is topology-sensitive: distributed erasure currently bypasses the metadata GET cache, so distributed baselines should assume full metadata fanout unless a later change records otherwise.
• Rollout percentage is not the primary safety gate for codec streaming or metadata early-stop in the current defaults; record the base enable flag and compatibility switches before interpreting any percentage value.
• Codec-streaming cost accounting must include both shard-to-output-buffer and output-buffer-to-client copies, plus the + Sync reader wrapper lock cost.

Artifact Contract

Every run directory must keep enough information to reproduce or reject the result:

• git_head or git rev-parse HEAD
• branch name and dirty-tree status
• full script command and arguments
• RustFS binary path and build mode
• object size, object class, bucket, concurrency, duration, rounds
• metric mode: metrics-off throughput or metrics-on attribution
• environment affecting GET path selection
• manifest.env and/or environment.txt
• warp/median_summary.csv and warp/round_results.csv for throughput runs
• service_metrics_summary.csv, service_metrics_stage_distribution.csv, or raw before/after Prometheus snapshots for attribution runs

Do not compare a metrics-on run against a metrics-off run as a throughput win or loss. The only valid cross-mode claim is attribution.

Baseline Dimensions

Object sizes:

• 1KiB
• 4KiB
• 10KiB
• 64KiB
• 100KiB
• 128KiB
• 1MiB
• 10MiB
• 128MiB

Object classes:

• default inline candidate
• forced non-inline candidate
• plain single-part
• multipart
• range request
• versioned bucket
• checksum-mode request
• CORS request

Run the small default-path proof first. Expand into large, range, multipart, and versioned dimensions only after the small matrix has stable artifacts.

Step 0: Script Syntax Gate

for s in \
  scripts/run_get_codec_streaming_smoke.sh \
  scripts/run_get_metrics_gate_smoke.sh \
  scripts/run_object_batch_bench_enhanced.sh \
  scripts/run_object_data_cache_bench.sh
do
  bash -n "$s"
done

Step 1: Metrics-Off Throughput Smoke

Use this run only for throughput and latency. Observability export and detailed GET stage attribution are disabled by the script.

scripts/run_get_metrics_gate_smoke.sh \
  --skip-build \
  --size 1KiB \
  --concurrency 32 \
  --duration 10s \
  --rounds 3 \
  --out-dir target/bench/get-default-metrics-off-1kib-c32

Required checks:

• target/bench/get-default-metrics-off-1kib-c32/manifest.env records observability export as disabled.
• warp/median_summary.csv and warp/round_results.csv exist.
• No statement about reader path is made from this run alone.

Step 2: Metrics-On Default Path Attribution

Use this run to prove the default reader path. It may be slower than the metrics-off smoke and that is expected.

scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 1KiB,4KiB,10MiB \
  --concurrency 32 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._](reader_path|stage_duration|reader_setup|disk_permit|request|reader)' \
  --out-dir target/bench/get-default-attribution-legacy

Required checks:

• manifest.env records mode=legacy and diagnostic metrics enabled.
• service_metrics_summary.csv exists.
• The default path proof reports the delta for rustfs_io_get_object_reader_path_total{path="legacy_duplex"}.
• If legacy_duplex is not positive, report the observed path instead of forcing the expected conclusion.

Step 3: Small-Object Matrix

Run the small sizes with metrics-off first:

for size in 1KiB 4KiB 10KiB 64KiB 100KiB 128KiB 1MiB; do
  scripts/run_get_metrics_gate_smoke.sh \
    --skip-build \
    --size "$size" \
    --concurrency 32 \
    --duration 10s \
    --rounds 3 \
    --out-dir "target/bench/get-default-metrics-off-${size}-c32"
done

Then run attribution only for representative sizes:

scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 1KiB,10KiB,128KiB,1MiB \
  --concurrency 32 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._](reader_path|stage_duration|reader_setup|disk_permit|request|reader)' \
  --out-dir target/bench/get-default-small-attribution

Step 4: Large Sequential and Range Matrix

Use the dedicated GT1G helper for very large sequential/ranged GETs when the target machine has enough local disk and time:

scripts/run_gt1g_get_http_matrix.sh \
  --skip-build \
  --out-dir target/bench/get-default-large-http-matrix

For a shorter release-gate matrix:

scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 10MiB,128MiB \
  --concurrency 16 \
  --duration 20s \
  --rounds 3 \
  --out-dir target/bench/get-default-large-metrics-off

Add --diagnostic-metrics only in a separate attribution run.

Step 5: Codec Streaming A/B Matrix

Codec streaming is not default-path proof. Use it only after the default legacy path has been proven.

scripts/run_get_codec_streaming_smoke.sh \
  --mode both \
  --profile-order reverse \
  --codec-engine rustfs \
  --sizes 1MiB,10MiB,128MiB \
  --concurrency 16 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --out-dir target/bench/get-codec-ab-attribution

Required checks:

• Legacy profile has a positive legacy_duplex path delta.
• Codec profile has a positive codec_streaming path delta only when codec settings are enabled.
• Compatibility summaries pass before any performance conclusion is accepted.

Step 6: Admission and Permit Fallback Stress

Use this to validate workload admission visibility under pressure:

RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=1 \
RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=1 \
scripts/run_get_codec_streaming_smoke.sh \
  --mode legacy \
  --sizes 10MiB \
  --concurrency 64 \
  --duration 20s \
  --rounds 3 \
  --diagnostic-metrics \
  --diagnostic-metrics-filter-regex 'rustfs[._]get_object[._]disk_permit[._]bypass|rustfs_io_disk_permit|rustfs_io_queue_|rustfs_io_get_object_' \
  --out-dir target/bench/get-permit-admission

Required checks:

• rustfs.get_object.disk_permit.bypass.total or its exported equivalent is captured when fallback happens.
• Queue/admission metrics are present in the attribution snapshot.
• Slow-client behavior does not deadlock the benchmark.

Reporting Template

Use this structure in the final issue or PR comment:

Commit:
Branch:
Dirty tree:
Host:
RustFS binary:
Metric mode:
Command:
Out dir:

Throughput evidence:
- median req/s:
- median latency:
- p90:
- p99:

Attribution evidence:
- reader_path legacy_duplex delta:
- reader_path codec_streaming delta:
- top stage durations:
- disk permit fallback delta:

Conclusion:
- Default path:
- Compatibility:
- Follow-up:

Acceptance Checklist

• Metrics-off throughput and metrics-on attribution are separate directories.
• Every artifact directory records commit, env, object layout, and arguments.
• Default path hit rate is explicitly reported from diagnostic metrics.
• Any default-path claim cites the metric delta used to prove it.
• Codec/direct-memory/metadata-early-stop claims include their enabling env.