gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-08 06:13:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: harden GET object performance paths (#4271)

Browse Source 
* fix: harden GET object performance paths

* fix: satisfy GET multipart layer guard

* fix: keep v1 list markers S3 compatible

* perf: tighten GET direct-memory decision

* ci: isolate s3tests from scanner workload

* refactor: simplify get object body lifecycle

* fix: satisfy get object clippy
This commit is contained in:
houseme
2026-07-05 12:03:22 +08:00
committed by GitHub
parent a134717249
commit db277b17a4
12 changed files with 1092 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/benchmark/issue824-get-pressure-results-2026-07-05.md
+86
View File
@@ -0,0 +1,86 @@
# Issue 824 GET Pressure Validation Results
Date: 2026-07-05
## Scope
This run validates the current branch `houseme/get-object-performance-hardening`
against the closest local historical GET metrics gate baseline.
Historical baseline:
- Path: `docs/benchmark/rustfs-target-bench/get-metrics-gate-baseline/warp/median_summary.csv`
- Git head: `92c50156a3cbbdfa454be1d14062098708851f51`
- Parameters: `10MiB`, warp, concurrency `32`, duration `10s`, rounds `3`
- Median throughput: `5734882344.960000` B/s
- Median req/s: `546.920000`
- Median latency: `48.700000` ms
Current run:
- Path: `target/bench/issue824-get-metrics-gate-current-20260705`
- Git head recorded by script: `55ad8df1c2f574178669052093aa185fae8185ca`
- Branch/worktree contained staged and unstaged issue-824 changes.
- Parameters: `10MiB`, warp, concurrency `32`, duration `10s`, rounds `3`
- RustFS binary: `target/release/rustfs`
- Observability export: disabled
- Scanner: disabled
- Command:
```bash
scripts/run_get_metrics_gate_smoke.sh \
--skip-build \
--rustfs-bin target/release/rustfs \
--size 10MiB \
--concurrency 32 \
--duration 10s \
--rounds 3 \
--round-cooldown-secs 1 \
--baseline-csv docs/benchmark/rustfs-target-bench/get-metrics-gate-baseline/warp/median_summary.csv \
--out-dir target/bench/issue824-get-metrics-gate-current-20260705 \
--data-root /private/tmp/issue824-get-metrics-gate-current-20260705
```
## Raw Rounds
| round | throughput | req/s | avg latency | p90 | p99 |
| --- | ---: | ---: | ---: | ---: | ---: |
| 1 | 6505.75 MiB/s | 650.57 | 47.9 ms | 73.0 ms | 103.9 ms |
| 2 | 5631.35 MiB/s | 563.13 | 52.1 ms | 80.1 ms | 106.3 ms |
| 3 | 5830.30 MiB/s | 583.03 | 52.9 ms | 72.2 ms | 92.7 ms |
Corrected median from raw round data:
- Median throughput: `6113512652.800000` B/s (`5830.30 MiB/s`)
- Median req/s: `583.030000`
- Median latency: `52.100000` ms
- Median p90: `73.000000` ms
- Median p99: `103.900000` ms
## Baseline Comparison
| metric | baseline | current | delta |
| --- | ---: | ---: | ---: |
| throughput B/s | 5734882344.960000 | 6113512652.800000 | +6.60% |
| req/s | 546.920000 | 583.030000 | +6.60% |
| avg latency ms | 48.700000 | 52.100000 | +6.98% |
Interpretation:
- No throughput regression was observed in this local release-binary smoke.
- Latency moved higher by 6.98% against the selected baseline. This is not a
blocking regression by itself because host state, cooldown, and branch state
differ, but it should be watched in the PR benchmark matrix.
- The run validates that the current changes do not obviously collapse the
default 10MiB/C32 GET hot path.
## Artifact Caveat
The script completed with exit code `0`, but its generated
`warp/median_summary.csv` and `warp/baseline_compare.csv` contain `N/A` values.
The terminal summary and `warp/round_results.csv` preserved the actual round
values. The corrected medians above are computed from those raw round logs.
Follow-up: fix `scripts/run_object_batch_bench_enhanced.sh` CSV parsing/report
generation so the persisted median and baseline comparison match the terminal
summary.
docs/benchmark/issue829-default-get-hot-path-runbook-2026-07-05.md
+283
View File
@@ -0,0 +1,283 @@
# Issue 829 Default GET Hot Path Benchmark and Attribution Runbook
Date: 2026-07-05
## Scope
This runbook defines the evidence package for proving the current default GET
hot path. It intentionally separates throughput evidence from path attribution:
- Metrics-off runs are throughput evidence.
- Metrics-on diagnostic runs are path attribution evidence.
- Codec and direct-memory experiments are A/B evidence, not default-path proof
unless their enabling environment is recorded.
The current review baseline is:
- Ordinary non-inline GETs are expected to use ECStore metadata/cache lookup and
the legacy duplex reader path.
- Metadata early-stop is not expected for normal `read_data` GETs.
- Direct-memory, codec streaming, and metadata early-stop are opt-in or gated.
- Diagnostic metrics add overhead and must not be used as pure throughput proof.
- Metadata cache is topology-sensitive: distributed erasure currently bypasses
the metadata GET cache, so distributed baselines should assume full metadata
fanout unless a later change records otherwise.
- Rollout percentage is not the primary safety gate for codec streaming or
metadata early-stop in the current defaults; record the base enable flag and
compatibility switches before interpreting any percentage value.
- Codec-streaming cost accounting must include both shard-to-output-buffer and
output-buffer-to-client copies, plus the `+ Sync` reader wrapper lock cost.
## Artifact Contract
Every run directory must keep enough information to reproduce or reject the
result:
- `git_head` or `git rev-parse HEAD`
- branch name and dirty-tree status
- full script command and arguments
- RustFS binary path and build mode
- object size, object class, bucket, concurrency, duration, rounds
- metric mode: metrics-off throughput or metrics-on attribution
- environment affecting GET path selection
- `manifest.env` and/or `environment.txt`
- `warp/median_summary.csv` and `warp/round_results.csv` for throughput runs
- `service_metrics_summary.csv`, `service_metrics_stage_distribution.csv`, or
raw before/after Prometheus snapshots for attribution runs
Do not compare a metrics-on run against a metrics-off run as a throughput win or
loss. The only valid cross-mode claim is attribution.
## Baseline Dimensions
Object sizes:
- `1KiB`
- `4KiB`
- `10KiB`
- `64KiB`
- `100KiB`
- `128KiB`
- `1MiB`
- `10MiB`
- `128MiB`
Object classes:
- default inline candidate
- forced non-inline candidate
- plain single-part
- multipart
- range request
- versioned bucket
- checksum-mode request
- CORS request
Run the small default-path proof first. Expand into large, range, multipart, and
versioned dimensions only after the small matrix has stable artifacts.
## Step 0: Script Syntax Gate
```bash
for s in \
scripts/run_get_codec_streaming_smoke.sh \
scripts/run_get_metrics_gate_smoke.sh \
scripts/run_object_batch_bench_enhanced.sh \
scripts/run_object_data_cache_bench.sh
do
bash -n "$s"
done
```
## Step 1: Metrics-Off Throughput Smoke
Use this run only for throughput and latency. Observability export and detailed
GET stage attribution are disabled by the script.
```bash
scripts/run_get_metrics_gate_smoke.sh \
--skip-build \
--size 1KiB \
--concurrency 32 \
--duration 10s \
--rounds 3 \
--out-dir target/bench/get-default-metrics-off-1kib-c32
```
Required checks:
- `target/bench/get-default-metrics-off-1kib-c32/manifest.env` records
observability export as disabled.
- `warp/median_summary.csv` and `warp/round_results.csv` exist.
- No statement about reader path is made from this run alone.
## Step 2: Metrics-On Default Path Attribution
Use this run to prove the default reader path. It may be slower than the
metrics-off smoke and that is expected.
```bash
scripts/run_get_codec_streaming_smoke.sh \
--mode legacy \
--sizes 1KiB,4KiB,10MiB \
--concurrency 32 \
--duration 20s \
--rounds 3 \
--diagnostic-metrics \
--diagnostic-metrics-filter-regex 'rustfs[._]get_object[._](reader_path|stage_duration|reader_setup|disk_permit|request|reader)' \
--out-dir target/bench/get-default-attribution-legacy
```
Required checks:
- `manifest.env` records `mode=legacy` and diagnostic metrics enabled.
- `service_metrics_summary.csv` exists.
- The default path proof reports the delta for
`rustfs_io_get_object_reader_path_total{path="legacy_duplex"}`.
- If `legacy_duplex` is not positive, report the observed path instead of
forcing the expected conclusion.
## Step 3: Small-Object Matrix
Run the small sizes with metrics-off first:
```bash
for size in 1KiB 4KiB 10KiB 64KiB 100KiB 128KiB 1MiB; do
scripts/run_get_metrics_gate_smoke.sh \
--skip-build \
--size "$size" \
--concurrency 32 \
--duration 10s \
--rounds 3 \
--out-dir "target/bench/get-default-metrics-off-${size}-c32"
done
```
Then run attribution only for representative sizes:
```bash
scripts/run_get_codec_streaming_smoke.sh \
--mode legacy \
--sizes 1KiB,10KiB,128KiB,1MiB \
--concurrency 32 \
--duration 20s \
--rounds 3 \
--diagnostic-metrics \
--diagnostic-metrics-filter-regex 'rustfs[._]get_object[._](reader_path|stage_duration|reader_setup|disk_permit|request|reader)' \
--out-dir target/bench/get-default-small-attribution
```
## Step 4: Large Sequential and Range Matrix
Use the dedicated GT1G helper for very large sequential/ranged GETs when the
target machine has enough local disk and time:
```bash
scripts/run_gt1g_get_http_matrix.sh \
--skip-build \
--out-dir target/bench/get-default-large-http-matrix
```
For a shorter release-gate matrix:
```bash
scripts/run_get_codec_streaming_smoke.sh \
--mode legacy \
--sizes 10MiB,128MiB \
--concurrency 16 \
--duration 20s \
--rounds 3 \
--out-dir target/bench/get-default-large-metrics-off
```
Add `--diagnostic-metrics` only in a separate attribution run.
## Step 5: Codec Streaming A/B Matrix
Codec streaming is not default-path proof. Use it only after the default legacy
path has been proven.
```bash
scripts/run_get_codec_streaming_smoke.sh \
--mode both \
--profile-order reverse \
--codec-engine rustfs \
--sizes 1MiB,10MiB,128MiB \
--concurrency 16 \
--duration 20s \
--rounds 3 \
--diagnostic-metrics \
--out-dir target/bench/get-codec-ab-attribution
```
Required checks:
- Legacy profile has a positive `legacy_duplex` path delta.
- Codec profile has a positive `codec_streaming` path delta only when codec
settings are enabled.
- Compatibility summaries pass before any performance conclusion is accepted.
## Step 6: Admission and Permit Fallback Stress
Use this to validate workload admission visibility under pressure:
```bash
RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=1 \
RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=1 \
scripts/run_get_codec_streaming_smoke.sh \
--mode legacy \
--sizes 10MiB \
--concurrency 64 \
--duration 20s \
--rounds 3 \
--diagnostic-metrics \
--diagnostic-metrics-filter-regex 'rustfs[._]get_object[._]disk_permit[._]bypass|rustfs_io_disk_permit|rustfs_io_queue_|rustfs_io_get_object_' \
--out-dir target/bench/get-permit-admission
```
Required checks:
- `rustfs.get_object.disk_permit.bypass.total` or its exported equivalent is
captured when fallback happens.
- Queue/admission metrics are present in the attribution snapshot.
- Slow-client behavior does not deadlock the benchmark.
## Reporting Template
Use this structure in the final issue or PR comment:
```text
Commit:
Branch:
Dirty tree:
Host:
RustFS binary:
Metric mode:
Command:
Out dir:
Throughput evidence:
- median req/s:
- median latency:
- p90:
- p99:
Attribution evidence:
- reader_path legacy_duplex delta:
- reader_path codec_streaming delta:
- top stage durations:
- disk permit fallback delta:
Conclusion:
- Default path:
- Compatibility:
- Follow-up:
```
## Acceptance Checklist
- Metrics-off throughput and metrics-on attribution are separate directories.
- Every artifact directory records commit, env, object layout, and arguments.
- Default path hit rate is explicitly reported from diagnostic metrics.
- Any default-path claim cites the metric delta used to prove it.
- Codec/direct-memory/metadata-early-stop claims include their enabling env.