mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
Zhengchao An
5294f36669
bench(rustfs): remove the fake s3_operations benchmark rustfs/benches/s3_operations.rs never measured S3: all three groups (put_object/get_object/list_objects) only black_box(data.len())/black_box(count) with a 'In a real benchmark, this would call the actual S3 client' comment. It gave a false 'S3 operations have benchmark coverage' signal and was dead weight on rustfs/Cargo.toml. Delete the file and its [[bench]] entry. S3-face performance is covered solely by the warp A/B gate (performance-ab.yml); the runbook scope note now says so explicitly. Micro-benchmarks stay at the function level (perf-8). Refs rustfs/backlog#1152 (perf-9).
165 lines
7.7 KiB
Markdown
165 lines
7.7 KiB
Markdown
# Hotpath warp A/B runbook
|
||
|
||
Relative-budget A/B gate for the hotpath series (rustfs/backlog#935 HP-14). It
|
||
runs the same warp workloads against a **baseline** binary and a **candidate**
|
||
binary, across the drive-sync on/off matrix, then applies a relative budget:
|
||
a metric regressing past the fail budget fails the gate, past the warn budget
|
||
warns. This is how the macOS profiling conclusions of the HP series get
|
||
confirmed or corrected on Linux — structural wins (call counts, read
|
||
amplification) should hold; absolute numbers are whatever the rig measures.
|
||
|
||
Pieces:
|
||
|
||
- `scripts/run_hotpath_warp_ab.sh` — orchestrator (baseline vs candidate,
|
||
workload × drive-sync matrix).
|
||
- `scripts/hotpath_warp_ab_gate.sh` — the budget gate over the
|
||
`baseline_compare.csv` deltas the load driver emits.
|
||
- `scripts/run_object_batch_bench_enhanced.sh` — the warp driver + median +
|
||
`baseline_compare.csv` (reused, not reimplemented).
|
||
- `.github/workflows/performance-ab.yml` — nightly on `main` (post-merge
|
||
detection) plus opt-in pre-merge via the `perf-ab` label.
|
||
|
||
Metric directions: `reqps` (put obj/s) and `throughput` (get MiB/s) are
|
||
higher-is-better; `latency` / p99 (mixed) is lower-is-better. warp is assumed
|
||
pre-installed, as elsewhere in `scripts/`.
|
||
|
||
## Workload matrix
|
||
|
||
Six workloads × the drive-sync on/off matrix × baseline/candidate = 24 cells:
|
||
|
||
| Workload | mode | size | why |
|
||
| --- | --- | --- | --- |
|
||
| `put-4kib` / `get-4kib` | put / get | 4KiB | the #4221 fsync regression size (~-10% @4KiB) — previously invisible |
|
||
| `put-4mib` / `get-4mib` | put / get | 4MiB | bulk obj/s and MiB/s |
|
||
| `get-10mib` | get | 10MiB | the historical large-GET EOF size |
|
||
| `mixed-256k` | mixed | 256KiB | p99 latency |
|
||
|
||
Sizes are passed to the load driver via `--sizes` (one size per cell); the
|
||
driver's `DEFAULT_SIZES` covers 1KiB..10MiB, so any of those can be added by
|
||
editing `WORKLOADS` in `scripts/run_hotpath_warp_ab.sh`. A 1KiB cell is left
|
||
out for now to keep the nightly matrix comfortably under budget; re-enable it
|
||
(one line in `WORKLOADS`) once perf-6 recalibrates the warp params.
|
||
|
||
CI runs a **short** warp matrix (`--duration`/`--rounds`/`--cooldown` tuned in
|
||
`.github/workflows/performance-ab.yml`) so all 24 cells fit the budget without
|
||
dropping cells. These params are deliberately noisy-but-fast for the Phase-0
|
||
"keep the pipeline alive" goal; perf-6 recalibrates them.
|
||
|
||
## Baseline binary cache (CI)
|
||
|
||
The nightly no longer builds both binaries from source. Every push to `main`
|
||
runs a `build-baseline-cache` job that builds the release binary once and stores
|
||
it in the actions cache under `rustfs-baseline-<sha>` (perf-3). The A/B job
|
||
restores the binary for `origin/main` by that key and passes it as
|
||
`--baseline-bin`; on the nightly, where the candidate commit equals the baseline
|
||
commit, the same cached binary serves both phases (`--skip-build`) and the run
|
||
does zero source builds — the common path finishes well under 50 minutes. A
|
||
cache miss (binary evicted, or not built for that SHA yet) transparently falls
|
||
back to the source double-build via `--baseline-ref origin/main`.
|
||
|
||
Each `gate.md` ends with a **Provenance** section recording the baseline and
|
||
candidate commit SHAs and whether each binary came from the cache or a source
|
||
build, plus the runner, warp version, and matrix params. `perf-5`/`perf-12`
|
||
reuse this contract for their archived baselines.
|
||
|
||
## Local mode (quick / CI smoke)
|
||
|
||
Builds both binaries and runs a throwaway single-node server on local disks.
|
||
|
||
```bash
|
||
scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main
|
||
# or with prebuilt binaries:
|
||
scripts/run_hotpath_warp_ab.sh --skip-build \
|
||
--baseline-bin ./rustfs-main --candidate-bin ./target/release/rustfs
|
||
```
|
||
|
||
Preview the full plan without running anything:
|
||
|
||
```bash
|
||
scripts/run_hotpath_warp_ab.sh --dry-run --skip-build \
|
||
--baseline-bin /tmp/base --candidate-bin /tmp/cand
|
||
```
|
||
|
||
## External mode (real cluster, ansible-deployed)
|
||
|
||
For the production-representative run, warp targets an already-running cluster
|
||
and a `--deploy-hook` swaps in each phase's binary and durability config
|
||
between the baseline and candidate phases. The hook receives context via the
|
||
environment:
|
||
|
||
- `HOTPATH_AB_PHASE` — `baseline` or `candidate`
|
||
- `HOTPATH_AB_BINARY` — binary path (or empty; the hook may build its own)
|
||
- `HOTPATH_AB_DRIVE_SYNC` — `true` or `false` for this matrix cell
|
||
|
||
This maps directly onto the team's ansible harness. Build the candidate with
|
||
the cross toolchain, stage both binaries, then let the hook drive
|
||
`rustfs-manage.yml`:
|
||
|
||
```bash
|
||
# 1. Build the candidate (cross-compile for the cluster target).
|
||
cargo zigbuild --release --target x86_64-unknown-linux-gnu -p rustfs --bins
|
||
|
||
# 2. Run the A/B against the cluster; the hook deploys the phase's binary and
|
||
# applies the drive-sync config, then restarts, before each phase.
|
||
scripts/run_hotpath_warp_ab.sh \
|
||
--endpoint "$CLUSTER_ENDPOINT" \
|
||
--deploy-hook '
|
||
set -euo pipefail
|
||
cd /home/xiaomage/xiaomage/ansible
|
||
# Select the phase binary and the drive-sync value for this cell.
|
||
cp "${HOTPATH_AB_BINARY:?}" ./roles/rustfs/files/rustfs
|
||
export RUSTFS_DRIVE_SYNC_ENABLE="$HOTPATH_AB_DRIVE_SYNC"
|
||
ansible-playbook -f 4 -l testing rustfs-manage.yml --tags stop
|
||
ansible-playbook -f 4 -l testing rustfs-manage.yml --tags config
|
||
ansible-playbook -f 4 -l testing rustfs-manage.yml --tags binary-copy
|
||
ansible-playbook -f 4 -l testing rustfs-manage.yml --tags start
|
||
' \
|
||
--baseline-bin /path/to/rustfs-main \
|
||
--candidate-bin ./target/x86_64-unknown-linux-gnu/release/rustfs
|
||
```
|
||
|
||
The `config` tag is responsible for threading `RUSTFS_DRIVE_SYNC_ENABLE` (or
|
||
the finer `RUSTFS_DURABILITY_MODE`) into the deployed unit — the hook exports
|
||
it so the config template can pick it up. The rig itself never restarts the
|
||
cluster; lifecycle stays with ansible.
|
||
|
||
## Budget and exemptions
|
||
|
||
Default budget: a metric regressing more than **10%** vs baseline fails,
|
||
more than **5%** warns. Tune with `--fail-pct` / `--warn-pct`.
|
||
|
||
Some regressions are the correct trade — #4221 deliberately paid a large write
|
||
cost to restore power-loss durability. For those, run with
|
||
`--allow-regression` (or add the `perf-deliberate-tradeoff` label in CI): the
|
||
FAIL is recorded and rendered as an exempted WARN, and the gate exits 0.
|
||
|
||
## Diagnosing a failed run
|
||
|
||
Each phase's server log and its startup environment are written under the run's
|
||
output dir (`target/hotpath-ab/<ts>/server-logs/<phase>-sync-<sync>.{log,env}`)
|
||
and uploaded in the `hotpath-warp-ab-<run>` artifact, so a failure is
|
||
diagnosable after the fact. On a health-check failure the rig also dumps the
|
||
last 50 log lines into the job log and the CI job writes the failing phase (or
|
||
the gate table) into the GitHub step summary.
|
||
|
||
Readiness polling waits up to `--health-timeout` seconds (default **180**),
|
||
which must outlast the server's own startup-readiness budget
|
||
(`RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS`, default 120s) — a shorter poll on a
|
||
slow shared runner misreports a slow cold start as a failure. In local mode the
|
||
rig also fails fast if the server process exits before becoming healthy instead
|
||
of polling out the full budget.
|
||
|
||
## Scope note
|
||
|
||
The gate logic is unit-validated across pass/warn/fail/exempt outcomes; the
|
||
orchestrator and workflow are shellcheck- and `--dry-run`-validated. The first
|
||
real warp measurement belongs on a Linux runner or the ansible cluster — there
|
||
is no warp/multi-disk rig in the repo's local checkout.
|
||
|
||
This warp A/B gate is the **only** entry point for S3-face (PutObject /
|
||
GetObject / ListObjects) performance coverage. There is deliberately no
|
||
in-process criterion benchmark for those operations: a criterion harness that
|
||
stands up an embedded server measures the harness, not the S3 path, so it would
|
||
report a number without guarding anything. Micro-benchmarks stay at the
|
||
function level (EC encode, `xl.meta` parse, `rename_data`; perf-8).
|