rustfs/docs/operations/hotpath-warp-abba-runbook.md

Hotpath warp ABBA validation runbook

This runbook describes how to collect formal Linux or production-cluster evidence for hotpath performance changes. Use it when a short local A/B smoke run is too noisy to decide whether a regression is real.

The ABBA runner executes each workload and drive-sync cell as:

A1 baseline -> B1 candidate -> B2 candidate -> A2 baseline

B1 and B2 are compared with A1 to measure the candidate delta. A2 is also compared with A1 to measure baseline drift. Treat a candidate regression as actionable only when the A2 drift is passing or materially smaller than the B1 and B2 delta for the same workload.

Scope

Use this runbook for hotpath profiling and performance validation of RustFS object I/O changes, especially when CPU, memory allocation, lock/channel wait time, request throughput, or tail latency is the review question.

The script validates the same workload matrix as the hotpath warp A/B gate:

Workload	mode	size
put-4kib	put	4KiB
put-4mib	put	4MiB
get-4kib	get	4KiB
get-4mib	get	4MiB
get-10mib	get	10MiB
mixed-256k	mixed	256KiB

Each workload runs with RUSTFS_DRIVE_SYNC_ENABLE=true and RUSTFS_DRIVE_SYNC_ENABLE=false, so a full ABBA pass produces 48 measurement cells: 6 workloads x 2 drive-sync modes x 4 ABBA legs.

Prerequisites

Run the formal pass on Linux, not on a laptop smoke environment.

Required tools on the bench host:

• bash, curl, git, and core GNU userland.
• warp on PATH, or pass --warp-bin.
• Two RustFS Linux binaries: one baseline and one candidate.
• Enough isolated disks or directories for the local runner, or an externally managed RustFS cluster for production-like validation.
• Stable host telemetry collection such as pidstat, mpstat, iostat, sar, perf, heaptrack, or the platform's equivalent observability stack.

Cluster-mode requirements:

• A deploy hook that can replace the RustFS binary on every node.
• The hook must apply RUSTFS_DRIVE_SYNC_ENABLE for the current ABBA leg.
• The hook must restart RustFS and return only after the rollout command has been accepted. The ABBA script performs the HTTP readiness wait.
• The benchmark client should run outside the RustFS nodes when possible.
• Do not run against a production data set unless the workload bucket and test credentials are isolated and approved for destructive benchmark traffic.

Build the binaries

Build the baseline from the comparison commit, usually origin/main or the previous accepted release:

git fetch origin main
git switch --detach origin/main
cargo build --release -p rustfs --bins
cp target/release/rustfs /tmp/rustfs-baseline

Build the candidate from the PR commit:

git switch <candidate-branch>
cargo build --release -p rustfs --bins
cp target/release/rustfs /tmp/rustfs-candidate

For cross-compiled cluster binaries, keep both outputs on the bench host and make the deploy hook copy the selected binary to the cluster. The ABBA runner passes the selected binary path through HOTPATH_ABBA_BINARY.

Local Linux runner

Use local mode for a dedicated Linux runner with disposable data paths. This is not a substitute for a production-like cluster, but it is useful before spending cluster time.

scripts/run_hotpath_warp_abba.sh \
  --baseline-bin /tmp/rustfs-baseline \
  --candidate-bin /tmp/rustfs-candidate \
  --address 127.0.0.1:9000 \
  --data-root /var/tmp/rustfs-hotpath-abba \
  --disks 4 \
  --duration 120s \
  --rounds 3 \
  --cooldown 30 \
  --concurrency 16 \
  --out-dir target/hotpath-abba/linux-local

The script starts and stops RustFS for each ABBA leg. The data root is throwaway and should not contain important data.

Production-like cluster runner

Use external mode when RustFS lifecycle is managed by ansible, systemd, a cluster scheduler, or a dedicated deployment harness. In this mode the ABBA script does not start RustFS directly; it calls --deploy-hook before each leg and then waits for http://<endpoint><health-path>.

The deploy hook receives:

Environment variable	Value
HOTPATH_ABBA_LEG	A1, B1, B2, or A2
HOTPATH_ABBA_PHASE	baseline or candidate
HOTPATH_ABBA_BINARY	selected baseline or candidate binary path
HOTPATH_ABBA_DRIVE_SYNC	true or false

Example ansible-shaped command:

scripts/run_hotpath_warp_abba.sh \
  --baseline-bin /srv/rustfs-binaries/rustfs-baseline \
  --candidate-bin /srv/rustfs-binaries/rustfs-candidate \
  --endpoint rustfs-bench.example.internal:9000 \
  --deploy-hook '
    set -euo pipefail
    cd /srv/rustfs-ansible
    cp "${HOTPATH_ABBA_BINARY:?}" roles/rustfs/files/rustfs
    export RUSTFS_DRIVE_SYNC_ENABLE="${HOTPATH_ABBA_DRIVE_SYNC:?}"
    ansible-playbook -f 4 -l bench rustfs-manage.yml --tags stop
    ansible-playbook -f 4 -l bench rustfs-manage.yml --tags config
    ansible-playbook -f 4 -l bench rustfs-manage.yml --tags binary-copy
    ansible-playbook -f 4 -l bench rustfs-manage.yml --tags start
  ' \
  --duration 180s \
  --rounds 5 \
  --cooldown 45 \
  --concurrency 32 \
  --out-dir target/hotpath-abba/cluster-pr-XXXX

For formal evidence, prefer --rounds 5 or higher when the cluster budget allows it. The script enforces --rounds >= 3.

CPU and memory evidence

ABBA warp output answers whether the candidate changed throughput or latency. Collect host telemetry at the same time to explain why.

Recommended minimum:

mkdir -p target/hotpath-abba/cluster-pr-XXXX/telemetry

pidstat -durh 5 > target/hotpath-abba/cluster-pr-XXXX/telemetry/pidstat.txt &
PIDSTAT_PID=$!

mpstat 5 > target/hotpath-abba/cluster-pr-XXXX/telemetry/mpstat.txt &
MPSTAT_PID=$!

iostat -xz 5 > target/hotpath-abba/cluster-pr-XXXX/telemetry/iostat.txt &
IOSTAT_PID=$!

Stop the collectors after the ABBA script exits:

kill "$PIDSTAT_PID" "$MPSTAT_PID" "$IOSTAT_PID"

For deeper CPU attribution, run perf record around one representative workload after the ABBA gate identifies a candidate regression or improvement:

perf record -F 99 -g -- sleep 180
perf report --stdio > target/hotpath-abba/cluster-pr-XXXX/telemetry/perf-report.txt

For allocation profiling, build the candidate with:

cargo build --release -p rustfs --bins --features hotpath-alloc

Then run the same ABBA command with that binary. Compare allocation-heavy function sections only within the same build mode. Do not compare hotpath-alloc binaries directly with default release binaries for throughput acceptance, because allocation instrumentation intentionally changes what is measured.

For CPU hotpath sections emitted by hotpath, build with:

cargo build --release -p rustfs --bins --features hotpath-cpu

Use the CPU-enabled report to explain hotspots after the default or plain hotpath ABBA gate shows a real effect.

Output layout

The ABBA runner writes:

<out-dir>/
  manifest.env
  abba_schedule.csv
  candidate_gate.md
  baseline_drift_gate.md
  summary.md
  <workload>/<sync>/<leg>/median_summary.csv
  <workload>/<sync>/<leg>/baseline_compare.csv

Attach or link at least these files in the issue or PR:

• summary.md
• candidate_gate.md
• baseline_drift_gate.md
• abba_schedule.csv
• every median_summary.csv and baseline_compare.csv for a failed or borderline workload
• host telemetry files used to explain CPU, memory, or disk saturation

Interpretation

Use this decision table:

Candidate gate	A2 drift gate	Interpretation
PASS	PASS	Candidate is acceptable for the measured matrix.
WARN	PASS	Candidate has a small measurable signal; inspect telemetry and decide if it is expected.
FAIL	PASS	Candidate likely regressed the affected workload; investigate before merge.
FAIL	FAIL on the same workload	Environment drift is high; rerun on a quieter runner or increase duration and rounds.
PASS	FAIL	Candidate did not exceed the budget, but the rig was unstable; avoid using the numbers as proof of improvement.

When B1 and B2 disagree, treat the result as inconclusive even if the gate passes. Increase duration, rounds, cooldown, or runner isolation before drawing a conclusion.

AI execution checklist

When delegating the run to an AI agent or an automation runner, provide these inputs explicitly:

• repository checkout and candidate branch or commit;
• baseline commit or binary path;
• candidate binary path;
• runner type: local Linux or external cluster;
• endpoint, access key, secret key source, and region;
• deploy hook path or exact command for cluster mode;
• output directory;
• required duration, rounds, cooldown, concurrency, and fail/warn budgets;
• where to upload artifacts after the run.

The AI agent should execute this sequence:

1. Confirm uname -a, RustFS commits, binary SHA256 sums, warp --version, CPU model, memory size, disk layout, and whether the run is local or cluster.
2. Run scripts/run_hotpath_warp_abba.sh --dry-run with the final arguments.
3. Run the real ABBA command with --rounds >= 3.
4. Preserve the full output directory without editing generated CSV files.
5. Read summary.md, candidate_gate.md, and baseline_drift_gate.md.
6. Summarize only measured facts: candidate deltas, baseline drift, CPU or memory saturation, and any failed workloads.
7. Post the summary and artifact location to the tracking issue or PR.

Do not report a performance win or loss when the baseline drift gate failed on the same workload and no rerun was collected.