ci(perf): warp A/B relative-budget gate for the hotpath series (#4480)

* ci(perf): warp A/B relative-budget gate for the hotpath series

performance.yml only uploads a samply profile and a cargo-bench artifact with
no pass/fail, so a 26x-class write-path regression like #4221 or the GET perf
churn would sail through CI and only surface in customer load tests. This adds
the missing gate — the acceptance surface every queued HP change (#922/#923/
#925/#927/#930/#932) needs before its default can flip.

- scripts/hotpath_warp_ab_gate.sh: applies the relative budget to the
  baseline_compare.csv that run_object_batch_bench_enhanced.sh already emits
  (it computes deltas but never gates). A metric regressing past --fail-pct
  fails, past --warn-pct warns; reqps/throughput are higher-is-better, latency
  lower-is-better. --allow-regression downgrades a FAIL to an exempted WARN so
  a deliberate correctness cost (e.g. #4221) is recorded, not blocked
  (rustfs/backlog#935 correction 1). Unit-checked across pass/warn/fail/exempt.
- scripts/run_hotpath_warp_ab.sh: single-host baseline-binary vs candidate-
  binary A/B over {put-4mib, get-4mib, mixed-256k} x {drive-sync on, off},
  reusing the enhanced bench as the warp driver and the single-node local-disk
  lifecycle. Has --dry-run; warp is assumed pre-installed as elsewhere in
  scripts/. Drive-sync on/off keeps a sync-semantics change from being masked
  by nosync numbers.
- .github/workflows/performance-ab.yml: nightly on main (post-merge detection)
  plus opt-in pre-merge via the `perf-ab` label; `perf-deliberate-tradeoff`
  runs the gate with --allow-regression; posts the gate table as a PR comment
  and uploads the run. A Linux runner answers "do the macOS conclusions hold".

The gate logic is unit-validated with synthetic compare CSVs; the orchestrator
and workflow are shellcheck- and --dry-run-validated. The first real warp
measurement belongs on the Linux runner (no warp/multi-disk rig locally).

Refs: rustfs/backlog#935 (HP-14 warp A/B gate, item 4), rustfs/backlog#725
(cooled A/B harness precedent), rustfs/backlog#936

Co-Authored-By: heihutu <heihutu@gmail.com>

* ci(perf): pin upload-artifact, add external-cluster A/B mode + runbook

- Pin actions/upload-artifact to the repo-standard full-length SHA (# v6);
  the previous @v4 float tripped the workflow-pin guard.
- Add an external deployment mode to run_hotpath_warp_ab.sh so warp can target
  an already-running cluster instead of a throwaway single-node server:
  --endpoint selects the cluster and --deploy-hook runs between phases to swap
  in the phase's binary and drive-sync config (context passed via
  HOTPATH_AB_PHASE / HOTPATH_AB_BINARY / HOTPATH_AB_DRIVE_SYNC). This maps onto
  the team's ansible harness (cargo zigbuild -> ansible rustfs-manage --tags
  stop,config,binary-copy,start -> warp).
- Restructure the matrix loop to bring a deployment up once per (phase,
  drive-sync) and run all three workloads against it, instead of restarting per
  workload — fewer server starts / cluster redeploys. Baseline medians are read
  from a deterministic path, dropping the bash-4-only associative array so the
  script (and its --dry-run self-check) runs on macOS bash 3.2 too.
- Add docs/operations/hotpath-warp-ab-runbook.md tying local mode, the ansible
  external mode, and the budget/exemption together.

Verification: shellcheck clean; --dry-run in both modes prints the expected
4 deployments x 3 workloads and passes exactly 6 compare CSVs to the gate;
check_workflow_pins.sh, check_doc_paths.sh, and make pre-commit all pass.

Refs: rustfs/backlog#935, rustfs/backlog#936

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

docs/operations/hotpath-warp-ab-runbook.md

@@ -0,0 +1,102 @@
+# Hotpath warp A/B runbook
+
+Relative-budget A/B gate for the hotpath series (rustfs/backlog#935 HP-14). It
+runs the same warp workloads against a **baseline** binary and a **candidate**
+binary, across the drive-sync on/off matrix, then applies a relative budget:
+a metric regressing past the fail budget fails the gate, past the warn budget
+warns. This is how the macOS profiling conclusions of the HP series get
+confirmed or corrected on Linux — structural wins (call counts, read
+amplification) should hold; absolute numbers are whatever the rig measures.
+
+Pieces:
+
+- `scripts/run_hotpath_warp_ab.sh` — orchestrator (baseline vs candidate,
+  workload × drive-sync matrix).
+- `scripts/hotpath_warp_ab_gate.sh` — the budget gate over the
+  `baseline_compare.csv` deltas the load driver emits.
+- `scripts/run_object_batch_bench_enhanced.sh` — the warp driver + median +
+  `baseline_compare.csv` (reused, not reimplemented).
+- `.github/workflows/performance-ab.yml` — nightly on `main` (post-merge
+  detection) plus opt-in pre-merge via the `perf-ab` label.
+
+Metric directions: `reqps` (put obj/s) and `throughput` (get MiB/s) are
+higher-is-better; `latency` / p99 (mixed) is lower-is-better. warp is assumed
+pre-installed, as elsewhere in `scripts/`.
+
+## Local mode (quick / CI smoke)
+
+Builds both binaries and runs a throwaway single-node server on local disks.
+
+```bash
+scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main
+# or with prebuilt binaries:
+scripts/run_hotpath_warp_ab.sh --skip-build \
+  --baseline-bin ./rustfs-main --candidate-bin ./target/release/rustfs
+```
+
+Preview the full plan without running anything:
+
+```bash
+scripts/run_hotpath_warp_ab.sh --dry-run --skip-build \
+  --baseline-bin /tmp/base --candidate-bin /tmp/cand
+```
+
+## External mode (real cluster, ansible-deployed)
+
+For the production-representative run, warp targets an already-running cluster
+and a `--deploy-hook` swaps in each phase's binary and durability config
+between the baseline and candidate phases. The hook receives context via the
+environment:
+
+- `HOTPATH_AB_PHASE` — `baseline` or `candidate`
+- `HOTPATH_AB_BINARY` — binary path (or empty; the hook may build its own)
+- `HOTPATH_AB_DRIVE_SYNC` — `true` or `false` for this matrix cell
+
+This maps directly onto the team's ansible harness. Build the candidate with
+the cross toolchain, stage both binaries, then let the hook drive
+`rustfs-manage.yml`:
+
+```bash
+# 1. Build the candidate (cross-compile for the cluster target).
+cargo zigbuild --release --target x86_64-unknown-linux-gnu -p rustfs --bins
+
+# 2. Run the A/B against the cluster; the hook deploys the phase's binary and
+#    applies the drive-sync config, then restarts, before each phase.
+scripts/run_hotpath_warp_ab.sh \
+  --endpoint "$CLUSTER_ENDPOINT" \
+  --deploy-hook '
+    set -euo pipefail
+    cd /home/xiaomage/xiaomage/ansible
+    # Select the phase binary and the drive-sync value for this cell.
+    cp "${HOTPATH_AB_BINARY:?}" ./roles/rustfs/files/rustfs
+    export RUSTFS_DRIVE_SYNC_ENABLE="$HOTPATH_AB_DRIVE_SYNC"
+    ansible-playbook -f 4 -l testing rustfs-manage.yml --tags stop
+    ansible-playbook -f 4 -l testing rustfs-manage.yml --tags config
+    ansible-playbook -f 4 -l testing rustfs-manage.yml --tags binary-copy
+    ansible-playbook -f 4 -l testing rustfs-manage.yml --tags start
+  ' \
+  --baseline-bin /path/to/rustfs-main \
+  --candidate-bin ./target/x86_64-unknown-linux-gnu/release/rustfs
+```
+
+The `config` tag is responsible for threading `RUSTFS_DRIVE_SYNC_ENABLE` (or
+the finer `RUSTFS_DURABILITY_MODE`) into the deployed unit — the hook exports
+it so the config template can pick it up. The rig itself never restarts the
+cluster; lifecycle stays with ansible.
+
+## Budget and exemptions
+
+Default budget: a metric regressing more than **10%** vs baseline fails,
+more than **5%** warns. Tune with `--fail-pct` / `--warn-pct`.
+
+Some regressions are the correct trade — #4221 deliberately paid a large write
+cost to restore power-loss durability. For those, run with
+`--allow-regression` (or add the `perf-deliberate-tradeoff` label in CI): the
+FAIL is recorded and rendered as an exempted WARN, and the gate exits 0.
+
+## Scope note
+
+The gate logic is unit-validated across pass/warn/fail/exempt outcomes; the
+orchestrator and workflow are shellcheck- and `--dry-run`-validated. The first
+real warp measurement belongs on a Linux runner or the ansible cluster — there
+is no warp/multi-disk rig in the repo's local checkout.