mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
ci(perf): fix nightly warp A/B startup failure, make it diagnosable, add small-object + 10MiB cells (#4669)
Both nightly runs of the warp A/B rig failed at server bring-up: `endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s after start. The server binds its public listener only after startup converges (erasure-format load + IAM); its own budget (RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s health poll gave up before a slow cold start on the shared sm-standard-2 runner finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/), so the root cause was invisible in the artifact. Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh): - Health poll is configurable via --health-timeout, default 180s (> the 120s server startup budget). Fails fast if the local server process exits before becoming healthy instead of polling out the full window. - Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they ride along in the CI artifact. On a health failure the rig dumps the last 50 log lines to the job log. Workflow (.github/workflows/performance-ab.yml): - On every run, write gate.md (or, on a pre-gate failure, the failing phase's server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention. - Short measurement matrix (--duration/--rounds/--cooldown) so the expanded matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3 caches the baseline binary (the ~65min double build dominates). - TODO(ci-8/perf-2) hook comment for the failure-alert composite action. Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the budget until perf-3. Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155.
This commit is contained in:
@@ -23,6 +23,29 @@ Metric directions: `reqps` (put obj/s) and `throughput` (get MiB/s) are
|
||||
higher-is-better; `latency` / p99 (mixed) is lower-is-better. warp is assumed
|
||||
pre-installed, as elsewhere in `scripts/`.
|
||||
|
||||
## Workload matrix
|
||||
|
||||
Six workloads × the drive-sync on/off matrix × baseline/candidate = 24 cells:
|
||||
|
||||
| Workload | mode | size | why |
|
||||
| --- | --- | --- | --- |
|
||||
| `put-4kib` / `get-4kib` | put / get | 4KiB | the #4221 fsync regression size (~-10% @4KiB) — previously invisible |
|
||||
| `put-4mib` / `get-4mib` | put / get | 4MiB | bulk obj/s and MiB/s |
|
||||
| `get-10mib` | get | 10MiB | the historical large-GET EOF size |
|
||||
| `mixed-256k` | mixed | 256KiB | p99 latency |
|
||||
|
||||
Sizes are passed to the load driver via `--sizes` (one size per cell); the
|
||||
driver's `DEFAULT_SIZES` covers 1KiB..10MiB, so any of those can be added by
|
||||
editing `WORKLOADS` in `scripts/run_hotpath_warp_ab.sh`. A 1KiB cell is left
|
||||
out for now to keep the nightly matrix under the 90-minute job budget (the
|
||||
baseline+candidate double build dominates until perf-3 caches it); re-enable it
|
||||
once that lands.
|
||||
|
||||
CI runs a **short** warp matrix (`--duration`/`--rounds`/`--cooldown` tuned in
|
||||
`.github/workflows/performance-ab.yml`) so all 24 cells fit the budget without
|
||||
dropping cells. These params are deliberately noisy-but-fast for the Phase-0
|
||||
"keep the pipeline alive" goal; perf-6 recalibrates them.
|
||||
|
||||
## Local mode (quick / CI smoke)
|
||||
|
||||
Builds both binaries and runs a throwaway single-node server on local disks.
|
||||
@@ -94,6 +117,22 @@ cost to restore power-loss durability. For those, run with
|
||||
`--allow-regression` (or add the `perf-deliberate-tradeoff` label in CI): the
|
||||
FAIL is recorded and rendered as an exempted WARN, and the gate exits 0.
|
||||
|
||||
## Diagnosing a failed run
|
||||
|
||||
Each phase's server log and its startup environment are written under the run's
|
||||
output dir (`target/hotpath-ab/<ts>/server-logs/<phase>-sync-<sync>.{log,env}`)
|
||||
and uploaded in the `hotpath-warp-ab-<run>` artifact, so a failure is
|
||||
diagnosable after the fact. On a health-check failure the rig also dumps the
|
||||
last 50 log lines into the job log and the CI job writes the failing phase (or
|
||||
the gate table) into the GitHub step summary.
|
||||
|
||||
Readiness polling waits up to `--health-timeout` seconds (default **180**),
|
||||
which must outlast the server's own startup-readiness budget
|
||||
(`RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS`, default 120s) — a shorter poll on a
|
||||
slow shared runner mis-reports a slow cold start as a failure. In local mode the
|
||||
rig also fails fast if the server process exits before becoming healthy instead
|
||||
of polling out the full budget.
|
||||
|
||||
## Scope note
|
||||
|
||||
The gate logic is unit-validated across pass/warn/fail/exempt outcomes; the
|
||||
|
||||
Reference in New Issue
Block a user