mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
Zhengchao An
fd18b1df49
Both nightly runs of the warp A/B rig failed at server bring-up: `endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s after start. The server binds its public listener only after startup converges (erasure-format load + IAM); its own budget (RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s health poll gave up before a slow cold start on the shared sm-standard-2 runner finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/), so the root cause was invisible in the artifact. Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh): - Health poll is configurable via --health-timeout, default 180s (> the 120s server startup budget). Fails fast if the local server process exits before becoming healthy instead of polling out the full window. - Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they ride along in the CI artifact. On a health failure the rig dumps the last 50 log lines to the job log. Workflow (.github/workflows/performance-ab.yml): - On every run, write gate.md (or, on a pre-gate failure, the failing phase's server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention. - Short measurement matrix (--duration/--rounds/--cooldown) so the expanded matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3 caches the baseline binary (the ~65min double build dominates). - TODO(ci-8/perf-2) hook comment for the failure-alert composite action. Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the budget until perf-3. Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155.