test(heal): relative disk target and fail fast on terminal-but-short (#6748)

* test(heal): relative disk target and fail fast on terminal-but-short

The absolute 40 GiB heal target was calibrated to the background scanner
(auto-heal), which is now disabled for determinism; with only the explicit
heal the recovered node lands at ~36 GiB for 40 GiB survivors. Make the
success criterion relative: the outage node must reach at least 90% of the
least-used surviving node (absolute HEAL_TARGET_GB floor optional, default
0 = relative only).

Also fail fast when the heal task reaches a terminal success but the disk
target is not met (previously the monitor kept polling until timeout), and
drop the misleading 'progress absent' warning on the final (cleaned) task
response — mid-run progress is reported correctly.

Validated live: heal summary=finished, 0 failed, vm000/vm001=40GB,
vm002=40GB (target 36GB), test PASSED.

* test(heal): gate success on server verdict + data read-back, drop disk GB gate

The per-node disk-usage target (40 GiB / 90% of survivors) is not a
code-level invariant: EC distributes different shards per node, so the
final GB per node depends on the layout, not on heal correctness. Gate the
test on what the server actually verifies:

- Heal task terminal success (finished/completed) with objectsFailed == 0
  (the server's per-object scan/repair verdict).
- S3 read-back verification: list the test bucket and GET a sample of
  objects, requiring HTTP 200 for every read (end-to-end proof the data is
  still reconstructable after repair). The GET uses a discard mode so
  binary bodies are not captured (no null-byte warnings / SIGPIPE).

Per-node disk usage stays in the output as observability (with a warning if
the outage node gained no usage), not as the pass/fail gate. Removes the
heal_target_gb input and the relative-target logic.

Validated live: heal summary=finished, 0 failed, 20/20 objects read back,
vm002_used=40GB, PASS.
This commit is contained in:
hector
2026-08-27 22:25:17 +08:00
committed by GitHub
parent d48dda5bdc
commit 2e6c820f53
4 changed files with 75 additions and 38 deletions
-5
View File
@@ -15,10 +15,6 @@ on:
description: 'Stop warp when surviving nodes reach N GiB'
required: false
default: '40'
heal_target_gb:
description: 'Outage node must reach N GiB after heal to pass'
required: false
default: '40'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
@@ -100,7 +96,6 @@ jobs:
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
--stop-node-gb "${{ inputs.stop_node_gb }}" \
--warp-stop-gb "${{ inputs.warp_stop_gb }}" \
--heal-target-gb "${{ inputs.heal_target_gb }}" \
--log-file /tmp/rustfs-heal-test.log
- name: Upload test logs