mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-29 00:17:11 +00:00
test(heal): relative disk target and fail fast on terminal-but-short (#6748)
* test(heal): relative disk target and fail fast on terminal-but-short The absolute 40 GiB heal target was calibrated to the background scanner (auto-heal), which is now disabled for determinism; with only the explicit heal the recovered node lands at ~36 GiB for 40 GiB survivors. Make the success criterion relative: the outage node must reach at least 90% of the least-used surviving node (absolute HEAL_TARGET_GB floor optional, default 0 = relative only). Also fail fast when the heal task reaches a terminal success but the disk target is not met (previously the monitor kept polling until timeout), and drop the misleading 'progress absent' warning on the final (cleaned) task response — mid-run progress is reported correctly. Validated live: heal summary=finished, 0 failed, vm000/vm001=40GB, vm002=40GB (target 36GB), test PASSED. * test(heal): gate success on server verdict + data read-back, drop disk GB gate The per-node disk-usage target (40 GiB / 90% of survivors) is not a code-level invariant: EC distributes different shards per node, so the final GB per node depends on the layout, not on heal correctness. Gate the test on what the server actually verifies: - Heal task terminal success (finished/completed) with objectsFailed == 0 (the server's per-object scan/repair verdict). - S3 read-back verification: list the test bucket and GET a sample of objects, requiring HTTP 200 for every read (end-to-end proof the data is still reconstructable after repair). The GET uses a discard mode so binary bodies are not captured (no null-byte warnings / SIGPIPE). Per-node disk usage stays in the output as observability (with a warning if the outage node gained no usage), not as the pass/fail gate. Removes the heal_target_gb input and the relative-target logic. Validated live: heal summary=finished, 0 failed, 20/20 objects read back, vm002_used=40GB, PASS.
This commit is contained in:
@@ -15,10 +15,6 @@ on:
|
||||
description: 'Stop warp when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '40'
|
||||
heal_target_gb:
|
||||
description: 'Outage node must reach N GiB after heal to pass'
|
||||
required: false
|
||||
default: '40'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
@@ -100,7 +96,6 @@ jobs:
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--stop-node-gb "${{ inputs.stop_node_gb }}" \
|
||||
--warp-stop-gb "${{ inputs.warp_stop_gb }}" \
|
||||
--heal-target-gb "${{ inputs.heal_target_gb }}" \
|
||||
--log-file /tmp/rustfs-heal-test.log
|
||||
|
||||
- name: Upload test logs
|
||||
|
||||
@@ -38,10 +38,6 @@ on:
|
||||
description: 'Heal: stop warp when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '40'
|
||||
heal_target_gb:
|
||||
description: 'Heal: outage node must reach N GiB after heal'
|
||||
required: false
|
||||
default: '40'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
@@ -218,7 +214,6 @@ jobs:
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--stop-node-gb "${{ inputs.stop_node_gb || '15' }}" \
|
||||
--warp-stop-gb "${{ inputs.warp_stop_gb || '40' }}" \
|
||||
--heal-target-gb "${{ inputs.heal_target_gb || '40' }}" \
|
||||
--log-file /tmp/rustfs-heal-test.log
|
||||
|
||||
- name: Upload test logs
|
||||
|
||||
Reference in New Issue
Block a user