A fixed /tmp path (log file, final heal status, warp log) can be owned by
another user on the shared runner (e.g. a previous root run), which made the
github-runner user fail: tee could not append the test log, the final heal
status write killed step 6 with EACCES, and upload-artifact could not read
stale root-owned warp logs. The heal scenario itself had passed
(summary=finished, vm002 reached the target) before the status-save died.
- Log files fall back to a unique mktemp path when the configured path is not
writable (heal + pool scripts).
- The final heal status is written to a mktemp file (best effort).
- Workflow artifact uploads use globs for the fallback names.
* test(heal): add node-outage heal E2E script and workflow
RustFS heal test on the 3x4 cluster (3 nodes x 4 disks, same
RUSTFS_VOLUMES expression on every node): write data with warp, stop the
outage node mid-write, restart it, start cluster heal via the admin API,
and pass only when the heal task finishes with 0 failures AND the outage
node's disk usage reaches the target.
Includes the GitHub Actions workflow (smoke-testing runner, nightly deb by
default) and a README. Validated end-to-end on the test environment:
40/40/16 GiB before heal -> 40/40/40 GiB after heal, summary=finished.
The script also writes RUSTFS_HEAL_TASK_TIMEOUT_SECS (default 6h) into the
node config because the server default (5 min) is far too short for
healing tens of GiB.
* ci(pool-test): chain heal regression after the pool test
The pool-expansion workflow is now triggered by the Nightly GNU Build
(workflow_run, replacing the schedule) and runs two sequential jobs on the
shared test environment:
1. pool-expansion-test (existing) — skipped if the nightly build failed.
2. heal-test — runs after the pool test regardless of its outcome
(if: always()): a pool failure makes the run red but does not block the
heal regression. Runs the heal script (reset -> install/start 3x4 ->
write/outage -> heal -> verify -> reset).
* test(heal): address review — camelCase progress, fail-closed, workflow hygiene
- Heal progress fields are camelCase in the API (objectsScanned/objectsHealed/
objectsFailed/progressPercentage); read them with a snake_case fallback and
distinguish null (absent) progress from zero, logging null as evidence
(rustfs/backlog#2035) instead of silently coercing.
- Fail closed in step 3: the outage node must actually be inactive after stop,
the write target must be reached, and an unobserved outage or incomplete
write fails the test instead of warning.
- Step 4 waits (bounded) for the cluster to report an active pool after the
outage-node restart instead of swallowing the verification error.
- Heal start fails fast on 400/403 (deterministic request/auth problems) and
only retries transient server errors.
- Disable the background scanner (RUSTFS_HEAL_AUTO_HEAL_ENABLE=false) so the
explicit heal is the only repair mechanism and the outage is observable.
- Workflows: heal and pool share one concurrency group; workflow_run requires
an exact successful nightly conclusion; checkout is pinned to the triggering
SHA; comma-separated step args are quoted (actionlint SC2054).
hector