* test(heal): add node-outage heal E2E script and workflow
RustFS heal test on the 3x4 cluster (3 nodes x 4 disks, same
RUSTFS_VOLUMES expression on every node): write data with warp, stop the
outage node mid-write, restart it, start cluster heal via the admin API,
and pass only when the heal task finishes with 0 failures AND the outage
node's disk usage reaches the target.
Includes the GitHub Actions workflow (smoke-testing runner, nightly deb by
default) and a README. Validated end-to-end on the test environment:
40/40/16 GiB before heal -> 40/40/40 GiB after heal, summary=finished.
The script also writes RUSTFS_HEAL_TASK_TIMEOUT_SECS (default 6h) into the
node config because the server default (5 min) is far too short for
healing tens of GiB.
* ci(pool-test): chain heal regression after the pool test
The pool-expansion workflow is now triggered by the Nightly GNU Build
(workflow_run, replacing the schedule) and runs two sequential jobs on the
shared test environment:
1. pool-expansion-test (existing) — skipped if the nightly build failed.
2. heal-test — runs after the pool test regardless of its outcome
(if: always()): a pool failure makes the run red but does not block the
heal regression. Runs the heal script (reset -> install/start 3x4 ->
write/outage -> heal -> verify -> reset).
* test(heal): address review — camelCase progress, fail-closed, workflow hygiene
- Heal progress fields are camelCase in the API (objectsScanned/objectsHealed/
objectsFailed/progressPercentage); read them with a snake_case fallback and
distinguish null (absent) progress from zero, logging null as evidence
(rustfs/backlog#2035) instead of silently coercing.
- Fail closed in step 3: the outage node must actually be inactive after stop,
the write target must be reached, and an unobserved outage or incomplete
write fails the test instead of warning.
- Step 4 waits (bounded) for the cluster to report an active pool after the
outage-node restart instead of swallowing the verification error.
- Heal start fails fast on 400/403 (deterministic request/auth problems) and
only retries transient server errors.
- Disable the background scanner (RUSTFS_HEAL_AUTO_HEAL_ENABLE=false) so the
explicit heal is the only repair mechanism and the outage is observable.
- Workflows: heal and pool share one concurrency group; workflow_run requires
an exact successful nightly conclusion; checkout is pinned to the triggering
SHA; comma-separated step args are quoted (actionlint SC2054).
* test(pool): fix warp log path and retry rebalance start
- warp writes now use a unique mktemp log file instead of a fixed
/tmp/rustfs-warp.log: the runner user could not write the stale
root-owned file, which made the background warp process die instantly
(warp never ran). The workflow uploads /tmp/rustfs-warp.*.log.
- rebalance start is retried (6x, 20s apart): nightly builds gate
rebalance activation on a live cross-pool fence fleet capability proof
that takes ~10-20s to re-establish after a pool joins. Verified live:
attempt 1 fails with 500 'pool activation requires a live fleet
capability proof', attempt 2 succeeds.
* test(pool): annotate known server-side issues in failure output
When a node fails to start, grab the rustfs journal tail and match known
server-side error signatures (e.g. the fleet capability proof cold-start
regression, rustfs/backlog#2031), printing a hint with the tracking issue.
Also annotate the rebalance-start retry exhaustion and the rc.3 decommission
metacache-listing failure with actionable guidance.
* fix(ecstore): defer rebalance activation without fleet proof
---------
Co-authored-by: 马登山 <cxymds@qq.com>
Co-authored-by: cxymds <cxymds@gmail.com>
ci(pool-test): fix scheduled runs and read env from secrets or vars
workflow_dispatch inputs are empty for schedule events, so the scheduled
pool test built a broken package URL (--version "") and failed preflight.
Fall back to the latest nightly deb (R2) when no version/package_url input
is given, default the thresholds/duration/pools, and default cleanup to
enabled. Also read RUSTFS_API_ENDPOINT / RUSTFS_NODES / RUSTFS_SSH_USER
from secrets first (variables as fallback) so either configuration works.
* ci(nightly): persist the nightly deb on Cloudflare R2
Upload the deb to artifacts/rustfs/packages/nightly/ (dated name plus a
rustfs-nightly-latest.deb alias) through the same R2 channel package.yml
uses, so the nightly package can be downloaded later with a stable URL.
The step is skipped when the R2 secrets are not configured, keeping the
artifact-only mode intact.
* test: add pool expansion / decommission E2E script and workflow
Add the admin-API based pool expansion, rebalance and decommission test
script (scripts/test/rustfs_pool_expand.sh) plus a workflow_dispatch /
nightly workflow that runs it on a self-hosted runner against real nodes.
The workflow accepts a release tag or a direct .deb URL (e.g. nightly/R2
package) via the package_url input.
* ci(pool-test): run the pool expansion test on the smoke-testing runner
hector