rustfs/scripts/test/rustfs_pool_expand.md

RustFS Pool Expansion / Decommission Test

End-to-end test for storage pool expansion, data rebalancing and pool decommission, driven by scripts/test/rustfs_pool_expand.sh.

All pool / rebalance / decommission checks talk to the RustFS admin API directly with SigV4-signed HTTP requests and assert on the JSON responses (jq), so the results are exact and independent of any client CLI output formatting. rc is not required.

What it does

1. Downloads the RustFS .deb package on all nodes (a release tag, or a direct URL such as a nightly/R2 package).
2. Installs it (dpkg -i).
3. Starts the first pool and verifies it via GET /rustfs/admin/v3/pools/list.
4. Writes data with warp and monitors storage usage (GET /rustfs/admin/v3/storageinfo) until the threshold.
5. Expands to a second pool (nodes started in parallel).
6. Starts rebalance (POST /rustfs/admin/v3/rebalance/start) and waits for all pools to report Completed.
7. (3-pool mode) Expands to a third pool, rebalances again.
8. (optional) Decommissions pool 0 (POST /rustfs/admin/v3/pools/decommission), with automatic clear + retry on failure, and waits for decommissionInfo.complete == true.

When an assertion fails, the script prints a per-pool summary and the full JSON response (pool state, progress, failure counters, waitingReason, unresolvedEntries, last rebalance error), so the GitHub Actions log shows exactly where the test stopped. Credentials never appear in the logs.

Self-hosted runner prerequisites

• Register the admin host (e.g. heal) as a runner with the smoke-testing label.
• Install jq, openssl, curl and warp (only needed for --with-warp) on the runner. rc is not required.
• The runner user must be able to SSH to all nodes without a password prompt (~/.ssh/config with keys).
• The nodes need passwordless sudo for the SSH user, resolvable rustfs-node* hostnames in /etc/hosts, and writable data directories.
• The admin API credentials must have the admin:server-info, admin:decommission and admin:rebalance actions.

Configuration

Set these in the repository (secrets/variables):

Kind	Name	Purpose
Secret	RUSTFS_ACCESS_KEY	RustFS access key
Secret	RUSTFS_SECRET_KEY	RustFS secret key
Var	RUSTFS_API_ENDPOINT	Admin API endpoint, e.g. http://10.0.0.7:9000 (RUSTFS_RC_ENDPOINT is used as a fallback)
Var	RUSTFS_NODES	Space-separated node names, e.g. vm000 vm001 vm002
Var	RUSTFS_SSH_USER	SSH user for the nodes, e.g. azureuser

Workflow inputs

Input	Default	Meaning
rustfs_version	1.0.0-rc.3	GitHub release tag to test
package_url	(empty)	Direct .deb URL (e.g. nightly/R2); overrides rustfs_version
pools	3	Expand to 2 or 3 pools
storage_threshold	50	Stop warp writes at N% usage
warp_duration	10m	warp write duration
run_decommission	true	Run decommission (3-pool mode only)
cleanup_before	true	Reset nodes before the test
cleanup_after	true	Reset nodes after the test

⚠️ cleanup_before / cleanup_after run the script's --reset mode, which stops the services and deletes the data directories and config on all nodes. Only use this workflow against a dedicated test environment.

Manual usage

# Full workflow with a release tag
./scripts/test/rustfs_pool_expand.sh --all --with-warp -y \
  --version 1.0.0-rc.3 --endpoint http://10.0.0.7:9000

# Use a direct .deb URL (e.g. nightly package on R2)
./scripts/test/rustfs_pool_expand.sh --all --with-warp -y \
  --package-url https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb \
  --endpoint http://10.0.0.7:9000

# Preflight / reset / single step
./scripts/test/rustfs_pool_expand.sh --preflight --version 1.0.0-rc.3
./scripts/test/rustfs_pool_expand.sh --reset -y
./scripts/test/rustfs_pool_expand.sh --step 9 --finalize-decommission -y

Known issues and caveats

• RustFS 1.0.0-rc.3 fails decommission with a large warp-written bucket ("metacache listing quorum failed / timeout"). If decommission repeatedly fails, reduce the written data (lower storage_threshold) or remove the test bucket, then re-run step 9. The script detects the failure, clears metadata and retries DECOMMISSION_RETRIES times before giving up.
• Multi-pool nodes must start simultaneously (the script does this) or the first node dies with not first disk.
• The admin API is SigV4-signed (host, x-amz-content-sha256: UNSIGNED-PAYLOAD, x-amz-date), matching the signer the RustFS server itself trusts. If the cluster requires a non-default region, set SIGV4_REGION in the script.