mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-29 08:27:06 +00:00
hector
766d88cc89
* ci(nightly): persist the nightly deb on Cloudflare R2 Upload the deb to artifacts/rustfs/packages/nightly/ (dated name plus a rustfs-nightly-latest.deb alias) through the same R2 channel package.yml uses, so the nightly package can be downloaded later with a stable URL. The step is skipped when the R2 secrets are not configured, keeping the artifact-only mode intact. * test: add pool expansion / decommission E2E script and workflow Add the admin-API based pool expansion, rebalance and decommission test script (scripts/test/rustfs_pool_expand.sh) plus a workflow_dispatch / nightly workflow that runs it on a self-hosted runner against real nodes. The workflow accepts a release tag or a direct .deb URL (e.g. nightly/R2 package) via the package_url input. * ci(pool-test): run the pool expansion test on the smoke-testing runner
107 lines
5.3 KiB
Markdown
107 lines
5.3 KiB
Markdown
# RustFS Pool Expansion / Decommission Test
|
|
|
|
End-to-end test for **storage pool expansion**, **data rebalancing** and
|
|
**pool decommission**, driven by
|
|
[`scripts/test/rustfs_pool_expand.sh`](rustfs_pool_expand.sh).
|
|
|
|
All pool / rebalance / decommission checks talk to the RustFS **admin API
|
|
directly** with SigV4-signed HTTP requests and assert on the JSON responses
|
|
(`jq`), so the results are exact and independent of any client CLI output
|
|
formatting. **`rc` is not required.**
|
|
|
|
## What it does
|
|
|
|
1. Downloads the RustFS `.deb` package on all nodes (a release tag, or a
|
|
direct URL such as a nightly/R2 package).
|
|
2. Installs it (`dpkg -i`).
|
|
3. Starts the first pool and verifies it via
|
|
`GET /rustfs/admin/v3/pools/list`.
|
|
4. Writes data with `warp` and monitors storage usage
|
|
(`GET /rustfs/admin/v3/storageinfo`) until the threshold.
|
|
5. Expands to a second pool (nodes started in parallel).
|
|
6. Starts rebalance (`POST /rustfs/admin/v3/rebalance/start`) and waits for
|
|
**all** pools to report `Completed`.
|
|
7. (3-pool mode) Expands to a third pool, rebalances again.
|
|
8. (optional) Decommissions pool 0
|
|
(`POST /rustfs/admin/v3/pools/decommission`), with automatic clear + retry
|
|
on failure, and waits for `decommissionInfo.complete == true`.
|
|
|
|
When an assertion fails, the script prints a per-pool summary **and the full
|
|
JSON response** (pool state, progress, failure counters, `waitingReason`,
|
|
`unresolvedEntries`, last rebalance error), so the GitHub Actions log shows
|
|
exactly where the test stopped. Credentials never appear in the logs.
|
|
|
|
## Self-hosted runner prerequisites
|
|
|
|
- Register the admin host (e.g. `heal`) as a runner with the
|
|
`smoke-testing` label.
|
|
- Install `jq`, `openssl`, `curl` and `warp` (only needed for `--with-warp`)
|
|
on the runner. `rc` is **not** required.
|
|
- The runner user must be able to SSH to all nodes without a password prompt
|
|
(`~/.ssh/config` with keys).
|
|
- The nodes need passwordless `sudo` for the SSH user, resolvable
|
|
`rustfs-node*` hostnames in `/etc/hosts`, and writable data directories.
|
|
- The admin API credentials must have the `admin:server-info`,
|
|
`admin:decommission` and `admin:rebalance` actions.
|
|
|
|
## Configuration
|
|
|
|
Set these in the repository (secrets/variables):
|
|
|
|
| Kind | Name | Purpose |
|
|
| ------ | --------------------- | ---------------------------------------------- |
|
|
| Secret | `RUSTFS_ACCESS_KEY` | RustFS access key |
|
|
| Secret | `RUSTFS_SECRET_KEY` | RustFS secret key |
|
|
| Var | `RUSTFS_API_ENDPOINT` | Admin API endpoint, e.g. `http://10.0.0.7:9000` (`RUSTFS_RC_ENDPOINT` is used as a fallback) |
|
|
| Var | `RUSTFS_NODES` | Space-separated node names, e.g. `vm000 vm001 vm002` |
|
|
| Var | `RUSTFS_SSH_USER` | SSH user for the nodes, e.g. `azureuser` |
|
|
|
|
## Workflow inputs
|
|
|
|
| Input | Default | Meaning |
|
|
| ------------------- | ------------- | ---------------------------------------- |
|
|
| `rustfs_version` | `1.0.0-rc.3` | GitHub release tag to test |
|
|
| `package_url` | *(empty)* | Direct `.deb` URL (e.g. nightly/R2); overrides `rustfs_version` |
|
|
| `pools` | `3` | Expand to 2 or 3 pools |
|
|
| `storage_threshold` | `50` | Stop warp writes at N% usage |
|
|
| `warp_duration` | `10m` | warp write duration |
|
|
| `run_decommission` | `true` | Run decommission (3-pool mode only) |
|
|
| `cleanup_before` | `true` | Reset nodes before the test |
|
|
| `cleanup_after` | `true` | Reset nodes after the test |
|
|
|
|
> ⚠️ `cleanup_before` / `cleanup_after` run the script's `--reset` mode, which
|
|
> **stops the services and deletes the data directories and config** on all
|
|
> nodes. Only use this workflow against a dedicated test environment.
|
|
|
|
## Manual usage
|
|
|
|
```bash
|
|
# Full workflow with a release tag
|
|
./scripts/test/rustfs_pool_expand.sh --all --with-warp -y \
|
|
--version 1.0.0-rc.3 --endpoint http://10.0.0.7:9000
|
|
|
|
# Use a direct .deb URL (e.g. nightly package on R2)
|
|
./scripts/test/rustfs_pool_expand.sh --all --with-warp -y \
|
|
--package-url https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb \
|
|
--endpoint http://10.0.0.7:9000
|
|
|
|
# Preflight / reset / single step
|
|
./scripts/test/rustfs_pool_expand.sh --preflight --version 1.0.0-rc.3
|
|
./scripts/test/rustfs_pool_expand.sh --reset -y
|
|
./scripts/test/rustfs_pool_expand.sh --step 9 --finalize-decommission -y
|
|
```
|
|
|
|
## Known issues and caveats
|
|
|
|
- RustFS `1.0.0-rc.3` fails decommission with a large `warp`-written bucket
|
|
("metacache listing quorum failed / timeout"). If decommission repeatedly
|
|
fails, reduce the written data (lower `storage_threshold`) or remove the test
|
|
bucket, then re-run step 9. The script detects the failure, clears metadata
|
|
and retries `DECOMMISSION_RETRIES` times before giving up.
|
|
- Multi-pool nodes must start **simultaneously** (the script does this) or the
|
|
first node dies with `not first disk`.
|
|
- The admin API is SigV4-signed (`host`, `x-amz-content-sha256:
|
|
UNSIGNED-PAYLOAD`, `x-amz-date`), matching the signer the RustFS server
|
|
itself trusts. If the cluster requires a non-default region, set
|
|
`SIGV4_REGION` in the script.
|