ci(nightly): persist the nightly deb on Cloudflare R2 (#6643)

* ci(nightly): persist the nightly deb on Cloudflare R2

Upload the deb to artifacts/rustfs/packages/nightly/ (dated name plus a
rustfs-nightly-latest.deb alias) through the same R2 channel package.yml
uses, so the nightly package can be downloaded later with a stable URL.
The step is skipped when the R2 secrets are not configured, keeping the
artifact-only mode intact.

* test: add pool expansion / decommission E2E script and workflow

Add the admin-API based pool expansion, rebalance and decommission test
script (scripts/test/rustfs_pool_expand.sh) plus a workflow_dispatch /
nightly workflow that runs it on a self-hosted runner against real nodes.
The workflow accepts a release tag or a direct .deb URL (e.g. nightly/R2
package) via the package_url input.

* ci(pool-test): run the pool expansion test on the smoke-testing runner

.github/workflows/nightly-gnu.yml

@@ -161,6 +161,49 @@ jobs:
           path: ${{ steps.deb.outputs.deb_file }}
           if-no-files-found: error
 
+      # Persist the nightly deb on Cloudflare R2 (same channel as package.yml)
+      # so it can be downloaded later with a stable, unauthenticated URL —
+      # e.g. https://dl.rustfs.com/artifacts/rustfs/packages/nightly/... .
+      # Skipped when the R2 secrets are not configured (artifact-only mode).
+      - name: Upload DEB to Cloudflare R2
+        if: env.R2_ACCESS_KEY_ID != ''
+        env:
+          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
+          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
+          R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
+          R2_BUCKET: ${{ secrets.R2_BUCKET }}
+          AWS_EC2_METADATA_DISABLED: true
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
+            echo "⚠️ R2 credentials missing, skipping upload"
+            exit 0
+          fi
+
+          if ! command -v aws >/dev/null 2>&1; then
+            sudo apt-get update && sudo apt-get install -y -qq awscli
+          fi
+
+          export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
+          export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
+          export AWS_DEFAULT_REGION="auto"
+
+          DEB_FILE="${{ steps.deb.outputs.deb_file }}"
+          R2_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/nightly/"
+
+          echo "📤 Uploading ${DEB_FILE} to ${R2_PREFIX}"
+          aws s3 cp "${DEB_FILE}" "${R2_PREFIX}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
+
+          # Stable "latest" alias so tests can fetch the newest nightly
+          # without knowing today's date.
+          echo "📤 Uploading latest alias"
+          aws s3 cp "${DEB_FILE}" "${R2_PREFIX}rustfs-nightly-latest.deb" \
+            --endpoint-url "$R2_ENDPOINT" --only-show-errors
+
+          echo "✅ R2 upload complete"
+
   # Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
   #
   # RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and

.github/workflows/rustfs-pool-expand-test.yml

@@ -0,0 +1,142 @@
+name: RustFS Pool Expansion / Decommission Test
+
+on:
+  workflow_dispatch:
+    inputs:
+      rustfs_version:
+        description: 'RustFS release tag to test (e.g. 1.0.0-rc.3)'
+        required: false
+        default: '1.0.0-rc.3'
+      package_url:
+        description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.'
+        required: false
+        type: string
+      pools:
+        description: 'Number of pools to expand to (2 = first rebalance only)'
+        type: choice
+        options:
+          - '2'
+          - '3'
+        default: '3'
+      storage_threshold:
+        description: 'Stop writing when storage usage reaches N%'
+        required: false
+        default: '50'
+      warp_duration:
+        description: 'warp write duration (e.g. 5m, 10m)'
+        required: false
+        default: '10m'
+      run_decommission:
+        description: 'Run the pool decommission step (3-pool topology only)'
+        type: boolean
+        default: true
+      cleanup_before:
+        description: 'Reset the nodes before the test (DESTROYS existing data/config)'
+        type: boolean
+        default: true
+      cleanup_after:
+        description: 'Reset the nodes after the test (DESTROYS test data/config)'
+        type: boolean
+        default: true
+  schedule:
+    # Nightly regression run; remove if you do not want a schedule.
+    - cron: '0 21 * * *'
+
+permissions:
+  contents: read
+
+# Only one pool-expansion test at a time: the workflow mutates a shared
+# test environment, so concurrent runs must not clobber each other.
+concurrency:
+  group: rustfs-pool-expansion-test
+  cancel-in-progress: false
+
+defaults:
+  run:
+    shell: bash
+
+env:
+  RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
+  RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
+  RUSTFS_API_ENDPOINT: ${{ vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
+  RUSTFS_NODES: ${{ vars.RUSTFS_NODES }}
+  RUSTFS_SSH_USER: ${{ vars.RUSTFS_SSH_USER }}
+
+jobs:
+  pool-expansion-test:
+    runs-on: smoke-testing
+    timeout-minutes: 360
+    steps:
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          persist-credentials: false
+
+      - name: Show environment
+        run: |
+          uname -a
+          jq --version
+          openssl version
+          warp --version || true
+          df -h /data | tail -1
+
+      - name: Reset test environment (before)
+        if: ${{ inputs.cleanup_before }}
+        run: |
+          chmod +x scripts/test/rustfs_pool_expand.sh
+          ./scripts/test/rustfs_pool_expand.sh --reset -y
+
+      - name: Preflight checks
+        run: |
+          ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
+          if [ -n "${{ inputs.package_url }}" ]; then
+            ARGS+=(--package-url "${{ inputs.package_url }}")
+          else
+            ARGS+=(--version "${{ inputs.rustfs_version }}")
+          fi
+          ./scripts/test/rustfs_pool_expand.sh "${ARGS[@]}"
+
+      - name: Run pool expansion & decommission test
+        id: pool_test
+        run: |
+          set -o pipefail
+          STEPS="1,2,3,4,5,6"
+          if [ "${{ inputs.pools }}" = "3" ]; then
+            STEPS="$STEPS,7,8"
+            if [ "${{ inputs.run_decommission }}" = "true" ]; then
+              STEPS="$STEPS,9"
+            fi
+          fi
+          ARGS=(--steps "$STEPS" --with-warp -y \
+            --endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
+            --storage-threshold "${{ inputs.storage_threshold }}" \
+            --warp-duration "${{ inputs.warp_duration }}" \
+            --log-file /tmp/rustfs-pool-test.log)
+          if [ -n "${{ inputs.package_url }}" ]; then
+            ARGS+=(--package-url "${{ inputs.package_url }}")
+          else
+            ARGS+=(--version "${{ inputs.rustfs_version }}")
+          fi
+          ./scripts/test/rustfs_pool_expand.sh "${ARGS[@]}"
+
+      - name: Upload test logs
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: rustfs-pool-test-${{ github.run_id }}
+          path: |
+            /tmp/rustfs-pool-test.log
+            /tmp/rustfs-warp.log
+          if-no-files-found: warn
+
+      - name: Reset test environment (after)
+        if: ${{ always() && inputs.cleanup_after }}
+        run: |
+          ./scripts/test/rustfs_pool_expand.sh --reset -y
+
+      - name: Notify on failure
+        if: failure()
+        run: |
+          echo "RustFS pool expansion test failed"
+          echo "Package source: ${{ inputs.package_url || format('release {0}', inputs.rustfs_version) }}"
+          echo "See the uploaded log artifact for details."

scripts/test/rustfs_pool_expand.md

@@ -0,0 +1,106 @@
+# RustFS Pool Expansion / Decommission Test
+
+End-to-end test for **storage pool expansion**, **data rebalancing** and
+**pool decommission**, driven by
+[`scripts/test/rustfs_pool_expand.sh`](rustfs_pool_expand.sh).
+
+All pool / rebalance / decommission checks talk to the RustFS **admin API
+directly** with SigV4-signed HTTP requests and assert on the JSON responses
+(`jq`), so the results are exact and independent of any client CLI output
+formatting. **`rc` is not required.**
+
+## What it does
+
+1. Downloads the RustFS `.deb` package on all nodes (a release tag, or a
+   direct URL such as a nightly/R2 package).
+2. Installs it (`dpkg -i`).
+3. Starts the first pool and verifies it via
+   `GET /rustfs/admin/v3/pools/list`.
+4. Writes data with `warp` and monitors storage usage
+   (`GET /rustfs/admin/v3/storageinfo`) until the threshold.
+5. Expands to a second pool (nodes started in parallel).
+6. Starts rebalance (`POST /rustfs/admin/v3/rebalance/start`) and waits for
+   **all** pools to report `Completed`.
+7. (3-pool mode) Expands to a third pool, rebalances again.
+8. (optional) Decommissions pool 0
+   (`POST /rustfs/admin/v3/pools/decommission`), with automatic clear + retry
+   on failure, and waits for `decommissionInfo.complete == true`.
+
+When an assertion fails, the script prints a per-pool summary **and the full
+JSON response** (pool state, progress, failure counters, `waitingReason`,
+`unresolvedEntries`, last rebalance error), so the GitHub Actions log shows
+exactly where the test stopped. Credentials never appear in the logs.
+
+## Self-hosted runner prerequisites
+
+- Register the admin host (e.g. `heal`) as a runner with the
+  `smoke-testing` label.
+- Install `jq`, `openssl`, `curl` and `warp` (only needed for `--with-warp`)
+  on the runner. `rc` is **not** required.
+- The runner user must be able to SSH to all nodes without a password prompt
+  (`~/.ssh/config` with keys).
+- The nodes need passwordless `sudo` for the SSH user, resolvable
+  `rustfs-node*` hostnames in `/etc/hosts`, and writable data directories.
+- The admin API credentials must have the `admin:server-info`,
+  `admin:decommission` and `admin:rebalance` actions.
+
+## Configuration
+
+Set these in the repository (secrets/variables):
+
+| Kind   | Name                  | Purpose                                        |
+| ------ | --------------------- | ---------------------------------------------- |
+| Secret | `RUSTFS_ACCESS_KEY`   | RustFS access key                              |
+| Secret | `RUSTFS_SECRET_KEY`   | RustFS secret key                              |
+| Var    | `RUSTFS_API_ENDPOINT` | Admin API endpoint, e.g. `http://10.0.0.7:9000` (`RUSTFS_RC_ENDPOINT` is used as a fallback) |
+| Var    | `RUSTFS_NODES`        | Space-separated node names, e.g. `vm000 vm001 vm002` |
+| Var    | `RUSTFS_SSH_USER`     | SSH user for the nodes, e.g. `azureuser`       |
+
+## Workflow inputs
+
+| Input               | Default       | Meaning                                  |
+| ------------------- | ------------- | ---------------------------------------- |
+| `rustfs_version`    | `1.0.0-rc.3`  | GitHub release tag to test               |
+| `package_url`       | *(empty)*     | Direct `.deb` URL (e.g. nightly/R2); overrides `rustfs_version` |
+| `pools`             | `3`           | Expand to 2 or 3 pools                   |
+| `storage_threshold` | `50`          | Stop warp writes at N% usage             |
+| `warp_duration`     | `10m`         | warp write duration                      |
+| `run_decommission`  | `true`        | Run decommission (3-pool mode only)      |
+| `cleanup_before`    | `true`        | Reset nodes before the test              |
+| `cleanup_after`     | `true`        | Reset nodes after the test               |
+
+> ⚠️ `cleanup_before` / `cleanup_after` run the script's `--reset` mode, which
+> **stops the services and deletes the data directories and config** on all
+> nodes. Only use this workflow against a dedicated test environment.
+
+## Manual usage
+
+```bash
+# Full workflow with a release tag
+./scripts/test/rustfs_pool_expand.sh --all --with-warp -y \
+  --version 1.0.0-rc.3 --endpoint http://10.0.0.7:9000
+
+# Use a direct .deb URL (e.g. nightly package on R2)
+./scripts/test/rustfs_pool_expand.sh --all --with-warp -y \
+  --package-url https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb \
+  --endpoint http://10.0.0.7:9000
+
+# Preflight / reset / single step
+./scripts/test/rustfs_pool_expand.sh --preflight --version 1.0.0-rc.3
+./scripts/test/rustfs_pool_expand.sh --reset -y
+./scripts/test/rustfs_pool_expand.sh --step 9 --finalize-decommission -y
+```
+
+## Known issues and caveats
+
+- RustFS `1.0.0-rc.3` fails decommission with a large `warp`-written bucket
+  ("metacache listing quorum failed / timeout"). If decommission repeatedly
+  fails, reduce the written data (lower `storage_threshold`) or remove the test
+  bucket, then re-run step 9. The script detects the failure, clears metadata
+  and retries `DECOMMISSION_RETRIES` times before giving up.
+- Multi-pool nodes must start **simultaneously** (the script does this) or the
+  first node dies with `not first disk`.
+- The admin API is SigV4-signed (`host`, `x-amz-content-sha256:
+  UNSIGNED-PAYLOAD`, `x-amz-date`), matching the signer the RustFS server
+  itself trusts. If the cluster requires a non-default region, set
+  `SIGV4_REGION` in the script.