ci(e2e): use tmpfs for distributed pool isolation (#7207)

* test(e2e): add distributed 4x4 validation

* ci(e2e): use tmpfs for distributed pool isolation

sm-standard-4 is an ARC pod without usable loop devices, so
mount -o loop fails with ENOENT before any pool filesystem is
attached. Sized tmpfs still gives each pool a distinct st_dev
and independent 1G statfs capacity.

Co-authored-by: RustFS <hello@rustfs.com>

* test(e2e): prove operations overlap data movement

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

.github/workflows/e2e-distributed.yml

@@ -20,6 +20,13 @@
 # `[profile.e2e-distributed]` in `.config/nextest.toml`. Storage-sensitive PRs,
 # nightly runs, and manual dispatches all execute the same fail-closed suite.
 # Upgrade cases download the same pinned previous release as e2e-upgrade.yml.
+#
+# Isolated pool filesystems: expand/decommission/rebalance cases require
+# independent `statfs` capacity. `sm-standard-4` is an ARC pod
+# (`scripts/ci/check_runner_ephemerality.sh`) and usually has no
+# `/dev/loop-control`, so `mount -o loop` fails with ENOENT ("mount failed:
+# No such file or directory"). The prepare step therefore mounts four 1 GiB
+# tmpfs instances and exports them as `RUSTFS_E2E_POOL_ROOTS`.
 
 name: e2e-distributed
 
@@ -101,21 +108,21 @@ jobs:
           mkdir -p "${mount_base}"
           roots=()
           for pool in 0 1 2 3; do
-            image="${mount_base}/pool-${pool}.img"
             mountpoint="${mount_base}/pool-${pool}"
-            truncate -s 1G "${image}"
-            mkfs.ext4 -q -F "${image}"
             mkdir -p "${mountpoint}"
-            sudo mount -o loop,nosuid,nodev "${image}" "${mountpoint}"
+            # sm-standard-4 is an ARC pod without usable loop devices, so
+            # `mount -o loop` fails with ENOENT. Sized tmpfs still reports a
+            # distinct st_dev and independent 1G statfs capacity.
+            sudo mount -t tmpfs -o size=1G,nosuid,nodev,mode=1777 tmpfs "${mountpoint}"
             sudo chmod 1777 "${mountpoint}"
             roots+=("${mountpoint}")
           done
           printf -v joined_roots '%s:' "${roots[@]}"
           echo "RUSTFS_E2E_POOL_ROOTS=${joined_roots%:}" >> "${GITHUB_ENV}"
-          findmnt --noheadings --output TARGET,SOURCE,FSTYPE --target "${roots[0]}"
-          findmnt --noheadings --output TARGET,SOURCE,FSTYPE --target "${roots[1]}"
-          findmnt --noheadings --output TARGET,SOURCE,FSTYPE --target "${roots[2]}"
-          findmnt --noheadings --output TARGET,SOURCE,FSTYPE --target "${roots[3]}"
+          findmnt --noheadings --output TARGET,SOURCE,FSTYPE,SIZE --target "${roots[0]}"
+          findmnt --noheadings --output TARGET,SOURCE,FSTYPE,SIZE --target "${roots[1]}"
+          findmnt --noheadings --output TARGET,SOURCE,FSTYPE,SIZE --target "${roots[2]}"
+          findmnt --noheadings --output TARGET,SOURCE,FSTYPE,SIZE --target "${roots[3]}"
 
       - name: Download pinned previous release
         env:

docs/testing/distributed-e2e.md

@@ -17,7 +17,7 @@ A multi-pool layout in which any pool spans several localhost ports is not expre
 
 Data-movement cases fail closed. A decommission or rebalance test must observe a successful start response, an active state, a clean terminal state, non-zero movement counters, and post-operation object integrity. An unsupported response, HTTP 5xx, missing status fields, cleanup warning, or zero-progress terminal response fails the case; pre/post S3 availability alone is not evidence that movement ran.
 
-The four expansion pools must report independent capacity. Four directories on one runner filesystem all return the same `statfs` totals, so RustFS correctly concludes that no pool is less free than the cluster average and performs no rebalance. The Actions job mounts four isolated ext4 loopback filesystems and exports their absolute paths through `RUSTFS_E2E_POOL_ROOTS`. The harness rejects missing, duplicate, relative, nonexistent, or same-device roots instead of allowing a vacuous movement pass. Planned pool additions stop every process with SIGTERM; hard process termination remains a chaos-only fault. After the fourth pool joins, the harness performs one full graceful persistent restart: this proves the expanded pool map survives restart and ensures movement begins only after every replica can load the converged metadata.
+The four expansion pools must report independent capacity. Four directories on one runner filesystem all return the same `statfs` totals, so RustFS correctly concludes that no pool is less free than the cluster average and performs no rebalance. The Actions job mounts four isolated 1 GiB tmpfs filesystems and exports their absolute paths through `RUSTFS_E2E_POOL_ROOTS`. It does not use ext4 loop devices: the `sm-standard-4` ARC pods have no `/dev/loop-control`, so `mount -o loop` fails with `No such file or directory`. Sized tmpfs still reports a distinct `st_dev` and independent 1 GiB `statfs` capacity. The harness rejects missing, duplicate, relative, nonexistent, or same-device roots instead of allowing a vacuous movement pass. Planned pool additions stop every process with SIGTERM; hard process termination remains a chaos-only fault. After the fourth pool joins, the harness performs one full graceful persistent restart: this proves the expanded pool map survives restart and ensures movement begins only after every replica can load the converged metadata.
 
 The expansion fixture is an all-current-binary fleet, so it initializes pool metadata with the documented V3 write and fleet-confirmation gates. Decommission cases write their baseline objects, version history, and multipart data into pool 0 before adding pools 1–3, then retire pool 0. This makes a passing result evidence of user-data movement rather than merely an internal-metadata counter changing.
 
@@ -58,6 +58,11 @@ Hardware power-loss, physical NIC pull, authenticated inter-node partition, firm
 ```bash
 cargo build -p rustfs --bins
 # Expansion/decommission/rebalance cases require four paths on distinct filesystems.
+# If you do not already have four disks, sized tmpfs is enough:
+#   for p in 0 1 2 3; do
+#     sudo mkdir -p /mnt/rustfs-pool-$p
+#     sudo mount -t tmpfs -o size=1G,nosuid,nodev,mode=1777 tmpfs /mnt/rustfs-pool-$p
+#   done
 export RUSTFS_E2E_POOL_ROOTS=/mnt/rustfs-pool-0:/mnt/rustfs-pool-1:/mnt/rustfs-pool-2:/mnt/rustfs-pool-3
 # Upgrade cases require the pinned previous binary (CI downloads it).
 export RUSTFS_UPGRADE_SOURCE_BINARY=/path/to/rustfs-1.0.0-rc.2