test(e2e): cluster harness drivesPerNode + 2-pool topology (backlog #1325) (#4937)

test(e2e): add drivesPerNode and 2-pool cluster harness topology (backlog #1325)

Extend the e2e cluster harness so tests can declare per-node multiple drives (drivesPerNode) and a two-pool topology, alongside a smoke suite that boots such a cluster and round-trips a PUT/GET.

A new `ClusterTopology` describes node_count, drives_per_node, and pool membership, and `RustFSTestClusterEnvironment::with_topology` builds the matching on-disk drive directories and `RUSTFS_VOLUMES` string. `new(node_count)` now delegates to a single-pool single-drive topology, so existing single-drive cluster tests are byte-for-byte unchanged. `ClusterNode` gains `data_dirs` (all drives, `data_dirs[0] == data_dir`) and `pool_idx`; multi-drive/multi-pool clusters automatically add `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` because their drives share the same temp filesystem.

The `RUSTFS_VOLUMES` assembly matches the two forms the server parser accepts on a single localhost machine (verified against ecstore `DisksLayout::from_volumes`): a single pool is the explicit enumeration of every `(node, drive)` endpoint (no ellipses, one legacy DistErasure pool), while multiple pools each contribute one ellipses argument `http://<addr><node-base>/drive{0...N-1}`. A pool argument is a single URL template, so it cannot enumerate multiple distinct-port hosts; a pool striped across several localhost nodes would need a host ellipses that forces a shared on-disk path and collides physically. The topology validator therefore requires every pool in a multi-pool layout to own exactly one node and requires drives_per_node >= 2 (the parser rejects a single-drive `drive{0...0}` ellipses pool). Genuine multi-node pools need real multi-host infrastructure and are deferred to the nightly cluster lane (backlog #1313/#1314).

Unit tests assert the volumes-string layout for single-pool single-drive (backward compatible), single-pool multi-drive (8 explicit endpoints), and two-pool (one ellipses arg per pool), plus the validator rejections. The smoke suite `cluster_multidrive_pool_test` boots a 4-node x 2-drive single pool and a 2-node/2-pool cluster, asserts the volumes layout, and round-trips a PUT/GET using the harness readiness handshake (no fixed sleeps). It joins the six existing RustFSTestClusterEnvironment suites excluded from the merge-gate `e2e-full` profile so it runs in the nightly 4-node lane.

Network fault injection (toxiproxy / socket proxy) and 5GiB large-object budgets remain out of scope for this block.
This commit is contained in:
Zhengchao An
2026-07-17 09:02:38 +08:00
committed by GitHub
parent be2e454d9d
commit b0b5ffb8e2
4 changed files with 493 additions and 18 deletions
+3 -3
View File
@@ -263,8 +263,8 @@ path = "junit.xml"
# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
# * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed
# ports; they join a scheduled lane once ci-6 randomises the ports (ci-7).
# * the 6 cluster suites that spin up a RustFSTestClusterEnvironment
# (cluster_concurrency, stale_multipart_cleanup_cluster,
# * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
# (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
# namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
# object_lambda) — too heavy for the merge budget; they run in ci-7's
# nightly 4-node lane.
@@ -301,7 +301,7 @@ path = "junit.xml"
default-filter = """
package(e2e_test)
& !test(/^protocols::/)
& !test(/^(admin_timeout_regression_test|cluster_concurrency_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
& !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
& !test(/^replication_extension_test::/)
& !test(/^multipart_auth_test::test_signed_put_object_extract_(accepts_compat_header|expands_tar_entries_with_prefix_headers|expands_tar_gz_archive|expands_tbz2_archive|expands_tgz_archive|expands_txz_archive|expands_tzst_archive|normalizes_prefix_header_value|preserves_directory_markers_by_default|preserves_object_lock_legal_hold|preserves_object_lock_retention|preserves_pax_metadata_and_version_id|preserves_request_metadata_on_extracted_objects|preserves_sse_c|preserves_sse_s3_and_redirect|preserves_storage_class|uses_bucket_default_sse_s3)$/)
& !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(prefers_exact_minio_prefix_over_suffix_fallback|supports_minio_prefix_and_directory_markers)$/)