gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-22 12:26:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(heal): add privileged replacement rebuild e2e (#5918)

Browse Source 
* test(heal): add privileged replacement rebuild e2e

Add ignored Linux-only 3x4 automatic replacement coverage for EC8+4 and EC6+6. The tests use real tmpfs mounts in an isolated mount namespace, wait for scanner-driven replacement recovery status, and verify the replacement target with per-version xl.meta and part.N physical census without invoking Admin deep heal.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(test): avoid unsafe in privileged replacement e2e

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(heal): harden privileged replacement e2e

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(heal): prove absent replacement recovery witness

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): prove absent replacement observation

Stop the target node before detaching the test mount so RustFS releases its mount lease instead of continuing to serve the old tmpfs through an open fd. Restart the node with the endpoint absent and wait for the scanner's real readiness rejection in that node's log.

Assert the absent window has no replacement intent, completion proof, checkpoint, healing marker, or Admin v4 durable record for the target before mounting the blank replacement and waiting for automatic recovery plus physical shard census.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): streamline cluster log capture

Move cluster-node log capture out of ClusterNode and into per-node cluster launch configuration so the privileged replacement E2E uses an explicit harness API instead of mutating node identity data.

Reuse the same stdout/stderr capture helper for single-node and cluster processes, and pin the per-node capture behavior with a focused common test.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): harden privileged replacement proof

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-08-11 16:31:32 +08:00
committed by GitHub
parent ddacce6e75
commit 4ac7c56c89
5 changed files with 944 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.config/nextest.toml
+5 -4
View File
@@ -34,7 +34,8 @@ e2e-vault = { max-threads = 1 }
# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
# server and manipulate its disk directories at runtime (crates/e2e_test:
# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
# replacement_privileged_e2e_test when explicitly run as root on Linux). They
# are correct in isolation but resource-heavy; serialize them under nextest's
# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
# servers never run at once. ci-7's nightly picks these up via the e2e suite;
@@ -90,7 +91,7 @@ test-group = 'ecstore-serial-flaky'
# e2e-reliability test-group note above). The matching ci-profile override is at
# the end of the file, after [profile.ci] is declared.
[[profile.default.overrides]]
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
test-group = 'e2e-reliability'
[[profile.default.overrides]]
@@ -155,7 +156,7 @@ retries = 2
# quarantine: no retries, just single-threaded so several 4-disk servers never
# run concurrently when ci-7's nightly runs the full e2e suite.
[[profile.ci.overrides]]
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
test-group = 'e2e-reliability'
# Serialize the multipart crash-consistency scenarios under the ci profile too
@@ -383,7 +384,7 @@ path = "junit.xml"
# quarantine: no retries, just single-threaded so several 4-disk servers never
# run concurrently.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
test-group = 'e2e-reliability'
[[profile.e2e-full.overrides]]