test(ecstore): isolate tier-free recovery fixture startup

The fixture constructor can acquire the production recovery lock and enter its 60-second idle wait before this test installs its queue and seeds metadata. The later production entrypoint then remains a follower until the original 30-second queue timeout expires. Disable recovery only around this test's setup_test_env await, and restore the environment before invoking the unchanged production entrypoint.

Only the existing test setup expression changes. Production and store/init.rs are byte-identical to the parent. All original assertions, timeouts, lock semantics, cancellation and cleanup remain intact; no assertions or tests were added.

Validation used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true. The following 22 gates ran on tree 8ec358cb42 with an unchanged source identity before and after every command.

Command:
shellcheck --version && actionlint
Exit 0; wall 7.430s.
Observed output:
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; wall 12.587s.
Observed output:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; wall 6.592s.
Observed output:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; wall 43.613s.
Observed output:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; wall 11.368s.
Observed output:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; wall 9.590s.
Observed output:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; wall 6.542s.
Observed output:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; wall 6.551s.
Observed output:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; wall 6.539s.
Observed output:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; wall 6.550s.
Observed output:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; wall 6.602s.
Observed output:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; wall 6.539s.
Observed output:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; wall 6.541s.
Observed output:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; wall 7.242s.
Observed output:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; wall 226.418s.
Observed output excerpt:
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Ran 29 tests in 146.556s
OK
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
fuzz runner seed manifest tests passed
Embedded secret guard self-test passed.
✅ s3s footprint self-test passed
Ran 52 tests in 12.037s
OK
Ran 35 tests in 0.807s
OK
Ran 19 tests in 9.479s
OK
Ran 10 tests in 0.080s
OK
Ran 13 tests in 1.732s
OK
security coverage self-test passed
Ran 10 tests in 0.044s
OK
Ran 30 tests in 10.863s
OK
Ran 10 tests in 1.780s
OK
Ran 33 tests in 0.116s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.100s
OK
Ran 3 tests in 0.651s
OK
PASS: cold stampede script self-test
Log SHA-256: 9d24e8082553ce61149d4aed14fea7db087cca77ba2755493c96b24420cd6fb6

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; wall 6.196s.
Observed output:
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.902s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
Log SHA-256: 09e339ca649a4e25d687119cea3c5f26acb3034d971c18123817a0d0596e7229

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; wall 13.291s.
Observed output:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; wall 6.253s.
Observed output:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; wall 6.638s.
Observed output:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 154.036s.
Observed summary:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 24s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.007s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.454s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.600s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 2b0fc63af63c3771ce924b668828b29fe84b1197edbc40f5fef0bb9a30c28819

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 6.761s.
Observed summary:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.37s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.071s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.669s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.872s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   1.012s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: 9cbd549c0aff7aca2a747fc05ea99cb1ebd3dff375ccbd0185b45f91632a203d

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1124.541s.
Observed summary:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 38s
Log SHA-256: 88054b4c84a3ba92ddfcb832c79f1e0a31fe1e21b598adbe4502b8488244aaf1

All 19 original Quick Checks passed. The unchanged lifecycle test passed 1/1 (0.454s; Summary 0.600s; 5871 original filter skips). The three existing controls passed 3/3 (Summary 1.012s; 5869 original filter skips). Strict RIO Clippy exited 0 (Finished dev profile in 18m 38s; wall 1124.541s). The existing native linker warning about __eh_frame exceeding 16MB was retained; flags, profiles and allowances were not changed.

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: Reverse only the test-local constructor flag from false to true.
The mutant compiled successfully and failed the original queue assertion in crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs:14287 (expected recovered tier-free item: Elapsed(())). Observer prerequisites passed; no earlier failure is counted as a kill.
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 27s
        FAIL [  30.481s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [  30.633s] 1 test run: 0 passed, 1 failed, 5871 skipped
        FAIL [  30.481s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
Red log SHA-256: 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293
The entire observed candidate source was restored byte-for-byte before the same command passed:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 30s
  SETUP PASS [   0.010s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.460s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.613s] 1 test run: 1 passed, 5871 skipped
Restored log SHA-256: 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52

M2: Replace only the actual awaited enqueue at line 1520 with false while consuming its original inputs.
The mutant compiled successfully and failed the original queue assertion in crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs:14290 (expected recovered tier-free item: Elapsed(())). Observer prerequisites passed; no earlier failure is counted as a kill.
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 24s
        FAIL [  30.516s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [  30.662s] 1 test run: 0 passed, 1 failed, 5871 skipped
        FAIL [  30.516s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
Red log SHA-256: 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd
The entire observed candidate source was restored byte-for-byte before the same command passed:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2m 24s
  SETUP PASS [   0.007s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.457s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.602s] 1 test run: 1 passed, 5871 skipped
Restored log SHA-256: e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b

All temporary observers and enqueue mutations were removed before the final gates. Reversing the final setup expression restores the entire parent lifecycle file; the entire init file is also restored. Historical unmodified two-test failures, the two diagnostic compilation failures and the true-intent runtime failure remain recorded and are not mutation credit. Object Lock intent retry policy remains unresolved. These native macOS gates do not establish Linux RIO CI recovery.

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -14196,7 +14196,11 @@ mod tests {
     #[tokio::test]
     #[serial]
     async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
-        let (disk_paths, ecstore) = setup_test_env().await;
+        let (disk_paths, ecstore) =
+            temp_env::async_with_vars([(super::ENV_TIER_FREE_VERSION_RECOVERY_ENABLED, Some("false"))], async {
+                setup_test_env().await
+            })
+            .await;
         let runtime_state = install_unconsumed_runtime_expiry_worker(&ecstore, 1).await;
         let recovery_rx = {
             let state = runtime_state.read().await;