`test_heal_resume_across_page_boundary_e2e` panicked whenever the
erasure-set healer deferred a single object version to a later heal
cycle. Under load a per-version `heal_object` can hit a transient error;
`ErasureSetHealer::heal_bucket_with_resume` then persists its
resume/checkpoint state and returns a terminal `Failed { .. "retry
scheduled" }`, expecting a fresh heal run to finish the job. In
production the background scanner is that next run — the e2e had no such
follow-up, so the first task's `Failed` state failed the test. This is a
pre-existing rare flake (the wiped-disk heal is otherwise correct); it is
unrelated to any policy/proptest work that happened to surface it in CI.
Drive the heal to a genuine `Completed` instead: on a `Failed` carrying
the `retry scheduled` marker (retry budget still remaining) re-submit the
idempotent heal (`force_start`), mirroring the production scanner, up to a
small bound. A `Failed` without that marker (e.g. `exhausted retries`) or
any other non-`Completed` terminal state still fails the test, and the
strict per-version data-restoration assertions still run only after a
real `Completed`. `wait_for_task` is refactored onto the shared
`await_terminal_status` poller; its panic-on-failure semantics for the
unversioned-bucket heal are unchanged.
Test-only change; no production heal code is touched.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(utils): add rustfs-test-utils crate, absorb heal/iam ECStore bootstrap
backlog#1153 infra-1. The ~50-line "build a real temp-disk ECStore"
bootstrap was copy-pasted (and drifting) across the heal and iam
integration tests. This adds crates/test-utils (rustfs-test-utils, a
dev-dependency-only crate) owning that bootstrap and converts the four
copies into thin wrappers:
- TestECStoreEnvBuilder: disk_count (default 4), prefix (uuid-suffixed
/tmp dir), base_dir (caller-owned dir, e.g. tempfile::TempDir),
init_bucket_metadata (default true; the iam bootstrap test opts out
to preserve its historical semantics). TestECStoreEnv exposes
temp_root/disk_paths/ecstore plus a versioned-bucket helper, and
init_tracing() replaces the per-file Once blocks.
- All rustfs_ecstore imports stay behind src/ecstore_test_compat.rs,
the sanctioned test-compat boundary pattern (mirrors
crates/iam/tests/ecstore_test_compat).
- heal: heal_integration_test / heal_b5_versioned_regression_test /
heal_b920_subquorum_union_test drop their setup_test_env{,_n} copies
for heal_env{,_n} wrappers; the tests/storage_api.rs integration
surface shrinks to what test bodies still touch.
- iam: iam_bootstrap_no_lock_test drops build_local_ecstore; its
ecstore_test_compat fixture shrinks to SetupType +
update_erasure_type.
rg 'async fn setup_test_env' crates/heal crates/iam now returns 0.
Scanner's lifecycle tests are deliberately NOT absorbed (gated on
ilm-1; 14 of 15 are #[ignore]d today). Net -230 lines.
* fix(heal): drop tokio::fs import orphaned by the b920 bootstrap move
* fix(heal): drop tokio::fs import orphaned by the b5 bootstrap move
Disk-replacement heal previously repaired only the latest version of each
object and never enumerated objects whose latest version is a delete marker,
so old versions were left unrepaired on a replaced drive.
Switch heal enumeration from list_objects_v2 (latest-only) to
list_object_versions (every version incl. delete markers), thread the concrete
version_id into the existing per-version heal_object, and make resume
cursor-based instead of positional: an opaque (marker, version_marker) paging
token persisted in ResumeState, a length-prefixed injective per-version dedup
key, schema_version bumps (v2) migrated independently in each of the two
persisted files, and a retry that resets both managers together (fixing a
latent rescan-skips-everything defect). Adds a real-disk-wipe e2e regression
suite proving old versions and delete-marker-latest objects are physically
restored.
Fixesrustfs/backlog#918Fixesrustfs/backlog#919Closesrustfs/backlog#854
houseme