Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(ecstore): drop the client shim, import rustfs-s3-client directly
Completes the migration window opened by the rustfs-s3-client extraction (rustfs/backlog#1842 PR3): every consumer now imports the client crate directly and the crate::client shim is deleted.
- All in-crate crate::client:: paths (tier warm backends, tier core, lifecycle tier_sweeper, replication storage boundary, set_disk) now import rustfs_s3_client::* directly; crates/ecstore/src/client/mod.rs and the lib.rs mod client declaration are gone.
- The two server-side modules historically misfiled under client/ move to their real homes: object_api_utils.rs to crates/ecstore/src/object_api/ (it builds engine-side object readers/writers), and object_handlers_common.rs to crates/ecstore/src/bucket/lifecycle/ (it is the lifecycle noncurrent-version cleanup helper). The latter now routes its replication calls through the lifecycle replication_sink boundary (schedule_delete wrapper and the sink's ReplicationObjectBridge re-export), as the lifecycle guard requires.
- The ecstore public facade drops api::client: object_api_utils is exposed as api::object_api_utils, and the rustfs crate takes admin_handler_utils (AdminError) from rustfs-s3-client directly (new dependency).
- Guard updates: the migration guard no longer pins mod client in ecstore's lib.rs or the admin_handler_utils facade module (it pins the new api::object_api_utils facade instead), and the module-lint register follows object_api_utils.rs to its new path.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs; cargo fmt --all; tier/transition/lifecycle-focused nextest (626 passed) and the decommission/rebalance/heal families in a filtered run (603 passed; the full-suite parallel run only fails on this machine's known decommission/rebalance baseline flakes, which pass in filtered reruns and fail identically on pristine origin/main); layer/migration/s3s/logging/error-format/doc-path guard scripts all pass.
* docs(architecture): record the S3 client extraction and reword invariant 4 (#6669)
Closes the documentation step of rustfs/backlog#1842. ARCHITECTURE.md invariant 4 now states the serving-vs-consuming distinction the adversarial ruling asked for: ecstore must not serve HTTP/S3 wire types, while consuming remote S3 endpoints is a legitimate engine capability that lives in the extracted rustfs-s3-client crate. The violation note is updated from the pre-extraction snapshot (58 files, embedded client) to the current ratcheted state (shrink-only S3S_ECSTORE_FILES_BASELINE in scripts/check_s3s_footprint.sh, object_lock converted first), and the crate map gains s3-client. ecstore-module-split-plan.md gets the client-directory entry the plan was missing: a Current Shape row and a completed-extraction section describing the pure-move + shim + direct-import sequence and the re-homing of the two misfiled server-side modules.
The storage engine embedded a ~8.4K-line hand-written S3 HTTP client under crates/ecstore/src/client (rustfs/backlog#1842). That client is a legitimate engine capability — it consumes remote S3-compatible endpoints for ILM tier warm backends and transition targets — but it was misfiled inside the engine, dragging s3s/hyper wire types into ecstore and blocking ARCHITECTURE.md invariant 4.
This PR is the pure-move step: 21 modules move verbatim to the new crates/s3-client crate (rustfs-s3-client), and crates/ecstore/src/client/mod.rs becomes a re-export shim so every in-crate crate::client:: path keeps working. The two server-side modules that were historically misfiled under client/ — object_api_utils.rs and object_handlers_common.rs — stay in ecstore.
Three reverse dependencies from the client into engine internals are severed so the move can be pure:
- transition_api::ReaderImpl::ObjectBody held ecstore's GetObjectReader; the client only ever reads the body, so the variant now holds an ObjectReader newtype over Box<dyn AsyncRead + Send + Sync + Unpin> with the same read_all() surface. The single production construction site (set_disk transition upload) and the two engine-side consumers were adjusted.
- api_list/api_remove used ecstore's storage_api_contracts / object_api types; api_list now imports BucketInfo from rustfs-storage-api directly, and api_remove uses the client's own transition_api::ObjectInfo (only .name/.version_id were read; the error-path bucket name is now threaded as a parameter instead of read from the deleted objects).
- the api_put_object_streaming regression tests built a GetObjectReader by hand; they now wrap the duplex stream in ObjectReader::new.
Guard updates: the s3s footprint ratchet gains an ecstore-scoped counter (42 files, shrink-only, per rustfs/backlog#1842), the ecstore module-lint-blanket register follows the moved files into crates/s3-client so the blanket ratchet keeps covering them, the logging guardrail path pin follows transition_api.rs, and the ::other(format!) baseline is regenerated (moved call sites left ecstore).
Verification: cargo check -p rustfs-s3-client -p rustfs-ecstore; cargo nextest run -p rustfs-s3-client (43 passed) and -p rustfs-ecstore (4515/4523; the 8 failures reproduce identically on pristine origin/main on the same machine); cargo clippy --all-targets; scripts/check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_s3s_footprint.sh, check_logging_guardrails.sh, check_error_other_format_ratchet.sh, check_doc_paths.sh, check_ci_paths_sync.sh all pass.
* fix(tier): gate exact remote version consumption
Reject non-empty remote tier versions before transitioned GET and remote delete backend I/O when the tier backend does not support exact version operations.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(tier): split mock remote version validation fault
Separate one-shot mock remote version validation failures from persistent unsupported-backend behavior so cleanup durability tests can still verify exact-version recovery while #1358 fail-closed gate tests keep asserting no backend I/O.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Advance transition transactions to UploadOutcomeUnknown before remote tier PUT so response-loss windows can be recovered through provider-authoritative probing.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(tier): add transition candidate probe contract
Add a fail-closed WarmBackend probe contract for provider-authoritative transition candidate state. Default providers report Unsupported, while the shared mock backend can now model missing, unversioned, and exact-version candidates for follow-up recovery tests.
This is a forward-compatible foundation for #1352/#1358 recovery work and does not change production cleanup behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(tier): probe transition candidates from providers
Implement provider-authoritative transition candidate probing for S3-family warm backends by querying ListObjectVersions with exact-key filtering and fail-closed classification for delete markers, multiple versions, truncation, and unknown versioning state.
This keeps non-S3 providers on the default Unsupported probe result and forwards MinIO, RustFS, and R2 through the S3 probe implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (rustfs/backlog#1303)
The excluded test's 'missing xl.meta ... on disk2' was NOT an EC
metadata-distribution issue: after a transition all four shard disks hold
a fully consistent xl.meta (verified by decoding each shard). The panic
came from the tier test util's open_disk, which hardcoded disk_index 0
for every disk path; LocalDisk::new validates the endpoint's
(set_idx, disk_idx) against the disk's own format.json and rejected every
non-slot-0 disk with InconsistentDisk, which read_transition_meta
collapsed into 'missing xl.meta'. Derive the real indices from
format.json instead. This also un-breaks free_version_count /
wait_for_free_version_absence for non-first disks (silently 0 before).
With that fixed, the test advanced to the #4877 restore self-deadlock,
whose main paths #4886 already fixed. Complete that fix on the one path
it missed: update_restore_metadata (the restore-failure metadata
rewrite) still rebuilt copy_object options with no_lock=false and would
re-acquire the object write lock the restore handler already holds.
Propagate the caller's no_lock there too.
Remove the test from the serial-lane exclusion list; the four remaining
exclusions are unrelated known issues and stay.
* test(ecstore): extract shared MockWarmBackend into a test-util feature (backlog#1148 ilm-6)
The tier/lifecycle integration tests carried two byte-for-byte copies of an
in-memory WarmBackend mock — one in crates/scanner/tests and one in
rustfs/src/app — plus duplicated register_mock_tier and polling helpers. Both
implemented the same ecstore WarmBackend trait.
Consolidate them into ecstore behind a new `test-util` feature, exposed via the
`rustfs_ecstore::api::tier::test_util` facade:
- MockWarmBackend: in-memory WarmBackend with an operation log (for ordering
assertions such as "local delete precedes remote remove") and fault injection
(FaultConfig): unreachable, HTTP 5xx, credential rejection, injected latency,
plus external_remove to simulate an out-of-band remote deletion.
- register_mock_tier / register_mock_tier_backend: register the mock into any
TierConfigMgr handle (the global manager used by scanner tests or a
per-instance one used by the app tests).
- xl.meta transition assertion helpers: read_transition_meta,
assert_transition_meta_consistent (cross-shard consistency of the
status/tier/remote-key/remote-version-id tuple plus free-version count), and
free_version_count.
- polling helpers: wait_for_remote_absence, wait_for_object_count,
wait_for_free_version_absence.
Both existing copies now consume this single definition; `rg 'struct
MockWarmBackend'` collapses to one. The feature is enabled only from
[dev-dependencies], so it never links into the production binary (resolver 3).
Designed for downstream ilm-8 (restore lifecycle) and ilm-11 (tier fault
injection matrix). Coordinates with #4706 (ilm-2), which adds op-logging to the
scanner mock — that op-logging is now part of this shared surface, so #4706
should rebase onto it.
Refs rustfs/backlog#1148 (ilm-6), rustfs/backlog#1155.
* test(ecstore): fix shared MockWarmBackend usage after main merge
- Access stored objects via MockWarmBackend::contains() instead of the now
private inner objects map (fixes E0609 after the shared test-util refactor).
- Drop dead ReadCloser/ReaderImpl/DiskAPI imports and the unused
transition_api test re-exports the mock extraction left behind.
- Reword the scanner/rustfs test-util dependency comments so they no longer
embed the literal rustfs_ecstore:: path that trips the ECStore
architecture-migration guard.
cxymds