Add canonical record-object naming and crate-private save, load, and idempotent delete helpers for tier mutation intents. Cover malformed record keys, mismatched persisted mutation IDs, and an ECStore config-object round trip under test-util.
Co-authored-by: heihutu <heihutu@gmail.com>
Retry cleanup-pending transition transaction recovery after restart, keeping committed remote bodies when local metadata already references them and preserving records when remote cleanup fails.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): recover uploaded transition transactions
Persist transition transaction records through production transition uploads, use transaction-scoped remote object names, and start a conservative recovery loop for uploaded candidates.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): reconcile committed transition records
Drop LocalCommitStarted transaction records only after object metadata confirms that the local transition commit already points at the same tier object and version.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): add fallible erasure construction
(cherry picked from commit bd148b20f7)
* fix(ecstore): resolve storage parity per pool
(cherry picked from commit c05c2cb24b)
* fix(ecstore): keep carved per-pool parity core self-contained on main
Fixups so the cherry-picked fallible-erasure + per-pool-parity core builds standalone on current main without the excluded scope-creep commits:
- runtime/sources.rs: re-add backend_storage_class_parities (removed by the per-pool commit; its rebalance caller was updated in an unrelated reporting commit that was left out). Reimplemented over the snapshot API, behavior-identical.
- config/mod.rs: rename the storage-class publish test module (main independently added a mod tests, so the cherry-pick collided).
- rustfs storage_api.rs + startup_storage.rs: route the storage-class ENV consts through the startup storage facade and use a local const for the erasure-set-drive-count env name, satisfying the layer/facade guardrail (main's guardrail is stricter than when the core was authored).
* fix(ecstore): use struct-init in erasure test helper to satisfy clippy field_reassign_with_default
The cherry-picked fallible-erasure commit's `erasure_with_invalid_dimensions` test helper built `Erasure` via `default()` then reassigned fields, which trips `clippy::field_reassign_with_default` under `-D warnings` (only surfaced by `--all-targets`, which lints test code). #4977 fixed this in a later commit that was not part of the carved core. Use struct-init with `..Default::default()`, matching #4977's final form.
* fix(rustfs): gate the test-only storage-class ENV facade re-export behind cfg(test)
The ENV constants (INLINE_BLOCK_ENV/OPTIMIZE_ENV/RRS_ENV/STANDARD_ENV) re-exported through the startup storage facade are only consumed by a #[cfg(test)] test in startup_storage.rs, so in a non-test lib build the re-export is unused and trips -D unused-imports under clippy --all-targets. Gate it with #[cfg(test)], matching #4977's final form.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
test(1306): pin Some(0) usage serialization, snapshot cache invalidation, and listing send classification
Follow-up test hardening for the merged admin-usage-snapshot work
(#4979/#4980/#4981/#4982, rustfs/backlog#1306). Tests only; no production
behavior change.
- B-1 madmin: pin that a scanned-but-empty bucket (Some(0)) serializes usage
stats as zeros, staying distinct from the no-snapshot (None) omitted case.
- B-2 gating: revert detector proving save_data_usage_in_backend invalidates
the 30s snapshot cache so a fresh save is visible to the next cached read.
- A-1 list_objects: pin that a successful gather_results send is never
misclassified as ConsumerGone (correct state + err sentinel delivered), and
document the wrapper Err arm invariant. A full wrapper-level producer-error
integration test is deferred as it needs the fake-disk list harness.
* fix(scanner): back off clean single-disk cycles
* fix(scanner): extend idle backoff across erasure clusters
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
fix(ilm): serialize RestoreObject accepts with a CAS guard, not the whole copy-back
Implements the backlog#1304 decision: replace #4877's object write lock
held across the entire tier copy-back with an atomic compare-and-set on
the restore ongoing flag.
- ecstore: add ECStore::acquire_restore_accept_guard + RestoreAcceptGuard
(opaque, purpose-scoped object write lock with an is_lock_lost fence);
handle_restore_transitioned_object no longer locks the copy-back, so
HEAD/get_object_info stay non-blocking during a restore and the inner
put_object/complete_multipart_upload commit locks no longer self-deadlock.
- API: execute_restore_object holds the guard across the restore-status
read-check-write (no_lock inside the scope), drops it before spawning
the copy-back; SELECT-type restores keep the plain read-locked path;
the copy-back pins the resolved version so a concurrent PUT cannot
strand the flagged version at ongoing=true; concurrent restores are
rejected with 409 RestoreAlreadyInProgress (was a retryable 500) and
guard contention maps to 503 SlowDown.
- tests: drop the #4877 entry-blocking unit test (semantics deliberately
reversed), pin accept-guard mutual exclusion, update the ilm-8 restore
integration test to the final semantics, and add a concurrent
double-POST test asserting exactly one acceptance and one tier GET.
fix(object-data-cache): make GET body cache key write-unique and dedup lookups
Address four object-data-cache GET-path findings (backlog#1107 batch):
ODC-06 (backlog#1111): the cache key was content-unique, not write-unique.
Extend ObjectDataCacheKey with the resolved version's modification time
(i128 unix nanoseconds, None -> 0), derived once in the shared planner so the
ecstore hook and the usecase layer produce an identical key. An unversioned
overwrite advances mod_time, so a stale node can no longer serve old bytes for
up to the TTL under an MD5 collision; etag + size stay as belt-and-braces.
ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in
the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes
and lookups. GetObjectReader now carries a GetObjectBodySource marker
(Unprobed / HookMissed / HookServed); the hook stamps it, and
build_get_object_body_with_cache serves a hook-served body directly and skips
its lookup whenever the hook already probed. One hook-served GET now records
exactly one lookup.
ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly,
which never fills and keeps a permanent 0% hit rate. Default to
FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is
selected explicitly.
ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was
silently inert. Clamp the planner's size eligibility to
min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible
sizes plan SkipTooLarge instead of being reported eligible, and warn at startup
when the excess is inert.
Co-authored-by: heihutu <heihutu@gmail.com>
Foreground S3 listings wrapped the entire walk_dir stream in a single
wall-clock timeout (RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS, default 5s). That
budget measured how much data the walk had to produce rather than whether
the drive was still answering, so a healthy but large prefix on slow media
returned 500 InternalError / "Io error: timeout" to the client. The timer
also kept running while the walk was blocked writing to a slow consumer,
charging merge-side backpressure to the producer.
WalkDirOptions::stall_timeout_ms already carried the right semantics but was
only honored by the remote-disk RPC walk; LocalDisk::walk_dir ignored it
entirely. A single-drive deployment therefore had no way to distinguish a
hung drive from a big directory.
Teach LocalDisk::walk_dir to bound each individual drive read with the stall
budget, defaulting it from RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS when the
caller does not pin one, and let the foreground listing path skip the
wrapper-level total timeout. A walk that keeps making progress now runs to
completion; a drive that stops answering still fails with DiskError::Timeout.
Time spent blocked on the consumer stays outside the budget.
Heal and rebalance walks already skipped the total timeout and previously ran
unbounded on local drives. Give them an explicit, generous 60s stall budget so
this change does not tighten them from "no bound" to the 5s default.
Fixes#4644
Refs #2999, #3001
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.
Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:
- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
(the ambient disk registry = the first published store's context), so
list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
a second server's bucket metadata landed in the first server's
.rustfs.sys; set/remove/get/created_at all used the ambient metadata
system.
Now the whole chain is bound to the owning store's InstanceContext:
- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
operate on that context's registered disks; ECStore::new (and the test
store builder) pass the store's context. The legacy constructors keep
the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
context's metadata system and fall back to the ambient default before
the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
table-bucket delete guard and emptiness check) use the per-context
variants and this instance's disks.
Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.
579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
backlog#1052 S3, third slice — the hard blocker for a second embedded
server's service stage. GLOBAL_BUCKET_METADATA_SYS was a process-global
OnceLock whose init ran .set().unwrap(): a second instance's service
startup panicked the whole process. The system it guards is inherently
per-store (it holds the store handle and that store's bucket→metadata
cache), so it now lives on the store's own InstanceContext:
- init_bucket_metadata_sys writes the cell of the store it was handed
(api.ctx) — no signature change — and keeps the double-init fail-fast,
now scoped to the instance (assert on the per-context set). The
dist-erasure check for the refresh loop reads the same context.
- get_global_bucket_metadata_sys / the crate-private getter behind the
~20 get_*_config helpers resolve through the current_ctx() facade —
the published store's context, or bootstrap before that — so every
existing reader keeps its exact single-instance behavior.
- New acceptance test: two real stores in one process each initialize
their own metadata system (the old global cell panicked right there),
plus a shared builder extracted from the store-graph test.
201 bucket/metadata regression tests green.
* refactor(app): give each context's server-config handle its own copy
backlog#1052 S3, first slice: establish the per-context state pattern on
the smallest subsystem. ServerConfigHandle was a unit struct forwarding
both get and set to the process-global GLOBAL_SERVER_CONFIG, so every
AppContext shared one config regardless of which context a caller held.
The handle now keeps its own copy: a set lands on the owning context and
on the process global (ambient readers — the config loader path, the
scanner — keep working), and a get prefers the owned copy, falling back
to the global while the initial load still publishes there. Single
instance: the owned copy and the global are written together, so reads
are unchanged. Two contexts that both published stay isolated even though
the shared global only remembers the last write (covered by a new test).
The global write in set() is the transitional bridge for ambient readers;
it goes away when those readers migrate and multi-instance flips on
(backlog#1052 S5).
* refactor(notify): hand out the notification system as an owned Arc
backlog#1052 S3, second slice. GLOBAL_NOTIFICATION_SYS stored the
NotificationSys by value and every accessor returned
Option<&'static NotificationSys> — a process-lifetime borrow that a
per-server AppContext can never hold. The global now stores an Arc and
the accessor chain (ecstore runtime sources, ECStore::notification_system,
the iam forwarders, the rustfs shims, NotificationSystemInterface and the
AppContext resolvers) hands out owned Arcs instead.
Call sites are unchanged apart from two borrow adjustments in the
rebalance admin handler (pass &Arc where a reference is expected; borrow
with as_ref() so the handle survives to the second use). Behavior is
identical — same single global instance, first init still wins — but the
type now permits a future per-server context to own its notification
system (backlog#1052 S3 follow-up).
The three store::bucket::tests::bucket_delete_* tests drive make_bucket /
delete_bucket through the process-global local-disk registry
(GLOBAL_LOCAL_DISK_MAP) and lock client, and through Sets::new which reads the
process-global erasure mode. Under `cargo test` (single process, many threads)
they race other global-touching tests and fail deterministically with
InsufficientWriteQuorum.
serial_test's #[serial] with the crate default key serializes them across the
in-process suite. Measured at --test-threads=16: baseline fails all three every
round; with #[serial] all three pass every round. nextest's per-test processes
are covered separately by the ecstore-serial-flaky test-group in
.config/nextest.toml (#4558). Full instance-level isolation remains a backlog
#939 (InstanceContext) concern and is not attempted here.
Co-authored-by: heihutu <heihutu@gmail.com>
Non-slash delimiter ListObjects/ListObjectVersions force a recursive walk and
defer common-prefix folding to from_meta_cache_entries_sorted_infos. When a page
of max_keys+1 raw candidates collapses into fewer than max_keys common prefixes,
list_objects_paginate left is_truncated=false with no next_marker even though the
walker had filled its raw candidate limit (disk_has_more=true). Clients were told
listing was complete and silently lost every key beyond the scan window, a
data-loss bug for backup/sync/mirror/reconcile consumers (backlog ECA-03 / #944).
Add a delimiter re-fold guard: when disk_has_more is true but the folded visible
entries are fewer than max_keys, set is_truncated=true and continue from the last
RAW scanned key (captured before folding via last_scanned_entry_name). Using the
last raw key rather than a folded prefix is required for correctness: forward_past
advances on name > marker, and a folded prefix such as "data-" is lexicographically
smaller than its own keys ("data-0001"), so it would re-list and re-fold the same
keys forever. A real scanned key strictly advances past the scanned window and
keeps pagination finite. The slash-delimiter and no-delimiter paths fold during
the walk and are unaffected.
Regression tests cover the direct re-fold case, the disk-exhausted no-marker case,
the no-delimiter path guard, and an end-to-end 5000 data-* + 100 other-* paging
loop asserting bounded termination and full common-prefix coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
is_all_not_found and is_all_volume_not_found delegated to
is_err_bucket_not_found, which matches StorageError::DiskNotFound. When
every erasure set in every pool was unreachable, the per-set aggregate
DiskNotFound slice was classified as "all not found", so list_merged
returned Ok(empty) and walk_result_from_set_errors was fed an
availability failure disguised as an empty listing. Clients saw a
successful empty ListObjects and mistook a full outage for an empty
bucket.
Introduce is_err_strict_not_found (FileNotFound / VolumeNotFound /
FileVersionNotFound / ObjectNotFound / VersionNotFound, excluding
DiskNotFound) for is_all_not_found, mirroring MinIO's isAllNotFound and
DiskError::is_all_not_found. Give is_all_volume_not_found its own
is_err_strict_volume_not_found (VolumeNotFound / BucketNotFound, minus
DiskNotFound) so genuine volume/bucket absence still surfaces while a
missing object under an existing volume is not escalated. Add
is_all_disk_not_found and a pre-check in walk_result_from_set_errors so
an all-offline slice surfaces DiskNotFound instead of being swallowed as
an empty listing, while partial outages (a healthy set present) stay
tolerated as before.
Regression tests cover all-DiskNotFound rejection, genuine not-found
acceptance, volume-not-found semantics, partial/empty short-circuit, and
the walk full-offline vs partial-offline paths.
Co-authored-by: heihutu <heihutu@gmail.com>
B5 switched heal enumeration to list_object_versions, which only reflects the
read-quorum metadata view: a version present on fewer than read-quorum disks was
never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION
enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw +
mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and
feeds each to the existing per-version heal_object.
- filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the
cross-disk version union at one tested seam.
- ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1,
dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator
+ HealWalkVersion.
- ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta
physically probes part files via check_parts; when >= data_blocks data shards
survive (meta lost but data recoverable) it regenerates xl.meta from a surviving
FileInfo with the correct per-disk shard index instead of dangling-deleting.
Genuine torn writes (< data_blocks) keep the current behavior — no resurrection.
- heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor,
idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk
trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume
selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else
the unchanged B5 path; anti-loop guard aborts on (empty && truncated).
Closesrustfs/backlog#920
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* fix(swift): replace assert!(false) with panic! in expiration_worker test
clippy::assertions_on_constants fails the swift clippy CI job under -D warnings.
* fix(ecstore): add missing ctx field to multipart lock test store
The multipart list-parts lock test (#4437) constructs ECStore without the
ctx field added by the Phase 5 InstanceContext work (backlog#939). Both
landed on main independently, leaving a semantic conflict that breaks the
ecstore lib-test build (E0063). Adopt the process bootstrap context,
matching the existing bootstrap_ctx() test in store/mod.rs.
* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type
Phase 5 of the global-singleton consolidation (backlog#939): begin moving
runtime identity state out of process globals so multiple ECStore instances
can coexist in one process. Isolation is carried by the object graph
(ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a
task-local, which does not propagate across the many internal tokio::spawn
boundaries in the data/background paths.
This first slice migrates the erasure setup type -- previously three
independent process-global bools -- into a single per-instance
RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd,
removing a triple source of truth that could drift out of sync.
- New runtime::instance module: InstanceContext + process bootstrap context.
- The legacy free-function facade (is_erasure/update_erasure_type/...) keeps
its signatures and forwards to the current instance's context, falling back
to the bootstrap context before a store is published.
- ECStore gains a pub(crate) ctx field and setup_is_* accessors; its
constructors adopt the bootstrap context (never mint a fresh one) so startup
writes and post-construction reads share one cell -- single-instance
behavior is byte-for-byte unchanged.
Tests: erasure predicate derivation vs the legacy behavior, object-graph
carrier isolation across two ECStore instances, and bootstrap adoption.
Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8)
* refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415)
* refactor(ecstore): source the namespace lock manager per-instance (#4417)
refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3)
Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by
sourcing SetDisks' lock manager from the instance context instead of the
process singleton. This removes the false cross-instance mutual exclusion (and
attendant ABBA risk) that a shared GlobalLockManager would cause once multiple
instances coexist.
- InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints
a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the
process singleton via get_global_lock_manager(), so a single-instance
deployment keeps exactly one shared namespace.
- SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx
it already adopts), not `runtime_sources::global_lock_manager()`. Single
instance: same Arc as before, so behavior is unchanged.
- Remove the now-unused `runtime_sources::global_lock_manager()` wrapper.
Tests: bootstrap lock manager aliases the process singleton; two fresh contexts
own distinct managers; a SetDisks' lock manager is the one from its context and
aliases the global singleton in a single-instance build.
Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking
regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean),
make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2).
Merge the hotpath-rs wall-time instrumentation from the backlog#936 analysis worktree behind an opt-in 'hotpath' cargo feature, keeping the default build at zero overhead and zero dependency.
- hotpath is an optional dependency everywhere (dep:hotpath feature syntax); the default dependency tree contains no hotpath crate at all
- 40+ measurement points across S3 handlers, ECStore/SetDisks object and multipart ops, erasure encode/decode, bitrot, LocalDisk I/O, FileMeta codec, and HashReader
- attribute sites use #[cfg_attr(feature = "hotpath", hotpath::measure)]; async_trait bodies use per-crate hp_guard! macros (ecstore + rustfs bin); rio gates measure_block! behind hp_measure_block!
- feature chain: rustfs -> rustfs-ecstore -> rustfs-rio / rustfs-filemeta, each crate owning its own gate
- hotpath-alloc is intentionally not wired up (hotpath 0.21.x TLS panic on cross-thread guard drop under tokio, see backlog#935); mimalloc stays the unconditional global allocator
- docs/development/hotpath-profiling.md documents building, HOTPATH_* env vars, SIGTERM report flow, and how to reproduce the backlog#936 timing reports
Refs: https://github.com/rustfs/backlog/issues/935 (HP-14, item 2), https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(migration): decrypt MinIO IAM & server config on drop-in migration
MinIO encrypts IAM identity/service-account files and the server config at
rest with a key derived from the root credentials. The drop-in migration
paths read those blobs from the legacy `.minio.sys` bucket and parsed them
as plaintext JSON, so any encrypted blob failed to parse and was silently
skipped with "incompatible format". This is why users migrating from MinIO
kept their buckets/objects/policies but lost users and access keys (#2212).
The IAM load path already knows how to decrypt these blobs (RustFS master
keys plus MinIO-compatible legacy keys derived from the root credentials),
but that logic lived behind a private method and was never used by the
migration paths. Expose it as `rustfs_iam::try_decrypt_iam_blob` and inject
it into both migration paths via a `LegacyBlobDecryptFn` callback (ecstore
cannot depend on the IAM crate, so the closure is wired in the binary crate).
When a blob cannot be decrypted the raw bytes are used as-is, preserving the
previous plaintext-only behavior with no regression.
Also improve object-layer migration observability without changing control
flow: `try_migrate_format` now distinguishes "no legacy format" (a normal
fresh install) from "legacy format present but incompatible", and the caller
logs a loud error before initializing a fresh format that would leave the
existing MinIO objects unreadable. Topology/version skip reasons are promoted
from debug to warn.
Fixes a pre-existing test isolation race by marking
`test_recovery_falls_back_to_default_config_when_blob_stays_corrupt` serial,
since it reads a process-wide env var toggled by a sibling test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(migration): box FormatV3 in LegacyFormatOutcome to satisfy clippy
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): stabilize concurrent multipart resend lock timeout
concurrent_resend_same_part_commits_one_generation spawns 6 same-part
resends whose cross-disk commits serialize on the per-uploadId commit
lock. Under the full nextest suite the parallel disk load pushes those
serialized commits past the small default lock-acquire timeout (5s),
producing a spurious `Lock(Timeout ...)` unrelated to the property under
test (observed on CI at 5.775s vs ~0.5s in isolation).
Raise RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT to the production default (30s)
for the concurrent-commit section via temp_env, so the regression guard
reflects correctness (exactly one intact generation) rather than disk
latency under CI load. The meaningful assertions are unchanged, and
#[serial] keeps the process-wide env override isolated.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(lock): bound fast-lock notification wait to prevent lost-wakeup stall
The real cause of the concurrent_resend_same_part_commits_one_generation
failures was a lost wakeup in the fast-lock slow path, not disk latency:
raising the acquire timeout to 30s only delayed the failure (it then timed
out at 30s), proving a genuine stall rather than overload.
In acquire_lock_slow_path a waiter that reaches the notification phase did a
single `timeout(remaining, wait_for_write())` spanning the whole acquire
budget, and treated that wait's elapse as a hard `Timeout`. But the release
path only notifies when `writer_waiters > 0`, so if the holder releases in
the gap after the waiter's `try_acquire` fails and before it registers as a
waiter, no notification (and no stored permit, since the pooled `Notify` is
gated) is produced. The waiter then blocks until the deadline even though the
lock is free and stays free — a spurious lock-acquire timeout. The shared
process-wide notify pool makes it worse: a wakeup can be consumed by a waiter
of a different lock hashing to the same slot.
Bound each notification wait (NOTIFY_WAIT_CAP = 50ms) and, on elapse, loop
back and re-`try_acquire` instead of returning `Timeout`; the deadline check
at the top of the loop is the single source of truth for timing out. A
lost/stolen wakeup now degrades to bounded re-polling (acquire within ~50ms
of the lock becoming free) instead of stalling for the whole timeout.
Correctness (mutual exclusion) is unchanged — acquisition still only happens
via `try_acquire_*`.
Add a regression test that reproduces the stall (holder + late waiter across
many keys): it times out without the fix and passes in ~1s with it. Revert
the earlier acquire-timeout workaround in the multipart test now that the
underlying stall is fixed, so it runs under the default timeout again.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(ecstore): replace linear merge scan with k-way heap and drop clone-to-parse (backlog#874 backlog#875)
merge_entry_channels advanced the k-way merge with a linear scan over all
channel heads (O(entries x channels)) and allocated two fresh Strings per
pairwise comparison via path::clean. Every step also cloned MetaCacheEntry
values, including entry.clone().xl_meta() clone-to-parse calls.
- Introduce MergeHead with a cached cleaned name (allocated only when the
raw name is not already clean) and drive the merge with a BinaryHeap of
boxed heads: O(log channels) per entry, allocation-free comparisons.
- Move entries through the merge instead of cloning; the winner is sent
without an intermediate copy.
- Remove the dead merge_file_meta_versions block: it only ran for
prefix-dir groups whose entries have empty metadata, so xl_meta() always
failed; cross-drive version merging happens in the resolve path.
- Keep legacy same-name semantics (dir groups collapse, objects shadow
prefix dirs, later object candidate wins) and add regression tests for
interleaved ordering, dir/object precedence, uncleaned-name grouping,
and prefix-dir collapse.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): honor ascending versions_sort in ListObjects walk (#4348)
* fix(ecstore): honor ascending versions_sort in walk and document ordering invariant (backlog#876)
The walk loop carried a bare `//TODO: SORT` inside the
`WalkVersionsSortOrder::Ascending` branch, so the requested ascending
order was silently ignored and versions streamed newest-first (the raw
FileMeta order). WalkOptions defaults to Ascending, so every default
walker -- notably replication resync, which replays versions and needs
oldest-first to preserve the version-stack order -- received the exact
opposite of the contract.
FileMeta maintains versions newest-first (sort_by_mod_time is
descending) and into_file_info_versions preserves that order, so
ascending emission is the exact reverse of file_info_versions output.
Reverse in place when ascending is requested and add a regression test
locking the newest-first invariant plus the reversal contract.
Key-ordering audit result (no gap found): per-disk walkers emit sorted
streams, merge_entry_channels performs an ordered k-way merge, and
gather_results only filters by marker/limit, so ListObjects key order is
guaranteed upstream and needs no post-sort.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): enable GET metadata early-stop by default (#4349)
* perf(ecstore): enable GET metadata early-stop by default with env opt-out (backlog#872)
The metadata early-stop fanout (read_all_fileinfo_early_stop) has been
implemented and instrumented for a while but stayed behind an opt-in
flag, so default GETs always waited for every disk to answer the
metadata read even after quorum agreement was reached.
Flip RUSTFS_GET_METADATA_EARLY_STOP_ENABLE to default-on. The gate stays
conservative: should_allow_metadata_early_stop only admits metadata-only
reads (read_data=false) without version_id, healing, or free-version
requirements, everything else falls back to the full-wait fanout, and
setting the env var to false restores the old behavior entirely. The
version-aware gate (RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE)
remains opt-in because versioned reads carry a higher stale-selection
risk profile.
Also replace the stale "optimize concurrency" TODO in
get_object_fileinfo with a pointer to the early-stop implementation and
add regression tests for the new default plus the explicit opt-out path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): lazily construct codec streaming multipart readers (#4350)
* perf(ecstore): lazily construct codec streaming multipart part readers (backlog#871)
get_object_decode_reader_with_fileinfo opened shard readers for every
part of a multipart object before returning the streaming reader, so
TTFB paid for parts x disks file opens up front and an early client
disconnect wasted the setup work for every unread part.
Replace the eager loop with LazyMultipartCodecStreamingReader: the first
part is still built eagerly so the dominant fallback conditions (missing
shards / read quorum) are detected before any byte is streamed and the
whole request can fall back to the legacy duplex path exactly as before.
Each subsequent part is built on demand -- when the previous part hits
EOF -- via a spawned task handle owned by the reader; dropping the
reader aborts an in-flight build so disconnects stop all further IO.
If a later part hits a fallback condition mid-stream (a shard vanished
after the request started), the reader surfaces an explicit read error
with a pipeline-failure metric instead of silently degrading; the
client's retry then detects the condition on the eager first-part setup
and takes the legacy path cleanly.
Adds unit tests for in-order streaming across lazy boundaries, deferred
construction (no build when the client stops within part 1), and the
mid-stream fallback error path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): prefetch next multipart part reader setup during decode (#4351)
* perf(ecstore): prefetch next multipart part reader setup during decode (backlog#870)
get_object_with_fileinfo processed multipart parts strictly serially:
the next part's bitrot reader setup (file opens + read-quorum wait
across all disks) only started after the current part finished
decoding, so large multipart reads paid full setup latency between
every part.
Overlap the two stages with a depth-one pipeline: right after the
current part's readers are obtained, the next part's setup is spawned
(shared inputs behind Arc) and joined when the loop reaches that part.
The shared setup_multipart_part_readers helper keeps stage-duration
metrics semantics identical for both paths; a failed or stale prefetch
falls back to the synchronous setup, and the PrefetchedReaderSetup
guard aborts the in-flight task on error returns, early breaks, or
caller drop so disconnects stop background disk IO.
Gate: RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH (default on, env
opt-out). Adds a three-part end-to-end read test covering the prefetch
hit path and cross-part content ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): move FileInfo through GET shuffle instead of cloning (#4352)
perf(ecstore): move FileInfo entries through the GET shuffle instead of cloning (backlog#873)
shuffle_disks_and_parts_metadata_by_index deep-cloned every valid
FileInfo (parts, erasure info, metadata map) once per disk on each GET.
Add an ownership-taking variant that runs the same by-index consistency
check as a read-only first pass and then moves entries into their
shuffled slots with mem::take, and switch get_object_with_fileinfo to
it -- that call site already owned the parts metadata vector. Disk
handles are Arc clones and stay cheap.
Scope notes from the backlog#873 audit:
- get_object_fileinfo's disks.clone() stays: DiskStore is Arc<Disk>, so
the clone is per-slot refcounting and correctly avoids holding the
RwLock read guard across the metadata fanout awaits.
- get_object_decode_reader_with_fileinfo keeps the borrowing shuffle:
its caller must retain files/disks for the legacy fallback path, so an
owned variant would just shift the same clone upstream.
- The metadata-cache hit path still clones parts_metadata; sharing the
cached entry via Arc changes the read-path return types and is left
as a follow-up.
Equivalence tests cover both the by-index placement and the mod-time
fallback against the borrowing variant.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): gate merge emission on cleaned key and clear clippy redundant_clone
Address review + CI findings on the ListObjects/GET optimization PR:
- merge_entry_channels gated emission on the raw entry name while the heap
orders by the cleaned key, so entries whose cleaned order and raw byte order
disagree (e.g. redundant slashes) could be dropped. Gate on the same cleaned
sort key the heap uses; add a regression test (`a//c` after `a/b`).
- Drop three redundant `.clone()` calls in test code flagged by
clippy::redundant_clone (owned-shuffle equivalence tests and the walk
ascending-versions contract test) that failed the CI clippy gate.
- Document the known mid-stream fallback limitation of the opt-in multipart
codec streaming reader (default off) and mark the in-place per-part legacy
degradation as a follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): force full metadata fanout for object tagging writes (backlog#872)
put_object_tags reads the object fileinfo with read_data=false and then
writes the updated tags to the online-disk set that read returned. With
metadata early-stop enabled by default, that read now returns as soon as
read quorum is reached, so the online-disk set is only a read-quorum
subset. Writing tags to that subset fails write quorum -> ErasureWriteQuorum
-> S3 SlowDown, which is exactly the s3-tests tagging failures
(PutObjectTagging/DeleteObjectTagging, reached max retries).
Thread a caller-controlled `allow_early_stop` gate through
read_all_fileinfo_observed/_inner and add get_object_fileinfo_gated;
put_object_tags calls it with allow_early_stop=false so the metadata read
does the full quorum fanout and returns the complete online-disk set as
the write target. Pure-read callers (GET/HEAD/tag read) keep the
early-stop fast path unchanged.
Extract metadata_early_stop_permitted() as the single gate and add a unit
test locking the invariant: caller opt-out (and observe=false, and data
reads) never early-stop even with the env flags on.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Three tests share the global list-objects mutation-sequence state (the
LIST_OBJECTS_MUTATION_SEQUENCE atomic and the per-bucket sequence map)
and all call reset_list_objects_mutation_sequences_for_test(), but only
observed_mutation_persists_namespace_journal_high_water was marked
#[serial_test::serial]. The serial lock only excludes tests that also
carry the attribute, so the unmarked tests still ran concurrently with
it: the journal test's persist of "bucket" -> 11 could land after the
recovery test's reset, making it observe 11 instead of the expected 9
(flaky under the default multi-threaded test runner, green with
--test-threads=1).
Add #[serial_test::serial] to the two unmarked tests:
- persistent_key_only_index_load_recovers_mutation_sequence_from_journal
- list_objects_mutation_sequence_advances_per_bucket
Unique bucket names alone would not fix this: the per-bucket assertions
in list_objects_mutation_sequence_advances_per_bucket check exact values
produced by the single global atomic counter, which any concurrent
observe call perturbs, and every reset zeroes that global state for
whichever peer is mid-flight.
Co-authored-by: heihutu <heihutu@gmail.com>
Second TODO-convergence round over the current tree (backlog#646). All
line numbers in the old inventory had gone stale after the set_disk /
diagnostics / cluster refactors, so this re-scans and reduces the marker
count from 144 to 99.
STALE removals (comment describes already-implemented behavior, or dead
commented-out blocks) across ecstore (set_disk ops/core, store,
cluster/rpc, bucket/metadata_sys, services), iam, filemeta, s3select and
rustfs auth/object_usecase. No behavior change.
Safe fills, each verified:
- filemeta: replication_info_equals now also compares
replication_state_internal (function currently has no callers; adds a
regression test).
- bitrot: drop the confirmed-unused `_want` parameter from bitrot_verify
and the now-unused `sum` on LocalDisk::bitrot_verify, removing a
Bytes::copy_from_slice allocation. Streaming verify uses the file's
embedded per-shard hash, never the passed sum.
- signer: rename v4_ignored_headers -> V4_IGNORED_HEADERS and drop the
non_upper_case_globals allow.
- admin/heal: test_decode was #[ignore]d and used serde_urlencoded on a
JSON body (would panic); rewire to serde_json::from_slice to match the
production decode path, add assertions, un-ignore.
Verified: cargo fmt; cargo check on touched crates; tests pass
(filemeta, signer, bitrot, heal::test_decode); arch guardrail scripts
pass.
PR #4220 added a purge for orphan empty-directory trees (folder keys with
no xl.meta) on the delete path, but the guard only accepted
object-not-found. Over the real HTTP DELETE path the guard is never
reached: `del_opts` pins `version_id = Uuid::nil()` for directory keys, so
the missing dir object fails the specific-version lookup with
version-not-found (FileVersionNotFound), not object-not-found. The guard
short-circuits, the store returns the error, and the API layer turns it
into a fake 204 — the ghost folder survives, exactly the #4189 symptom.
The existing unit tests passed because they call
`purge_orphan_dir_object` directly, bypassing the nil-version lookup.
Accept both misses in the guard (extracted as
`should_purge_orphan_dir_on_missing`) so the folder delete actually takes
effect. Non-directory keys and non-miss errors (e.g. quorum failures) are
unaffected; the cross-disk data-safety refusal in
`purge_orphan_dir_object` is unchanged.
Verified end-to-end against a running 4-disk erasure server: DELETE of a
planted orphan `ghost/` tree now purges it on all drives and clears the
listing, a real object under the same prefix is untouched, and a prefix
holding real data on any drive is refused (all drives preserved).
Adds predicate regression tests covering version-not-found /
object-not-found on directory keys, and the negative cases.
houseme