BucketMetadataSys::get_config lazily fabricated a default BucketMetadata
(object-lock off) for any bucket whose .metadata.bin was ConfigNotFound and
cached it in the map that the map-only, fail-closed metadata_sys::get()
serves. The object-lock batch-delete gate (object_lock_delete_check_required,
backlog#929 / #4297) treats that map as authoritative, so a metadata miss
became a cached "no lock" answer: a versioning peek could poison the cache
and let delete_objects skip the per-object retention/legal-hold stat. The
same fabrication raced make_bucket (lost update overwriting freshly
persisted lock-enabled metadata) and let the 15-minute refresh loop replace
good cached metadata on a transient quorum dip.
Production changes:
- get_config caches only metadata actually read from disk; misses are
recorded in a bounded negative cache (30s TTL, 10k entries, invalidated by
set()) so repeated lookups for metadata-less names cost no extra
namespace-lock + erasure-set fanout (reachable pre-auth via CORS
preflight and per-key in DeleteObjects).
- concurrent_load never lets a fabricated default REPLACE an existing map
entry; startup insert-if-vacant behavior for legacy buckets is preserved.
- delete_objects and new_ns_lock resolve dist-erasure, versioning, and the
object-lock gate from the set's own instance context (backlog#1052)
instead of the ambient facade, so a second in-process instance (or, in
tests, another test's transient DistErasure window) cannot reroute
locking onto an empty dist locker list or answer with the wrong
instance's bucket state.
Test-isolation changes (the bug that surfaced all of the above: the
delete_objects lock-gating test failed deterministically when sharing a
process with the lifecycle env tests):
- The MinIO-migration test builds on an isolated InstanceContext instead of
registering soon-deleted disks in the shared bootstrap registry.
- The cached lifecycle env re-registers its disks on every use, surviving
other serial tests' reset_local_disk_test_state.
- Hermetic SetDisks helpers gain isolated-context variants pinned to plain
erasure; tier-free non-serial test modules use them, guard-based
SetupTypeGuard tests stay on the bootstrap context.
- Three deterministic pin tests (nextest-safe) cover the caching contract,
the delete gate resolution source, and the ns-lock resolution source.
Verification:
- cargo test -p rustfs-ecstore --lib -- --exact <4-test combo from the
report> (previously failing, now green)
- cargo test -p rustfs-ecstore --lib: 3169 passed / 0 failed across
repeated runs; cargo fmt --check and cargo clippy --lib --tests clean
- Adversarial validation (high-risk tier, all seven roles) run per
AGENTS.md; all findings fixed or rebutted with evidence
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.
Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:
- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
(the ambient disk registry = the first published store's context), so
list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
a second server's bucket metadata landed in the first server's
.rustfs.sys; set/remove/get/created_at all used the ambient metadata
system.
Now the whole chain is bound to the owning store's InstanceContext:
- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
operate on that context's registered disks; ECStore::new (and the test
store builder) pass the store's context. The legacy constructors keep
the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
context's metadata system and fall back to the ambient default before
the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
table-bucket delete guard and emptiness check) use the per-context
variants and this instance's disks.
Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.
579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
The OnceLock introduced in 5fbd49a80 for bucket_metadata_sys panics when
multiple test modules initialize their own ECStore against the shared
bootstrap context. Switch to Mutex<Option> so test fixtures can
reinitialize safely.
Extract shared gating test environment to gating_test_env.rs to avoid
duplicate ECStore setup across delete_objects_stat_gating_test and
put_prelookup_gating_test.
Fixes 8 test failures:
- capacity_dirty_scope_test (2 tests)
- delete_objects_stat_gating_test (3 tests)
- put_prelookup_gating_test (3 tests)
backlog#1052 S3, third slice — the hard blocker for a second embedded
server's service stage. GLOBAL_BUCKET_METADATA_SYS was a process-global
OnceLock whose init ran .set().unwrap(): a second instance's service
startup panicked the whole process. The system it guards is inherently
per-store (it holds the store handle and that store's bucket→metadata
cache), so it now lives on the store's own InstanceContext:
- init_bucket_metadata_sys writes the cell of the store it was handed
(api.ctx) — no signature change — and keeps the double-init fail-fast,
now scoped to the instance (assert on the per-context set). The
dist-erasure check for the refresh loop reads the same context.
- get_global_bucket_metadata_sys / the crate-private getter behind the
~20 get_*_config helpers resolve through the current_ctx() facade —
the published store's context, or bootstrap before that — so every
existing reader keeps its exact single-instance behavior.
- New acceptance test: two real stores in one process each initialize
their own metadata system (the old global cell panicked right there),
plus a shared builder extracted from the store-graph test.
201 bucket/metadata regression tests green.
* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)
Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.
- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
with `init_background_cancel_token` (set-once) and `background_cancel_token()`
returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
route through the current instance's context; the static is removed. The
getter now returns an owned `Option<CancellationToken>` instead of a
`Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
the lifecycle worker/loops take the owned token (a shared fallback is cloned
when the token is somehow uninitialized). No Arc cycle is introduced —
workers hold a token clone, not the instance context.
Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.
Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.
Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 13)
* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)
Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.
- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
that constructs two independent InstanceContexts and verifies NONE of the
migrated state is shared — erasure setup, lock manager, region, deployment id,
the four service handles (tier/notifier/expiry/transition), the local disk
registry, the bucket monitor, and the background cancel token. This is the
object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
intentionally retained: storage startup still publishes into the process-level
bootstrap context (write-once region/endpoints/deployment id) and the single
GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
state. Lifting the guard requires threading a per-instance context through
startup — a follow-up beyond migrating the globals. The guard is NOT removed:
rejecting the second start is safer than the panic it would otherwise become.
Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).
perf(ecstore): per-bucket durability tier overrides (HP-5 phase 2)
Let a bucket override the process-wide RUSTFS_DURABILITY_MODE with its own
strict/relaxed/none tier, stored as a durability.json extension entry in the
bucket metadata file and resolved at commit points via effective_durability.
System-critical buckets (.rustfs.sys, .minio.sys) can never carry an override
and stay pinned to strict; the legacy full-off switch keeps its historical
semantics and per-bucket overrides do not apply under it. Overrides are
published and cleared through the existing bucket metadata cache-invalidation
path, and an admin GET/PUT handler exposes the configuration. Default behavior
is unchanged: with no override a bucket follows the global mode, which defaults
to strict and stays byte-for-byte identical to before.
Refs: https://github.com/rustfs/backlog/issues/938, https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
The peer `DeleteBucketMetadata` RPC handler was a stub that returned
success without doing anything, and the delete-bucket flow never sent
the notification in the first place. As a result, after a bucket was
deleted other nodes kept serving its stale cached metadata.
Wire the whole path end to end:
- ecstore: add an in-memory `remove_bucket_metadata` (free fn) and
`BucketMetadataSys::remove`, the counterpart to `set_bucket_metadata`,
and export it through the `api::bucket::metadata_sys` facade.
- node_service: `handle_delete_bucket_metadata` now validates the bucket
name and actually drops the cached metadata for it.
- bucket_usecase: after a successful delete_bucket, notify peers via
`notification_sys.delete_bucket_metadata` in the background, symmetric
to the existing `notify_bucket_metadata_reload` path.
Also update the delete-bucket-metadata unit test to assert the
empty-bucket rejection instead of the old always-success stub, and drop
an unused `tracing::debug` test import left over from #4322.
Verified: cargo fmt; cargo check -p rustfs-ecstore; cargo test -p rustfs
--lib --features rio-v2 test_delete_bucket_metadata_empty_bucket; arch
guardrail scripts pass.
Second TODO-convergence round over the current tree (backlog#646). All
line numbers in the old inventory had gone stale after the set_disk /
diagnostics / cluster refactors, so this re-scans and reduces the marker
count from 144 to 99.
STALE removals (comment describes already-implemented behavior, or dead
commented-out blocks) across ecstore (set_disk ops/core, store,
cluster/rpc, bucket/metadata_sys, services), iam, filemeta, s3select and
rustfs auth/object_usecase. No behavior change.
Safe fills, each verified:
- filemeta: replication_info_equals now also compares
replication_state_internal (function currently has no callers; adds a
regression test).
- bitrot: drop the confirmed-unused `_want` parameter from bitrot_verify
and the now-unused `sum` on LocalDisk::bitrot_verify, removing a
Bytes::copy_from_slice allocation. Streaming verify uses the file's
embedded per-shard hash, never the passed sum.
- signer: rename v4_ignored_headers -> V4_IGNORED_HEADERS and drop the
non_upper_case_globals allow.
- admin/heal: test_decode was #[ignore]d and used serde_urlencoded on a
JSON body (would panic); rewire to serde_json::from_slice to match the
production decode path, add assertions, un-ignore.
Verified: cargo fmt; cargo check on touched crates; tests pass
(filemeta, signer, bitrot, heal::test_decode); arch guardrail scripts
pass.
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.
Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
cxymds