* fix: address rc.1 release blockers
* fix: route release guards through architecture boundaries
* fix: close remaining rc.1 regression gaps
* refactor: group multipart listing options
* fix: resolve rc.1 CI regressions
* fix(ecstore): keep bucket-config writes off the caller's stack
A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.
Two CI lanes aborted with SIGABRT on this:
ILM Integration (serial)
rustfs app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions
Test and Lint (swift)
rustfs-protocols::swift_metadata_persistence::
swift_metadata_writes_are_durable
Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.
Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.
* fix(lifecycle): unbreak restore under the bucket generation fence
The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.
1. RestoreObject and ListMultipartUploads hard-required
`opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
deliberately leaves it unset when no guard extension is present — only the
S3 access layer installs one. Every direct caller therefore got
`InternalError: ... bucket generation guard is missing`. Resolve the
current generation instead, the way the copy path already does. The fence
is unaffected: RestoreObject still re-reads the incarnation from disk and
compares before admitting the restore, and the multipart listing is
filtered by the value it resolves.
2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
restored-copy expiry whose `restore_expires` had not already elapsed.
Whether the restored copy is due to expire is the ILM evaluator's
decision, made when it emitted DeleteRestoredAction; re-deriving it in
the set layer only adds a way for a legitimate action to be rejected.
The stale-event risk it appears to guard is already covered by the
surrounding snapshot match — a re-restore rewrites `restore_expires`,
so a replayed event fails the equality check. Drop the clause; the
fifteen identity clauses are unchanged.
Fixed:
rustfs app::lifecycle_transition_api_test::
restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
restore_object_usecase_completes_suspended_null_version_in_place
restore_object_usecase_reports_ongoing_conflict
rustfs-scanner::lifecycle_integration_test serial_tests::
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore
Verification: the CI ILM lane filter now runs 53/53 green locally.
* chore: address review follow-ups on this branch
Four items from the adversarial review that were still open.
- Restore the assertion `test_bucket_replication_replayed_delete_marker_
preserves_source_mtime_without_source_restart` is named for. The branch
had replaced the backlog#867 mtime check with `assert_replication_
converged`, which any successful replication satisfies, and deleted the
two helpers it needed — so the regression the test exists to catch would
now pass. This matters here specifically because the branch changes the
flag feeding `replication_delete_remove_options` and routes replay
through a new file and ordering.
- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
one real consumer calls the `_with_metadata` variant directly). Its test
stanza now exercises that variant, so the coverage moves to live code
rather than being deleted.
- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
2.7.0 before this branch and is untouched, so the caret range already
resolved there. Nothing in the diff uses the crate.
- Split the AGENTS.md "Adversarial Validation" policy change out of this
branch. The edit is defensible on its own, but it relaxes the review gate
that this branch has to pass, so it should land as its own PR reviewed on
its own merits rather than bundled with the change that benefits from it.
The reverted hunks are unchanged and ready to re-apply.
Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.
* test(ci): serialize the new bucket-incarnation tests
The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.
Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.
Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.
* test(ecstore): pin that a drained bucket is actually deletable
`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.
That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.
Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.
The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.
* fix(ecstore): let an explicit null-version delete purge its delete marker
Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.
On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:
explicit null-version purge of the null delete marker must succeed,
got [Some(MethodNotAllowed)]
so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.
The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.
This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.
Normalize both sides through `delete_file_info_version_id`.
The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.
* fix(app): drop the now-needless struct update
Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.
Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.
* test(e2e): assert the fresh-volume classification
four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.
Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.
* fix(bucket): apply the requested Object Lock to existing buckets
Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.
Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.
An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.
Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".
cargo nextest run -p rustfs-ecstore --lib: 3633 passed.
* fix(ecstore): box the generation-checked config mutation paths too
The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.
* fix(restore): keep the nil-version normalization the strip removed
Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:
- self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
+ self.version_id == fi.version_id
For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.
Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.
Only replication-target writes took the bucket transaction lock. Every other
config write (policy, tagging, lifecycle, versioning, ...) went straight to
the process-local metadata-system guard, which serializes nothing across
nodes.
Each config write is a read-modify-write of one whole BucketMetadata blob:
load the blob, replace one field, save the blob back. The namespace locks
inside read_config/save_config are taken and released separately, so they do
not span that cycle. Two nodes updating different config files of the same
bucket therefore both load the same blob, each set their own field, and the
later save drops the other's -- with both clients already told 2xx. This is
not last-writer-wins on one document; an orthogonal config silently vanishes.
Route update(), delete() and update_config_with() through
acquire_config_write_guards(), which takes the cluster-wide transaction lock
first and the metadata-system write guard second. That order is load-bearing:
taking the process-local guard first would park every local reader and writer
of every bucket behind a lock whose holder may be another node, turning
remote contention into a local stall.
The lock is per bucket rather than per config file, since a per-file key
would let exactly the offending pair run concurrently. Rename the helper to
acquire_bucket_metadata_transaction_lock to match, but deliberately keep the
lock resource string as "bucket-targets/{bucket}/transaction.lock": the key
is what nodes agree on, so renaming it would leave a mixed-version cluster
with two disjoint keys and stop old and new nodes from excluding each other
on the very writes that are serialized today.
update_config_with() already narrowed its staleness window to a single load
and save, but its exclusion was explicitly process-local; it is now
cluster-wide, so its doc comment no longer disclaims cross-node races.
Also make update_and_parse load through self.api instead of the ambient store
handle, so the read and the write of one read-modify-write cannot resolve to
different instances.
The new tests drive two BucketMetadataSys instances over one ECStore -- the
in-process stand-in for two nodes, since they share no RwLock and can only be
serialized by the namespace lock. Verified the lost-update test has teeth by
removing the lock and confirming it fails on round 0, with the tagging config
clobbered to empty by the concurrent policy write.
The LoadBucketMetadata peer-notification handler loaded bucket metadata
with the fabricating loader (ConfigNotFound -> BucketMetadata::new) and
unconditionally cached the result. On a transient read-quorum dip during
a reload notification, a peer cached an authoritative "no Object Lock"
default for a lock-enabled bucket, disabling the batch-delete retention
gate (object_lock_delete_check_required) on that node until the next
refresh, and wiping its bucket-target/durability sync state.
Production changes:
- New BucketMetadataSys::reload_from_store (metadata_sys::
reload_bucket_metadata): the peer reload path uses the presence-aware
loader and installs only metadata actually read from persisted
storage. A load miss returns an error (surfaced to the notifying peer
as success=false) and leaves the cache untouched; deletion still
propagates only through the dedicated DeleteBucketMetadata
notification.
- The reload runs under the outer metadata-sys write guard, load
included, mirroring update(): every other cache installer holds that
lock, so a reload snapshot can never land after - and roll back - a
newer concurrent install (the stale-load lost-update from the
review), and the install-plus-registry-sync sequence stays atomic
against concurrent removes and reloads (previously only the set call
was write-guarded, with the load outside any lock).
- The peer-visible miss error is a fixed string: the notifying peer
substring-matches error text against network-failure needles
(is_network_like_error), so interpolating a bucket name (e.g. a legal
bucket literally named "unavailable") could mark a healthy peer
offline.
- get_config's lazy insert routes through set(), picking up the
negative-cache invalidation.
An earlier draft instead guarded set() with a per-config updated_at
freshness comparison. Adversarial validation rejected it (three roles
independently): update_config stamps with the handling node's wall
clock, so within the skew the cluster already tolerates (+/-300s RPC
auth window) a config rewritten with an earlier stamp - e.g. revoking a
public-read policy through a second node, or any same-field rewrite
after an NTP step-back - would be skipped by every peer forever,
silently pinning the revoked permissive config with no re-convergence
path (the 15-minute refresh also routed through the guard). Race
staleness is second-scale while skew is minute-scale, so no tolerance
bound can separate them; the write-guard serialization closes the same
race without clocks and preserves the refresh loop's unconditional
converge-to-disk property, which is the cluster's self-healing
mechanism.
Startup audit (BucketMetadataSys::init): concurrent_load's
insert-if-vacant still installs a fabricated default when a transient
miss hits at boot - indistinguishable from a legacy bucket without a
metadata file at this layer - bounded by the next successful persisted
load. Making the object-lock gate fail closed on such entries is filed
as a follow-up, alongside the bare "unavailable" needle in
is_network_like_error and the Swift cache-only metadata writes.
Tests:
- bucket::metadata_sys::tests::
peer_reload_never_caches_fabricated_defaults_as_authoritative:
miss installs nothing / miss keeps the existing entry intact
(asserting the dedicated non-persisted error) / persisted reload
converges the cache over a stale entry.
- node_service::tests::
test_load_bucket_metadata_failure_skips_scanner_maintenance:
a failed reload reports failure and does not advance scanner
maintenance activity (previously recorded even on a miss).
- The handler success path stays uncovered at the RPC layer (needs an
isolated global object layer, like the pre-existing ignored test);
the composition is pinned at the sys level instead.
Verification:
- cargo fmt --check and cargo clippy --lib --tests clean on
rustfs-ecstore and rustfs.
- Targeted suites green; full cargo test -p rustfs-ecstore --lib:
3198/3200 with two parallelism-sensitive lock-test flakes from the
known baseline (pass in isolation; a different pair flakes per run).
- Adversarial validation (high-risk tier, all seven roles as
independent parallel reviewers) run per AGENTS.md; all findings
fixed or rebutted with evidence, three out-of-scope findings filed
as follow-up tasks.
Co-authored-by: houseme <housemecn@gmail.com>
Swift container and account metadata handlers cloned the cached
BucketMetadata, set the tagging fields, and called set_bucket_metadata,
which only updates the in-memory cache map. Nothing reached
.metadata.bin, so every Swift metadata POST was lost on restart and
silently overwritten by the next disk-truth reload (a peer
LoadBucketMetadata notification or the 15-minute refresh loop) — while
the client had already been told 2xx.
Route these writes through a new metadata_sys::update_config_with: a
read-modify-write that loads the on-disk metadata and persists the
result under the same write guard metadata_sys::update uses, so the
rewrite merges against disk truth instead of a possibly stale cache and
cannot clobber a concurrent update to another config file. Peers are
notified afterwards, matching the S3 config handlers.
Persisting these writes required hardening the paths that now produce
durable state:
- Account metadata writes validate account ownership. This metadata
holds the account's TempURL signing key, so an unauthenticated write
for someone else's account would have become a durable, cluster-wide
takeover of that account's pre-signed URLs. Reads stay open because
TempURL signature validation runs before credentials exist.
- disable_versioning verifies the container exists. Without it the
metadata loader's "no metadata on disk" default would be persisted,
creating an orphan metadata file and caching a fabricated default as
authoritative.
- Container and account metadata are size- and count-limited, reusing
the Swift limits object metadata already enforces; these tags land in
the bucket metadata file that every later config write rewrites whole.
- A rewrite refuses to run when the persisted tagging config is
unreadable, instead of merging onto an empty set and wiping the
container ACL and versioning tags. It reports 409 naming the remedy.
- Storage errors are logged in full and reported generically, since they
now carry real disk and quorum detail.
The tagging arm of BucketMetadata::update_config also clears the parsed
config, as the lifecycle arm does: parse_all_configs skips empty XML
rather than clearing, so a cleared config kept serving the old tags.
Tagging is serialized with the S3 XML serializer the loader can parse
back, not quick_xml, whose output was never round-trippable.
BucketMetadataSys::get_config lazily fabricated a default BucketMetadata
(object-lock off) for any bucket whose .metadata.bin was ConfigNotFound and
cached it in the map that the map-only, fail-closed metadata_sys::get()
serves. The object-lock batch-delete gate (object_lock_delete_check_required,
backlog#929 / #4297) treats that map as authoritative, so a metadata miss
became a cached "no lock" answer: a versioning peek could poison the cache
and let delete_objects skip the per-object retention/legal-hold stat. The
same fabrication raced make_bucket (lost update overwriting freshly
persisted lock-enabled metadata) and let the 15-minute refresh loop replace
good cached metadata on a transient quorum dip.
Production changes:
- get_config caches only metadata actually read from disk; misses are
recorded in a bounded negative cache (30s TTL, 10k entries, invalidated by
set()) so repeated lookups for metadata-less names cost no extra
namespace-lock + erasure-set fanout (reachable pre-auth via CORS
preflight and per-key in DeleteObjects).
- concurrent_load never lets a fabricated default REPLACE an existing map
entry; startup insert-if-vacant behavior for legacy buckets is preserved.
- delete_objects and new_ns_lock resolve dist-erasure, versioning, and the
object-lock gate from the set's own instance context (backlog#1052)
instead of the ambient facade, so a second in-process instance (or, in
tests, another test's transient DistErasure window) cannot reroute
locking onto an empty dist locker list or answer with the wrong
instance's bucket state.
Test-isolation changes (the bug that surfaced all of the above: the
delete_objects lock-gating test failed deterministically when sharing a
process with the lifecycle env tests):
- The MinIO-migration test builds on an isolated InstanceContext instead of
registering soon-deleted disks in the shared bootstrap registry.
- The cached lifecycle env re-registers its disks on every use, surviving
other serial tests' reset_local_disk_test_state.
- Hermetic SetDisks helpers gain isolated-context variants pinned to plain
erasure; tier-free non-serial test modules use them, guard-based
SetupTypeGuard tests stay on the bootstrap context.
- Three deterministic pin tests (nextest-safe) cover the caching contract,
the delete gate resolution source, and the ns-lock resolution source.
Verification:
- cargo test -p rustfs-ecstore --lib -- --exact <4-test combo from the
report> (previously failing, now green)
- cargo test -p rustfs-ecstore --lib: 3169 passed / 0 failed across
repeated runs; cargo fmt --check and cargo clippy --lib --tests clean
- Adversarial validation (high-risk tier, all seven roles) run per
AGENTS.md; all findings fixed or rebutted with evidence
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.
Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:
- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
(the ambient disk registry = the first published store's context), so
list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
a second server's bucket metadata landed in the first server's
.rustfs.sys; set/remove/get/created_at all used the ambient metadata
system.
Now the whole chain is bound to the owning store's InstanceContext:
- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
operate on that context's registered disks; ECStore::new (and the test
store builder) pass the store's context. The legacy constructors keep
the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
context's metadata system and fall back to the ambient default before
the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
table-bucket delete guard and emptiness check) use the per-context
variants and this instance's disks.
Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.
579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
The OnceLock introduced in 5fbd49a80 for bucket_metadata_sys panics when
multiple test modules initialize their own ECStore against the shared
bootstrap context. Switch to Mutex<Option> so test fixtures can
reinitialize safely.
Extract shared gating test environment to gating_test_env.rs to avoid
duplicate ECStore setup across delete_objects_stat_gating_test and
put_prelookup_gating_test.
Fixes 8 test failures:
- capacity_dirty_scope_test (2 tests)
- delete_objects_stat_gating_test (3 tests)
- put_prelookup_gating_test (3 tests)
backlog#1052 S3, third slice — the hard blocker for a second embedded
server's service stage. GLOBAL_BUCKET_METADATA_SYS was a process-global
OnceLock whose init ran .set().unwrap(): a second instance's service
startup panicked the whole process. The system it guards is inherently
per-store (it holds the store handle and that store's bucket→metadata
cache), so it now lives on the store's own InstanceContext:
- init_bucket_metadata_sys writes the cell of the store it was handed
(api.ctx) — no signature change — and keeps the double-init fail-fast,
now scoped to the instance (assert on the per-context set). The
dist-erasure check for the refresh loop reads the same context.
- get_global_bucket_metadata_sys / the crate-private getter behind the
~20 get_*_config helpers resolve through the current_ctx() facade —
the published store's context, or bootstrap before that — so every
existing reader keeps its exact single-instance behavior.
- New acceptance test: two real stores in one process each initialize
their own metadata system (the old global cell panicked right there),
plus a shared builder extracted from the store-graph test.
201 bucket/metadata regression tests green.
* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)
Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.
- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
with `init_background_cancel_token` (set-once) and `background_cancel_token()`
returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
route through the current instance's context; the static is removed. The
getter now returns an owned `Option<CancellationToken>` instead of a
`Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
the lifecycle worker/loops take the owned token (a shared fallback is cloned
when the token is somehow uninitialized). No Arc cycle is introduced —
workers hold a token clone, not the instance context.
Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.
Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.
Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 13)
* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)
Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.
- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
that constructs two independent InstanceContexts and verifies NONE of the
migrated state is shared — erasure setup, lock manager, region, deployment id,
the four service handles (tier/notifier/expiry/transition), the local disk
registry, the bucket monitor, and the background cancel token. This is the
object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
intentionally retained: storage startup still publishes into the process-level
bootstrap context (write-once region/endpoints/deployment id) and the single
GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
state. Lifting the guard requires threading a per-instance context through
startup — a follow-up beyond migrating the globals. The guard is NOT removed:
rejecting the second start is safer than the panic it would otherwise become.
Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).
perf(ecstore): per-bucket durability tier overrides (HP-5 phase 2)
Let a bucket override the process-wide RUSTFS_DURABILITY_MODE with its own
strict/relaxed/none tier, stored as a durability.json extension entry in the
bucket metadata file and resolved at commit points via effective_durability.
System-critical buckets (.rustfs.sys, .minio.sys) can never carry an override
and stay pinned to strict; the legacy full-off switch keeps its historical
semantics and per-bucket overrides do not apply under it. Overrides are
published and cleared through the existing bucket metadata cache-invalidation
path, and an admin GET/PUT handler exposes the configuration. Default behavior
is unchanged: with no override a bucket follows the global mode, which defaults
to strict and stays byte-for-byte identical to before.
Refs: https://github.com/rustfs/backlog/issues/938, https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
The peer `DeleteBucketMetadata` RPC handler was a stub that returned
success without doing anything, and the delete-bucket flow never sent
the notification in the first place. As a result, after a bucket was
deleted other nodes kept serving its stale cached metadata.
Wire the whole path end to end:
- ecstore: add an in-memory `remove_bucket_metadata` (free fn) and
`BucketMetadataSys::remove`, the counterpart to `set_bucket_metadata`,
and export it through the `api::bucket::metadata_sys` facade.
- node_service: `handle_delete_bucket_metadata` now validates the bucket
name and actually drops the cached metadata for it.
- bucket_usecase: after a successful delete_bucket, notify peers via
`notification_sys.delete_bucket_metadata` in the background, symmetric
to the existing `notify_bucket_metadata_reload` path.
Also update the delete-bucket-metadata unit test to assert the
empty-bucket rejection instead of the old always-success stub, and drop
an unused `tracing::debug` test import left over from #4322.
Verified: cargo fmt; cargo check -p rustfs-ecstore; cargo test -p rustfs
--lib --features rio-v2 test_delete_bucket_metadata_empty_bucket; arch
guardrail scripts pass.
Second TODO-convergence round over the current tree (backlog#646). All
line numbers in the old inventory had gone stale after the set_disk /
diagnostics / cluster refactors, so this re-scans and reduces the marker
count from 144 to 99.
STALE removals (comment describes already-implemented behavior, or dead
commented-out blocks) across ecstore (set_disk ops/core, store,
cluster/rpc, bucket/metadata_sys, services), iam, filemeta, s3select and
rustfs auth/object_usecase. No behavior change.
Safe fills, each verified:
- filemeta: replication_info_equals now also compares
replication_state_internal (function currently has no callers; adds a
regression test).
- bitrot: drop the confirmed-unused `_want` parameter from bitrot_verify
and the now-unused `sum` on LocalDisk::bitrot_verify, removing a
Bytes::copy_from_slice allocation. Streaming verify uses the file's
embedded per-shard hash, never the passed sum.
- signer: rename v4_ignored_headers -> V4_IGNORED_HEADERS and drop the
non_upper_case_globals allow.
- admin/heal: test_decode was #[ignore]d and used serde_urlencoded on a
JSON body (would panic); rewire to serde_json::from_slice to match the
production decode path, add assertions, un-ignore.
Verified: cargo fmt; cargo check on touched crates; tests pass
(filemeta, signer, bitrot, heal::test_decode); arch guardrail scripts
pass.
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.
Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
cxymds