fix: address rc.1 release blockers (#5648)

* fix: address rc.1 release blockers

* fix: route release guards through architecture boundaries

* fix: close remaining rc.1 regression gaps

* refactor: group multipart listing options

* fix: resolve rc.1 CI regressions

* fix(ecstore): keep bucket-config writes off the caller's stack

A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.

Two CI lanes aborted with SIGABRT on this:

  ILM Integration (serial)
    rustfs app::lifecycle_transition_api_test::
      compensation_driven_complete_multipart_upload_still_transitions
  Test and Lint (swift)
    rustfs-protocols::swift_metadata_persistence::
      swift_metadata_writes_are_durable

Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.

Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.

* fix(lifecycle): unbreak restore under the bucket generation fence

The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.

1. RestoreObject and ListMultipartUploads hard-required
   `opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
   deliberately leaves it unset when no guard extension is present — only the
   S3 access layer installs one. Every direct caller therefore got
   `InternalError: ... bucket generation guard is missing`. Resolve the
   current generation instead, the way the copy path already does. The fence
   is unaffected: RestoreObject still re-reads the incarnation from disk and
   compares before admitting the restore, and the multipart listing is
   filtered by the value it resolves.

2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
   restored-copy expiry whose `restore_expires` had not already elapsed.
   Whether the restored copy is due to expire is the ILM evaluator's
   decision, made when it emitted DeleteRestoredAction; re-deriving it in
   the set layer only adds a way for a legitimate action to be rejected.
   The stale-event risk it appears to guard is already covered by the
   surrounding snapshot match — a re-restore rewrites `restore_expires`,
   so a replayed event fails the equality check. Drop the clause; the
   fifteen identity clauses are unchanged.

Fixed:
  rustfs app::lifecycle_transition_api_test::
    restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
    restore_object_usecase_completes_suspended_null_version_in_place
    restore_object_usecase_reports_ongoing_conflict
  rustfs-scanner::lifecycle_integration_test serial_tests::
    test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore

Verification: the CI ILM lane filter now runs 53/53 green locally.

* chore: address review follow-ups on this branch

Four items from the adversarial review that were still open.

- Restore the assertion `test_bucket_replication_replayed_delete_marker_
  preserves_source_mtime_without_source_restart` is named for. The branch
  had replaced the backlog#867 mtime check with `assert_replication_
  converged`, which any successful replication satisfies, and deleted the
  two helpers it needed — so the regression the test exists to catch would
  now pass. This matters here specifically because the branch changes the
  flag feeding `replication_delete_remove_options` and routes replay
  through a new file and ordering.

- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
  one real consumer calls the `_with_metadata` variant directly). Its test
  stanza now exercises that variant, so the coverage moves to live code
  rather than being deleted.

- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
  2.7.0 before this branch and is untouched, so the caret range already
  resolved there. Nothing in the diff uses the crate.

- Split the AGENTS.md "Adversarial Validation" policy change out of this
  branch. The edit is defensible on its own, but it relaxes the review gate
  that this branch has to pass, so it should land as its own PR reviewed on
  its own merits rather than bundled with the change that benefits from it.
  The reverted hunks are unchanged and ready to re-apply.

Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.

* test(ci): serialize the new bucket-incarnation tests

The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.

Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.

Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.

* test(ecstore): pin that a drained bucket is actually deletable

`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.

That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.

Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.

The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.

* fix(ecstore): let an explicit null-version delete purge its delete marker

Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.

On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:

    explicit null-version purge of the null delete marker must succeed,
    got [Some(MethodNotAllowed)]

so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.

The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.

This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.

Normalize both sides through `delete_file_info_version_id`.

The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.

* fix(app): drop the now-needless struct update

Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.

Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.

* test(e2e): assert the fresh-volume classification

four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.

Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.

* fix(bucket): apply the requested Object Lock to existing buckets

Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.

Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.

An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.

Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".

cargo nextest run -p rustfs-ecstore --lib: 3633 passed.

* fix(ecstore): box the generation-checked config mutation paths too

The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.

* fix(restore): keep the nil-version normalization the strip removed

Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:

    -  self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
    +  self.version_id == fi.version_id

For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.

Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.

rustfs/src/admin/handlers/durability.rs

@@ -160,6 +160,9 @@ impl Operation for SetBucketDurabilityHandler {
         authenticate_admin(&req).await?;
 
         let bucket = bucket_from_params(&params)?;
+        let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(&bucket)
+            .await
+            .map_err(|e| s3_error!(InternalError, "failed to capture bucket incarnation: {}", e))?;
 
         let body = req
             .input
@@ -175,7 +178,7 @@ impl Operation for SetBucketDurabilityHandler {
 
         // System buckets are rejected by the metadata layer (and pinned to
         // strict by the disk layer regardless).
-        metadata_sys::update(&bucket, BUCKET_DURABILITY_CONFIG, json)
+        metadata_sys::update_if_incarnation(&bucket, BUCKET_DURABILITY_CONFIG, json, expected_incarnation_id)
             .await
             .map_err(|e| s3_error!(InternalError, "failed to set bucket durability: {}", e))?;
 
@@ -218,8 +221,11 @@ impl Operation for DeleteBucketDurabilityHandler {
         authenticate_admin(&req).await?;
 
         let bucket = bucket_from_params(&params)?;
+        let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(&bucket)
+            .await
+            .map_err(|e| s3_error!(InternalError, "failed to capture bucket incarnation: {}", e))?;
 
-        metadata_sys::delete(&bucket, BUCKET_DURABILITY_CONFIG)
+        metadata_sys::delete_if_incarnation(&bucket, BUCKET_DURABILITY_CONFIG, expected_incarnation_id)
             .await
             .map_err(|e| s3_error!(InternalError, "failed to clear bucket durability: {}", e))?;
 

rustfs/src/admin/handlers/quota.rs

@@ -18,7 +18,7 @@ use crate::admin::auth::{validate_admin_request, validate_admin_request_with_buc
 use crate::admin::handlers::site_replication::site_replication_bucket_meta_hook;
 use crate::admin::router::{AdminOperation, Operation, S3Router};
 use crate::admin::runtime_sources::{current_bucket_metadata_handle, current_object_store_handle};
-use crate::admin::storage_api::bucket::metadata_sys::BucketMetadataSys;
+use crate::admin::storage_api::bucket::metadata_sys::{self, BucketMetadataSys};
 use crate::admin::storage_api::bucket::quota::checker::QuotaChecker;
 use crate::admin::storage_api::bucket::quota::{BucketQuota, QuotaError, QuotaOperation};
 use crate::auth::{check_key_valid, get_session_token};
@@ -277,6 +277,9 @@ impl Operation for SetBucketQuotaHandler {
         if bucket.is_empty() {
             return Err(s3_error!(InvalidRequest, "bucket name is required"));
         }
+        let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(&bucket)
+            .await
+            .map_err(|e| s3_error!(InternalError, "failed to capture bucket incarnation: {}", e))?;
 
         let body = req
             .input
@@ -297,7 +300,7 @@ impl Operation for SetBucketQuotaHandler {
         let mut quota_checker = QuotaChecker::new(metadata_sys_lock.clone());
 
         let updated_at = quota_checker
-            .set_quota_config(&bucket, quota.clone())
+            .set_quota_config_if_incarnation(&bucket, quota.clone(), expected_incarnation_id)
             .await
             .map_err(|e| s3_error!(InternalError, "failed to set quota: {}", e))?;
 
@@ -369,7 +372,6 @@ impl Operation for GetBucketQuotaHandler {
         if bucket.is_empty() {
             return Err(s3_error!(InvalidRequest, "bucket name is required"));
         }
-
         validate_admin_request_with_bucket(
             &req.headers,
             &cred,
@@ -452,6 +454,9 @@ impl Operation for ClearBucketQuotaHandler {
         if bucket.is_empty() {
             return Err(s3_error!(InvalidRequest, "bucket name is required"));
         }
+        let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(&bucket)
+            .await
+            .map_err(|e| s3_error!(InternalError, "failed to capture bucket incarnation: {}", e))?;
 
         info!(
             event = EVENT_ADMIN_QUOTA_STATE,
@@ -471,7 +476,7 @@ impl Operation for ClearBucketQuotaHandler {
         // Clear quota (set to None)
         let quota = BucketQuota::new(None);
         let updated_at = quota_checker
-            .set_quota_config(&bucket, quota.clone())
+            .set_quota_config_if_incarnation(&bucket, quota.clone(), expected_incarnation_id)
             .await
             .map_err(|e| s3_error!(InternalError, "failed to clear quota: {}", e))?;
 

rustfs/src/admin/handlers/site_replication.rs

@@ -6667,6 +6667,7 @@ async fn ensure_site_replication_bucket_targets_with_runtime(
     local_peer: &PeerInfo,
     config: Option<&s3s::dto::ReplicationConfiguration>,
     service_account_secret_key: &str,
+    expected_incarnation_id: Uuid,
 ) -> S3Result<()> {
     let existing = match metadata_sys::list_bucket_targets(bucket).await {
         Ok(targets) => targets,
@@ -6690,11 +6691,9 @@ async fn ensure_site_replication_bucket_targets_with_runtime(
     if json_targets == existing_json {
         return Ok(());
     }
-    metadata_sys::update(bucket, BUCKET_TARGETS_FILE, json_targets)
+    metadata_sys::update_if_incarnation(bucket, BUCKET_TARGETS_FILE, json_targets, expected_incarnation_id)
         .await
         .map_err(ApiError::from)?;
-    BucketTargetSys::get().update_all_targets(bucket, Some(&updated)).await;
-
     Ok(())
 }
 
@@ -6707,6 +6706,9 @@ async fn bucket_replication_config_for_target_refresh(bucket: &str) -> S3Result<
 }
 
 async fn ensure_site_replication_bucket_targets(bucket: &str) -> S3Result<()> {
+    let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(bucket)
+        .await
+        .map_err(ApiError::from)?;
     let _targets_guard = lock_bucket_targets_metadata(bucket).await;
     let Some(runtime) = runtime_site_replication_targets().await? else {
         return Ok(());
@@ -6718,6 +6720,7 @@ async fn ensure_site_replication_bucket_targets(bucket: &str) -> S3Result<()> {
         &runtime.local_peer,
         config.as_ref(),
         &runtime.service_account_secret_key,
+        expected_incarnation_id,
     )
     .await
 }
@@ -6727,6 +6730,7 @@ async fn ensure_site_replication_bucket_replication_config_with_runtime(
     state: &SiteReplicationState,
     local_peer: &PeerInfo,
     service_account_secret_key: &str,
+    expected_incarnation_id: Uuid,
 ) -> S3Result<()> {
     let existing = match metadata_sys::get_replication_config(bucket).await {
         Ok((existing, _)) => Some(existing),
@@ -6771,7 +6775,7 @@ async fn ensure_site_replication_bucket_replication_config_with_runtime(
 
     let data = serialize(&ReplicationConfiguration { role, rules })
         .map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize replication failed: {e}")))?;
-    metadata_sys::update(bucket, BUCKET_REPLICATION_CONFIG, data)
+    metadata_sys::update_if_incarnation(bucket, BUCKET_REPLICATION_CONFIG, data, expected_incarnation_id)
         .await
         .map_err(ApiError::from)?;
 
@@ -6782,11 +6786,33 @@ async fn ensure_site_replication_bucket_setup(bucket: &str) -> S3Result<bool> {
     let Some(runtime) = runtime_site_replication_targets().await? else {
         return Ok(false);
     };
-    ensure_site_replication_bucket_setup_with_runtime(bucket, &runtime).await?;
+    let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(bucket)
+        .await
+        .map_err(ApiError::from)?;
+    ensure_site_replication_bucket_setup_with_runtime_for_incarnation(bucket, &runtime, expected_incarnation_id).await?;
+    Ok(true)
+}
+
+async fn ensure_site_replication_bucket_setup_for_incarnation(bucket: &str, incarnation_id: Uuid) -> S3Result<bool> {
+    let Some(runtime) = runtime_site_replication_targets().await? else {
+        return Ok(false);
+    };
+    ensure_site_replication_bucket_setup_with_runtime_for_incarnation(bucket, &runtime, incarnation_id).await?;
     Ok(true)
 }
 
 async fn ensure_site_replication_bucket_setup_with_runtime(bucket: &str, runtime: &SiteReplicationRuntime) -> S3Result<()> {
+    let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(bucket)
+        .await
+        .map_err(ApiError::from)?;
+    ensure_site_replication_bucket_setup_with_runtime_for_incarnation(bucket, runtime, expected_incarnation_id).await
+}
+
+async fn ensure_site_replication_bucket_setup_with_runtime_for_incarnation(
+    bucket: &str,
+    runtime: &SiteReplicationRuntime,
+    expected_incarnation_id: Uuid,
+) -> S3Result<()> {
     let _targets_guard = lock_bucket_targets_metadata(bucket).await;
     let config = bucket_replication_config_for_target_refresh(bucket).await?;
     ensure_site_replication_bucket_targets_with_runtime(
@@ -6795,6 +6821,7 @@ async fn ensure_site_replication_bucket_setup_with_runtime(bucket: &str, runtime
         &runtime.local_peer,
         config.as_ref(),
         &runtime.service_account_secret_key,
+        expected_incarnation_id,
     )
     .await?;
     ensure_site_replication_bucket_replication_config_with_runtime(
@@ -6802,12 +6829,16 @@ async fn ensure_site_replication_bucket_setup_with_runtime(bucket: &str, runtime
         &runtime.state,
         &runtime.local_peer,
         &runtime.service_account_secret_key,
+        expected_incarnation_id,
     )
     .await?;
     Ok(())
 }
 
 async fn cleanup_removed_site_replication_bucket(bucket: &str, removed_deployment_ids: &HashSet<String>) -> S3Result<usize> {
+    let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(bucket)
+        .await
+        .map_err(ApiError::from)?;
     let _targets_guard = lock_bucket_targets_metadata(bucket).await;
     let mut removed = 0usize;
 
@@ -6819,12 +6850,9 @@ async fn cleanup_removed_site_replication_bucket(bucket: &str, removed_deploymen
                 let json_targets = serde_json::to_vec(&updated_targets).map_err(|e| {
                     S3Error::with_message(S3ErrorCode::InternalError, format!("serialize bucket targets failed: {e}"))
                 })?;
-                metadata_sys::update(bucket, BUCKET_TARGETS_FILE, json_targets)
+                metadata_sys::update_if_incarnation(bucket, BUCKET_TARGETS_FILE, json_targets, expected_incarnation_id)
                     .await
                     .map_err(ApiError::from)?;
-                BucketTargetSys::get()
-                    .update_all_targets(bucket, Some(&updated_targets))
-                    .await;
                 removed = removed.saturating_add(removed_targets);
             }
         }
@@ -6840,11 +6868,11 @@ async fn cleanup_removed_site_replication_bucket(bucket: &str, removed_deploymen
                     let data = serialize(&updated_config).map_err(|e| {
                         S3Error::with_message(S3ErrorCode::InternalError, format!("serialize replication failed: {e}"))
                     })?;
-                    metadata_sys::update(bucket, BUCKET_REPLICATION_CONFIG, data)
+                    metadata_sys::update_if_incarnation(bucket, BUCKET_REPLICATION_CONFIG, data, expected_incarnation_id)
                         .await
                         .map_err(ApiError::from)?;
                 } else {
-                    metadata_sys::delete(bucket, BUCKET_REPLICATION_CONFIG)
+                    metadata_sys::delete_if_incarnation(bucket, BUCKET_REPLICATION_CONFIG, expected_incarnation_id)
                         .await
                         .map_err(ApiError::from)?;
                 }
@@ -7121,6 +7149,9 @@ async fn refresh_bucket_targets_after_endpoint_edit(pending_id: &str, service_ac
     let buckets = store.list_bucket(&BucketOptions::default()).await.map_err(ApiError::from)?;
 
     for bucket in buckets {
+        let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(&bucket.name)
+            .await
+            .map_err(ApiError::from)?;
         let _state_guard = SITE_REPLICATION_STATE_LOCK.lock().await;
         let state = load_site_replication_state().await?;
         let Some(pending) = pending_endpoint_refresh(&state).filter(|pending| pending.id == pending_id) else {
@@ -7136,6 +7167,7 @@ async fn refresh_bucket_targets_after_endpoint_edit(pending_id: &str, service_ac
             &local_peer,
             replication_config.as_ref(),
             service_account_secret_key,
+            expected_incarnation_id,
         )
         .await?;
     }
@@ -7205,7 +7237,7 @@ async fn start_site_bucket_resync(bucket: &str, target_arn: &str, resync_id: &st
         return bucket_status;
     };
     let _targets_guard = lock_bucket_targets_metadata(bucket).await;
-    let _transaction_guard = match metadata_sys::acquire_bucket_metadata_transaction_lock(bucket).await {
+    let transaction_guard = match metadata_sys::acquire_bucket_metadata_transaction_lock(bucket).await {
         Ok(guard) => guard,
         Err(_) => {
             bucket_status.status = "failed".to_string();
@@ -7273,12 +7305,12 @@ async fn start_site_bucket_resync(bucket: &str, target_arn: &str, resync_id: &st
     let opts = replication::resync_opts(bucket, target_arn.clone(), resync_id, reset_before);
     let admission_pool = pool.clone();
     let activation_pool = pool.clone();
-    let committed_targets = match replication::commit_resync_target(
+    let _committed_targets = match replication::commit_resync_target(
         targets,
         opts,
         move |opts| async move { admission_pool.admit_bucket_resync(opts).await },
         move |encoded| async move {
-            metadata_sys::update_bucket_targets_under_transaction_lock(bucket, encoded)
+            metadata_sys::update_bucket_targets_under_transaction_lock(&transaction_guard, bucket, encoded)
                 .await
                 .map(|_| ())
                 .map_err(|_| {
@@ -7304,10 +7336,6 @@ async fn start_site_bucket_resync(bucket: &str, target_arn: &str, resync_id: &st
             return bucket_status;
         }
     };
-    BucketTargetSys::get()
-        .update_all_targets(bucket, Some(&committed_targets))
-        .await;
-
     bucket_status
 }
 
@@ -7318,6 +7346,14 @@ async fn cancel_site_bucket_resync(bucket: &str, target_arn: &str, resync_id: &s
         status: "canceled".to_string(),
         ..Default::default()
     };
+    let expected_incarnation_id = match metadata_sys::capture_bucket_metadata_incarnation(bucket).await {
+        Ok(incarnation_id) => incarnation_id,
+        Err(err) => {
+            bucket_status.status = "failed".to_string();
+            bucket_status.err_detail = err.to_string();
+            return bucket_status;
+        }
+    };
     let targets_guard = lock_bucket_targets_metadata(bucket).await;
 
     let mut targets = match metadata_sys::list_bucket_targets(bucket).await {
@@ -7366,12 +7402,13 @@ async fn cancel_site_bucket_resync(bucket: &str, target_arn: &str, resync_id: &s
         }
     };
 
-    if let Err(err) = metadata_sys::update(bucket, BUCKET_TARGETS_FILE, json_targets).await {
+    if let Err(err) =
+        metadata_sys::update_if_incarnation(bucket, BUCKET_TARGETS_FILE, json_targets, expected_incarnation_id).await
+    {
         bucket_status.status = "failed".to_string();
         bucket_status.err_detail = err.to_string();
         return bucket_status;
     }
-    BucketTargetSys::get().update_all_targets(bucket, Some(&targets)).await;
     drop(targets_guard);
 
     bucket_status
@@ -7491,13 +7528,16 @@ fn bucket_versioning_xml() -> S3Result<Vec<u8>> {
 }
 
 async fn ensure_site_replication_bucket_versioning(bucket: &str) -> S3Result<()> {
+    let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(bucket)
+        .await
+        .map_err(ApiError::from)?;
     match metadata_sys::get_versioning_config(bucket).await {
         Ok((config, _)) if config.enabled() => return Ok(()),
         Ok(_) | Err(StorageError::ConfigNotFound) => {}
         Err(err) => return Err(ApiError::from(err).into()),
     }
 
-    metadata_sys::update(bucket, BUCKET_VERSIONING_CONFIG, bucket_versioning_xml()?)
+    metadata_sys::update_if_incarnation(bucket, BUCKET_VERSIONING_CONFIG, bucket_versioning_xml()?, expected_incarnation_id)
         .await
         .map_err(ApiError::from)?;
 
@@ -7530,6 +7570,9 @@ async fn apply_bucket_meta_item(item: SRBucketMeta) -> S3Result<()> {
     let Some(store) = current_object_store_handle() else {
         return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
     };
+    let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(&item.bucket)
+        .await
+        .map_err(ApiError::from)?;
 
     store
         .get_bucket_info(&item.bucket, &BucketOptions::default())
@@ -7630,11 +7673,11 @@ async fn apply_bucket_meta_item(item: SRBucketMeta) -> S3Result<()> {
 
     if !skip_config_write {
         if let Some(data) = data {
-            metadata_sys::update(&item.bucket, config_file, data)
+            metadata_sys::update_if_incarnation(&item.bucket, config_file, data, expected_incarnation_id)
                 .await
                 .map_err(ApiError::from)?;
         } else {
-            metadata_sys::delete(&item.bucket, config_file)
+            metadata_sys::delete_if_incarnation(&item.bucket, config_file, expected_incarnation_id)
                 .await
                 .map_err(ApiError::from)?;
         }
@@ -7644,7 +7687,7 @@ async fn apply_bucket_meta_item(item: SRBucketMeta) -> S3Result<()> {
     if item.r#type == "replication-config" {
         // Rebuild the local outbound rules too: a site that joined an already-replicated
         // bucket receives this item before it has any `site-repl-*` rule of its own.
-        ensure_site_replication_bucket_setup(&item.bucket).await?;
+        ensure_site_replication_bucket_setup_for_incarnation(&item.bucket, expected_incarnation_id).await?;
     }
 
     if item.r#type == "version-config"
@@ -7653,7 +7696,7 @@ async fn apply_bucket_meta_item(item: SRBucketMeta) -> S3Result<()> {
             .ok()
             .is_some_and(|(config, _)| config.enabled())
     {
-        ensure_site_replication_bucket_setup(&item.bucket).await?;
+        ensure_site_replication_bucket_setup_for_incarnation(&item.bucket, expected_incarnation_id).await?;
     }
 
     Ok(())
@@ -8569,9 +8612,17 @@ impl Operation for SRPeerBucketOpsHandler {
                     )
                     .await
                     .map_err(ApiError::from)?;
-                metadata_sys::update(&bucket, BUCKET_VERSIONING_CONFIG, bucket_versioning_xml()?)
+                let expected_incarnation_id = metadata_sys::capture_bucket_metadata_incarnation(&bucket)
                     .await
                     .map_err(ApiError::from)?;
+                metadata_sys::update_if_incarnation(
+                    &bucket,
+                    BUCKET_VERSIONING_CONFIG,
+                    bucket_versioning_xml()?,
+                    expected_incarnation_id,
+                )
+                .await
+                .map_err(ApiError::from)?;
             }
             "configure-replication" => {
                 store

rustfs/src/admin/router.rs

@@ -2478,7 +2478,7 @@ async fn start_replication_resync(bucket: &str, reset: &ReplicationResetStartReq
     };
 
     let _targets_guard = lock_bucket_targets_metadata(bucket).await;
-    let _transaction_guard = metadata_sys::acquire_bucket_metadata_transaction_lock(bucket)
+    let transaction_guard = metadata_sys::acquire_bucket_metadata_transaction_lock(bucket)
         .await
         .map_err(ApiError::from)?;
     let (config, _) = metadata_sys::get_replication_config(bucket).await.map_err(ApiError::from)?;
@@ -2494,7 +2494,7 @@ async fn start_replication_resync(bucket: &str, reset: &ReplicationResetStartReq
         opts,
         move |opts| async move { admission_pool.admit_bucket_resync(opts).await },
         move |encoded| async move {
-            metadata_sys::update_bucket_targets_under_transaction_lock(bucket, encoded)
+            metadata_sys::update_bucket_targets_under_transaction_lock(&transaction_guard, bucket, encoded)
                 .await
                 .map(|_| ())
                 .map_err(|_| {

rustfs/src/admin/storage_api.rs

@@ -286,16 +286,39 @@ pub(crate) mod metadata_sys {
         crate::storage::storage_api::update_bucket_metadata_config(bucket, config_file, data).await
     }
 
-    pub(crate) async fn acquire_bucket_metadata_transaction_lock(bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard> {
+    pub(crate) async fn update_if_incarnation(
+        bucket: &str,
+        config_file: &str,
+        data: Vec<u8>,
+        expected_incarnation_id: uuid::Uuid,
+    ) -> Result<OffsetDateTime> {
+        super::ecstore_bucket::metadata_sys::update_if_incarnation(bucket, config_file, data, expected_incarnation_id).await
+    }
+
+    pub(crate) async fn capture_bucket_metadata_incarnation(bucket: &str) -> Result<uuid::Uuid> {
+        super::ecstore_bucket::metadata_sys::capture_bucket_metadata_incarnation(bucket).await
+    }
+
+    pub(crate) async fn acquire_bucket_metadata_transaction_lock(
+        bucket: &str,
+    ) -> Result<super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard> {
         crate::storage::storage_api::acquire_bucket_metadata_transaction_lock(bucket).await
     }
 
-    pub(crate) async fn update_bucket_targets_under_transaction_lock(bucket: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
-        crate::storage::storage_api::update_bucket_targets_under_transaction_lock(bucket, data).await
+    pub(crate) async fn update_bucket_targets_under_transaction_lock(
+        guard: &super::ecstore_bucket::metadata_sys::BucketMetadataMutationGuard,
+        bucket: &str,
+        data: Vec<u8>,
+    ) -> Result<OffsetDateTime> {
+        crate::storage::storage_api::update_bucket_targets_under_transaction_lock(guard, bucket, data).await
     }
 
-    pub(crate) async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
-        crate::storage::storage_api::delete_bucket_metadata_config(bucket, config_file).await
+    pub(crate) async fn delete_if_incarnation(
+        bucket: &str,
+        config_file: &str,
+        expected_incarnation_id: uuid::Uuid,
+    ) -> Result<OffsetDateTime> {
+        super::ecstore_bucket::metadata_sys::delete_if_incarnation(bucket, config_file, expected_incarnation_id).await
     }
 
     pub(crate) async fn get_bucket_policy(bucket: &str) -> Result<(BucketPolicy, OffsetDateTime)> {

rustfs/src/app/bucket_usecase.rs

@@ -16,12 +16,11 @@
 
 use super::storage_api::bucket_usecase::ECStore;
 use super::storage_api::bucket_usecase::StorageObjectInfo as ObjectInfo;
-use super::storage_api::bucket_usecase::access::{ReqInfo, authorize_request, req_info_ref};
+use super::storage_api::bucket_usecase::access::{ReqInfo, authorize_request, bucket_config_mutation_incarnation, req_info_ref};
 #[cfg(test)]
 use super::storage_api::bucket_usecase::bucket::target::BucketTarget;
 use super::storage_api::bucket_usecase::bucket::{
     ObjectLockConfigExt as _, VersioningConfigExt as _,
-    bucket_target_sys::BucketTargetSys,
     lifecycle::bucket_lifecycle_ops::{
         enqueue_expiry_for_existing_objects, enqueue_transition_for_existing_objects, run_stale_multipart_upload_cleanup_once,
         validate_lifecycle_config, validate_transition_tier,
@@ -383,6 +382,29 @@ fn serialize_config<T: xml::Serialize>(value: &T) -> S3Result<Vec<u8>> {
     serialize(value).map_err(to_internal_error)
 }
 
+async fn update_bucket_config_for_incarnation(
+    bucket: &str,
+    config_file: &str,
+    data: Vec<u8>,
+    expected_incarnation_id: Option<uuid::Uuid>,
+) -> Result<time::OffsetDateTime, StorageError> {
+    match expected_incarnation_id {
+        Some(incarnation_id) => metadata_sys::update_if_incarnation(bucket, config_file, data, incarnation_id).await,
+        None => metadata_sys::update(bucket, config_file, data).await,
+    }
+}
+
+async fn delete_bucket_config_for_incarnation(
+    bucket: &str,
+    config_file: &str,
+    expected_incarnation_id: Option<uuid::Uuid>,
+) -> Result<time::OffsetDateTime, StorageError> {
+    match expected_incarnation_id {
+        Some(incarnation_id) => metadata_sys::delete_if_incarnation(bucket, config_file, incarnation_id).await,
+        None => metadata_sys::delete(bucket, config_file).await,
+    }
+}
+
 fn to_internal_error(err: impl Display) -> S3Error {
     S3Error::with_message(S3ErrorCode::InternalError, format!("{err}"))
 }
@@ -604,10 +626,7 @@ async fn replication_targets_without_config_targets(
 
     let mut targets = match metadata_sys::get_bucket_targets_config(bucket).await {
         Ok(targets) => targets,
-        Err(StorageError::ConfigNotFound) => {
-            BucketTargetSys::get().update_all_targets(bucket, None).await;
-            return Ok(None);
-        }
+        Err(StorageError::ConfigNotFound) => return Ok(None),
         Err(err) => return Err(ApiError::from(err).into()),
     };
 
@@ -632,12 +651,16 @@ fn remove_replication_targets_from_config_targets(targets: &mut BucketTargets, t
     original_len - targets.targets.len()
 }
 
-async fn write_replication_targets_after_config_delete(bucket: &str, targets: &BucketTargets, removed: usize) -> S3Result<()> {
+async fn write_replication_targets_after_config_delete(
+    bucket: &str,
+    targets: &BucketTargets,
+    removed: usize,
+    expected_incarnation_id: Option<uuid::Uuid>,
+) -> S3Result<()> {
     let json_targets = serde_json::to_vec(&targets).map_err(to_internal_error)?;
-    metadata_sys::update(bucket, BUCKET_TARGETS_FILE, json_targets)
+    update_bucket_config_for_incarnation(bucket, BUCKET_TARGETS_FILE, json_targets, expected_incarnation_id)
         .await
         .map_err(ApiError::from)?;
-    BucketTargetSys::get().update_all_targets(bucket, Some(targets)).await;
     info!(bucket = %bucket, removed, "removed replication remote targets referenced by deleted bucket replication config");
 
     Ok(())
@@ -647,10 +670,13 @@ async fn restore_replication_config_after_target_cleanup_failure(
     bucket: &str,
     config: &ReplicationConfiguration,
     cleanup_err: S3Error,
+    expected_incarnation_id: Option<uuid::Uuid>,
 ) -> S3Error {
     match serialize(config) {
         Ok(data) => {
-            if let Err(restore_err) = metadata_sys::update(bucket, BUCKET_REPLICATION_CONFIG, data).await {
+            if let Err(restore_err) =
+                update_bucket_config_for_incarnation(bucket, BUCKET_REPLICATION_CONFIG, data, expected_incarnation_id).await
+            {
                 error!(
                     bucket = %bucket,
                     error = ?restore_err,
@@ -1412,6 +1438,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<DeleteBucketEncryptionInput>,
     ) -> S3Result<S3Response<DeleteBucketEncryptionOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let DeleteBucketEncryptionInput { bucket, .. } = req.input;
 
@@ -1424,7 +1451,7 @@ impl DefaultBucketUsecase {
             .await
             .map_err(ApiError::from)?;
 
-        metadata_sys::delete(&bucket, BUCKET_SSECONFIG)
+        delete_bucket_config_for_incarnation(&bucket, BUCKET_SSECONFIG, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -1443,6 +1470,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<DeleteBucketCorsInput>,
     ) -> S3Result<S3Response<DeleteBucketCorsOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let DeleteBucketCorsInput { bucket, .. } = req.input;
 
@@ -1455,7 +1483,7 @@ impl DefaultBucketUsecase {
             .await
             .map_err(ApiError::from)?;
 
-        metadata_sys::delete(&bucket, BUCKET_CORS_CONFIG)
+        delete_bucket_config_for_incarnation(&bucket, BUCKET_CORS_CONFIG, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -1474,6 +1502,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<DeleteBucketLifecycleInput>,
     ) -> S3Result<S3Response<DeleteBucketLifecycleOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let DeleteBucketLifecycleInput { bucket, .. } = req.input;
 
@@ -1486,7 +1515,7 @@ impl DefaultBucketUsecase {
             .await
             .map_err(ApiError::from)?;
 
-        metadata_sys::delete(&bucket, BUCKET_LIFECYCLE_CONFIG)
+        delete_bucket_config_for_incarnation(&bucket, BUCKET_LIFECYCLE_CONFIG, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -1505,6 +1534,7 @@ impl DefaultBucketUsecase {
         req: S3Request<DeleteBucketPolicyInput>,
     ) -> S3Result<S3Response<DeleteBucketPolicyOutput>> {
         record_s3_op(S3Operation::DeleteBucketPolicy);
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let DeleteBucketPolicyInput { bucket, .. } = req.input;
 
@@ -1517,7 +1547,7 @@ impl DefaultBucketUsecase {
             .await
             .map_err(ApiError::from)?;
 
-        metadata_sys::delete(&bucket, BUCKET_POLICY_CONFIG)
+        delete_bucket_config_for_incarnation(&bucket, BUCKET_POLICY_CONFIG, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -1535,6 +1565,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<DeleteBucketReplicationInput>,
     ) -> S3Result<S3Response<DeleteBucketReplicationOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let DeleteBucketReplicationInput { bucket, .. } = req.input;
 
@@ -1558,14 +1589,21 @@ impl DefaultBucketUsecase {
             None
         };
 
-        metadata_sys::delete(&bucket, BUCKET_REPLICATION_CONFIG)
+        delete_bucket_config_for_incarnation(&bucket, BUCKET_REPLICATION_CONFIG, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
         if let Some((targets, removed)) = updated_targets
-            && let Err(err) = write_replication_targets_after_config_delete(&bucket, &targets, removed).await
+            && let Err(err) =
+                write_replication_targets_after_config_delete(&bucket, &targets, removed, expected_incarnation_id).await
         {
             if let Some(config) = replication_config.as_ref() {
-                return Err(restore_replication_config_after_target_cleanup_failure(&bucket, config, err).await);
+                return Err(restore_replication_config_after_target_cleanup_failure(
+                    &bucket,
+                    config,
+                    err,
+                    expected_incarnation_id,
+                )
+                .await);
             }
             return Err(err);
         }
@@ -1588,10 +1626,11 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<DeleteBucketTaggingInput>,
     ) -> S3Result<S3Response<DeleteBucketTaggingOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let DeleteBucketTaggingInput { bucket, .. } = req.input;
 
-        metadata_sys::delete(&bucket, BUCKET_TAGGING_CONFIG)
+        delete_bucket_config_for_incarnation(&bucket, BUCKET_TAGGING_CONFIG, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -1611,6 +1650,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<DeletePublicAccessBlockInput>,
     ) -> S3Result<S3Response<DeletePublicAccessBlockOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let DeletePublicAccessBlockInput { bucket, .. } = req.input;
 
@@ -1623,7 +1663,7 @@ impl DefaultBucketUsecase {
             .await
             .map_err(ApiError::from)?;
 
-        metadata_sys::delete(&bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG)
+        delete_bucket_config_for_incarnation(&bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -2039,6 +2079,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<PutBucketEncryptionInput>,
     ) -> S3Result<S3Response<PutBucketEncryptionOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let PutBucketEncryptionInput {
             bucket,
@@ -2077,7 +2118,7 @@ impl DefaultBucketUsecase {
             .map_err(ApiError::from)?;
 
         let data = serialize_config(&server_side_encryption_configuration)?;
-        metadata_sys::update(&bucket, BUCKET_SSECONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_SSECONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -2099,6 +2140,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<PutBucketLifecycleConfigurationInput>,
     ) -> S3Result<S3Response<PutBucketLifecycleConfigurationOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let PutBucketLifecycleConfigurationInput {
             bucket,
@@ -2138,7 +2180,7 @@ impl DefaultBucketUsecase {
 
         input_cfg.expiry_updated_at = Some(Timestamp::from(time::OffsetDateTime::now_utc()));
         let data = serialize_config(&input_cfg)?;
-        metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_LIFECYCLE_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -2194,6 +2236,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<PutBucketNotificationConfigurationInput>,
     ) -> S3Result<S3Response<PutBucketNotificationConfigurationOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_region = req.region.clone();
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
 
@@ -2215,7 +2258,7 @@ impl DefaultBucketUsecase {
             .map_err(ApiError::from)?;
 
         let data = serialize_config(&notification_configuration)?;
-        metadata_sys::update(&bucket, BUCKET_NOTIFICATION_CONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_NOTIFICATION_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -2269,6 +2312,7 @@ impl DefaultBucketUsecase {
         req: S3Request<PutBucketPolicyInput>,
     ) -> S3Result<S3Response<PutBucketPolicyOutput>> {
         record_s3_op(S3Operation::PutBucketPolicy);
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let PutBucketPolicyInput { bucket, policy, .. } = req.input;
 
@@ -2319,7 +2363,7 @@ impl DefaultBucketUsecase {
 
         let data = policy.as_bytes().to_vec();
 
-        metadata_sys::update(&bucket, BUCKET_POLICY_CONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_POLICY_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -2336,6 +2380,7 @@ impl DefaultBucketUsecase {
 
     #[instrument(level = "debug", skip(self))]
     pub async fn execute_put_bucket_cors(&self, req: S3Request<PutBucketCorsInput>) -> S3Result<S3Response<PutBucketCorsOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let PutBucketCorsInput {
             bucket,
@@ -2353,7 +2398,7 @@ impl DefaultBucketUsecase {
             .map_err(ApiError::from)?;
 
         let data = serialize_config(&cors_configuration)?;
-        metadata_sys::update(&bucket, BUCKET_CORS_CONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_CORS_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -2373,6 +2418,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<PutBucketReplicationInput>,
     ) -> S3Result<S3Response<PutBucketReplicationOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let PutBucketReplicationInput {
             bucket,
@@ -2395,7 +2441,7 @@ impl DefaultBucketUsecase {
         let targets_guard = lock_bucket_targets_metadata(&bucket).await;
         validate_bucket_replication_update(&bucket, &replication_configuration).await?;
         let data = serialize_config(&replication_configuration)?;
-        metadata_sys::update(&bucket, BUCKET_REPLICATION_CONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_REPLICATION_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
         drop(targets_guard);
@@ -2418,6 +2464,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<PutPublicAccessBlockInput>,
     ) -> S3Result<S3Response<PutPublicAccessBlockOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let PutPublicAccessBlockInput {
             bucket,
@@ -2435,7 +2482,7 @@ impl DefaultBucketUsecase {
             .map_err(ApiError::from)?;
 
         let data = serialize_config(&public_access_block_configuration)?;
-        metadata_sys::update(&bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -2449,6 +2496,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<PutBucketTaggingInput>,
     ) -> S3Result<S3Response<PutBucketTaggingOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let PutBucketTaggingInput { bucket, tagging, .. } = req.input;
 
@@ -2463,7 +2511,7 @@ impl DefaultBucketUsecase {
 
         let data = serialize_config(&tagging)?;
 
-        metadata_sys::update(&bucket, BUCKET_TAGGING_CONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_TAGGING_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -2484,6 +2532,7 @@ impl DefaultBucketUsecase {
         &self,
         req: S3Request<PutBucketVersioningInput>,
     ) -> S3Result<S3Response<PutBucketVersioningOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
         let PutBucketVersioningInput {
             bucket,
@@ -2495,7 +2544,7 @@ impl DefaultBucketUsecase {
 
         let data = serialize_config(&versioning_configuration)?;
 
-        metadata_sys::update(&bucket, BUCKET_VERSIONING_CONFIG, data)
+        update_bucket_config_for_incarnation(&bucket, BUCKET_VERSIONING_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -3297,7 +3346,7 @@ mod tests {
         let req = build_request(input, Method::PUT);
         let usecase = DefaultBucketUsecase::without_context();
 
-        let err = usecase.execute_create_bucket(req).await.unwrap_err();
+        let err = Box::pin(usecase.execute_create_bucket(req)).await.unwrap_err();
         assert_eq!(err.code(), &S3ErrorCode::InternalError);
     }
 

rustfs/src/app/delete_objects_stat_gating_test.rs

@@ -21,16 +21,15 @@
 //! - buckets created with Object Lock keep the held-lock stat and the #4297
 //!   delete protection (explicit-version deletes of retained objects are
 //!   rejected);
-//! - buckets without Object Lock take the gated (stat-skipping) path and must
-//!   behave exactly as before: unversioned batch deletes remove objects and
-//!   report per-key results, versioned batch deletes still create delete
-//!   markers and preserve the underlying version.
+//! - delete-marker creation can skip the held-lock stat, while destructive
+//!   deletes still inspect legacy or corrupt explicit Object Lock metadata even
+//!   when the bucket configuration is confirmed absent.
 
 use super::gating_test_env::shared_gating_ecstore;
 use super::storage_api::test::contract::bucket::{BucketOperations, MakeBucketOptions};
 use super::storage_api::test::contract::object::{ObjectIO as _, ObjectOperations as _};
 use super::storage_api::test::{StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader};
-use crate::storage::storage_api::{StorageObjectLockDeleteOptions, StorageObjectToDelete as ObjectToDelete};
+use crate::storage::storage_api::{StorageError, StorageObjectLockDeleteOptions, StorageObjectToDelete as ObjectToDelete};
 use serial_test::serial;
 use uuid::Uuid;
 
@@ -98,10 +97,7 @@ async fn object_lock_bucket_batch_delete_keeps_held_lock_protection() {
         )
         .await;
 
-    assert!(
-        errs[0].is_some(),
-        "explicit-version delete of a COMPLIANCE-retained object must be rejected on lock buckets"
-    );
+    assert!(matches!(errs[0], Some(StorageError::PrefixAccessDenied(_, _))));
 
     ecstore
         .get_object_info(
@@ -117,6 +113,454 @@ async fn object_lock_bucket_batch_delete_keeps_held_lock_protection() {
         .expect("retained version must survive the batch delete");
 }
 
+#[tokio::test]
+#[serial]
+async fn object_lock_batch_delete_preserves_explicit_null_version_protection() {
+    let ecstore = shared_gating_ecstore().await;
+    let bucket = format!("explicit-null-lock-{}", Uuid::new_v4());
+
+    ecstore
+        .make_bucket(
+            &bucket,
+            &MakeBucketOptions {
+                lock_enabled: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("create object-lock bucket");
+    let mut reader = PutObjReader::from_vec(b"retained null version".to_vec());
+    ecstore
+        .put_object(
+            &bucket,
+            "null.bin",
+            &mut reader,
+            &ObjectOptions {
+                version_suspended: true,
+                user_defined: compliance_retention_metadata(),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("put retained null version");
+
+    let (_deleted, errs) = ecstore
+        .delete_objects(
+            &bucket,
+            vec![ObjectToDelete {
+                object_name: "null.bin".to_string(),
+                version_id: Some(Uuid::nil()),
+                ..Default::default()
+            }],
+            ObjectOptions {
+                versioned: true,
+                object_lock_delete: Some(StorageObjectLockDeleteOptions {
+                    bypass_governance: false,
+                }),
+                ..Default::default()
+            },
+        )
+        .await;
+
+    assert!(matches!(errs[0], Some(StorageError::PrefixAccessDenied(_, _))));
+    ecstore
+        .get_object_info(
+            &bucket,
+            "null.bin",
+            &ObjectOptions {
+                version_id: Some(Uuid::nil().to_string()),
+                versioned: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("retained null version must survive the batch delete");
+}
+
+#[tokio::test]
+#[serial]
+async fn recursive_force_delete_is_blocked_for_object_lock_bucket() {
+    let ecstore = shared_gating_ecstore().await;
+    let bucket = format!("force-delete-lock-{}", Uuid::new_v4());
+
+    ecstore
+        .make_bucket(
+            &bucket,
+            &MakeBucketOptions {
+                lock_enabled: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("create object-lock bucket");
+
+    let mut reader = PutObjReader::from_vec(b"protected payload".to_vec());
+    ecstore
+        .put_object(
+            &bucket,
+            "protected/object.bin",
+            &mut reader,
+            &ObjectOptions {
+                versioned: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("put object under protected prefix");
+
+    let err = ecstore
+        .delete_object(
+            &bucket,
+            "protected",
+            ObjectOptions {
+                delete_prefix: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("recursive force-delete must be rejected for Object Lock buckets");
+
+    assert!(matches!(err, StorageError::InvalidArgument(_, _, _)));
+    ecstore
+        .get_object_info(&bucket, "protected/object.bin", &ObjectOptions::default())
+        .await
+        .expect("rejected recursive delete must preserve the protected object");
+}
+
+#[tokio::test]
+#[serial]
+async fn lifecycle_style_delete_all_versions_rechecks_each_retained_version() {
+    let ecstore = shared_gating_ecstore().await;
+    let bucket = format!("delete-all-lock-{}", Uuid::new_v4());
+
+    ecstore
+        .make_bucket(
+            &bucket,
+            &MakeBucketOptions {
+                lock_enabled: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("create object-lock bucket");
+
+    let mut reader = PutObjReader::from_vec(b"retained payload".to_vec());
+    let put_info = ecstore
+        .put_object(
+            &bucket,
+            "retained.bin",
+            &mut reader,
+            &ObjectOptions {
+                versioned: true,
+                user_defined: compliance_retention_metadata(),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("put retained object");
+    let version_id = put_info.version_id.expect("lock bucket writes must be versioned");
+
+    let err = ecstore
+        .delete_object(
+            &bucket,
+            "retained.bin",
+            ObjectOptions {
+                delete_prefix: true,
+                delete_prefix_object: true,
+                versioned: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("delete-all must recheck every retained version under the object lock");
+
+    assert!(matches!(err, StorageError::PrefixAccessDenied(_, _)));
+    ecstore
+        .get_object_info(
+            &bucket,
+            "retained.bin",
+            &ObjectOptions {
+                version_id: Some(version_id.to_string()),
+                versioned: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("retained version must survive rejected delete-all");
+}
+
+#[tokio::test]
+#[serial]
+async fn malformed_persisted_retention_metadata_blocks_version_delete() {
+    let ecstore = shared_gating_ecstore().await;
+    let bucket = format!("malformed-retention-{}", Uuid::new_v4());
+
+    ecstore
+        .make_bucket(
+            &bucket,
+            &MakeBucketOptions {
+                lock_enabled: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("create object-lock bucket");
+
+    let mut malformed = std::collections::HashMap::new();
+    malformed.insert("x-amz-object-lock-mode".to_string(), "COMPLIANCE".to_string());
+    let mut reader = PutObjReader::from_vec(b"must survive".to_vec());
+    let put_info = ecstore
+        .put_object(
+            &bucket,
+            "malformed.bin",
+            &mut reader,
+            &ObjectOptions {
+                versioned: true,
+                user_defined: malformed,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("storage fixture should persist the malformed boundary value");
+    let version_id = put_info.version_id.expect("lock bucket writes must be versioned");
+
+    ecstore
+        .delete_object(
+            &bucket,
+            "malformed.bin",
+            ObjectOptions {
+                version_id: Some(version_id.to_string()),
+                versioned: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("incomplete persisted retention metadata must fail closed");
+
+    ecstore
+        .get_object_info(
+            &bucket,
+            "malformed.bin",
+            &ObjectOptions {
+                version_id: Some(version_id.to_string()),
+                versioned: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("malformed retained object must survive the rejected delete");
+}
+
+#[tokio::test]
+#[serial]
+async fn recursive_force_delete_remains_allowed_for_plain_bucket() {
+    let ecstore = shared_gating_ecstore().await;
+    let bucket = format!("force-delete-plain-{}", Uuid::new_v4());
+
+    ecstore
+        .make_bucket(&bucket, &MakeBucketOptions::default())
+        .await
+        .expect("create plain bucket");
+
+    for object in ["prefix/a.bin", "prefix/b.bin"] {
+        let mut reader = PutObjReader::from_vec(b"plain payload".to_vec());
+        ecstore
+            .put_object(&bucket, object, &mut reader, &ObjectOptions::default())
+            .await
+            .expect("put object under plain prefix");
+    }
+
+    ecstore
+        .delete_object(
+            &bucket,
+            "prefix",
+            ObjectOptions {
+                delete_prefix: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("recursive force-delete should remain allowed for a plain bucket");
+
+    for object in ["prefix/a.bin", "prefix/b.bin"] {
+        ecstore
+            .get_object_info(&bucket, object, &ObjectOptions::default())
+            .await
+            .expect_err("recursive force-delete should remove every matching object");
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn plain_bucket_explicit_retention_blocks_every_destructive_delete_shape() {
+    let ecstore = shared_gating_ecstore().await;
+    let bucket = format!("plain-explicit-lock-{}", Uuid::new_v4());
+    ecstore
+        .make_bucket(&bucket, &MakeBucketOptions::default())
+        .await
+        .expect("create plain bucket fixture");
+
+    for object in ["batch.bin", "delete-all.bin", "prefix/retained.bin"] {
+        let mut reader = PutObjReader::from_vec(b"legacy retained payload".to_vec());
+        ecstore
+            .put_object(
+                &bucket,
+                object,
+                &mut reader,
+                &ObjectOptions {
+                    user_defined: compliance_retention_metadata(),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("storage fixture should persist explicit retention metadata");
+    }
+
+    let (_deleted, errors) = ecstore
+        .delete_objects(
+            &bucket,
+            vec![ObjectToDelete {
+                object_name: "batch.bin".to_string(),
+                ..Default::default()
+            }],
+            ObjectOptions::default(),
+        )
+        .await;
+    assert!(matches!(errors[0], Some(StorageError::PrefixAccessDenied(_, _))));
+
+    let delete_all_error = ecstore
+        .delete_object(
+            &bucket,
+            "delete-all.bin",
+            ObjectOptions {
+                delete_prefix: true,
+                delete_prefix_object: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("delete-all must inspect explicit retention in a plain bucket");
+    assert!(matches!(delete_all_error, StorageError::PrefixAccessDenied(_, _)));
+
+    let prefix_error = ecstore
+        .delete_object(
+            &bucket,
+            "prefix",
+            ObjectOptions {
+                delete_prefix: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("recursive force-delete must inspect every matching object");
+    assert!(matches!(prefix_error, StorageError::PrefixAccessDenied(_, _)));
+
+    for object in ["batch.bin", "delete-all.bin", "prefix/retained.bin"] {
+        ecstore
+            .get_object_info(&bucket, object, &ObjectOptions::default())
+            .await
+            .expect("retained object must survive every rejected delete shape");
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn batch_delete_can_purge_an_explicit_delete_marker_version() {
+    let ecstore = shared_gating_ecstore().await;
+    let bucket = format!("batch-delete-marker-{}", Uuid::new_v4());
+    ecstore
+        .make_bucket(
+            &bucket,
+            &MakeBucketOptions {
+                versioning_enabled: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("create versioned bucket");
+
+    let deleted = ecstore
+        .delete_object(
+            &bucket,
+            "marker.bin",
+            ObjectOptions {
+                versioned: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("create a delete marker");
+    let marker_version = deleted.version_id.expect("delete marker should have a version id");
+
+    let (_deleted, errors) = ecstore
+        .delete_objects(
+            &bucket,
+            vec![ObjectToDelete {
+                object_name: "marker.bin".to_string(),
+                version_id: Some(marker_version),
+                ..Default::default()
+            }],
+            ObjectOptions {
+                versioned: true,
+                ..Default::default()
+            },
+        )
+        .await;
+    assert!(errors[0].is_none(), "explicit delete-marker purge must succeed: {:?}", errors[0]);
+}
+
+#[tokio::test]
+#[serial]
+async fn delete_all_exact_object_preserves_a_retained_child_key() {
+    let ecstore = shared_gating_ecstore().await;
+    let bucket = format!("delete-all-child-{}", Uuid::new_v4());
+    ecstore
+        .make_bucket(&bucket, &MakeBucketOptions::default())
+        .await
+        .expect("create plain bucket fixture");
+
+    let mut parent_reader = PutObjReader::from_vec(b"parent".to_vec());
+    ecstore
+        .put_object(&bucket, "foo", &mut parent_reader, &ObjectOptions::default())
+        .await
+        .expect("put exact parent object");
+    let mut child_reader = PutObjReader::from_vec(b"retained child".to_vec());
+    ecstore
+        .put_object(
+            &bucket,
+            "foo/bar",
+            &mut child_reader,
+            &ObjectOptions {
+                user_defined: compliance_retention_metadata(),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("put retained child object");
+
+    ecstore
+        .delete_object(
+            &bucket,
+            "foo",
+            ObjectOptions {
+                delete_prefix: true,
+                delete_prefix_object: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("delete-all should remove only the exact object");
+
+    ecstore
+        .get_object_info(&bucket, "foo", &ObjectOptions::default())
+        .await
+        .expect_err("exact parent should be deleted");
+    ecstore
+        .get_object_info(&bucket, "foo/bar", &ObjectOptions::default())
+        .await
+        .expect("retained child key must survive exact delete-all");
+}
+
 #[tokio::test]
 #[serial]
 async fn non_lock_versioned_bucket_batch_delete_still_creates_delete_marker() {

rustfs/src/app/multipart_usecase.rs

@@ -15,7 +15,10 @@
 //! Multipart application use-case contracts.
 
 use super::storage_api::multipart_usecase::ECStore;
-use super::storage_api::multipart_usecase::access::{has_bypass_governance_header, replication_request_authorized};
+use super::storage_api::multipart_usecase::access::{
+    apply_bucket_generation_guard, apply_copy_source_bucket_generation_guard, has_bypass_governance_header,
+    replication_request_authorized,
+};
 use super::storage_api::multipart_usecase::bucket::quota::checker::QuotaChecker;
 use super::storage_api::multipart_usecase::bucket::{
     lifecycle::{bucket_lifecycle_audit::LcEventSrc, bucket_lifecycle_ops::enqueue_transition_immediate},
@@ -52,8 +55,8 @@ use super::storage_api::multipart_usecase::sse::{
     DecryptionRequest, EncryptionKeyKind, EncryptionRequest, PrepareEncryptionRequest, SseKmsPrincipal,
     apply_bucket_default_lock_retention, authorize_sse_kms_object_read, build_ssec_read_headers, encryption_material_to_metadata,
     extract_server_side_encryption_from_headers, extract_ssec_params_from_headers, extract_ssekms_context_from_headers,
-    get_buffer_size_opt_in, map_get_object_reader_error, mark_encrypted_multipart_metadata, sse_decryption,
-    sse_prepare_encryption,
+    get_buffer_size_opt_in, load_bucket_object_lock_config_state, map_get_object_reader_error, mark_encrypted_multipart_metadata,
+    sse_decryption, sse_prepare_encryption,
 };
 use super::storage_api::multipart_usecase::{StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader};
 use crate::app::object_data_cache::{
@@ -61,7 +64,8 @@ use crate::app::object_data_cache::{
     invalidate_object_data_cache_after_delete_success, invalidate_object_data_cache_before_mutation,
 };
 use crate::app::object_usecase::{
-    build_put_like_object_lock_metadata, map_quota_check_outcome, validate_existing_object_lock_for_write,
+    acquire_copy_bucket_lifecycle_locks, build_put_like_object_lock_metadata, map_quota_check_outcome,
+    validate_existing_object_lock_for_write,
 };
 use crate::app::runtime_sources::{
     AppContext, current_app_context, current_object_data_cache_for_context, current_object_store_handle_for_context,
@@ -343,6 +347,8 @@ impl DefaultMultipartUsecase {
         req: S3Request<AbortMultipartUploadInput>,
     ) -> S3Result<S3Response<AbortMultipartUploadOutput>> {
         record_s3_op(S3Operation::AbortMultipartUpload);
+        let mut opts = ObjectOptions::default();
+        apply_bucket_generation_guard(&req, &req.input.bucket, &mut opts)?;
         let AbortMultipartUploadInput {
             bucket, key, upload_id, ..
         } = req.input;
@@ -351,8 +357,6 @@ impl DefaultMultipartUsecase {
             return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
         };
 
-        let opts = &ObjectOptions::default();
-
         // Special handling for abort_multipart_upload: Per AWS S3 API specification, this operation
         // should return NoSuchUpload (404) when the upload_id doesn't exist, even if the format
         // appears invalid. This differs from other multipart operations (upload_part, list_parts,
@@ -360,7 +364,7 @@ impl DefaultMultipartUsecase {
         // The lenient validation matches AWS S3 behavior where format validation is relaxed for
         // abort operations to avoid leaking information about upload_id format requirements.
         match store
-            .abort_multipart_upload(bucket.as_str(), key.as_str(), upload_id.as_str(), opts)
+            .abort_multipart_upload(bucket.as_str(), key.as_str(), upload_id.as_str(), &opts)
             .await
         {
             Ok(_) => {
@@ -388,7 +392,7 @@ impl DefaultMultipartUsecase {
             S3Operation::CompleteMultipartUpload,
         );
         let replication_authorized = replication_request_authorized(&req);
-        let input = req.input;
+        let input = req.input.clone();
         let CompleteMultipartUploadInput {
             multipart_upload,
             bucket,
@@ -454,6 +458,7 @@ impl DefaultMultipartUsecase {
 
         let mut opts = get_complete_multipart_upload_opts_with_replication_authorization(&req.headers, replication_authorized)
             .map_err(ApiError::from)?;
+        apply_bucket_generation_guard(&req, &bucket, &mut opts)?;
         let versioned = BucketVersioningSys::prefix_enabled(&bucket, &key).await;
         let version_suspended = BucketVersioningSys::prefix_suspended(&bucket, &key).await;
         opts.versioned = versioned;
@@ -495,7 +500,7 @@ impl DefaultMultipartUsecase {
         };
 
         let multipart_info = store
-            .get_multipart_info(&bucket, &key, &upload_id, &ObjectOptions::default())
+            .get_multipart_info(&bucket, &key, &upload_id, &opts)
             .await
             .map_err(ApiError::from)?;
         EncryptionRequest {
@@ -715,17 +720,22 @@ impl DefaultMultipartUsecase {
         let mut metadata = create_multipart_upload_metadata(input_metadata, &req.headers, tagging, storage_class.as_ref());
 
         let has_explicit_object_lock_retention = object_lock_mode.is_some() || object_lock_retain_until_date.is_some();
+        let object_lock_config_state = load_bucket_object_lock_config_state(&bucket).await?;
         if let Some(object_lock_metadata) = build_put_like_object_lock_metadata(
             &bucket,
+            &object_lock_config_state,
             object_lock_legal_hold_status,
             object_lock_mode,
             object_lock_retain_until_date,
-        )
-        .await?
-        {
+        )? {
             metadata.extend(object_lock_metadata);
         }
-        apply_bucket_default_lock_retention(&bucket, &mut metadata, has_explicit_object_lock_retention).await?;
+        apply_bucket_default_lock_retention(
+            &bucket,
+            &object_lock_config_state,
+            &mut metadata,
+            has_explicit_object_lock_retention,
+        )?;
         let (header_sse_customer_algorithm, header_sse_customer_key, header_sse_customer_key_md5) =
             extract_ssec_params_from_headers(&req.headers)?;
         let sse_customer_algorithm = sse_customer_algorithm.or(header_sse_customer_algorithm);
@@ -771,6 +781,7 @@ impl DefaultMultipartUsecase {
             put_opts_with_replication_authorization(&bucket, &key, version_id, &req.headers, metadata, replication_authorized)
                 .await
                 .map_err(ApiError::from)?;
+        apply_bucket_generation_guard(&req, &bucket, &mut opts)?;
 
         let dsc =
             must_replicate_object(&bucket, &key, &mt2, "".to_string(), opts.delete_marker_replication_status(), opts.clone())
@@ -835,6 +846,8 @@ impl DefaultMultipartUsecase {
 
     #[instrument(level = "debug", skip(self, req))]
     pub async fn execute_upload_part(&self, req: S3Request<UploadPartInput>) -> S3Result<S3Response<UploadPartOutput>> {
+        let mut opts = ObjectOptions::default();
+        apply_bucket_generation_guard(&req, &req.input.bucket, &mut opts)?;
         let input = req.input;
         let UploadPartInput {
             body,
@@ -881,7 +894,6 @@ impl DefaultMultipartUsecase {
             return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
         };
 
-        let mut opts = ObjectOptions::default();
         let fi = store
             .get_multipart_info(&bucket, &key, &upload_id, &opts)
             .await
@@ -1095,6 +1107,8 @@ impl DefaultMultipartUsecase {
         &self,
         req: S3Request<ListMultipartUploadsInput>,
     ) -> S3Result<S3Response<ListMultipartUploadsOutput>> {
+        let mut opts = ObjectOptions::default();
+        apply_bucket_generation_guard(&req, &req.input.bucket, &mut opts)?;
         let ListMultipartUploadsInput {
             bucket,
             prefix,
@@ -1110,13 +1124,29 @@ impl DefaultMultipartUsecase {
             key_marker,
             max_uploads,
         } = parse_list_multipart_uploads_params(prefix, key_marker, max_uploads)?;
-
         let Some(store) = self.object_store() else {
             return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
         };
 
+        // `apply_bucket_generation_guard` tolerates a missing guard (only the S3
+        // access layer installs one), so resolve the current generation rather
+        // than failing the request. Listing is filtered by this value, so a
+        // stale one simply hides foreign-incarnation uploads, as intended.
+        let expected_incarnation_id = match opts.expected_bucket_incarnation_id {
+            Some(incarnation_id) => incarnation_id,
+            None => store.bucket_incarnation_id_from_disk(&bucket).await.map_err(ApiError::from)?,
+        };
+
         let result = store
-            .list_multipart_uploads(&bucket, &prefix, delimiter, key_marker, upload_id_marker, max_uploads)
+            .list_multipart_uploads_for_bucket_incarnation(
+                &bucket,
+                &prefix,
+                key_marker,
+                upload_id_marker,
+                delimiter,
+                max_uploads,
+                expected_incarnation_id,
+            )
             .await
             .map_err(ApiError::from)?;
 
@@ -1124,6 +1154,8 @@ impl DefaultMultipartUsecase {
     }
 
     pub async fn execute_list_parts(&self, req: S3Request<ListPartsInput>) -> S3Result<S3Response<ListPartsOutput>> {
+        let mut opts = ObjectOptions::default();
+        apply_bucket_generation_guard(&req, &req.input.bucket, &mut opts)?;
         let ListPartsInput {
             bucket,
             key,
@@ -1140,14 +1172,7 @@ impl DefaultMultipartUsecase {
         };
 
         let res = store
-            .list_object_parts(
-                &bucket,
-                &key,
-                &upload_id,
-                params.part_number_marker,
-                params.max_parts,
-                &ObjectOptions::default(),
-            )
+            .list_object_parts(&bucket, &key, &upload_id, params.part_number_marker, params.max_parts, &opts)
             .await
             .map_err(ApiError::from)?;
 
@@ -1161,6 +1186,17 @@ impl DefaultMultipartUsecase {
     ) -> S3Result<S3Response<UploadPartCopyOutput>> {
         // Captured before `req.input` is destructured below.
         let copy_principal = SseKmsPrincipal::from_request(&req);
+        let source_bucket = match &req.input.copy_source {
+            CopySource::AccessPoint { .. } => return Err(s3_error!(NotImplemented)),
+            CopySource::Outpost { .. } => return Err(s3_error!(NotImplemented)),
+            CopySource::Bucket { bucket, .. } => bucket.to_string(),
+        };
+        let mut source_generation_opts = ObjectOptions::default();
+        apply_copy_source_bucket_generation_guard(&req, &source_bucket, &mut source_generation_opts)?;
+        let expected_source_incarnation_id = source_generation_opts.expected_bucket_incarnation_id;
+        let mut destination_generation_opts = ObjectOptions::default();
+        apply_bucket_generation_guard(&req, &req.input.bucket, &mut destination_generation_opts)?;
+        let expected_destination_incarnation_id = destination_generation_opts.expected_bucket_incarnation_id;
         let UploadPartCopyInput {
             bucket,
             key,
@@ -1203,8 +1239,47 @@ impl DefaultMultipartUsecase {
             return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
         };
 
+        let (source_bucket_lifecycle_guard, destination_bucket_lifecycle_guard_storage) =
+            acquire_copy_bucket_lifecycle_locks(store.as_ref(), &src_bucket, &bucket).await?;
+        let current_source_incarnation_id = store
+            .bucket_incarnation_id_from_disk(&src_bucket)
+            .await
+            .map_err(ApiError::from)?;
+        if expected_source_incarnation_id.is_some_and(|expected| expected != current_source_incarnation_id) {
+            return Err(ApiError::from(StorageError::BucketNotFound(src_bucket.clone())).into());
+        }
+        let current_destination_incarnation_id = if src_bucket == bucket {
+            current_source_incarnation_id
+        } else {
+            store.bucket_incarnation_id_from_disk(&bucket).await.map_err(ApiError::from)?
+        };
+        if expected_destination_incarnation_id.is_some_and(|expected| expected != current_destination_incarnation_id) {
+            return Err(ApiError::from(StorageError::BucketNotFound(bucket.clone())).into());
+        }
+        let destination_bucket_lifecycle_guard = destination_bucket_lifecycle_guard_storage
+            .as_ref()
+            .unwrap_or(&source_bucket_lifecycle_guard);
+        if source_bucket_lifecycle_guard.is_lock_lost() || destination_bucket_lifecycle_guard.is_lock_lost() {
+            return Err(ApiError::from(StorageError::NamespaceLockQuorumUnavailable {
+                mode: "copy_bucket_generation",
+                bucket: bucket.clone(),
+                object: key.clone(),
+                required: 1,
+                achieved: 0,
+            })
+            .into());
+        }
+        let mut dst_opts = ObjectOptions {
+            expected_bucket_incarnation_id: Some(current_destination_incarnation_id),
+            ..Default::default()
+        };
+        if src_bucket != bucket {
+            dst_opts.add_bucket_lifecycle_lock_guard(&source_bucket_lifecycle_guard);
+        }
+        dst_opts.add_bucket_lifecycle_lock_guard(destination_bucket_lifecycle_guard);
+
         let mp_info = store
-            .get_multipart_info(&bucket, &key, &upload_id, &ObjectOptions::default())
+            .get_multipart_info(&bucket, &key, &upload_id, &dst_opts)
             .await
             .map_err(ApiError::from)?;
         EncryptionRequest {
@@ -1233,8 +1308,19 @@ impl DefaultMultipartUsecase {
             version_id: src_opts.version_id.clone(),
             versioned: src_opts.versioned,
             version_suspended: src_opts.version_suspended,
+            expected_bucket_incarnation_id: Some(current_source_incarnation_id),
             ..Default::default()
         };
+        if source_bucket_lifecycle_guard.is_lock_lost() {
+            return Err(ApiError::from(StorageError::NamespaceLockQuorumUnavailable {
+                mode: "copy_source_bucket_generation",
+                bucket: src_bucket.clone(),
+                object: src_key.clone(),
+                required: 1,
+                achieved: 0,
+            })
+            .into());
+        }
 
         let src_reader = store
             .get_object_reader(&src_bucket, &src_key, rs.clone(), h, &get_opts)
@@ -1400,10 +1486,7 @@ impl DefaultMultipartUsecase {
 
         let mut reader = PutObjReader::new(reader);
 
-        let dst_opts = ObjectOptions {
-            user_defined: dst_user_defined,
-            ..Default::default()
-        };
+        dst_opts.user_defined = dst_user_defined;
 
         let part_info = store
             .put_object_part(&bucket, &key, &upload_id, part_id, &mut reader, &dst_opts)

rustfs/src/app/put_prelookup_gating_test.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 //! Regression coverage for rustfs/backlog#1009: the PUT path skips its
-//! pre-PUT `get_object_info` only when `put_prelookup_worm_gate` proves the
+//! pre-PUT `get_object_info` only when `object_lock_checks_required` proves the
 //! existing-object WORM validation is a no-op. These tests pin the gate's
 //! truth table against a real 4-disk `ECStore` with the bucket metadata sys
 //! initialized, mirroring the HP-8 delete-gating fixture:
@@ -24,7 +24,7 @@
 //!   so a degraded metadata subsystem can never silently drop the WORM check.
 
 use super::gating_test_env::shared_gating_ecstore;
-use super::object_usecase::put_prelookup_worm_gate;
+use super::object_usecase::object_lock_checks_required;
 use super::storage_api::test::contract::bucket::{BucketOperations, MakeBucketOptions};
 use serial_test::serial;
 use uuid::Uuid;
@@ -47,7 +47,7 @@ async fn worm_gate_keeps_prelookup_for_object_lock_bucket() {
         .expect("create object-lock bucket");
 
     assert!(
-        put_prelookup_worm_gate(&bucket).await,
+        object_lock_checks_required(&bucket).await,
         "an object-lock bucket must keep the pre-PUT lookup"
     );
 }
@@ -64,7 +64,7 @@ async fn worm_gate_allows_skip_for_plain_bucket() {
         .expect("create plain bucket");
 
     assert!(
-        !put_prelookup_worm_gate(&bucket).await,
+        !object_lock_checks_required(&bucket).await,
         "a bucket without object locking must take the prelookup-skip path"
     );
 }
@@ -76,7 +76,7 @@ async fn worm_gate_fails_closed_when_bucket_metadata_is_unavailable() {
     let missing_bucket = format!("put-gate-missing-{}", Uuid::new_v4());
 
     assert!(
-        put_prelookup_worm_gate(&missing_bucket).await,
+        object_lock_checks_required(&missing_bucket).await,
         "a bucket-metadata lookup failure must fail closed and keep the pre-PUT lookup"
     );
 }

rustfs/src/app/runtime_sources.rs

@@ -16,6 +16,8 @@ use crate::app::object_data_cache::ObjectDataCacheAdapter;
 #[cfg(test)]
 use crate::app::storage_api::runtime_sources::TierConfigMgr;
 use crate::runtime_sources as root_runtime_sources;
+#[cfg(test)]
+pub(crate) use crate::runtime_sources::ServerContextSlot;
 pub(crate) use crate::runtime_sources::{
     AppContext, current_encryption_service, current_endpoints_handle, current_notification_system,
     current_object_data_cache_handle_for_context, current_object_store_handle_for_context,

rustfs/src/app/storage_api.rs

@@ -216,8 +216,9 @@ pub(crate) mod runtime_sources {
 
 pub(crate) mod access {
     pub(crate) use crate::storage::storage_api::access_consumer::{
-        PostObjectRequestMarker, ReqInfo, authorize_request, has_bypass_governance_header, recursive_force_delete_is_authorized,
-        replication_request_authorized, req_info_mut, req_info_ref,
+        PostObjectRequestMarker, ReqInfo, apply_bucket_generation_guard, apply_copy_source_bucket_generation_guard,
+        authorize_request, bucket_config_mutation_incarnation, has_bypass_governance_header, load_bucket_generation_from_store,
+        recursive_force_delete_is_authorized, replication_request_authorized, req_info_mut, req_info_ref,
     };
 }
 
@@ -267,10 +268,6 @@ pub(crate) mod bucket {
         )
     }
 
-    pub(crate) mod bucket_target_sys {
-        pub(crate) type BucketTargetSys = crate::storage::storage_api::ecstore_bucket::bucket_target_sys::BucketTargetSys;
-    }
-
     pub(crate) mod lifecycle {
         pub(crate) mod bucket_lifecycle_audit {
             pub(crate) type LcEventSrc =
@@ -409,6 +406,7 @@ pub(crate) mod bucket {
         use super::target::BucketTargets;
 
         pub(crate) type BucketMetadataSys = crate::storage::storage_api::ecstore_bucket::metadata_sys::BucketMetadataSys;
+        pub(crate) type ObjectLockConfigState = crate::storage::storage_api::ecstore_bucket::metadata_sys::ObjectLockConfigState;
 
         #[cfg(test)]
         pub(crate) async fn init_bucket_metadata_sys(api: Arc<crate::storage::storage_api::ECStore>, buckets: Vec<String>) {
@@ -422,6 +420,19 @@ pub(crate) mod bucket {
             crate::storage::storage_api::delete_bucket_metadata_config(bucket, config_file).await
         }
 
+        pub(crate) async fn delete_if_incarnation(
+            bucket: &str,
+            config_file: &str,
+            expected_incarnation_id: uuid::Uuid,
+        ) -> Result<OffsetDateTime, crate::storage::storage_api::StorageError> {
+            crate::storage::storage_api::delete_bucket_metadata_config_if_incarnation(
+                bucket,
+                config_file,
+                Some(expected_incarnation_id),
+            )
+            .await
+        }
+
         pub(crate) async fn get_bucket_policy(
             bucket: &str,
         ) -> Result<(BucketPolicy, OffsetDateTime), crate::storage::storage_api::StorageError> {
@@ -499,6 +510,21 @@ pub(crate) mod bucket {
         ) -> Result<OffsetDateTime, crate::storage::storage_api::StorageError> {
             crate::storage::storage_api::update_bucket_metadata_config(bucket, config_file, data).await
         }
+
+        pub(crate) async fn update_if_incarnation(
+            bucket: &str,
+            config_file: &str,
+            data: Vec<u8>,
+            expected_incarnation_id: uuid::Uuid,
+        ) -> Result<OffsetDateTime, crate::storage::storage_api::StorageError> {
+            crate::storage::storage_api::update_bucket_metadata_config_if_incarnation(
+                bucket,
+                config_file,
+                data,
+                Some(expected_incarnation_id),
+            )
+            .await
+        }
     }
 
     pub(crate) mod object_lock {
@@ -517,14 +543,11 @@ pub(crate) mod bucket {
         }
 
         pub(crate) mod objectlock_sys {
-            pub(crate) type ObjectLockBlockReason =
-                crate::storage::storage_api::ecstore_bucket::object_lock::objectlock_sys::ObjectLockBlockReason;
-
             pub(crate) async fn check_object_lock_for_deletion(
                 bucket: &str,
-                obj_info: &crate::storage::storage_api::StorageObjectInfo,
+                obj_info: &crate::storage::storage_api::ObjectInfo,
                 bypass_governance: bool,
-            ) -> Option<ObjectLockBlockReason> {
+            ) -> Option<crate::storage::storage_api::ObjectLockBlockReason> {
                 crate::storage::storage_api::ecstore_bucket::object_lock::objectlock_sys::check_object_lock_for_deletion(
                     bucket,
                     obj_info,
@@ -899,8 +922,7 @@ pub(crate) mod ecfs {
 
 pub(crate) mod error {
     pub(crate) use crate::storage::storage_api::{
-        DiskError, StorageError, is_all_buckets_not_found, is_err_bucket_not_found, is_err_object_not_found,
-        is_err_version_not_found,
+        StorageError, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found,
     };
 
     pub(crate) type Error = StorageError;
@@ -938,7 +960,7 @@ pub(crate) mod options {
         extract_metadata_from_mime, extract_metadata_from_mime_with_object_name, filter_object_metadata,
         get_complete_multipart_upload_opts_with_replication_authorization, get_content_sha256_with_query, get_opts,
         namespace_reserved_user_metadata, normalize_content_encoding_for_storage, parse_copy_source_range,
-        put_opts_with_replication_authorization, validate_archive_content_encoding,
+        preserve_unclassified_user_metadata, put_opts_with_replication_authorization, validate_archive_content_encoding,
     };
 }
 
@@ -949,8 +971,9 @@ pub(crate) mod request_context {
 pub(crate) mod sse {
     pub(crate) use crate::storage::storage_api::sse_consumer::{
         DecryptionRequest, EncryptionRequest, PrepareEncryptionRequest, SseKmsPrincipal, apply_bucket_default_lock_retention,
-        authorize_sse_kms_object_read, extract_server_side_encryption_from_headers, get_buffer_size_opt_in, sse_decryption,
-        sse_encryption, sse_prepare_encryption,
+        authorize_sse_kms_object_read, extract_server_side_encryption_from_headers, get_buffer_size_opt_in,
+        load_bucket_object_lock_config_state, sse_decryption, sse_encryption, sse_prepare_encryption,
+        validate_bucket_object_lock_enabled_state,
     };
     pub(crate) use crate::storage::storage_api::sse_consumer::{
         EncryptionKeyKind, SSEType, build_ssec_read_headers, encryption_material_to_metadata, extract_ssec_params_from_headers,
@@ -1030,6 +1053,8 @@ pub(crate) mod bucket_usecase {
 }
 
 pub(crate) mod object_usecase {
+    pub(crate) use super::storage_contracts::BUCKET_LIFECYCLE_LOCK_OBJECT;
+
     pub(crate) mod object_cache {
         #[cfg(test)]
         pub(crate) use crate::storage::storage_api::ecstore_object::GetObjectBodySource;
@@ -1067,9 +1092,8 @@ pub(crate) mod object_usecase {
         ECStore, GetObjectReader, OldCurrentSize, RFC1123, StorageDeletedObject, StorageObjectInfo,
         StorageObjectLockDeleteOptions, StorageObjectOptions, StorageObjectToDelete, StoragePutObjReader, check_preconditions,
         parse_object_lock_legal_hold, parse_object_lock_retention, parse_part_number_i32_to_usize,
-        remove_object_lock_metadata_for_copy, strip_managed_encryption_metadata, validate_bucket_exists,
-        validate_bucket_object_lock_enabled, validate_object_key, validate_sse_headers_for_read, validate_sse_headers_for_write,
-        validate_ssec_for_read, wrap_response_with_cors,
+        remove_object_lock_metadata_for_copy, strip_managed_encryption_metadata, validate_bucket_exists, validate_object_key,
+        validate_sse_headers_for_read, validate_sse_headers_for_write, validate_ssec_for_read, wrap_response_with_cors,
     };
 }
 
@@ -1123,7 +1147,9 @@ pub(crate) mod test {
     pub(crate) use super::EndpointServerPools;
     pub(crate) mod contract {
         pub(crate) mod bucket {
-            pub(crate) use super::super::super::storage_contracts::{BucketOperations, BucketOptions, MakeBucketOptions};
+            pub(crate) use super::super::super::storage_contracts::{
+                BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions,
+            };
         }
 
         pub(crate) mod heal {

rustfs/src/storage/ecfs.rs

@@ -15,15 +15,15 @@
 use super::{
     BUCKET_ACCELERATE_CONFIG, BUCKET_LOGGING_CONFIG, BUCKET_REQUEST_PAYMENT_CONFIG, BUCKET_VERSIONING_CONFIG,
     BUCKET_WEBSITE_CONFIG, BucketVersioningSys, OBJECT_LOCK_CONFIG, StorageError, check_retention_for_modification, decode_tags,
-    decode_tags_to_map, delete_bucket_metadata_config, encode_tags, get_bucket_accelerate_config, get_bucket_logging_config,
-    get_bucket_object_lock_config, get_bucket_replication_config, get_bucket_request_payment_config, get_bucket_website_config,
-    is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found, record_replication_proxy, serialize,
-    update_bucket_metadata_config,
+    decode_tags_to_map, delete_bucket_metadata_config_if_incarnation, encode_tags, get_bucket_accelerate_config,
+    get_bucket_logging_config, get_bucket_object_lock_config, get_bucket_replication_config, get_bucket_request_payment_config,
+    get_bucket_website_config, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found,
+    record_replication_proxy, serialize, update_bucket_metadata_config_if_incarnation,
 };
 use super::{StorageReplicationConfigExt as _, StorageVersioningConfigExt as _};
 use crate::admin::handlers::site_replication::site_replication_bucket_meta_hook;
 use crate::error::ApiError;
-use crate::storage::access::has_bypass_governance_header;
+use crate::storage::access::{apply_bucket_generation_guard, bucket_config_mutation_incarnation, has_bypass_governance_header};
 use crate::storage::helper::OperationHelper;
 use crate::storage::options::get_opts;
 use crate::storage::s3_api::{self, acl};
@@ -135,20 +135,17 @@ impl FS {
         let tags = match store.get_object_tags(bucket, object, &opts).await {
             Ok(t) => t,
             Err(e) => {
-                if is_err_object_not_found(&e) || is_err_version_not_found(&e) {
+                if is_err_object_not_found(&e) || is_err_version_not_found(&e) || is_err_bucket_not_found(&e) {
                     debug!(
                         target: "rustfs::storage::ecfs",
                         bucket = %bucket,
                         object = %object,
                         version_id = ?version_id,
                         error = %e,
-                        "object or version not found when fetching tags for policy; treating as no tags"
+                        "object, version, or bucket not found when fetching tags for policy; treating as no tags"
                     );
                     return Ok(std::collections::HashMap::new());
                 }
-                if is_err_bucket_not_found(&e) {
-                    return Err(s3_error!(NoSuchBucket, "The specified bucket does not exist"));
-                }
                 warn!(
                     target: "rustfs::storage::ecfs",
                     bucket = %bucket,
@@ -194,6 +191,12 @@ fn invalid_object_lock_configuration(message: impl Into<String>) -> S3Error {
     S3Error::with_message(S3ErrorCode::MalformedXML, message.into())
 }
 
+pub(crate) fn propagate_object_lock_peer_reload(result: std::result::Result<(), StorageError>) -> S3Result<()> {
+    result.map_err(|err| {
+        S3Error::with_message(S3ErrorCode::InternalError, format!("Failed to publish Object Lock metadata: {err}"))
+    })
+}
+
 fn invalid_retention_period(message: impl Into<String>) -> S3Error {
     let mut err = S3Error::with_message(S3ErrorCode::Custom("InvalidRetentionPeriod".into()), message.into());
     err.set_status_code(StatusCode::BAD_REQUEST);
@@ -371,6 +374,7 @@ impl S3 for FS {
         &self,
         req: S3Request<DeleteBucketWebsiteInput>,
     ) -> S3Result<S3Response<DeleteBucketWebsiteOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let Some(store) = self.server_ctx.object_store() else {
             return Err(s3_error!(InternalError, "Not init"));
         };
@@ -380,7 +384,7 @@ impl S3 for FS {
             .await
             .map_err(crate::error::ApiError::from)?;
 
-        delete_bucket_metadata_config(&req.input.bucket, BUCKET_WEBSITE_CONFIG)
+        delete_bucket_metadata_config_if_incarnation(&req.input.bucket, BUCKET_WEBSITE_CONFIG, expected_incarnation_id)
             .await
             .map_err(crate::error::ApiError::from)?;
 
@@ -1058,6 +1062,7 @@ impl S3 for FS {
         &self,
         req: S3Request<PutBucketAccelerateConfigurationInput>,
     ) -> S3Result<S3Response<PutBucketAccelerateConfigurationOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let Some(store) = self.server_ctx.object_store() else {
             return Err(s3_error!(InternalError, "Not init"));
         };
@@ -1068,9 +1073,14 @@ impl S3 for FS {
 
         let accelerate_config = serialize(&req.input.accelerate_configuration)
             .map_err(|err| S3Error::with_message(S3ErrorCode::MalformedXML, format!("{err}")))?;
-        update_bucket_metadata_config(&req.input.bucket, BUCKET_ACCELERATE_CONFIG, accelerate_config)
-            .await
-            .map_err(crate::error::ApiError::from)?;
+        update_bucket_metadata_config_if_incarnation(
+            &req.input.bucket,
+            BUCKET_ACCELERATE_CONFIG,
+            accelerate_config,
+            expected_incarnation_id,
+        )
+        .await
+        .map_err(crate::error::ApiError::from)?;
 
         Ok(S3Response::new(PutBucketAccelerateConfigurationOutput::default()))
     }
@@ -1101,6 +1111,7 @@ impl S3 for FS {
     }
 
     async fn put_bucket_logging(&self, req: S3Request<PutBucketLoggingInput>) -> S3Result<S3Response<PutBucketLoggingOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         record_s3_op(S3Operation::PutBucketLogging);
         let Some(store) = self.server_ctx.object_store() else {
             return Err(s3_error!(InternalError, "Not init"));
@@ -1112,9 +1123,14 @@ impl S3 for FS {
 
         let logging_config = serialize(&req.input.bucket_logging_status)
             .map_err(|err| S3Error::with_message(S3ErrorCode::MalformedXML, format!("{err}")))?;
-        update_bucket_metadata_config(&req.input.bucket, BUCKET_LOGGING_CONFIG, logging_config)
-            .await
-            .map_err(crate::error::ApiError::from)?;
+        update_bucket_metadata_config_if_incarnation(
+            &req.input.bucket,
+            BUCKET_LOGGING_CONFIG,
+            logging_config,
+            expected_incarnation_id,
+        )
+        .await
+        .map_err(crate::error::ApiError::from)?;
 
         Ok(S3Response::new(PutBucketLoggingOutput::default()))
     }
@@ -1161,6 +1177,7 @@ impl S3 for FS {
         &self,
         req: S3Request<PutBucketRequestPaymentInput>,
     ) -> S3Result<S3Response<PutBucketRequestPaymentOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let Some(store) = self.server_ctx.object_store() else {
             return Err(s3_error!(InternalError, "Not init"));
         };
@@ -1171,9 +1188,14 @@ impl S3 for FS {
 
         let payment_config = serialize(&req.input.request_payment_configuration)
             .map_err(|err| S3Error::with_message(S3ErrorCode::MalformedXML, format!("{err}")))?;
-        update_bucket_metadata_config(&req.input.bucket, BUCKET_REQUEST_PAYMENT_CONFIG, payment_config)
-            .await
-            .map_err(crate::error::ApiError::from)?;
+        update_bucket_metadata_config_if_incarnation(
+            &req.input.bucket,
+            BUCKET_REQUEST_PAYMENT_CONFIG,
+            payment_config,
+            expected_incarnation_id,
+        )
+        .await
+        .map_err(crate::error::ApiError::from)?;
 
         Ok(S3Response::new(PutBucketRequestPaymentOutput::default()))
     }
@@ -1203,6 +1225,7 @@ impl S3 for FS {
     }
 
     async fn put_bucket_website(&self, req: S3Request<PutBucketWebsiteInput>) -> S3Result<S3Response<PutBucketWebsiteOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let Some(store) = self.server_ctx.object_store() else {
             return Err(s3_error!(InternalError, "Not init"));
         };
@@ -1213,9 +1236,14 @@ impl S3 for FS {
 
         let website_config = serialize(&req.input.website_configuration)
             .map_err(|err| S3Error::with_message(S3ErrorCode::MalformedXML, format!("{err}")))?;
-        update_bucket_metadata_config(&req.input.bucket, BUCKET_WEBSITE_CONFIG, website_config)
-            .await
-            .map_err(crate::error::ApiError::from)?;
+        update_bucket_metadata_config_if_incarnation(
+            &req.input.bucket,
+            BUCKET_WEBSITE_CONFIG,
+            website_config,
+            expected_incarnation_id,
+        )
+        .await
+        .map_err(crate::error::ApiError::from)?;
 
         Ok(S3Response::new(PutBucketWebsiteOutput::default()))
     }
@@ -1295,6 +1323,7 @@ impl S3 for FS {
             version_id: opts.version_id.clone(),
             ..Default::default()
         };
+        apply_bucket_generation_guard(&req, &bucket, &mut popts)?;
 
         // PutObjectLegalHold only rewrites metadata, so replication is not scheduled by the
         // object PUT path. Schedule it explicitly, otherwise the legal hold never reaches the
@@ -1353,6 +1382,7 @@ impl S3 for FS {
         &self,
         req: S3Request<PutObjectLockConfigurationInput>,
     ) -> S3Result<S3Response<PutObjectLockConfigurationOutput>> {
+        let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
         let PutObjectLockConfigurationInput {
             bucket,
             object_lock_configuration,
@@ -1396,7 +1426,7 @@ impl S3 for FS {
         let object_lock_config =
             String::from_utf8(data.clone()).map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, format!("{err}")))?;
 
-        let updated_at = update_bucket_metadata_config(&bucket, OBJECT_LOCK_CONFIG, data)
+        let updated_at = update_bucket_metadata_config_if_incarnation(&bucket, OBJECT_LOCK_CONFIG, data, expected_incarnation_id)
             .await
             .map_err(ApiError::from)?;
 
@@ -1410,9 +1440,20 @@ impl S3 for FS {
             };
             let versioning_data = serialize(&enable_versioning_config)
                 .map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, format!("{err}")))?;
-            update_bucket_metadata_config(&bucket, BUCKET_VERSIONING_CONFIG, versioning_data)
-                .await
-                .map_err(ApiError::from)?;
+            update_bucket_metadata_config_if_incarnation(
+                &bucket,
+                BUCKET_VERSIONING_CONFIG,
+                versioning_data,
+                expected_incarnation_id,
+            )
+            .await
+            .map_err(ApiError::from)?;
+        }
+
+        if let Some(notification_sys) =
+            runtime_sources::current_notification_system_for_context(self.server_ctx.app_context().as_deref())
+        {
+            propagate_object_lock_peer_reload(notification_sys.load_bucket_metadata(&bucket).await)?;
         }
 
         if let Err(err) = site_replication_bucket_meta_hook(SRBucketMeta {
@@ -1494,6 +1535,7 @@ impl S3 for FS {
         let mut opts: ObjectOptions = get_opts(&bucket, &key, version_id, None, &req.headers)
             .await
             .map_err(ApiError::from)?;
+        apply_bucket_generation_guard(&req, &bucket, &mut opts)?;
         opts.object_lock_retention = Some(ObjectLockRetentionOptions {
             mode: new_mode,
             retain_until: new_retain_until,

rustfs/src/storage/ecfs_extend.rs

@@ -12,13 +12,14 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use super::{StorageError, add_object_lock_years, get_bucket_cors_config, get_bucket_object_lock_config};
+use super::{StorageError, add_object_lock_years, get_bucket_cors_config};
 use crate::config::{RustFSBufferConfig, WorkloadProfile, is_buffer_profile_enabled};
 use crate::error::ApiError;
 use crate::server::cors;
 use crate::storage::ecfs::ListObjectUnorderedQuery;
 use crate::storage::storage_api::ecfs_extend_consumer::contract::bucket::{BucketOperations, BucketOptions};
 use crate::storage::storage_api::ecfs_extend_consumer::contract::multipart::MAX_MULTIPART_PART_NUMBER;
+use crate::storage::storage_api::ecstore_bucket::metadata_sys::{self, ObjectLockConfigState};
 use http::header::{IF_MATCH, IF_MODIFIED_SINCE, IF_NONE_MATCH, IF_UNMODIFIED_SINCE};
 use http::{HeaderMap, HeaderValue, StatusCode};
 use metrics::counter;
@@ -87,7 +88,7 @@ pub(crate) fn remove_object_lock_metadata_for_copy(metadata: &mut HashMap<String
 }
 
 /// Apply bucket default Object Lock retention to object metadata if no explicit retention is set.
-pub(crate) fn apply_lock_retention(object_lock_config: Option<ObjectLockConfiguration>, metadata: &mut HashMap<String, String>) {
+pub(crate) fn apply_lock_retention(object_lock_config: Option<&ObjectLockConfiguration>, metadata: &mut HashMap<String, String>) {
     if has_object_lock_retention_metadata(metadata) {
         return;
     }
@@ -98,8 +99,10 @@ pub(crate) fn apply_lock_retention(object_lock_config: Option<ObjectLockConfigur
         return;
     }
 
-    let Some(default_retention) = config.rule.and_then(|r| r.default_retention) else { return };
-    let Some(mode) = default_retention.mode else { return };
+    let Some(default_retention) = config.rule.as_ref().and_then(|r| r.default_retention.as_ref()) else {
+        return;
+    };
+    let Some(mode) = default_retention.mode.as_ref() else { return };
 
     let now = OffsetDateTime::now_utc();
     let retain_until = match (default_retention.days, default_retention.years) {
@@ -116,7 +119,7 @@ pub(crate) fn apply_lock_retention(object_lock_config: Option<ObjectLockConfigur
 }
 
 pub(crate) fn apply_default_lock_retention_metadata(
-    object_lock_configuration: Option<ObjectLockConfiguration>,
+    object_lock_configuration: Option<&ObjectLockConfiguration>,
     metadata: &mut HashMap<String, String>,
 ) -> bool {
     if has_object_lock_retention_metadata(metadata) {
@@ -133,41 +136,74 @@ pub(crate) fn apply_default_lock_retention_metadata(
     true
 }
 
-pub(crate) async fn apply_bucket_default_lock_retention(
+pub(crate) async fn load_bucket_object_lock_config_state(bucket: &str) -> S3Result<ObjectLockConfigState> {
+    map_bucket_object_lock_config_state(bucket, metadata_sys::get_object_lock_config_state(bucket).await)
+}
+
+pub(crate) fn map_bucket_object_lock_config_state(
     bucket: &str,
+    result: Result<ObjectLockConfigState, StorageError>,
+) -> S3Result<ObjectLockConfigState> {
+    match result {
+        Ok(ObjectLockConfigState::Fabricated) => {
+            warn!(
+                component = LOG_COMPONENT_STORAGE,
+                subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
+                event = "object_lock_config_not_authoritative",
+                bucket = %bucket,
+                "Bucket Object Lock configuration is not authoritative"
+            );
+            Err(S3Error::with_message(
+                S3ErrorCode::InternalError,
+                "Failed to load Object Lock configuration".to_string(),
+            ))
+        }
+        Ok(state) => Ok(state),
+        Err(err) => {
+            warn!(
+                component = LOG_COMPONENT_STORAGE,
+                subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
+                event = "object_lock_config_load_failed",
+                bucket = %bucket,
+                error = ?err,
+                "Failed to load bucket object lock configuration"
+            );
+            Err(S3Error::with_message(
+                S3ErrorCode::InternalError,
+                "Failed to load Object Lock configuration".to_string(),
+            ))
+        }
+    }
+}
+
+pub(crate) fn apply_bucket_default_lock_retention(
+    bucket: &str,
+    state: &ObjectLockConfigState,
     metadata: &mut HashMap<String, String>,
     has_explicit_retention: bool,
 ) -> S3Result<()> {
-    if has_explicit_retention {
-        return Ok(());
-    }
-
-    if has_object_lock_retention_metadata(metadata) {
-        return Ok(());
-    }
-
-    let object_lock_configuration = match get_bucket_object_lock_config(bucket).await {
-        Ok((cfg, _created)) => Some(cfg),
-        Err(err) => {
-            if err == StorageError::ConfigNotFound {
-                None
-            } else {
-                warn!(
-                    component = LOG_COMPONENT_STORAGE,
-                    subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
-                    event = "object_lock_config_load_failed",
-                    bucket = %bucket,
-                    error = ?err,
-                    "Failed to load bucket object lock configuration"
-                );
-                return Err(S3Error::with_message(
-                    S3ErrorCode::InternalError,
-                    "Failed to load Object Lock configuration".to_string(),
-                ));
-            }
+    let object_lock_configuration = match state {
+        ObjectLockConfigState::Configured { config, .. } => Some(config),
+        ObjectLockConfigState::ConfirmedAbsent => None,
+        ObjectLockConfigState::Fabricated => {
+            warn!(
+                component = LOG_COMPONENT_STORAGE,
+                subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
+                event = "object_lock_config_not_authoritative",
+                bucket = %bucket,
+                "Bucket Object Lock configuration is not authoritative"
+            );
+            return Err(S3Error::with_message(
+                S3ErrorCode::InternalError,
+                "Failed to load Object Lock configuration".to_string(),
+            ));
         }
     };
 
+    if has_explicit_retention || has_object_lock_retention_metadata(metadata) {
+        return Ok(());
+    }
+
     apply_default_lock_retention_metadata(object_lock_configuration, metadata);
     Ok(())
 }
@@ -429,37 +465,39 @@ pub(crate) fn parse_object_lock_legal_hold(legal_hold: Option<ObjectLockLegalHol
 }
 
 pub(crate) async fn validate_bucket_object_lock_enabled(bucket: &str) -> S3Result<()> {
-    match get_bucket_object_lock_config(bucket).await {
-        Ok((cfg, _created)) => {
-            if cfg.object_lock_enabled != Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)) {
-                return Err(S3Error::with_message(
-                    S3ErrorCode::InvalidRequest,
-                    "Object Lock is not enabled for this bucket".to_string(),
-                ));
-            }
+    let state = load_bucket_object_lock_config_state(bucket).await?;
+    validate_bucket_object_lock_enabled_state(bucket, &state)
+}
+
+pub(crate) fn validate_bucket_object_lock_enabled_state(bucket: &str, state: &ObjectLockConfigState) -> S3Result<()> {
+    match state {
+        ObjectLockConfigState::Configured { config, .. }
+            if config.object_lock_enabled == Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)) =>
+        {
+            Ok(())
         }
-        Err(err) => {
-            if err == StorageError::ConfigNotFound {
-                return Err(S3Error::with_message(
-                    S3ErrorCode::InvalidRequest,
-                    "Bucket is missing ObjectLockConfiguration".to_string(),
-                ));
-            }
+        ObjectLockConfigState::Configured { .. } => Err(S3Error::with_message(
+            S3ErrorCode::InvalidRequest,
+            "Object Lock is not enabled for this bucket".to_string(),
+        )),
+        ObjectLockConfigState::ConfirmedAbsent => Err(S3Error::with_message(
+            S3ErrorCode::InvalidRequest,
+            "Bucket is missing ObjectLockConfiguration".to_string(),
+        )),
+        ObjectLockConfigState::Fabricated => {
             warn!(
                 component = LOG_COMPONENT_STORAGE,
                 subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
-                event = "object_lock_config_load_failed",
+                event = "object_lock_config_not_authoritative",
                 bucket = %bucket,
-                error = ?err,
-                "Failed to load bucket object lock configuration"
+                "Bucket Object Lock configuration is not authoritative"
             );
-            return Err(S3Error::with_message(
+            Err(S3Error::with_message(
                 S3ErrorCode::InternalError,
                 "Failed to get bucket ObjectLockConfiguration".to_string(),
-            ));
+            ))
         }
     }
-    Ok(())
 }
 
 /// Validates HTTP conditional request headers for a single object according to

rustfs/src/storage/ecfs_test.rs

@@ -16,8 +16,11 @@
 mod tests {
     use crate::config::WorkloadProfile;
     use crate::server::cors;
-    use crate::storage::ecfs::{FS, validate_object_lock_configuration_input};
+    use crate::storage::StorageError;
+    use crate::storage::ecfs::{FS, propagate_object_lock_peer_reload, validate_object_lock_configuration_input};
+    use crate::storage::ecfs_extend::{apply_bucket_default_lock_retention, map_bucket_object_lock_config_state};
     use crate::storage::s3_api::common::{rustfs_initiator, rustfs_owner};
+    use crate::storage::storage_api::ecstore_bucket::metadata_sys::ObjectLockConfigState;
     use crate::storage::storage_api::test_consumer::{
         BucketMetadata, DEFAULT_READ_BUFFER_SIZE, StorageObjectInfo as ObjectInfo, apply_cors_headers,
         apply_default_lock_retention_metadata, bucket_metadata_sys_initialized, check_preconditions, decode_tags_to_map,
@@ -31,7 +34,8 @@ mod tests {
     use rustfs_config::MI_B;
     use rustfs_utils::http::{
         AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER,
-        SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, contains_key_str, get_str, insert_str,
+        MINIO_INTERNAL_PREFIX, RUSTFS_INTERNAL_PREFIX, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP,
+        SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, contains_key_str, get_str, insert_str,
     };
     use rustfs_zip::CompressionFormat;
     use s3s::dto::{
@@ -43,6 +47,7 @@ mod tests {
         PutObjectTaggingInput, QueueConfiguration, S3KeyFilter, Tag, Tagging, TopicConfiguration,
     };
     use s3s::{S3, S3Error, S3ErrorCode, S3Request, s3_error};
+    use std::collections::HashMap;
     use time::{OffsetDateTime, format_description::well_known::Rfc3339};
 
     fn build_request<T>(input: T, method: Method) -> S3Request<T> {
@@ -409,9 +414,16 @@ mod tests {
     }
 
     #[test]
-    fn test_apply_default_lock_retention_metadata_applies_bucket_default() {
-        use std::collections::HashMap;
+    fn test_object_lock_peer_reload_failure_is_returned_to_the_client() {
+        let err = propagate_object_lock_peer_reload(Err(StorageError::other("peer reload failed")))
+            .expect_err("peer reload failure must prevent a successful Object Lock update response");
 
+        assert_eq!(err.code(), &S3ErrorCode::InternalError);
+        assert!(err.message().is_some_and(|message| message.contains("peer reload failed")));
+    }
+
+    #[test]
+    fn test_apply_default_lock_retention_metadata_applies_bucket_default() {
         let cfg = ObjectLockConfiguration {
             object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: Some(ObjectLockRule {
@@ -422,9 +434,9 @@ mod tests {
                 }),
             }),
         };
-        let mut metadata = HashMap::new();
+        let mut metadata = std::collections::HashMap::new();
 
-        assert!(apply_default_lock_retention_metadata(Some(cfg), &mut metadata));
+        assert!(apply_default_lock_retention_metadata(Some(&cfg), &mut metadata));
         assert_eq!(metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER), Some(&"COMPLIANCE".to_string()));
         let retain_until = metadata
             .get(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER)
@@ -435,6 +447,8 @@ mod tests {
             .and_then(|value| OffsetDateTime::parse(&value, &Rfc3339).ok())
             .expect("default retention should write a valid internal timestamp");
         assert!(retention_timestamp <= OffsetDateTime::now_utc());
+        assert!(metadata.contains_key(&format!("{RUSTFS_INTERNAL_PREFIX}{SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP}")));
+        assert!(metadata.contains_key(&format!("{MINIO_INTERNAL_PREFIX}{SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP}")));
     }
 
     #[test]
@@ -456,7 +470,7 @@ mod tests {
             (AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), "2030-01-01T00:00:00Z".to_string()),
         ]);
 
-        assert!(!apply_default_lock_retention_metadata(Some(cfg), &mut metadata));
+        assert!(!apply_default_lock_retention_metadata(Some(&cfg), &mut metadata));
         assert_eq!(metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER), Some(&"GOVERNANCE".to_string()));
         assert_eq!(
             metadata.get(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER),
@@ -466,18 +480,99 @@ mod tests {
 
     #[test]
     fn test_apply_default_lock_retention_metadata_ignores_bucket_without_default() {
-        use std::collections::HashMap;
-
         let cfg = ObjectLockConfiguration {
             object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: None,
         };
+        let mut metadata = std::collections::HashMap::new();
+
+        assert!(!apply_default_lock_retention_metadata(Some(&cfg), &mut metadata));
+        assert!(metadata.is_empty());
+    }
+
+    #[test]
+    fn test_apply_bucket_default_lock_retention_applies_authoritative_default() {
+        let config = ObjectLockConfiguration {
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
+            rule: Some(ObjectLockRule {
+                default_retention: Some(DefaultRetention {
+                    mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::GOVERNANCE)),
+                    days: Some(1),
+                    years: None,
+                }),
+            }),
+        };
+        let state = ObjectLockConfigState::Configured {
+            config,
+            updated_at: OffsetDateTime::now_utc(),
+        };
+        let mut metadata = std::collections::HashMap::new();
+
+        apply_bucket_default_lock_retention("locked-bucket", &state, &mut metadata, false)
+            .expect("authoritative bucket default should be applied");
+
+        assert_eq!(metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER).map(String::as_str), Some("GOVERNANCE"));
+        assert!(metadata.contains_key(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER));
+    }
+
+    #[test]
+    fn test_apply_bucket_default_lock_retention_preserves_explicit_retention_path() {
+        let config = ObjectLockConfiguration {
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
+            rule: Some(ObjectLockRule {
+                default_retention: Some(DefaultRetention {
+                    mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
+                    days: Some(1),
+                    years: None,
+                }),
+            }),
+        };
+        let state = ObjectLockConfigState::Configured {
+            config,
+            updated_at: OffsetDateTime::now_utc(),
+        };
         let mut metadata = HashMap::new();
 
-        assert!(!apply_default_lock_retention_metadata(Some(cfg), &mut metadata));
+        apply_bucket_default_lock_retention("locked-bucket", &state, &mut metadata, true)
+            .expect("explicit retention should suppress only the bucket default");
+
         assert!(metadata.is_empty());
     }
 
+    #[test]
+    fn test_apply_bucket_default_lock_retention_allows_confirmed_absence() {
+        let mut metadata = HashMap::new();
+
+        apply_bucket_default_lock_retention("legacy-bucket", &ObjectLockConfigState::ConfirmedAbsent, &mut metadata, false)
+            .expect("confirmed absence should keep legacy buckets writable");
+
+        assert!(metadata.is_empty());
+    }
+
+    #[test]
+    fn test_apply_bucket_default_lock_retention_rejects_fabricated_metadata_even_with_explicit_retention() {
+        let mut metadata = std::collections::HashMap::from([
+            (AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), "COMPLIANCE".to_string()),
+            (AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), "2030-01-01T00:00:00Z".to_string()),
+        ]);
+
+        let err =
+            apply_bucket_default_lock_retention("untrusted-bucket", &ObjectLockConfigState::Fabricated, &mut metadata, true)
+                .expect_err("fabricated metadata must fail closed before accepting explicit retention");
+
+        assert_eq!(err.code(), &S3ErrorCode::InternalError);
+        assert_eq!(err.message(), Some("Failed to load Object Lock configuration"));
+    }
+
+    #[test]
+    fn test_bucket_object_lock_config_load_error_fails_closed() {
+        let err = map_bucket_object_lock_config_state("corrupt-bucket", Err(StorageError::other("corrupt Object Lock metadata")))
+            .expect_err("metadata read or parse failures must fail closed");
+
+        assert_eq!(err.code(), &S3ErrorCode::InternalError);
+        assert_eq!(err.message(), Some("Failed to load Object Lock configuration"));
+    }
+
     #[test]
     fn test_remove_object_lock_retention_metadata_clears_only_retention_fields() {
         use std::collections::HashMap;

rustfs/src/storage/mod.rs

@@ -61,20 +61,21 @@ pub(crate) use storage_api::{
     TONIC_RPC_PREFIX, TierConfigMgr, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, WorkloadAdmissionSnapshotProviderRef,
     WriteEncryption, WritePlan, access_consumer, add_object_lock_years, all_local_disk, all_local_disk_path,
     check_retention_for_modification, collect_local_metrics, compression_metadata_value, contract, decode_tags,
-    decode_tags_to_map, delete_bucket_metadata_config, disk_drive_path, disk_endpoint, ecfs_consumer, ecfs_extend_consumer,
-    ecstore_admin, ecstore_bucket, ecstore_capacity, ecstore_client, ecstore_cluster, ecstore_compression, ecstore_config,
-    ecstore_data_usage, ecstore_disk, ecstore_error, ecstore_event, ecstore_layout, ecstore_metrics, ecstore_notification,
-    ecstore_rebalance, ecstore_rio, ecstore_rpc, ecstore_set_disk, ecstore_storage, ecstore_tier, encode_tags,
-    find_local_disk_by_ref, get_bucket_accelerate_config, get_bucket_cors_config, get_bucket_logging_config, get_bucket_metadata,
-    get_bucket_notification_config, get_bucket_object_lock_config, get_bucket_policy_raw, get_bucket_replication_config,
-    get_bucket_request_payment_config, get_bucket_sse_config, get_bucket_website_config, get_local_server_property,
-    get_lock_acquire_timeout, get_public_access_block_config, head_prefix_consumer, helper_consumer, init_background_replication,
-    init_bucket_metadata_sys, init_ecstore_config, init_local_disks_with_instance_ctx, init_lock_clients,
-    is_all_buckets_not_found, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found, is_valid_storage_class,
-    options_consumer, prewarm_local_disk_id_map_with_instance_ctx, read_config, record_replication_proxy, rpc_consumer,
-    runtime_sources_consumer, s3_api_consumer, serialize, table_catalog_path_hash, to_s3s_etag,
-    topology_snapshot_from_endpoint_pools_with_capabilities, try_migrate_bucket_metadata, try_migrate_iam_config,
-    try_migrate_server_config, update_bucket_metadata_config, verify_rpc_signature, wrap_reader,
+    decode_tags_to_map, delete_bucket_metadata_config, delete_bucket_metadata_config_if_incarnation, disk_drive_path,
+    disk_endpoint, ecfs_consumer, ecfs_extend_consumer, ecstore_admin, ecstore_bucket, ecstore_capacity, ecstore_client,
+    ecstore_cluster, ecstore_compression, ecstore_config, ecstore_data_usage, ecstore_disk, ecstore_error, ecstore_event,
+    ecstore_layout, ecstore_metrics, ecstore_notification, ecstore_rebalance, ecstore_rio, ecstore_rpc, ecstore_set_disk,
+    ecstore_storage, ecstore_tier, encode_tags, find_local_disk_by_ref, get_bucket_accelerate_config, get_bucket_cors_config,
+    get_bucket_logging_config, get_bucket_metadata, get_bucket_notification_config, get_bucket_object_lock_config,
+    get_bucket_policy_raw, get_bucket_replication_config, get_bucket_request_payment_config, get_bucket_sse_config,
+    get_bucket_website_config, get_local_server_property, get_lock_acquire_timeout, get_public_access_block_config,
+    head_prefix_consumer, helper_consumer, init_background_replication, init_bucket_metadata_sys, init_ecstore_config,
+    init_local_disks_with_instance_ctx, init_lock_clients, is_err_bucket_not_found, is_err_object_not_found,
+    is_err_version_not_found, is_valid_storage_class, options_consumer, prewarm_local_disk_id_map_with_instance_ctx, read_config,
+    record_replication_proxy, rpc_consumer, runtime_sources_consumer, s3_api_consumer, serialize, table_catalog_path_hash,
+    to_s3s_etag, topology_snapshot_from_endpoint_pools_with_capabilities, try_migrate_bucket_metadata, try_migrate_iam_config,
+    try_migrate_server_config, update_bucket_metadata_config, update_bucket_metadata_config_if_incarnation, verify_rpc_signature,
+    wrap_reader,
 };
 
 #[cfg(test)]

rustfs/src/storage/options.rs

@@ -583,6 +583,17 @@ pub(crate) fn namespace_reserved_user_metadata(metadata: &mut HashMap<String, St
         .collect();
 }
 
+pub(crate) fn preserve_unclassified_user_metadata(metadata: &mut HashMap<String, String>, key: &str, value: &str) {
+    let classified_user_metadata = USER_METADATA_PREFIXES
+        .iter()
+        .any(|prefix| key.strip_prefix(prefix).is_some_and(|suffix| !suffix.is_empty()));
+    if classified_user_metadata || SUPPORTED_HEADERS.iter().any(|header| key.eq_ignore_ascii_case(header)) {
+        return;
+    }
+
+    metadata.insert(stored_user_metadata_key(key), value.to_owned());
+}
+
 /// Extracts metadata from headers and returns it as a HashMap with object name for MIME type detection.
 pub fn extract_metadata_from_mime_with_object_name(
     headers: &HeaderMap<HeaderValue>,

rustfs/src/storage/rpc/http_service.rs

@@ -14,6 +14,7 @@
 
 use crate::server::RPC_PREFIX;
 use crate::storage::request_context::spawn_traced;
+use crate::storage::storage_api::DiskError;
 use crate::storage::storage_api::rpc_consumer::http_service::{
     DEFAULT_READ_BUFFER_SIZE, NS_SCANNER_PROTOCOL_VERSION, NsScannerCapabilityResponse, StorageDiskRpcExt as _,
     WALK_DIR_STREAM_COMPLETION_V1, WalkDirOptions, find_local_disk_by_ref, sign_ns_scanner_capability, verify_rpc_signature,
@@ -541,7 +542,7 @@ async fn handle_read_file(req: Request<Incoming>) -> Response<Body> {
                 error = %e,
                 "internode rpc request failed"
             );
-            return response_with_status(StatusCode::INTERNAL_SERVER_ERROR, message);
+            return response_with_disk_error(&e, message);
         }
     };
 
@@ -1252,6 +1253,21 @@ fn response_with_status(status: StatusCode, message: impl Into<String>) -> Respo
         .expect("failed to build rpc error response")
 }
 
+fn response_with_disk_error(error: &DiskError, message: impl Into<String>) -> Response<Body> {
+    let missing = match error {
+        DiskError::FileNotFound => Some(rustfs_rio::INTERNODE_FILE_NOT_FOUND),
+        DiskError::VolumeNotFound => Some(rustfs_rio::INTERNODE_VOLUME_NOT_FOUND),
+        _ => None,
+    };
+    let mut response = response_with_status(StatusCode::INTERNAL_SERVER_ERROR, message);
+    if let Some(missing) = missing {
+        response
+            .headers_mut()
+            .insert(rustfs_rio::INTERNODE_DISK_ERROR_HEADER, HeaderValue::from_static(missing));
+    }
+    response
+}
+
 fn internode_rpc_subsystem(operation: Option<&'static str>) -> &'static str {
     match operation {
         Some(INTERNODE_OPERATION_WALK_DIR) => LOG_SUBSYSTEM_DIRECTORY_WALK,
@@ -1279,19 +1295,19 @@ fn put_file_stage_error_message(stage: &str, query: &PutFileQuery, err: &dyn std
 #[cfg(test)]
 mod tests {
     use super::{
-        LOG_SUBSYSTEM_DIRECTORY_WALK, LOG_SUBSYSTEM_FILE_TRANSFER, LOG_SUBSYSTEM_NAMESPACE_SCANNER, LOG_SUBSYSTEM_ROUTING,
-        NS_SCANNER_BODY_SHA256_QUERY, NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CYCLE_QUERY,
+        DiskError, LOG_SUBSYSTEM_DIRECTORY_WALK, LOG_SUBSYSTEM_FILE_TRANSFER, LOG_SUBSYSTEM_NAMESPACE_SCANNER,
+        LOG_SUBSYSTEM_ROUTING, NS_SCANNER_BODY_SHA256_QUERY, NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CYCLE_QUERY,
         NS_SCANNER_LEADER_EPOCH_QUERY, NS_SCANNER_PATH, NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY,
         NS_SCANNER_SESSION_ID_QUERY, NS_SCANNER_SESSION_SEQUENCE_QUERY, NsScannerQuery, PUT_FILE_STREAM_PATH, PutFileQuery,
         READ_FILE_STREAM_PATH, WALK_DIR_BODY_SHA256_QUERY, WALK_DIR_PATH, WalkDirQuery, append_walk_dir_completion,
         internode_http_operation, internode_rpc_subsystem, is_internode_rpc_path, ns_scanner_response_body,
         ns_scanner_server_epoch_matches, put_body_size_mismatch, put_file_stage_error_message, read_file_body_stream,
-        remote_scanner_claim_rejection, supports_walk_dir_stream_completion, validate_walk_dir_completion_request,
-        verify_internode_rpc_signature, verify_ns_scanner_body_digest, verify_walk_dir_body_digest, walk_dir_response_body,
-        write_body_chunks_to_writer,
+        remote_scanner_claim_rejection, response_with_disk_error, supports_walk_dir_stream_completion,
+        validate_walk_dir_completion_request, verify_internode_rpc_signature, verify_ns_scanner_body_digest,
+        verify_walk_dir_body_digest, walk_dir_response_body, write_body_chunks_to_writer,
     };
     use bytes::Bytes;
-    use http::{HeaderMap, Method, StatusCode, Uri};
+    use http::{HeaderMap, HeaderValue, Method, StatusCode, Uri};
     use http_body_util::BodyExt;
     use rustfs_io_metrics::internode_metrics::{
         INTERNODE_OPERATION_NS_SCANNER, INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM,
@@ -1701,4 +1717,21 @@ mod tests {
 
         assert_eq!(out, b"hello");
     }
+
+    #[test]
+    fn read_file_error_response_marks_only_missing_disk_errors() {
+        for (error, expected) in [
+            (DiskError::FileNotFound, rustfs_rio::INTERNODE_FILE_NOT_FOUND),
+            (DiskError::VolumeNotFound, rustfs_rio::INTERNODE_VOLUME_NOT_FOUND),
+        ] {
+            let response = response_with_disk_error(&error, error.to_string());
+            assert_eq!(
+                response.headers().get(rustfs_rio::INTERNODE_DISK_ERROR_HEADER),
+                Some(&HeaderValue::from_static(expected))
+            );
+        }
+
+        let response = response_with_disk_error(&DiskError::DiskAccessDenied, "permission denied");
+        assert!(response.headers().get(rustfs_rio::INTERNODE_DISK_ERROR_HEADER).is_none());
+    }
 }

rustfs/src/storage/runtime_sources.rs

@@ -14,6 +14,7 @@
 
 use crate::config::RustFSBufferConfig;
 use crate::runtime_sources as root_runtime_sources;
+use crate::storage::storage_api::NotificationSys;
 use crate::storage::storage_api::runtime_sources_consumer::ECStore;
 use rustfs_credentials::Credentials;
 use rustfs_iam::{error::Result as IamResult, store::object::ObjectStore, sys::IamSys};
@@ -26,6 +27,8 @@ use rustfs_lock::LockClient;
 use std::sync::Arc;
 
 pub(crate) use crate::runtime_sources::{AppContext, ServerContextSlot};
+#[cfg(test)]
+pub(crate) use crate::runtime_sources::{IamInterface, KmsInterface};
 
 pub(crate) fn current_app_context() -> Option<Arc<AppContext>> {
     root_runtime_sources::current_app_context()
@@ -39,6 +42,10 @@ pub(crate) fn current_object_store_handle_for_context(context: Option<&AppContex
     root_runtime_sources::current_object_store_handle_for_context(context)
 }
 
+pub(crate) fn current_notification_system_for_context(context: Option<&AppContext>) -> Option<Arc<NotificationSys>> {
+    root_runtime_sources::current_notification_system_for_context(context)
+}
+
 pub(crate) fn current_replication_stats_handle_for_context(
     context: Option<&AppContext>,
 ) -> Option<Arc<crate::storage::storage_api::ReplicationStats>> {

rustfs/src/storage/storage_api.rs

@@ -49,10 +49,14 @@ pub(crate) mod contract {
 
     pub(crate) mod bucket {
         pub(crate) use super::super::storage_contracts::{
-            BucketInfo, BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions,
+            BUCKET_LIFECYCLE_LOCK_OBJECT, BucketInfo, BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions,
         };
     }
 
+    pub(crate) mod namespace {
+        pub(crate) use super::super::storage_contracts::NamespaceLocking;
+    }
+
     pub(crate) mod list {
         pub(crate) use super::super::storage_contracts::{ListObjectVersionsInfo, ListObjectsV2Info, ListOperations};
     }
@@ -61,7 +65,7 @@ pub(crate) mod contract {
         pub(crate) const MAX_MULTIPART_PART_NUMBER: i32 = 10000;
         pub(crate) use super::super::storage_contracts::{ListMultipartsInfo, ListPartsInfo};
         #[cfg(test)]
-        pub(crate) use super::super::storage_contracts::{MultipartInfo, PartInfo};
+        pub(crate) use super::super::storage_contracts::{MultipartInfo, MultipartOperations, PartInfo};
     }
 
     pub(crate) mod object {
@@ -89,9 +93,9 @@ pub(crate) type StorageObjectToDelete = contract::object::ObjectToDelete;
 pub(crate) type StoragePutObjReader = super::PutObjReader;
 pub(crate) use super::ecfs_extend::{
     RFC1123, apply_bucket_default_lock_retention, apply_cors_headers, check_preconditions, get_buffer_size_opt_in,
-    get_validated_store, parse_object_lock_legal_hold, parse_object_lock_retention, parse_part_number_i32_to_usize,
-    process_lambda_configurations, process_queue_configurations, process_topic_configurations,
-    remove_object_lock_metadata_for_copy, validate_bucket_exists, validate_bucket_object_lock_enabled,
+    get_validated_store, load_bucket_object_lock_config_state, parse_object_lock_legal_hold, parse_object_lock_retention,
+    parse_part_number_i32_to_usize, process_lambda_configurations, process_queue_configurations, process_topic_configurations,
+    remove_object_lock_metadata_for_copy, validate_bucket_exists, validate_bucket_object_lock_enabled_state,
     validate_list_object_unordered_with_delimiter, validate_object_key, wrap_response_with_cors,
 };
 pub(crate) use super::sse::{
@@ -102,15 +106,10 @@ pub(crate) use super::sse::{
 
 pub(crate) mod access_consumer {
     pub(crate) use super::super::access::{
-        PostObjectRequestMarker, ReqInfo, authorize_request, has_bypass_governance_header, recursive_force_delete_is_authorized,
-        replication_request_authorized, req_info_mut, req_info_ref,
+        PostObjectRequestMarker, ReqInfo, apply_bucket_generation_guard, apply_copy_source_bucket_generation_guard,
+        authorize_request, bucket_config_mutation_incarnation, has_bypass_governance_header, load_bucket_generation_from_store,
+        recursive_force_delete_is_authorized, replication_request_authorized, req_info_mut, req_info_ref,
     };
-
-    pub(crate) mod contract {
-        pub(crate) mod bucket {
-            pub(crate) use super::super::super::contract::bucket::BucketOperations;
-        }
-    }
 }
 
 pub(crate) mod concurrency_consumer {
@@ -184,7 +183,7 @@ pub(crate) mod options_consumer {
         extract_metadata_from_mime, extract_metadata_from_mime_with_object_name, filter_object_metadata,
         get_complete_multipart_upload_opts_with_replication_authorization, get_content_sha256_with_query, get_opts,
         namespace_reserved_user_metadata, normalize_content_encoding_for_storage, parse_copy_source_range,
-        put_opts_with_replication_authorization, validate_archive_content_encoding,
+        preserve_unclassified_user_metadata, put_opts_with_replication_authorization, validate_archive_content_encoding,
     };
 
     pub(crate) mod contract {
@@ -261,7 +260,10 @@ pub(crate) mod rpc_consumer {
 pub(crate) mod runtime_sources_consumer {
     pub(crate) type ECStore = super::ECStore;
     pub(crate) type EndpointServerPools = super::EndpointServerPools;
+    pub(crate) type ServerContextSlot = super::ServerContextSlot;
     pub(crate) use crate::storage::runtime_sources;
+    #[cfg(test)]
+    pub(crate) use crate::storage::runtime_sources::{AppContext, IamInterface, KmsInterface};
 }
 
 pub(crate) mod heal_control_startup_consumer {
@@ -334,8 +336,9 @@ pub(crate) mod sse_consumer {
     };
     pub(crate) use super::{
         DecryptionRequest, EncryptionRequest, PrepareEncryptionRequest, SseKmsPrincipal, apply_bucket_default_lock_retention,
-        authorize_sse_kms_object_read, extract_server_side_encryption_from_headers, get_buffer_size_opt_in, sse_decryption,
-        sse_encryption, sse_prepare_encryption,
+        authorize_sse_kms_object_read, extract_server_side_encryption_from_headers, get_buffer_size_opt_in,
+        load_bucket_object_lock_config_state, sse_decryption, sse_encryption, sse_prepare_encryption,
+        validate_bucket_object_lock_enabled_state,
     };
 }
 
@@ -844,7 +847,7 @@ pub(crate) async fn reconcile_bucket_resync_target_intents(buckets: &[String]) -
     };
 
     for bucket in buckets {
-        let _transaction_guard = ecstore_bucket::metadata_sys::acquire_bucket_metadata_transaction_lock(bucket).await?;
+        let transaction_guard = ecstore_bucket::metadata_sys::acquire_bucket_metadata_transaction_lock(bucket).await?;
         let status = pool.get_bucket_resync_status(bucket).await?;
         if status.targets_map.is_empty() {
             continue;
@@ -859,7 +862,7 @@ pub(crate) async fn reconcile_bucket_resync_target_intents(buckets: &[String]) -
             continue;
         }
         let encoded = serde_json::to_vec(&targets).map_err(Error::other)?;
-        ecstore_bucket::metadata_sys::update_bucket_targets_under_transaction_lock(bucket, encoded).await?;
+        ecstore_bucket::metadata_sys::update_bucket_targets_under_transaction_lock(&transaction_guard, bucket, encoded).await?;
     }
 
     Ok(())
@@ -1396,7 +1399,18 @@ pub(crate) fn get_global_bucket_metadata_sys() -> Option<Arc<tokio::sync::RwLock
 }
 
 pub(crate) async fn delete_bucket_metadata_config(bucket: &str, config_file: &str) -> Result<time::OffsetDateTime> {
-    let updated_at = ecstore_bucket::metadata_sys::delete(bucket, config_file).await?;
+    delete_bucket_metadata_config_if_incarnation(bucket, config_file, None).await
+}
+
+pub(crate) async fn delete_bucket_metadata_config_if_incarnation(
+    bucket: &str,
+    config_file: &str,
+    expected_incarnation_id: Option<uuid::Uuid>,
+) -> Result<time::OffsetDateTime> {
+    let updated_at = match expected_incarnation_id {
+        Some(incarnation_id) => ecstore_bucket::metadata_sys::delete_if_incarnation(bucket, config_file, incarnation_id).await?,
+        None => ecstore_bucket::metadata_sys::delete(bucket, config_file).await?,
+    };
     record_scanner_maintenance_config_change(bucket, config_file);
     Ok(updated_at)
 }
@@ -1490,17 +1504,37 @@ pub(crate) async fn update_bucket_metadata_config(
     config_file: &str,
     data: Vec<u8>,
 ) -> Result<time::OffsetDateTime> {
-    let updated_at = ecstore_bucket::metadata_sys::update(bucket, config_file, data).await?;
+    update_bucket_metadata_config_if_incarnation(bucket, config_file, data, None).await
+}
+
+pub(crate) async fn update_bucket_metadata_config_if_incarnation(
+    bucket: &str,
+    config_file: &str,
+    data: Vec<u8>,
+    expected_incarnation_id: Option<uuid::Uuid>,
+) -> Result<time::OffsetDateTime> {
+    let updated_at = match expected_incarnation_id {
+        Some(incarnation_id) => {
+            ecstore_bucket::metadata_sys::update_if_incarnation(bucket, config_file, data, incarnation_id).await?
+        }
+        None => ecstore_bucket::metadata_sys::update(bucket, config_file, data).await?,
+    };
     record_scanner_maintenance_config_change(bucket, config_file);
     Ok(updated_at)
 }
 
-pub(crate) async fn acquire_bucket_metadata_transaction_lock(bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard> {
+pub(crate) async fn acquire_bucket_metadata_transaction_lock(
+    bucket: &str,
+) -> Result<ecstore_bucket::metadata_sys::BucketMetadataMutationGuard> {
     ecstore_bucket::metadata_sys::acquire_bucket_metadata_transaction_lock(bucket).await
 }
 
-pub(crate) async fn update_bucket_targets_under_transaction_lock(bucket: &str, data: Vec<u8>) -> Result<time::OffsetDateTime> {
-    ecstore_bucket::metadata_sys::update_bucket_targets_under_transaction_lock(bucket, data).await
+pub(crate) async fn update_bucket_targets_under_transaction_lock(
+    guard: &ecstore_bucket::metadata_sys::BucketMetadataMutationGuard,
+    bucket: &str,
+    data: Vec<u8>,
+) -> Result<time::OffsetDateTime> {
+    ecstore_bucket::metadata_sys::update_bucket_targets_under_transaction_lock(guard, bucket, data).await
 }
 
 fn record_scanner_maintenance_config_change(bucket: &str, config_file: &str) {
@@ -1568,10 +1602,6 @@ pub(crate) fn is_err_version_not_found(err: &Error) -> bool {
     ecstore_error::is_err_version_not_found(err)
 }
 
-pub(crate) fn is_all_buckets_not_found(errs: &[Option<DiskError>]) -> bool {
-    ecstore_disk::error_reduce::is_all_buckets_not_found(errs)
-}
-
 pub(crate) fn get_global_lock_client() -> Option<Arc<dyn rustfs_lock::client::LockClient>> {
     ecstore_runtime::global_lock_client()
 }

rustfs/src/table_catalog.rs

@@ -20837,17 +20837,27 @@ mod tests {
     #[test]
     fn object_mutation_entrypoints_call_reserved_prefix_guard() {
         let source = include_str!("app/object_usecase.rs");
+        let delete_object = source
+            .split_once("pub async fn execute_delete_object")
+            .and_then(|(_, remainder)| remainder.split_once("pub async fn execute_head_object"))
+            .map(|(delete_object, _)| delete_object)
+            .expect("delete object entrypoint should remain in the object usecase");
+
         for expected in [
             "validate_object_key(&key, request_method_name)?;\n        validate_table_catalog_object_mutation(&bucket, &key).await?;",
             "validate_object_key(&key, \"COPY (dest)\")?;\n        validate_table_catalog_object_mutation(&bucket, &key).await?;",
             "if let Err(err) = validate_table_catalog_object_mutation(&bucket, &obj_id.key).await",
-            "validate_object_key(&key, \"DELETE\")?;\n        validate_table_catalog_object_mutation(&bucket, &key).await?;",
             "validate_table_catalog_object_mutation(&bucket, &object).await?;",
             "validate_object_key(&key, \"PUT\")?;\n        validate_table_catalog_object_mutation(&bucket, &key).await?;",
             "validate_table_catalog_object_mutation(&bucket, &fpath).await?;",
         ] {
             assert!(source.contains(expected), "missing object mutation guard: {expected}");
         }
+        assert!(
+            delete_object.contains("validate_object_key(&key, \"DELETE\")?;")
+                && delete_object.contains("validate_table_catalog_object_mutation(&bucket, &key).await?;"),
+            "delete object entrypoint must validate the object key and reserved catalog prefix"
+        );
     }
 
     #[test]

rustfs/tests/common/mod.rs

@@ -0,0 +1,25 @@
+use std::future::Future;
+
+const EMBEDDED_TEST_STACK_SIZE: usize = 8 * 1024 * 1024;
+
+pub fn run_embedded_test<F, Fut>(test: F)
+where
+    F: FnOnce() -> Fut + Send + 'static,
+    Fut: Future<Output = ()> + 'static,
+{
+    let handle = std::thread::Builder::new()
+        .name("embedded-test".to_string())
+        .stack_size(EMBEDDED_TEST_STACK_SIZE)
+        .spawn(move || {
+            tokio::runtime::Builder::new_current_thread()
+                .enable_all()
+                .build()
+                .expect("build embedded test runtime")
+                .block_on(Box::pin(test()));
+        })
+        .expect("spawn embedded test thread");
+
+    if let Err(payload) = handle.join() {
+        std::panic::resume_unwind(payload);
+    }
+}

rustfs/tests/embedded_deferred_iam_test.rs

@@ -20,6 +20,8 @@ use rustfs_config::{ENV_TEST_IAM_FAIL_INIT_ATTEMPTS, ENV_TEST_IAM_RETRY_INTERVAL
 use std::time::Duration;
 use temp_env::async_with_vars;
 
+mod common;
+
 fn response_preview(body: &str) -> String {
     body.chars().take(512).collect()
 }
@@ -37,8 +39,13 @@ fn s3_client(endpoint: &str, access_key: &str, secret_key: &str) -> Client {
 }
 
 #[cfg(debug_assertions)]
-#[tokio::test]
-async fn test_embedded_server_recovers_after_deferred_iam_bootstrap() {
+#[test]
+fn test_embedded_server_recovers_after_deferred_iam_bootstrap() {
+    common::run_embedded_test(test_embedded_server_recovers_after_deferred_iam_bootstrap_body);
+}
+
+#[cfg(debug_assertions)]
+async fn test_embedded_server_recovers_after_deferred_iam_bootstrap_body() {
     async_with_vars(
         [
             (ENV_TEST_IAM_FAIL_INIT_ATTEMPTS, Some("1")),

rustfs/tests/embedded_multi_instance_test.rs

@@ -28,6 +28,8 @@ use reqwest::StatusCode;
 #[cfg(feature = "e2e-test-hooks")]
 use rustfs::embedded::pause_embedded_startup_after_http_bind;
 use rustfs::embedded::{RustFSServerBuilder, find_available_port};
+
+mod common;
 #[cfg(feature = "e2e-test-hooks")]
 use sha2::{Digest, Sha256};
 #[cfg(feature = "e2e-test-hooks")]
@@ -111,8 +113,12 @@ fn signed_admin_request(
 // startup pipeline lifts; a follow-up will widen the request path to route
 // per-server so the two servers can also serve different data planes end-to-
 // end without the shared-IAM caveat.
-#[tokio::test]
-async fn two_embedded_servers_start_and_shutdown_independently() {
+#[test]
+fn two_embedded_servers_start_and_shutdown_independently() {
+    common::run_embedded_test(two_embedded_servers_start_and_shutdown_independently_body);
+}
+
+async fn two_embedded_servers_start_and_shutdown_independently_body() {
     let port_a = match find_available_port() {
         Ok(port) => port,
         Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return,
@@ -181,8 +187,12 @@ async fn two_embedded_servers_start_and_shutdown_independently() {
 // rejects the other's) AND data plane (each server's buckets/objects are
 // invisible to the other; each lists/creates/deletes only on its own disks
 // and bucket-metadata system).
-#[tokio::test]
-async fn two_embedded_servers_isolate_auth_and_data_planes() {
+#[test]
+fn two_embedded_servers_isolate_auth_and_data_planes() {
+    common::run_embedded_test(two_embedded_servers_isolate_auth_and_data_planes_body);
+}
+
+async fn two_embedded_servers_isolate_auth_and_data_planes_body() {
     let port_a = match find_available_port() {
         Ok(port) => port,
         Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return,

rustfs/tests/embedded_notification_lifecycle_test.rs

@@ -18,6 +18,8 @@ use aws_sdk_s3::{Client, Config};
 use rustfs::embedded::{RustFSServerBuilder, find_available_port};
 use rustfs_notify::{NotificationRuntimeState, notification_system};
 
+mod common;
+
 fn s3_client(endpoint: &str, access_key: &str, secret_key: &str) -> Client {
     let credentials = Credentials::new(access_key, secret_key, None, None, "test");
     let config = Config::builder()
@@ -30,8 +32,12 @@ fn s3_client(endpoint: &str, access_key: &str, secret_key: &str) -> Client {
     Client::from_conf(config)
 }
 
-#[tokio::test]
-async fn notification_runtime_stays_enabled_until_the_last_embedded_owner_drains() {
+#[test]
+fn notification_runtime_stays_enabled_until_the_last_embedded_owner_drains() {
+    common::run_embedded_test(notification_runtime_stays_enabled_until_the_last_embedded_owner_drains_body);
+}
+
+async fn notification_runtime_stays_enabled_until_the_last_embedded_owner_drains_body() {
     temp_env::async_with_vars([(rustfs_config::ENV_NOTIFY_ENABLE, Some("true"))], async {
         let port_a = match find_available_port() {
             Ok(port) => port,

rustfs/tests/embedded_test.rs

@@ -22,6 +22,8 @@ use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::{Client, Config};
 use rustfs::embedded::{RustFSServerBuilder, find_available_port};
 
+mod common;
+
 /// Helper: create an S3 client pointed at the embedded server.
 fn s3_client(endpoint: &str, access_key: &str, secret_key: &str) -> Client {
     let creds = Credentials::new(access_key, secret_key, None, None, "test");
@@ -35,8 +37,12 @@ fn s3_client(endpoint: &str, access_key: &str, secret_key: &str) -> Client {
     Client::from_conf(config)
 }
 
-#[tokio::test]
-async fn test_embedded_server_basic_s3_operations() {
+#[test]
+fn test_embedded_server_basic_s3_operations() {
+    common::run_embedded_test(test_embedded_server_basic_s3_operations_body);
+}
+
+async fn test_embedded_server_basic_s3_operations_body() {
     // 1. Pick a free port and start the embedded server.
     let port = match find_available_port() {
         Ok(port) => port,