refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13) (#4586)

* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)

Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.

- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
  with `init_background_cancel_token` (set-once) and `background_cancel_token()`
  returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
  route through the current instance's context; the static is removed. The
  getter now returns an owned `Option<CancellationToken>` instead of a
  `Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
  the lifecycle worker/loops take the owned token (a shared fallback is cloned
  when the token is somehow uninitialized). No Arc cycle is introduced —
  workers hold a token clone, not the instance context.

Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.

Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.

Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).

Refs: backlog#939 (Phase 5, Slice 13)

* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)

Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.

- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
  that constructs two independent InstanceContexts and verifies NONE of the
  migrated state is shared — erasure setup, lock manager, region, deployment id,
  the four service handles (tier/notifier/expiry/transition), the local disk
  registry, the bucket monitor, and the background cancel token. This is the
  object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
  intentionally retained: storage startup still publishes into the process-level
  bootstrap context (write-once region/endpoints/deployment id) and the single
  GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
  state. Lifting the guard requires threading a per-instance context through
  startup — a follow-up beyond migrating the globals. The guard is NOT removed:
  rejecting the second start is safer than the panic it would otherwise become.

Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).

Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -618,7 +618,7 @@ impl ExpiryState {
     async fn worker(rx: &mut Receiver<Option<ExpiryOpType>>, api: Arc<ECStore>, stats: Arc<ExpiryStats>) {
         let cancel_token = runtime_sources::background_services_cancel_token().unwrap_or_else(|| {
             static FALLBACK: std::sync::OnceLock<tokio_util::sync::CancellationToken> = std::sync::OnceLock::new();
-            FALLBACK.get_or_init(tokio_util::sync::CancellationToken::new)
+            FALLBACK.get_or_init(tokio_util::sync::CancellationToken::new).clone()
         });
 
         loop {
@@ -1352,9 +1352,7 @@ fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>) {
     }
 
     tokio::spawn(async move {
-        let cancel_token = runtime_sources::background_services_cancel_token()
-            .cloned()
-            .unwrap_or_else(CancellationToken::new);
+        let cancel_token = runtime_sources::background_services_cancel_token().unwrap_or_default();
         let mut interval = tokio::time::interval(StdDuration::from_secs(60));
         let mut bucket_marker: Option<String> = None;
         let mut object_marker: Option<String> = None;
@@ -1427,9 +1425,7 @@ fn spawn_tier_delete_journal_recovery_once(api: Arc<ECStore>) {
     }
 
     tokio::spawn(async move {
-        let cancel_token = runtime_sources::background_services_cancel_token()
-            .cloned()
-            .unwrap_or_else(CancellationToken::new);
+        let cancel_token = runtime_sources::background_services_cancel_token().unwrap_or_default();
         run_tier_delete_journal_recovery_loop(api, cancel_token).await;
     });
 }

crates/ecstore/src/bucket/lifecycle/runtime_boundary.rs

@@ -35,7 +35,7 @@ pub(crate) fn tier_config_mgr_handle() -> Arc<RwLock<TierConfigMgr>> {
     sources::tier_config_mgr_handle()
 }
 
-pub(crate) fn background_services_cancel_token() -> Option<&'static CancellationToken> {
+pub(crate) fn background_services_cancel_token() -> Option<CancellationToken> {
     sources::background_services_cancel_token()
 }
 

crates/ecstore/src/bucket/metadata_sys.rs

@@ -93,7 +93,7 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
 }
 
 fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
-    let Some(cancel_token) = runtime_sources::background_services_cancel_token().cloned() else {
+    let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
         warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
         return;
     };