refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type (#4413)

* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type

Phase 5 of the global-singleton consolidation (backlog#939): begin moving
runtime identity state out of process globals so multiple ECStore instances
can coexist in one process. Isolation is carried by the object graph
(ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a
task-local, which does not propagate across the many internal tokio::spawn
boundaries in the data/background paths.

This first slice migrates the erasure setup type -- previously three
independent process-global bools -- into a single per-instance
RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd,
removing a triple source of truth that could drift out of sync.

- New runtime::instance module: InstanceContext + process bootstrap context.
- The legacy free-function facade (is_erasure/update_erasure_type/...) keeps
  its signatures and forwards to the current instance's context, falling back
  to the bootstrap context before a store is published.
- ECStore gains a pub(crate) ctx field and setup_is_* accessors; its
  constructors adopt the bootstrap context (never mint a fresh one) so startup
  writes and post-construction reads share one cell -- single-instance
  behavior is byte-for-byte unchanged.

Tests: erasure predicate derivation vs the legacy behavior, object-graph
carrier isolation across two ECStore instances, and bootstrap adoption.

Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8)

* refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415)

* refactor(ecstore): source the namespace lock manager per-instance (#4417)

refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3)

Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by
sourcing SetDisks' lock manager from the instance context instead of the
process singleton. This removes the false cross-instance mutual exclusion (and
attendant ABBA risk) that a shared GlobalLockManager would cause once multiple
instances coexist.

- InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints
  a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the
  process singleton via get_global_lock_manager(), so a single-instance
  deployment keeps exactly one shared namespace.
- SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx
  it already adopts), not `runtime_sources::global_lock_manager()`. Single
  instance: same Arc as before, so behavior is unchanged.
- Remove the now-unused `runtime_sources::global_lock_manager()` wrapper.

Tests: bootstrap lock manager aliases the process singleton; two fresh contexts
own distinct managers; a SetDisks' lock manager is the one from its context and
aliases the global singleton in a single-instance build.

Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking
regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean),
make pre-commit (pass).

Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2).

crates/ecstore/src/runtime/global.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use super::instance::{InstanceContext, bootstrap_ctx};
 use crate::bucket::bandwidth::monitor::Monitor;
 use crate::{
     bucket::lifecycle::bucket_lifecycle_ops::LifecycleSys,
@@ -42,16 +43,17 @@ pub const DISK_RESERVE_FRACTION: f64 = 0.15;
 // These should be migrated to AppContext over time.
 // See issue #730 for migration plan.
 //
-// Tier A (needs migration): GLOBAL_OBJECT_API, GLOBAL_IS_ERASURE*, GLOBAL_LOCAL_DISK_*,
+// Tier A (needs migration): GLOBAL_OBJECT_API, GLOBAL_LOCAL_DISK_*,
 //   GLOBAL_ROOT_DISK_THRESHOLD, GLOBAL_LIFECYCLE_SYS, GLOBAL_EVENT_NOTIFIER, etc.
 // Tier B (keep as static): GLOBAL_RUSTFS_PORT, GLOBAL_REGION, env var caches, etc.
+//
+// Phase 5 (backlog#939): the erasure setup type moved into the per-instance
+// `InstanceContext` (see `super::instance`); the erasure predicates below now
+// forward to the current instance's context.
 lazy_static! {
     static ref GLOBAL_RUSTFS_PORT: OnceLock<u16> = OnceLock::new();
     static ref GLOBAL_DEPLOYMENT_ID: OnceLock<Uuid> = OnceLock::new();
     pub static ref GLOBAL_OBJECT_API: OnceLock<Arc<ECStore>> = OnceLock::new();
-    pub static ref GLOBAL_IS_ERASURE: RwLock<bool> = RwLock::new(false);
-    pub static ref GLOBAL_IS_DIST_ERASURE: RwLock<bool> = RwLock::new(false);
-    pub static ref GLOBAL_IS_ERASURE_SD: RwLock<bool> = RwLock::new(false);
     pub static ref GLOBAL_LOCAL_DISK_MAP: Arc<RwLock<HashMap<String, Option<DiskStore>>>> = Arc::new(RwLock::new(HashMap::new()));
     pub static ref GLOBAL_LOCAL_DISK_ID_MAP: Arc<RwLock<HashMap<Uuid, String>>> = Arc::new(RwLock::new(HashMap::new()));
     pub static ref GLOBAL_LOCAL_DISK_SET_DRIVES: Arc<RwLock<TypeLocalDiskSetDrives>> = Arc::new(RwLock::new(Vec::new()));
@@ -207,6 +209,19 @@ pub fn resolve_object_store_handle() -> Option<Arc<ECStore>> {
         .or_else(new_object_layer_fn)
 }
 
+/// Resolve the instance context for the legacy free-function facade.
+///
+/// Prefers the currently-published `ECStore`'s own context; before any store
+/// is published (e.g. during storage startup, or in unit tests) it falls back
+/// to the process-level [`bootstrap_ctx`]. Because `ECStore::new` adopts the
+/// bootstrap `Arc`, single-instance callers always observe one and the same
+/// context — behavior is unchanged from the previous process-global bools.
+pub(crate) fn current_ctx() -> Arc<InstanceContext> {
+    resolve_object_store_handle()
+        .map(|store| store.ctx.clone())
+        .unwrap_or_else(bootstrap_ctx)
+}
+
 /// Set the global object layer
 ///
 /// # Arguments
@@ -226,8 +241,7 @@ pub async fn set_object_layer(o: Arc<ECStore>) {
 /// * `bool` - True if the setup type is distributed erasure coding, false otherwise
 ///
 pub async fn is_dist_erasure() -> bool {
-    let lock = GLOBAL_IS_DIST_ERASURE.read().await;
-    *lock
+    current_ctx().is_dist_erasure().await
 }
 
 /// Check if the setup type is erasure coding with single data center
@@ -236,8 +250,7 @@ pub async fn is_dist_erasure() -> bool {
 /// * `bool` - True if the setup type is erasure coding with single data center, false otherwise
 ///
 pub async fn is_erasure_sd() -> bool {
-    let lock = GLOBAL_IS_ERASURE_SD.read().await;
-    *lock
+    current_ctx().is_erasure_sd().await
 }
 
 /// Check if the setup type is erasure coding
@@ -246,8 +259,7 @@ pub async fn is_erasure_sd() -> bool {
 /// * `bool` - True if the setup type is erasure coding, false otherwise
 ///
 pub async fn is_erasure() -> bool {
-    let lock = GLOBAL_IS_ERASURE.read().await;
-    *lock
+    current_ctx().is_erasure().await
 }
 
 /// Update the global erasure type based on the setup type
@@ -258,18 +270,7 @@ pub async fn is_erasure() -> bool {
 /// # Returns
 /// * None
 pub async fn update_erasure_type(setup_type: SetupType) {
-    let mut is_erasure = GLOBAL_IS_ERASURE.write().await;
-    *is_erasure = setup_type == SetupType::Erasure;
-
-    let mut is_dist_erasure = GLOBAL_IS_DIST_ERASURE.write().await;
-    *is_dist_erasure = setup_type == SetupType::DistErasure;
-
-    if *is_dist_erasure {
-        *is_erasure = true
-    }
-
-    let mut is_erasure_sd = GLOBAL_IS_ERASURE_SD.write().await;
-    *is_erasure_sd = setup_type == SetupType::ErasureSD;
+    current_ctx().update_erasure_type(setup_type).await;
 }
 
 // pub fn is_legacy() -> bool {

crates/ecstore/src/runtime/instance.rs

@@ -0,0 +1,232 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Per-instance runtime state ("instance context").
+//!
+//! Phase 5 of the global-singleton consolidation (backlog#939). State that
+//! carries an `ECStore` instance's identity/runtime is being moved out of the
+//! process-level statics in [`super::global`] into this context so that
+//! multiple `ECStore` instances can coexist in one process without
+//! cross-contaminating each other's state.
+//!
+//! ## Isolation carrier
+//!
+//! Multi-instance disambiguation is carried by the **object graph**
+//! (`ECStore` → `Vec<Arc<Sets>>` → `SetDisks`), each holding an
+//! `Arc<InstanceContext>`. Instance-scoped reads resolve through `self.ctx`,
+//! so no call-site signatures change. (A `task_local` was rejected during
+//! design review: it does not propagate across the many internal
+//! `tokio::spawn` boundaries in the data/background paths.)
+//!
+//! ## Backward compatibility
+//!
+//! The legacy free-function facade in [`super::global`]
+//! (`is_erasure`/`update_erasure_type`/…) resolves the *current* instance via
+//! the object-store resolver and falls back to the process-level
+//! [`bootstrap_ctx`] when no store is published yet. Because `ECStore::new`
+//! **adopts** the same bootstrap `Arc` (rather than minting a fresh one),
+//! startup writes and post-construction reads hit the same cell and
+//! single-instance behavior is byte-for-byte unchanged.
+
+use crate::layout::endpoints::SetupType;
+use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
+use std::sync::{Arc, OnceLock};
+use tokio::sync::RwLock;
+
+/// Runtime state owned by a single `ECStore` instance.
+///
+/// This is intentionally minimal in the first migration slice; subsequent
+/// slices move additional identity/runtime state (topology, disk registry,
+/// service handles, cancellation token) into this struct.
+#[derive(Debug)]
+pub struct InstanceContext {
+    /// The deployment's erasure setup type.
+    ///
+    /// Single source of truth for the derived `is_erasure` /
+    /// `is_dist_erasure` / `is_erasure_sd` predicates, replacing the three
+    /// previously-independent process-global erasure-mode bools (which could
+    /// drift out of sync). Stored as one value so the three predicates can
+    /// never observe a torn intermediate state.
+    erasure_kind: RwLock<SetupType>,
+    /// This instance's namespace lock manager (Phase 5 Slice 3, backlog#939).
+    ///
+    /// Owned per-instance so two instances no longer share a lock namespace
+    /// (which would make same-named objects in different instances falsely
+    /// contend). Single-instance aliases the process singleton, so behavior is
+    /// unchanged; see [`bootstrap_ctx`].
+    lock_manager: Arc<GlobalLockManager>,
+}
+
+impl InstanceContext {
+    /// Create a fresh instance context in the initial [`SetupType::Unknown`]
+    /// state with its own lock manager — byte-for-byte equivalent to the old
+    /// all-`false` erasure globals.
+    pub fn new() -> Self {
+        Self::with_lock_manager(Arc::new(GlobalLockManager::new()))
+    }
+
+    /// Build a context bound to a specific lock manager.
+    ///
+    /// [`bootstrap_ctx`] uses this to alias the process-global lock manager so
+    /// single-instance deployments keep one shared namespace; `new` mints a
+    /// fresh manager for a genuinely independent instance.
+    fn with_lock_manager(lock_manager: Arc<GlobalLockManager>) -> Self {
+        Self {
+            erasure_kind: RwLock::new(SetupType::Unknown),
+            lock_manager,
+        }
+    }
+
+    /// This instance's namespace lock manager.
+    pub fn lock_manager(&self) -> Arc<GlobalLockManager> {
+        self.lock_manager.clone()
+    }
+
+    /// Update this instance's erasure setup type.
+    pub async fn update_erasure_type(&self, setup_type: SetupType) {
+        *self.erasure_kind.write().await = setup_type;
+    }
+
+    /// Whether this instance uses erasure coding.
+    ///
+    /// True for both single-node ([`SetupType::Erasure`]) and distributed
+    /// ([`SetupType::DistErasure`]) erasure setups, matching the original
+    /// `update_erasure_type` derivation where `DistErasure` implied
+    /// `is_erasure == true`.
+    pub async fn is_erasure(&self) -> bool {
+        matches!(*self.erasure_kind.read().await, SetupType::Erasure | SetupType::DistErasure)
+    }
+
+    /// Whether this instance uses distributed erasure coding.
+    pub async fn is_dist_erasure(&self) -> bool {
+        *self.erasure_kind.read().await == SetupType::DistErasure
+    }
+
+    /// Whether this instance uses single-drive erasure coding.
+    pub async fn is_erasure_sd(&self) -> bool {
+        *self.erasure_kind.read().await == SetupType::ErasureSD
+    }
+}
+
+impl Default for InstanceContext {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+/// Process-level bootstrap instance context.
+///
+/// Storage startup writes erasure state before any `ECStore` exists (see
+/// `init_startup_storage_foundation`), so the context must exist ahead of the
+/// object graph. `ECStore::new` adopts this same `Arc` via [`bootstrap_ctx`],
+/// guaranteeing that startup writes and post-construction reads share one cell.
+static BOOTSTRAP_CTX: OnceLock<Arc<InstanceContext>> = OnceLock::new();
+
+/// Return the process-level bootstrap instance context, creating it on first
+/// access.
+///
+/// This is the single-instance default that the legacy free-function facade
+/// falls back to before an `ECStore` is published, and the `Arc` that
+/// `ECStore::new` adopts as its own `ctx`.
+pub fn bootstrap_ctx() -> Arc<InstanceContext> {
+    BOOTSTRAP_CTX
+        .get_or_init(|| Arc::new(InstanceContext::with_lock_manager(get_global_lock_manager())))
+        .clone()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    // The SetupType inputs must derive the exact (is_erasure,
+    // is_dist_erasure, is_erasure_sd) triples that the original three
+    // process-global erasure bools produced via update_erasure_type().
+    #[tokio::test]
+    async fn erasure_predicates_match_legacy_derivation() {
+        let cases = [
+            // (input, is_erasure, is_dist_erasure, is_erasure_sd)
+            (SetupType::Unknown, false, false, false),
+            (SetupType::FS, false, false, false),
+            (SetupType::Erasure, true, false, false),
+            // DistErasure implies is_erasure == true (legacy global.rs:267-269).
+            (SetupType::DistErasure, true, true, false),
+            (SetupType::ErasureSD, false, false, true),
+        ];
+
+        for (input, want_erasure, want_dist, want_sd) in cases {
+            let ctx = InstanceContext::new();
+            ctx.update_erasure_type(input.clone()).await;
+            assert_eq!(ctx.is_erasure().await, want_erasure, "is_erasure for {input:?}");
+            assert_eq!(ctx.is_dist_erasure().await, want_dist, "is_dist_erasure for {input:?}");
+            assert_eq!(ctx.is_erasure_sd().await, want_sd, "is_erasure_sd for {input:?}");
+        }
+    }
+
+    // A fresh context (before any update) reflects the initial all-false state.
+    #[tokio::test]
+    async fn fresh_context_is_all_false() {
+        let ctx = InstanceContext::new();
+        assert!(!ctx.is_erasure().await);
+        assert!(!ctx.is_dist_erasure().await);
+        assert!(!ctx.is_erasure_sd().await);
+    }
+
+    // bootstrap_ctx() is a stable process singleton: repeated calls return the
+    // same Arc, so a startup write is visible to a later read through it.
+    #[tokio::test]
+    async fn bootstrap_ctx_is_stable_singleton() {
+        let a = bootstrap_ctx();
+        let b = bootstrap_ctx();
+        assert!(Arc::ptr_eq(&a, &b), "bootstrap_ctx must return the same Arc");
+    }
+
+    // Two independent contexts do not share erasure state — the property that
+    // lets two ECStore instances (each carrying its own ctx) stay isolated.
+    #[tokio::test]
+    async fn distinct_contexts_do_not_share_state() {
+        let ctx_a = Arc::new(InstanceContext::new());
+        let ctx_b = Arc::new(InstanceContext::new());
+        ctx_a.update_erasure_type(SetupType::DistErasure).await;
+        ctx_b.update_erasure_type(SetupType::ErasureSD).await;
+
+        assert!(ctx_a.is_dist_erasure().await && ctx_a.is_erasure().await);
+        assert!(!ctx_a.is_erasure_sd().await);
+
+        assert!(ctx_b.is_erasure_sd().await);
+        assert!(!ctx_b.is_erasure().await && !ctx_b.is_dist_erasure().await);
+    }
+
+    // Single-instance zero-change: the bootstrap context's lock manager is the
+    // very same Arc the process singleton hands out, so the lock namespace is
+    // unchanged from before Slice 3.
+    #[tokio::test]
+    async fn bootstrap_lock_manager_aliases_process_singleton() {
+        assert!(
+            Arc::ptr_eq(&bootstrap_ctx().lock_manager(), &get_global_lock_manager()),
+            "bootstrap context must alias the process lock-manager singleton"
+        );
+    }
+
+    // Two independent contexts own distinct lock managers — the property that
+    // stops two instances from falsely contending on same-named objects.
+    #[tokio::test]
+    async fn distinct_contexts_have_distinct_lock_managers() {
+        let ctx_a = InstanceContext::new();
+        let ctx_b = InstanceContext::new();
+        assert!(
+            !Arc::ptr_eq(&ctx_a.lock_manager(), &ctx_b.lock_manager()),
+            "fresh contexts must not share a lock manager"
+        );
+    }
+}

crates/ecstore/src/runtime/mod.rs

@@ -16,4 +16,5 @@
 #![allow(dead_code)]
 
 pub(crate) mod global;
+pub(crate) mod instance;
 pub(crate) mod sources;

crates/ecstore/src/runtime/sources.rs

@@ -32,13 +32,13 @@ use crate::{
     error::Result,
     layout::endpoints::{EndpointServerPools, SetupType},
     runtime::global::{
-        GLOBAL_BOOT_TIME, GLOBAL_EVENT_NOTIFIER, GLOBAL_IS_ERASURE_SD, GLOBAL_LIFECYCLE_SYS, GLOBAL_LOCAL_DISK_ID_MAP,
-        GLOBAL_LOCAL_DISK_MAP, GLOBAL_LOCAL_DISK_SET_DRIVES, GLOBAL_LOCAL_NODE_NAME_FALLBACK, GLOBAL_ROOT_DISK_THRESHOLD,
-        GLOBAL_TIER_CONFIG_MGR, TypeLocalDiskSetDrives, get_background_services_cancel_token, get_global_bucket_monitor,
-        get_global_deployment_id, get_global_endpoints, get_global_endpoints_opt, get_global_lock_client,
-        get_global_lock_clients, get_global_region, get_global_tier_config_mgr, global_rustfs_port, init_global_bucket_monitor,
-        is_dist_erasure, is_erasure, is_first_cluster_node_local, resolve_object_store_handle, set_global_deployment_id,
-        set_global_lock_client, set_global_lock_clients, set_object_layer, update_erasure_type,
+        GLOBAL_BOOT_TIME, GLOBAL_EVENT_NOTIFIER, GLOBAL_LIFECYCLE_SYS, GLOBAL_LOCAL_DISK_ID_MAP, GLOBAL_LOCAL_DISK_MAP,
+        GLOBAL_LOCAL_DISK_SET_DRIVES, GLOBAL_LOCAL_NODE_NAME_FALLBACK, GLOBAL_ROOT_DISK_THRESHOLD, GLOBAL_TIER_CONFIG_MGR,
+        TypeLocalDiskSetDrives, get_background_services_cancel_token, get_global_bucket_monitor, get_global_deployment_id,
+        get_global_endpoints, get_global_endpoints_opt, get_global_lock_client, get_global_lock_clients, get_global_region,
+        get_global_tier_config_mgr, global_rustfs_port, init_global_bucket_monitor, is_dist_erasure, is_erasure, is_erasure_sd,
+        is_first_cluster_node_local, resolve_object_store_handle, set_global_deployment_id, set_global_lock_client,
+        set_global_lock_clients, set_object_layer, update_erasure_type,
     },
     services::batch_processor::{GlobalBatchProcessors, get_global_processors},
     services::event_notification::EventNotifier,
@@ -148,7 +148,7 @@ pub async fn setup_is_dist_erasure() -> bool {
 }
 
 pub async fn setup_is_erasure_sd() -> bool {
-    *GLOBAL_IS_ERASURE_SD.read().await
+    is_erasure_sd().await
 }
 
 pub(crate) async fn current_setup_type() -> SetupType {
@@ -215,7 +215,7 @@ pub(crate) async fn scanner_init_time() -> Option<chrono::DateTime<chrono::Utc>>
 }
 
 pub(crate) async fn root_disk_threshold_for_erasure_disk() -> Option<u64> {
-    if *GLOBAL_IS_ERASURE_SD.read().await {
+    if is_erasure_sd().await {
         None
     } else {
         Some(*GLOBAL_ROOT_DISK_THRESHOLD.read().await)
@@ -288,10 +288,6 @@ pub(crate) fn ensure_deployment_id(deployment_id: Uuid) {
     }
 }
 
-pub(crate) fn global_lock_manager() -> Arc<rustfs_lock::GlobalLockManager> {
-    rustfs_lock::get_global_lock_manager()
-}
-
 pub fn global_lock_client() -> Option<Arc<dyn LockClient>> {
     get_global_lock_client()
 }