gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-03 10:48:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type (#4413)

Browse Source 
* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type

Phase 5 of the global-singleton consolidation (backlog#939): begin moving
runtime identity state out of process globals so multiple ECStore instances
can coexist in one process. Isolation is carried by the object graph
(ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a
task-local, which does not propagate across the many internal tokio::spawn
boundaries in the data/background paths.

This first slice migrates the erasure setup type -- previously three
independent process-global bools -- into a single per-instance
RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd,
removing a triple source of truth that could drift out of sync.

- New runtime::instance module: InstanceContext + process bootstrap context.
- The legacy free-function facade (is_erasure/update_erasure_type/...) keeps
  its signatures and forwards to the current instance's context, falling back
  to the bootstrap context before a store is published.
- ECStore gains a pub(crate) ctx field and setup_is_* accessors; its
  constructors adopt the bootstrap context (never mint a fresh one) so startup
  writes and post-construction reads share one cell -- single-instance
  behavior is byte-for-byte unchanged.

Tests: erasure predicate derivation vs the legacy behavior, object-graph
carrier isolation across two ECStore instances, and bootstrap adoption.

Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8)

* refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415)

* refactor(ecstore): source the namespace lock manager per-instance (#4417)

refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3)

Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by
sourcing SetDisks' lock manager from the instance context instead of the
process singleton. This removes the false cross-instance mutual exclusion (and
attendant ABBA risk) that a shared GlobalLockManager would cause once multiple
instances coexist.

- InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints
  a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the
  process singleton via get_global_lock_manager(), so a single-instance
  deployment keeps exactly one shared namespace.
- SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx
  it already adopts), not `runtime_sources::global_lock_manager()`. Single
  instance: same Arc as before, so behavior is unchanged.
- Remove the now-unused `runtime_sources::global_lock_manager()` wrapper.

Tests: bootstrap lock manager aliases the process singleton; two fresh contexts
own distinct managers; a SetDisks' lock manager is the one from its context and
aliases the global singleton in a single-instance build.

Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking
regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean),
make pre-commit (pass).

Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2).
This commit is contained in:
Zhengchao An
2026-07-08 15:01:40 +08:00
committed by GitHub
parent cda7688909
commit 91dec123d9
10 changed files with 444 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/ecstore/src/runtime/global.rs
+23 -22
View File
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::instance::{InstanceContext, bootstrap_ctx};
use crate::bucket::bandwidth::monitor::Monitor;
use crate::{
bucket::lifecycle::bucket_lifecycle_ops::LifecycleSys,
@@ -42,16 +43,17 @@ pub const DISK_RESERVE_FRACTION: f64 = 0.15;
// These should be migrated to AppContext over time.
// See issue #730 for migration plan.
//
// Tier A (needs migration): GLOBAL_OBJECT_API, GLOBAL_IS_ERASURE*, GLOBAL_LOCAL_DISK_*,
// Tier A (needs migration): GLOBAL_OBJECT_API, GLOBAL_LOCAL_DISK_*,
// GLOBAL_ROOT_DISK_THRESHOLD, GLOBAL_LIFECYCLE_SYS, GLOBAL_EVENT_NOTIFIER, etc.
// Tier B (keep as static): GLOBAL_RUSTFS_PORT, GLOBAL_REGION, env var caches, etc.
//
// Phase 5 (backlog#939): the erasure setup type moved into the per-instance
// `InstanceContext` (see `super::instance`); the erasure predicates below now
// forward to the current instance's context.
lazy_static! {
static ref GLOBAL_RUSTFS_PORT: OnceLock<u16> = OnceLock::new();
static ref GLOBAL_DEPLOYMENT_ID: OnceLock<Uuid> = OnceLock::new();
pub static ref GLOBAL_OBJECT_API: OnceLock<Arc<ECStore>> = OnceLock::new();
pub static ref GLOBAL_IS_ERASURE: RwLock<bool> = RwLock::new(false);
pub static ref GLOBAL_IS_DIST_ERASURE: RwLock<bool> = RwLock::new(false);
pub static ref GLOBAL_IS_ERASURE_SD: RwLock<bool> = RwLock::new(false);
pub static ref GLOBAL_LOCAL_DISK_MAP: Arc<RwLock<HashMap<String, Option<DiskStore>>>> = Arc::new(RwLock::new(HashMap::new()));
pub static ref GLOBAL_LOCAL_DISK_ID_MAP: Arc<RwLock<HashMap<Uuid, String>>> = Arc::new(RwLock::new(HashMap::new()));
pub static ref GLOBAL_LOCAL_DISK_SET_DRIVES: Arc<RwLock<TypeLocalDiskSetDrives>> = Arc::new(RwLock::new(Vec::new()));
@@ -207,6 +209,19 @@ pub fn resolve_object_store_handle() -> Option<Arc<ECStore>> {
.or_else(new_object_layer_fn)
}
/// Resolve the instance context for the legacy free-function facade.
///
/// Prefers the currently-published `ECStore`'s own context; before any store
/// is published (e.g. during storage startup, or in unit tests) it falls back
/// to the process-level [`bootstrap_ctx`]. Because `ECStore::new` adopts the
/// bootstrap `Arc`, single-instance callers always observe one and the same
/// context — behavior is unchanged from the previous process-global bools.
pub(crate) fn current_ctx() -> Arc<InstanceContext> {
resolve_object_store_handle()
.map(|store| store.ctx.clone())
.unwrap_or_else(bootstrap_ctx)
}
/// Set the global object layer
///
/// # Arguments
@@ -226,8 +241,7 @@ pub async fn set_object_layer(o: Arc<ECStore>) {
/// * `bool` - True if the setup type is distributed erasure coding, false otherwise
///
pub async fn is_dist_erasure() -> bool {
let lock = GLOBAL_IS_DIST_ERASURE.read().await;
*lock
current_ctx().is_dist_erasure().await
}
/// Check if the setup type is erasure coding with single data center
@@ -236,8 +250,7 @@ pub async fn is_dist_erasure() -> bool {
/// * `bool` - True if the setup type is erasure coding with single data center, false otherwise
///
pub async fn is_erasure_sd() -> bool {
let lock = GLOBAL_IS_ERASURE_SD.read().await;
*lock
current_ctx().is_erasure_sd().await
}
/// Check if the setup type is erasure coding
@@ -246,8 +259,7 @@ pub async fn is_erasure_sd() -> bool {
/// * `bool` - True if the setup type is erasure coding, false otherwise
///
pub async fn is_erasure() -> bool {
let lock = GLOBAL_IS_ERASURE.read().await;
*lock
current_ctx().is_erasure().await
}
/// Update the global erasure type based on the setup type
@@ -258,18 +270,7 @@ pub async fn is_erasure() -> bool {
/// # Returns
/// * None
pub async fn update_erasure_type(setup_type: SetupType) {
let mut is_erasure = GLOBAL_IS_ERASURE.write().await;
*is_erasure = setup_type == SetupType::Erasure;
let mut is_dist_erasure = GLOBAL_IS_DIST_ERASURE.write().await;
*is_dist_erasure = setup_type == SetupType::DistErasure;
if *is_dist_erasure {
*is_erasure = true
}
let mut is_erasure_sd = GLOBAL_IS_ERASURE_SD.write().await;
*is_erasure_sd = setup_type == SetupType::ErasureSD;
current_ctx().update_erasure_type(setup_type).await;
}
// pub fn is_legacy() -> bool {