refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type (#4413)

* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type

Phase 5 of the global-singleton consolidation (backlog#939): begin moving
runtime identity state out of process globals so multiple ECStore instances
can coexist in one process. Isolation is carried by the object graph
(ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a
task-local, which does not propagate across the many internal tokio::spawn
boundaries in the data/background paths.

This first slice migrates the erasure setup type -- previously three
independent process-global bools -- into a single per-instance
RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd,
removing a triple source of truth that could drift out of sync.

- New runtime::instance module: InstanceContext + process bootstrap context.
- The legacy free-function facade (is_erasure/update_erasure_type/...) keeps
  its signatures and forwards to the current instance's context, falling back
  to the bootstrap context before a store is published.
- ECStore gains a pub(crate) ctx field and setup_is_* accessors; its
  constructors adopt the bootstrap context (never mint a fresh one) so startup
  writes and post-construction reads share one cell -- single-instance
  behavior is byte-for-byte unchanged.

Tests: erasure predicate derivation vs the legacy behavior, object-graph
carrier isolation across two ECStore instances, and bootstrap adoption.

Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8)

* refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415)

* refactor(ecstore): source the namespace lock manager per-instance (#4417)

refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3)

Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by
sourcing SetDisks' lock manager from the instance context instead of the
process singleton. This removes the false cross-instance mutual exclusion (and
attendant ABBA risk) that a shared GlobalLockManager would cause once multiple
instances coexist.

- InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints
  a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the
  process singleton via get_global_lock_manager(), so a single-instance
  deployment keeps exactly one shared namespace.
- SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx
  it already adopts), not `runtime_sources::global_lock_manager()`. Single
  instance: same Arc as before, so behavior is unchanged.
- Remove the now-unused `runtime_sources::global_lock_manager()` wrapper.

Tests: bootstrap lock manager aliases the process singleton; two fresh contexts
own distinct managers; a SetDisks' lock manager is the one from its context and
aliases the global singleton in a single-instance build.

Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking
regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean),
make pre-commit (pass).

Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2).
This commit is contained in:
Zhengchao An
2026-07-08 15:01:40 +08:00
committed by GitHub
parent cda7688909
commit 91dec123d9
10 changed files with 444 additions and 37 deletions
+20
View File
@@ -33,6 +33,7 @@ use crate::{
error::StorageError,
layout::endpoints::{Endpoints, PoolEndpoints},
object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader},
runtime::instance::{InstanceContext, bootstrap_ctx},
runtime::sources as runtime_sources,
set_disk::SetDisks,
store::init_format::{check_format_erasure_values, get_format_erasure_in_quorum, load_format_erasure_all, save_format_file},
@@ -77,6 +78,12 @@ pub struct Sets {
pub default_parity_count: usize,
pub distribution_algo: DistributionAlgoVersion,
exit_signal: Option<Sender<()>>,
/// Per-instance runtime context (Phase 5, backlog#939).
///
/// Carried down the object graph (ECStore → Sets → SetDisks) so that
/// instance-scoped state resolves through the owning instance rather than a
/// process global. Consumed starting Slice 3 (lock-namespace isolation).
ctx: Arc<InstanceContext>,
}
impl Drop for Sets {
@@ -186,6 +193,10 @@ impl Sets {
default_parity_count: parity_count,
distribution_algo: fm.erasure.distribution_algo.clone(),
exit_signal: Some(tx),
// Single-instance: same bootstrap context the owning ECStore adopts
// (constructed before the store, so sourced here directly). Slice 8
// threads a per-instance context in for true multi-instance.
ctx: bootstrap_ctx(),
});
let asets = sets.clone();
@@ -200,6 +211,12 @@ impl Sets {
self.set_drive_count
}
/// This pool's per-instance runtime context (Phase 5, backlog#939).
#[allow(dead_code)] // Consumed starting Slice 3 (lock-namespace isolation).
pub(crate) fn instance_ctx(&self) -> &Arc<InstanceContext> {
&self.ctx
}
pub async fn monitor_and_connect_endpoints(&self, mut rx: Receiver<()>) {
tokio::time::sleep(Duration::from_secs(5)).await;
@@ -1165,6 +1182,7 @@ mod tests {
default_parity_count: 1,
distribution_algo: DistributionAlgoVersion::V1,
exit_signal: None,
ctx: bootstrap_ctx(),
};
let result = sets
@@ -1244,6 +1262,7 @@ mod tests {
default_parity_count: 1,
distribution_algo: DistributionAlgoVersion::V1,
exit_signal: None,
ctx: bootstrap_ctx(),
});
let bucket = format!("bucket-{}", Uuid::new_v4().simple());
@@ -1290,6 +1309,7 @@ mod tests {
default_parity_count: 0,
distribution_algo: DistributionAlgoVersion::V1,
exit_signal: None,
ctx: bootstrap_ctx(),
};
let err = sets