Files
rustfs/rustfs/src/app/context/global.rs
T
Zhengchao An f403ed1c2e feat(embedded): allow multiple embedded servers to coexist in one process
* feat(embedded): allow multiple embedded servers to coexist in one process

backlog#1052 S5: turn the embedded startup guard into a sequential lock
and make every write-once shared cell tolerate a second embedded start.
Two RustFSServers on different ports and volumes now start, run, and
shut down independently in the same process.

- EMBEDDED_SERVER_STARTED is released once each startup hands off; a
  second startup that runs after the first is no longer rejected.
- The second embedded server constructs its own InstanceContext instead
  of adopting the process bootstrap one, so region/endpoints/deployment
  id land on that context (the first server keeps adopting bootstrap to
  keep single-instance ambient facades unchanged).
- Startup-time tolerant paths:
  - action_credentials publish treats AlreadyInitialized as success
    (per-server ActionCredentialHandle already holds the real creds).
  - GLOBAL_RUSTFS_PORT warns instead of panicking on a second set.
  - Observability install returns Ok when the process subscriber is
    already set — the second server reuses it.
- New acceptance test proves two servers start, respond on their own
  ports, and one can be shut down without disturbing the other; the
  survivor keeps serving S3 requests. IAM and root-credential lookup
  still share a process domain (a second server whose creds differ from
  the first will fail signature validation), tracked as a follow-up.
- Embedded doc rewritten: 'Limitations' → 'Multi-instance status'; the
  AlreadyStarted error is now scoped to concurrent startups only.

The remaining work in #1052 is the auth path per-server dispatch and the
matching data-plane routing so two servers with different credentials
serve independent buckets end-to-end.

* feat(app): per-server auth and application context for multiple embedded servers (#4633)

backlog#1052 S6: each embedded server now authenticates against — and
its request path resolves — its OWN application context, so two servers
with different root credentials each accept their own access key and
reject the other's.

- AppContext is per-server: ensure_startup_after_iam constructs a fresh
  context around this server's store + IAM + KMS and installs it into the
  server's own ServerContextSlot, then publishes it as the process
  default first-writer-wins (publish_global_app_context) for legacy
  ambient readers. The old 'reuse the global if present' path is gone.
- FS::check (the S3 data-plane access gate) resolves auth against
  self.server_ctx's context: check_key_valid gains a _with_context
  variant that takes the root credentials and IAM system from an explicit
  context (None = ambient, unchanged for all 140+ existing callers). The
  region and the server context slot are published into the request
  extensions for downstream handlers.
- Each embedded server seeds its own root credentials into its context
  (ActionCredentialHandle.publish) at startup, so credential validation
  no longer falls back to the first server's process-global identity.
- The bucket/object/multipart use-cases resolve their store from the
  server's context (bucket_usecase_for/object_usecase_for/... take &FS).

New acceptance test: two servers with distinct credentials each
authenticate with their own key and reject the other's.

KNOWN FOLLOW-UP: full bucket-namespace isolation still requires threading
the instance context through the lower ecstore data plane (peer_sys /
disk registry / bucket-metadata reads still resolve via the process
GLOBAL_OBJECT_API), so the two servers do not yet present independent
bucket listings even though each holds its own store. That deeper pass —
a continuation of the #939 object-graph ctx threading — is the remaining
work on #1052.

Stacked on the S5 guard change.
2026-07-10 02:00:59 +08:00

372 lines
16 KiB
Rust

// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::super::storage_api::context::ECStore;
use super::super::storage_api::context::runtime::set_object_store_resolver;
use super::handles::{
IamHandle, KmsHandle, default_action_credential_interface, default_boot_time_interface, default_bucket_metadata_interface,
default_bucket_monitor_interface, default_buffer_config_interface, default_deployment_id_interface,
default_endpoints_interface, default_expiry_state_interface, default_internode_metrics_interface,
default_kms_runtime_interface, default_local_node_name_interface, default_lock_client_interface,
default_lock_clients_interface, default_notification_system_interface, default_notify_interface, default_oidc_interface,
default_outbound_tls_runtime_interface, default_performance_metrics_interface, default_region_interface,
default_replication_pool_interface, default_replication_stats_interface, default_runtime_port_interface,
default_s3select_db_interface, default_scanner_metrics_interface, default_server_config_interface,
default_storage_class_interface, default_tier_config_interface, default_transition_state_interface, oidc_interface,
};
use super::interfaces::{
ActionCredentialInterface, BootTimeInterface, BucketMetadataInterface, BucketMonitorInterface, BufferConfigInterface,
DeploymentIdInterface, EndpointsInterface, ExpiryStateInterface, IamInterface, InternodeMetricsInterface, KmsInterface,
KmsRuntimeInterface, LocalNodeNameInterface, LockClientInterface, LockClientsInterface, NotificationSystemInterface,
NotifyInterface, OidcInterface, OutboundTlsRuntimeInterface, PerformanceMetricsInterface, RegionInterface,
ReplicationPoolInterface, ReplicationStatsInterface, RuntimePortInterface, S3SelectDbInterface, ScannerMetricsInterface,
ServerConfigInterface, StorageClassInterface, TierConfigInterface, TransitionStateInterface,
};
use crate::app::object_data_cache::ObjectDataCacheAdapter;
use rustfs_iam::{oidc::OidcSys, store::object::ObjectStore, sys::IamSys};
use rustfs_kms::KmsServiceManager;
use std::sync::{Arc, OnceLock};
/// Application-layer context with explicit dependencies.
#[derive(Clone)]
pub struct AppContext {
object_store: Arc<ECStore>,
iam: Arc<dyn IamInterface>,
oidc: Arc<dyn OidcInterface>,
#[allow(dead_code)]
kms: Arc<dyn KmsInterface>,
kms_runtime: Arc<dyn KmsRuntimeInterface>,
outbound_tls_runtime: Arc<dyn OutboundTlsRuntimeInterface>,
notify: Arc<dyn NotifyInterface>,
notification_system: Arc<dyn NotificationSystemInterface>,
bucket_metadata: Arc<dyn BucketMetadataInterface>,
bucket_monitor: Arc<dyn BucketMonitorInterface>,
replication_pool: Arc<dyn ReplicationPoolInterface>,
replication_stats: Arc<dyn ReplicationStatsInterface>,
boot_time: Arc<dyn BootTimeInterface>,
scanner_metrics: Arc<dyn ScannerMetricsInterface>,
endpoints: Arc<dyn EndpointsInterface>,
deployment_id: Arc<dyn DeploymentIdInterface>,
runtime_port: Arc<dyn RuntimePortInterface>,
lock_client: Arc<dyn LockClientInterface>,
lock_clients: Arc<dyn LockClientsInterface>,
performance_metrics: Arc<dyn PerformanceMetricsInterface>,
internode_metrics: Arc<dyn InternodeMetricsInterface>,
s3select_db: Arc<dyn S3SelectDbInterface>,
local_node_name: Arc<dyn LocalNodeNameInterface>,
action_credentials: Arc<dyn ActionCredentialInterface>,
region: Arc<dyn RegionInterface>,
tier_config: Arc<dyn TierConfigInterface>,
expiry_state: Arc<dyn ExpiryStateInterface>,
transition_state: Arc<dyn TransitionStateInterface>,
server_config: Arc<dyn ServerConfigInterface>,
storage_class: Arc<dyn StorageClassInterface>,
buffer_config: Arc<dyn BufferConfigInterface>,
object_data_cache: Arc<ObjectDataCacheAdapter>,
}
impl AppContext {
pub fn new(object_store: Arc<ECStore>, iam: Arc<dyn IamInterface>, kms: Arc<dyn KmsInterface>) -> Self {
let object_data_cache = ObjectDataCacheAdapter::from_env_or_disabled();
// Let ecstore probe this cache inside get_object_reader, after
// metadata resolution but before the erasure data read (backlog#802).
crate::app::object_data_cache::register_object_data_cache_body_hook(Arc::clone(&object_data_cache));
Self {
object_store,
iam,
oidc: default_oidc_interface(),
kms,
kms_runtime: default_kms_runtime_interface(),
outbound_tls_runtime: default_outbound_tls_runtime_interface(),
notify: default_notify_interface(),
notification_system: default_notification_system_interface(),
bucket_metadata: default_bucket_metadata_interface(),
bucket_monitor: default_bucket_monitor_interface(),
replication_pool: default_replication_pool_interface(),
replication_stats: default_replication_stats_interface(),
boot_time: default_boot_time_interface(),
scanner_metrics: default_scanner_metrics_interface(),
endpoints: default_endpoints_interface(),
deployment_id: default_deployment_id_interface(),
runtime_port: default_runtime_port_interface(),
lock_client: default_lock_client_interface(),
lock_clients: default_lock_clients_interface(),
performance_metrics: default_performance_metrics_interface(),
internode_metrics: default_internode_metrics_interface(),
s3select_db: default_s3select_db_interface(),
local_node_name: default_local_node_name_interface(),
action_credentials: default_action_credential_interface(),
region: default_region_interface(),
tier_config: default_tier_config_interface(),
expiry_state: default_expiry_state_interface(),
transition_state: default_transition_state_interface(),
server_config: default_server_config_interface(),
storage_class: default_storage_class_interface(),
buffer_config: default_buffer_config_interface(),
object_data_cache,
}
}
pub fn with_default_interfaces(
object_store: Arc<ECStore>,
iam: Arc<IamSys<ObjectStore>>,
kms: Arc<KmsServiceManager>,
) -> Self {
let mut context = Self::new(object_store, Arc::new(IamHandle::new(iam)), Arc::new(KmsHandle::new(kms)));
context.oidc = oidc_interface(super::runtime_sources::oidc_handle());
context
}
pub fn object_store(&self) -> Arc<ECStore> {
self.object_store.clone()
}
pub fn iam(&self) -> Arc<dyn IamInterface> {
self.iam.clone()
}
pub fn oidc(&self) -> Arc<dyn OidcInterface> {
self.oidc.clone()
}
pub fn publish_oidc_handle(&self, oidc: Arc<OidcSys>) -> bool {
self.oidc.publish_handle(oidc)
}
#[allow(dead_code)]
pub fn kms(&self) -> Arc<dyn KmsInterface> {
self.kms.clone()
}
pub fn kms_runtime(&self) -> Arc<dyn KmsRuntimeInterface> {
self.kms_runtime.clone()
}
pub fn outbound_tls_runtime(&self) -> Arc<dyn OutboundTlsRuntimeInterface> {
self.outbound_tls_runtime.clone()
}
pub fn notify(&self) -> Arc<dyn NotifyInterface> {
self.notify.clone()
}
pub fn notification_system(&self) -> Arc<dyn NotificationSystemInterface> {
self.notification_system.clone()
}
pub fn bucket_metadata(&self) -> Arc<dyn BucketMetadataInterface> {
self.bucket_metadata.clone()
}
pub fn bucket_monitor(&self) -> Arc<dyn BucketMonitorInterface> {
self.bucket_monitor.clone()
}
pub(crate) fn replication_pool(&self) -> Arc<dyn ReplicationPoolInterface> {
self.replication_pool.clone()
}
pub(crate) fn replication_stats(&self) -> Arc<dyn ReplicationStatsInterface> {
self.replication_stats.clone()
}
pub fn boot_time(&self) -> Arc<dyn BootTimeInterface> {
self.boot_time.clone()
}
pub fn scanner_metrics(&self) -> Arc<dyn ScannerMetricsInterface> {
self.scanner_metrics.clone()
}
pub fn endpoints(&self) -> Arc<dyn EndpointsInterface> {
self.endpoints.clone()
}
pub fn deployment_id(&self) -> Arc<dyn DeploymentIdInterface> {
self.deployment_id.clone()
}
pub fn runtime_port(&self) -> Arc<dyn RuntimePortInterface> {
self.runtime_port.clone()
}
pub fn lock_client(&self) -> Arc<dyn LockClientInterface> {
self.lock_client.clone()
}
pub fn lock_clients(&self) -> Arc<dyn LockClientsInterface> {
self.lock_clients.clone()
}
pub fn performance_metrics(&self) -> Arc<dyn PerformanceMetricsInterface> {
self.performance_metrics.clone()
}
pub fn internode_metrics(&self) -> Arc<dyn InternodeMetricsInterface> {
self.internode_metrics.clone()
}
pub fn s3select_db(&self) -> Arc<dyn S3SelectDbInterface> {
self.s3select_db.clone()
}
pub fn local_node_name(&self) -> Arc<dyn LocalNodeNameInterface> {
self.local_node_name.clone()
}
pub fn action_credentials(&self) -> Arc<dyn ActionCredentialInterface> {
self.action_credentials.clone()
}
/// Publish this context's own root credentials (backlog#1052 S6).
pub fn publish_action_credentials(&self, credentials: rustfs_credentials::Credentials) -> bool {
self.action_credentials.publish(credentials)
}
pub fn region(&self) -> Arc<dyn RegionInterface> {
self.region.clone()
}
pub fn tier_config(&self) -> Arc<dyn TierConfigInterface> {
self.tier_config.clone()
}
pub fn expiry_state(&self) -> Arc<dyn ExpiryStateInterface> {
self.expiry_state.clone()
}
pub fn transition_state(&self) -> Arc<dyn TransitionStateInterface> {
self.transition_state.clone()
}
pub fn server_config(&self) -> Arc<dyn ServerConfigInterface> {
self.server_config.clone()
}
pub fn storage_class(&self) -> Arc<dyn StorageClassInterface> {
self.storage_class.clone()
}
pub fn buffer_config(&self) -> Arc<dyn BufferConfigInterface> {
self.buffer_config.clone()
}
pub(crate) fn object_data_cache(&self) -> Arc<ObjectDataCacheAdapter> {
Arc::clone(&self.object_data_cache)
}
}
#[cfg(test)]
pub(super) struct AppContextTestInterfaces {
pub(super) iam: Arc<dyn IamInterface>,
pub(super) oidc: Arc<dyn OidcInterface>,
pub(super) kms: Arc<dyn KmsInterface>,
pub(super) kms_runtime: Arc<dyn KmsRuntimeInterface>,
pub(super) outbound_tls_runtime: Arc<dyn OutboundTlsRuntimeInterface>,
pub(super) notify: Arc<dyn NotifyInterface>,
pub(super) notification_system: Arc<dyn NotificationSystemInterface>,
pub(super) bucket_metadata: Arc<dyn BucketMetadataInterface>,
pub(super) bucket_monitor: Arc<dyn BucketMonitorInterface>,
pub(super) replication_pool: Arc<dyn ReplicationPoolInterface>,
pub(super) replication_stats: Arc<dyn ReplicationStatsInterface>,
pub(super) boot_time: Arc<dyn BootTimeInterface>,
pub(super) scanner_metrics: Arc<dyn ScannerMetricsInterface>,
pub(super) endpoints: Arc<dyn EndpointsInterface>,
pub(super) deployment_id: Arc<dyn DeploymentIdInterface>,
pub(super) runtime_port: Arc<dyn RuntimePortInterface>,
pub(super) lock_client: Arc<dyn LockClientInterface>,
pub(super) lock_clients: Arc<dyn LockClientsInterface>,
pub(super) performance_metrics: Arc<dyn PerformanceMetricsInterface>,
pub(super) internode_metrics: Arc<dyn InternodeMetricsInterface>,
pub(super) s3select_db: Arc<dyn S3SelectDbInterface>,
pub(super) local_node_name: Arc<dyn LocalNodeNameInterface>,
pub(super) action_credentials: Arc<dyn ActionCredentialInterface>,
pub(super) region: Arc<dyn RegionInterface>,
pub(super) tier_config: Arc<dyn TierConfigInterface>,
pub(super) expiry_state: Arc<dyn ExpiryStateInterface>,
pub(super) transition_state: Arc<dyn TransitionStateInterface>,
pub(super) server_config: Arc<dyn ServerConfigInterface>,
pub(super) storage_class: Arc<dyn StorageClassInterface>,
pub(super) buffer_config: Arc<dyn BufferConfigInterface>,
}
#[cfg(test)]
impl AppContext {
pub(super) fn with_test_interfaces(object_store: Arc<ECStore>, interfaces: AppContextTestInterfaces) -> Self {
Self {
object_store,
iam: interfaces.iam,
oidc: interfaces.oidc,
kms: interfaces.kms,
kms_runtime: interfaces.kms_runtime,
outbound_tls_runtime: interfaces.outbound_tls_runtime,
notify: interfaces.notify,
notification_system: interfaces.notification_system,
bucket_metadata: interfaces.bucket_metadata,
bucket_monitor: interfaces.bucket_monitor,
replication_pool: interfaces.replication_pool,
replication_stats: interfaces.replication_stats,
boot_time: interfaces.boot_time,
scanner_metrics: interfaces.scanner_metrics,
endpoints: interfaces.endpoints,
deployment_id: interfaces.deployment_id,
runtime_port: interfaces.runtime_port,
lock_client: interfaces.lock_client,
lock_clients: interfaces.lock_clients,
performance_metrics: interfaces.performance_metrics,
internode_metrics: interfaces.internode_metrics,
s3select_db: interfaces.s3select_db,
local_node_name: interfaces.local_node_name,
action_credentials: interfaces.action_credentials,
region: interfaces.region,
tier_config: interfaces.tier_config,
expiry_state: interfaces.expiry_state,
transition_state: interfaces.transition_state,
server_config: interfaces.server_config,
storage_class: interfaces.storage_class,
buffer_config: interfaces.buffer_config,
object_data_cache: ObjectDataCacheAdapter::disabled_arc(),
}
}
}
static APP_CONTEXT_SINGLETON: OnceLock<Arc<AppContext>> = OnceLock::new();
/// Initialize global application context once and return the canonical instance.
pub fn init_global_app_context(context: AppContext) -> Arc<AppContext> {
publish_global_app_context(Arc::new(context))
}
/// Publish an already-constructed application context as the process default,
/// first-writer-wins (backlog#1052 S6).
///
/// A per-server startup constructs its own context and installs it into its
/// own [`ServerContextSlot`](super::server_slot::ServerContextSlot); it also
/// calls this so the *first* server's context becomes the ambient default that
/// legacy free-function readers resolve. Later servers publish too, but the
/// `OnceLock` keeps the first — their own slot already carries their context,
/// so their request path stays isolated regardless.
pub fn publish_global_app_context(context: Arc<AppContext>) -> Arc<AppContext> {
let canonical = APP_CONTEXT_SINGLETON.get_or_init(|| context).clone();
let resolver_context = canonical.clone();
let _ = set_object_store_resolver(Arc::new(move || Some(resolver_context.object_store())));
canonical
}
/// Get global application context if it has been initialized.
pub fn get_global_app_context() -> Option<Arc<AppContext>> {
APP_CONTEXT_SINGLETON.get().cloned()
}