refactor: route runtime consumers through app context (#3785)

This commit is contained in:
Zhengchao An
2026-06-23 17:21:34 +08:00
committed by GitHub
parent a62e2b6c94
commit 61cfd4fc13
21 changed files with 723 additions and 85 deletions
+237 -6
View File
@@ -5,13 +5,15 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
## Current Context
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
- Branch: `overtrue/arch-auth-credentials-context`
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170`.
- Based on: API-169 PR #3782 after merge.
- Branch: `overtrue/arch-runtime-context-consumer-batch`
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177`.
- Based on: API-170 merged in PR #3783; this branch batches API-171 through
API-177 on top of latest `main`.
- PR type for this branch: `consumer-migration`
- Runtime behavior changes: none.
- Rust code changes: route auth, protocol storage client, and storage helper
action-credential reads through the AppContext action-credential resolver.
- Rust code changes: route replication pool, outbound TLS generation, runtime
region, KMS encryption service, runtime support handles, S3 Select DB, and
internode RPC metrics through AppContext-first resolvers.
- CI/script changes: lock completed owner and test/fuzz boundaries against
bare/glob imports, scattered raw ECStore facade subpaths, and startup
runtime/root-server/table/S3/app shared/app bucket/app ECStore/admin facade
@@ -20,7 +22,7 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
and storage owner thin bridge regressions, plus app context and notify
event-bridge thin module regressions; accept the reviewed AppContext resolver
reverse dependencies in the layer baseline.
- Docs changes: record the API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170 owner facade cleanup.
- Docs changes: record the API-136 through API-177 owner facade cleanup.
## Phase 0 Tasks
@@ -4460,6 +4462,100 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
credential scan, Rust risk scan, branch freshness check, and three-expert
review.
- [x] `API-171` Route runtime replication pool reads through AppContext.
- Do: route bucket metadata startup resync and workload admission
replication pool reads through the AppContext replication-pool resolver.
- Acceptance: production startup/workload admission consumers no longer read
the replication pool directly from the storage global facade, while the
AppContext default adapter preserves the legacy global fallback.
- Must preserve: bucket metadata resync initialization, replication admission
active worker counts, queued replication counts, and unknown-runtime
reporting.
- Verification: RustFS compile coverage, targeted context resolver tests,
migration guard, layer guard, formatting, diff hygiene, residual
replication-pool scan, Rust risk scan, branch freshness check, and
three-expert review.
- [x] `API-172` Route outbound TLS generation reads through AppContext.
- Do: route startup TLS material initialization and TLS reload loop
generation reads through the AppContext outbound TLS generation resolver.
- Acceptance: production startup/reload paths no longer read outbound TLS
generation directly from `rustfs_common`, while the AppContext default
adapter preserves the legacy global fallback.
- Must preserve: generation increment semantics, outbound TLS state publish,
TLS generation metrics, reload-loop enrichment, and TLS acceptor rebuilds.
- Verification: RustFS compile coverage, targeted context resolver tests,
migration guard, layer guard, formatting, diff hygiene, residual outbound
TLS generation scan, Rust risk scan, branch freshness check, and
three-expert review.
- [x] `API-173` Route runtime region reads through AppContext.
- Do: route bucket notification setup and S3 request context region reads
through the AppContext region resolver.
- Acceptance: production init/storage request paths no longer read region
directly from the storage global facade, while the AppContext default
adapter preserves the legacy global fallback.
- Must preserve: notification ARN target mapping fallback region behavior,
request context region propagation, auth/policy request construction, and
existing startup region setters.
- Verification: RustFS compile coverage, targeted context resolver tests,
migration guard, layer guard, formatting, diff hygiene, residual region
scan, Rust risk scan, branch freshness check, and three-expert review.
- [x] `API-174` Route KMS encryption service reads through AppContext.
- Do: route app bucket encryption defaults and storage SSE managed
encryption/decryption provider reads through an AppContext-first KMS
encryption service resolver.
- Acceptance: production app/storage paths no longer read the KMS encryption
service directly from the global service manager, while the resolver
preserves the legacy global fallback.
- Must preserve: default SSE-KMS key population, managed SSE encryption and
decryption metadata handling, DEK provider selection, and KMS service
initialization fallback semantics.
- Verification: RustFS compile coverage, targeted SSE/KMS tests, migration
guard, layer guard, formatting, diff hygiene, residual encryption-service
scan, Rust risk scan, branch freshness check, and three-expert review.
- [x] `API-175` Route runtime support reads through AppContext.
- Do: route runtime readiness lock-client collections, storage concurrency
performance metrics, and config-info buffer profile reads through
AppContext-first resolvers.
- Acceptance: production readiness, storage concurrency, and config-info paths
no longer read those runtime globals directly, while default adapters
preserve the legacy global fallbacks.
- Must preserve: distributed readiness lock quorum aggregation, performance
metric singleton sharing, workload profile display output, and existing
buffer-profile enablement behavior.
- Verification: RustFS compile coverage, targeted context resolver tests,
migration guard, layer guard, formatting, diff hygiene, residual global-read
scans, Rust risk scan, branch freshness check, and three-expert review.
- [x] `API-176` Route S3 Select DB factory reads through AppContext.
- Do: route S3 Select object execution database creation through an
AppContext-first S3 Select DB resolver.
- Acceptance: production S3 Select object execution no longer reads the
S3 Select DB factory directly, while the default adapter preserves the
cached global component behavior.
- Must preserve: request validation, preflight object metadata checks,
DataFusion execution flow, output event streaming, and cached S3 Select
component reuse.
- Verification: RustFS compile coverage, targeted context resolver tests,
migration guard, layer guard, formatting, diff hygiene, residual S3 Select
DB scan, Rust risk scan, branch freshness check, and three-expert review.
- [x] `API-177` Route internode RPC metrics through AppContext.
- Do: route HTTP and gRPC internode RPC metric recording through an
AppContext-first internode metrics resolver.
- Acceptance: production internode HTTP and disk RPC paths no longer read
the internode metrics singleton directly, while the default adapter
preserves the shared global metrics instance.
- Must preserve: HTTP read/write/walk counters, gRPC read/write counters,
byte accounting, classified transport backend labels, and error recording.
- Verification: RustFS compile coverage, targeted context resolver tests,
migration guard, layer guard, formatting, diff hygiene, residual internode
metrics scan, Rust risk scan, branch freshness check, and three-expert
review.
## Next PRs
1. `consumer-migration`: continue reducing direct global reads behind AppContext resolver boundaries.
@@ -4526,11 +4622,146 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
| Quality/architecture | pass | API-170 keeps action-credential reads behind the AppContext action-credential resolver across auth, protocols, and storage helper paths. |
| Migration preservation | pass | Owner checks, claim validation, policy principal classification, protocol metadata, and audit access-key enrichment preserve existing fallback behavior. |
| Testing/verification | pass | RustFS focused compile, targeted context tests, formatting, migration/layer guards, diff hygiene, residual action credential scan, and Rust risk scan passed for API-170. |
| Quality/architecture | pass | API-171 keeps replication pool reads behind the AppContext replication-pool resolver in startup and workload admission paths. |
| Migration preservation | pass | Bucket metadata resync, replication worker counts, queue counts, and unknown-runtime reporting preserve existing fallback behavior. |
| Testing/verification | pass | RustFS focused compile, workload admission tests, targeted context tests, formatting, migration/layer guards, diff hygiene, residual replication-pool scan, and Rust risk scan passed for API-171. |
| Quality/architecture | pass | API-172 keeps outbound TLS generation reads behind the AppContext outbound TLS generation resolver in startup and reload paths. |
| Migration preservation | pass | Generation increments, outbound TLS publication, generation metrics, reload enrichment, and acceptor rebuild behavior preserve existing semantics. |
| Testing/verification | pass | RustFS focused compile, TLS generation test, targeted context test, formatting, migration/layer guards, diff hygiene, residual outbound TLS generation scan, and Rust risk scan passed for API-172. |
| Quality/architecture | pass | API-173 keeps region reads behind the AppContext region resolver in notification setup and storage request context paths. |
| Migration preservation | pass | Notification fallback region, request context propagation, auth/policy request construction, and startup setters preserve existing behavior. |
| Testing/verification | pass | RustFS focused compile, targeted context test, formatting, migration/layer guards, diff hygiene, residual region scan, and Rust risk scan passed for API-173. |
| Quality/architecture | pass | API-174 keeps app/storage KMS encryption service reads behind an AppContext-first resolver using the existing KMS runtime manager boundary. |
| Migration preservation | pass | Default SSE-KMS key population, managed SSE metadata handling, DEK provider selection, and legacy global fallback behavior are preserved. |
| Testing/verification | pass | RustFS focused compile, targeted SSE/KMS and bucket encryption tests, formatting, migration/layer guards, diff hygiene, residual encryption-service scan, and Rust risk scan passed for API-174. |
| Quality/architecture | pass | API-175 keeps readiness lock clients, storage performance metrics, and config-info buffer profile reads behind AppContext-first resolvers. |
| Migration preservation | pass | Lock quorum aggregation, performance metrics sharing, workload profile display, and legacy global fallback behavior are preserved. |
| Testing/verification | pass | RustFS focused compile, targeted context resolver test, formatting, migration/layer guards, diff hygiene, residual global-read scans, and Rust risk scan passed for API-175. |
| Quality/architecture | pass | API-176 keeps S3 Select DB factory access behind an AppContext-first resolver in object select execution. |
| Migration preservation | pass | Request validation, object preflight, query execution, event streaming, and cached component fallback behavior are preserved. |
| Testing/verification | pass | RustFS focused compile, targeted context resolver test, formatting, migration/layer guards, diff hygiene, residual S3 Select DB scan, and Rust risk scan passed for API-176. |
| Quality/architecture | pass | API-177 keeps internode RPC metrics behind an AppContext-first resolver across HTTP and gRPC RPC paths. |
| Migration preservation | pass | Request counters, byte accounting, transport backend labels, and error metrics preserve existing global fallback behavior. |
| Testing/verification | pass | RustFS focused compile, targeted context resolver test, formatting, migration/layer guards, diff hygiene, residual internode metrics scan, and Rust risk scan passed for API-177. |
## Verification Notes
Passed before push:
- Issue #660 API-176 current slice:
- `cargo check --tests -p rustfs`: passed.
- `cargo test -p rustfs resolver_helpers_are_context_first_and_fallback_when_context_is_absent --lib`:
passed.
- `cargo fmt --all`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- S3 Select DB scan: passed; direct production `get_global_db` reads are
removed from S3 Select object execution.
- Rust risk scan: no new production unwrap/expect, panic/todo/unsafe, or cast
risks added.
- Issue #660 API-177 current slice:
- `cargo check --tests -p rustfs`: passed.
- `cargo test -p rustfs resolver_helpers_are_context_first_and_fallback_when_context_is_absent --lib`:
passed.
- `cargo fmt --all`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- Internode metrics scan: passed; direct production
`global_internode_metrics` reads are removed from RustFS HTTP and disk RPC
paths.
- Rust risk scan: no new production unwrap/expect, panic/todo/unsafe, or cast
risks added.
- Issue #660 API-175 current slice:
- `cargo check --tests -p rustfs`: passed.
- `cargo test -p rustfs resolver_helpers_are_context_first_and_fallback_when_context_is_absent --lib`:
passed.
- `cargo fmt --all`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- Runtime support scan: passed; direct production lock-client collection,
performance metrics, and config-info buffer profile reads now go through
AppContext resolvers.
- Rust risk scan: no new production unwrap/expect, panic/todo/unsafe, or cast
risks added.
- Issue #660 API-174 current slice:
- `cargo check --tests -p rustfs`: passed.
- `cargo test -p rustfs test_kms_sse_dek_provider_uses_latest_reconfigured_service --lib`:
passed.
- `cargo test -p rustfs test_sse_encryption_fails_closed_without_local_sse_master_key --lib`:
passed.
- `cargo test -p rustfs execute_put_bucket_encryption_returns_internal_error_when_store_uninitialized --lib`:
passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- AppContext KMS encryption service scan: passed; direct production
`get_global_encryption_service` reads are removed from app bucket
encryption and storage SSE paths.
- Rust risk scan: no new production unwrap/expect, panic/todo/unsafe, or cast
risks added.
- Issue #660 API-173 current slice:
- `cargo check --tests -p rustfs`: passed.
- `cargo test -p rustfs resolver_helpers_are_context_first_and_fallback_when_context_is_absent --lib`:
passed.
- `cargo fmt --all`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- AppContext region scan: passed; direct production `get_global_region`
reads are removed from notification setup and storage request context
paths.
- Rust risk scan: no new production unwrap/expect, panic/todo/unsafe, or cast
risks added.
- Issue #660 API-172 current slice:
- `cargo check --tests -p rustfs`: passed.
- `cargo test -p rustfs next_tls_generation --lib`: passed.
- `cargo test -p rustfs resolver_helpers_are_context_first_and_fallback_when_context_is_absent --lib`:
passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- AppContext outbound TLS generation scan: passed; direct production
`get_global_outbound_tls_generation` reads are removed from startup TLS
material and TLS reload paths.
- Rust risk scan: no new production unwrap/expect, panic/todo/unsafe, or cast
risks added.
- Issue #660 API-171 current slice:
- `cargo check --tests -p rustfs`: passed.
- `cargo test -p rustfs workload_admission --lib`: passed.
- `cargo test -p rustfs resolver_helpers_are_context_first_and_fallback_when_context_is_absent --lib`:
passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- AppContext replication pool scan: passed; direct production
`get_global_replication_pool` reads are removed from bucket metadata
startup and workload admission paths.
- Rust risk scan: no new production unwrap/expect, panic/todo/unsafe, or cast
risks added.
- Issue #660 API-170 current slice:
- `cargo check --tests -p rustfs`: passed.
- `cargo test -p rustfs resolver_helpers_are_context_first_and_fallback_when_context_is_absent --lib`:
+3 -3
View File
@@ -39,8 +39,8 @@ use crate::admin::handlers::site_replication::{
site_replication_bucket_meta_hook, site_replication_delete_bucket_hook, site_replication_make_bucket_hook,
};
use crate::app::context::{
AppContext, get_global_app_context, resolve_notification_system, resolve_notify_interface_for_context,
resolve_object_store_handle_for_context,
AppContext, get_global_app_context, resolve_encryption_service, resolve_notification_system,
resolve_notify_interface_for_context, resolve_object_store_handle_for_context,
};
use crate::auth::get_condition_values_with_client_info;
use crate::error::ApiError;
@@ -1560,7 +1560,7 @@ impl DefaultBucketUsecase {
&& by_default.sse_algorithm.as_str() == ServerSideEncryption::AWS_KMS
&& by_default.kms_master_key_id.as_deref().is_none_or(str::is_empty)
{
let service = rustfs_kms::service_manager::get_global_encryption_service()
let service = resolve_encryption_service()
.await
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "KMS service not initialized".to_string()))?;
let default_key = service
+256 -5
View File
@@ -38,10 +38,13 @@ use crate::config::RustFSBufferConfig;
use rustfs_config::server_config::Config;
use rustfs_credentials::Credentials;
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
use rustfs_kms::{KmsServiceManager, init_global_kms_service_manager};
use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics};
use rustfs_kms::{KmsServiceManager, ObjectEncryptionService, init_global_kms_service_manager};
use rustfs_lock::LockClient;
use rustfs_s3select_api::{QueryResult, server::dbms::DatabaseManagerSystem};
use rustfs_tls_runtime::{GlobalPublishedOutboundTlsState, TlsGeneration};
use std::{future::Future, sync::Arc, time::SystemTime};
use s3s::dto::SelectObjectContentInput;
use std::{collections::HashMap, future::Future, sync::Arc, time::SystemTime};
use tokio::sync::RwLock;
/// Resolve KMS runtime service manager using AppContext-first precedence.
@@ -54,6 +57,11 @@ pub fn resolve_or_init_kms_runtime_service_manager() -> Arc<KmsServiceManager> {
resolve_or_init_kms_runtime_service_manager_with(get_global_app_context(), init_global_kms_service_manager)
}
/// Resolve KMS encryption service using AppContext-first precedence.
pub async fn resolve_encryption_service() -> Option<Arc<ObjectEncryptionService>> {
resolve_encryption_service_with(get_global_app_context(), rustfs_kms::get_global_encryption_service).await
}
/// Resolve outbound TLS generation using AppContext-first precedence.
pub fn resolve_outbound_tls_generation() -> TlsGeneration {
resolve_outbound_tls_generation_with(get_global_app_context(), || default_outbound_tls_runtime_interface().generation())
@@ -155,6 +163,32 @@ pub fn resolve_lock_client() -> Option<Arc<dyn LockClient>> {
resolve_lock_client_with(get_global_app_context(), || default_lock_client_interface().handle())
}
/// Resolve lock clients using AppContext-first precedence.
pub fn resolve_lock_clients_handle() -> Option<HashMap<String, Arc<dyn LockClient>>> {
resolve_lock_clients_handle_with(get_global_app_context(), || default_lock_clients_interface().handle())
}
/// Resolve performance metrics using AppContext-first precedence.
pub fn resolve_performance_metrics() -> Arc<PerformanceMetrics> {
resolve_performance_metrics_with(get_global_app_context(), || default_performance_metrics_interface().handle())
}
/// Resolve internode metrics using AppContext-first precedence.
pub fn resolve_internode_metrics() -> Arc<InternodeMetrics> {
resolve_internode_metrics_with(get_global_app_context(), || default_internode_metrics_interface().handle())
}
/// Resolve S3 Select database using AppContext-first precedence.
pub async fn resolve_s3select_db(
input: SelectObjectContentInput,
enable_debug: bool,
) -> QueryResult<Arc<dyn DatabaseManagerSystem + Send + Sync>> {
resolve_s3select_db_with(get_global_app_context(), input, enable_debug, |input, enable_debug| async move {
default_s3select_db_interface().get(input, enable_debug).await
})
.await
}
/// Resolve local node name using AppContext-first precedence.
pub async fn resolve_local_node_name() -> String {
resolve_local_node_name_with(get_global_app_context(), rustfs_common::get_global_local_node_name).await
@@ -215,6 +249,21 @@ fn resolve_or_init_kms_runtime_service_manager_with(
.unwrap_or_else(fallback)
}
async fn resolve_encryption_service_with<F, Fut>(
context: Option<Arc<AppContext>>,
fallback: F,
) -> Option<Arc<ObjectEncryptionService>>
where
F: FnOnce() -> Fut,
Fut: Future<Output = Option<Arc<ObjectEncryptionService>>>,
{
if let Some(manager) = context.and_then(|context| context.kms_runtime().service_manager()) {
return manager.get_encryption_service().await;
}
fallback().await
}
fn resolve_outbound_tls_generation_with(
context: Option<Arc<AppContext>>,
fallback: impl FnOnce() -> TlsGeneration,
@@ -339,6 +388,44 @@ fn resolve_lock_client_with(
context.and_then(|context| context.lock_client().handle()).or_else(fallback)
}
fn resolve_lock_clients_handle_with(
context: Option<Arc<AppContext>>,
fallback: impl FnOnce() -> Option<HashMap<String, Arc<dyn LockClient>>>,
) -> Option<HashMap<String, Arc<dyn LockClient>>> {
context.and_then(|context| context.lock_clients().handle()).or_else(fallback)
}
fn resolve_performance_metrics_with(
context: Option<Arc<AppContext>>,
fallback: impl FnOnce() -> Arc<PerformanceMetrics>,
) -> Arc<PerformanceMetrics> {
context.map_or_else(fallback, |context| context.performance_metrics().handle())
}
fn resolve_internode_metrics_with(
context: Option<Arc<AppContext>>,
fallback: impl FnOnce() -> Arc<InternodeMetrics>,
) -> Arc<InternodeMetrics> {
context.map_or_else(fallback, |context| context.internode_metrics().handle())
}
async fn resolve_s3select_db_with<F, Fut>(
context: Option<Arc<AppContext>>,
input: SelectObjectContentInput,
enable_debug: bool,
fallback: F,
) -> QueryResult<Arc<dyn DatabaseManagerSystem + Send + Sync>>
where
F: FnOnce(SelectObjectContentInput, bool) -> Fut,
Fut: Future<Output = QueryResult<Arc<dyn DatabaseManagerSystem + Send + Sync>>>,
{
if let Some(context) = context {
return context.s3select_db().get(input, enable_debug).await;
}
fallback(input, enable_debug).await
}
async fn resolve_local_node_name_with<F, Fut>(context: Option<Arc<AppContext>>, fallback: F) -> String
where
F: FnOnce() -> Fut,
@@ -419,14 +506,21 @@ mod tests {
};
use crate::app::context::interfaces::{
ActionCredentialInterface, BootTimeInterface, BucketMetadataInterface, BufferConfigInterface, DeploymentIdInterface,
EndpointsInterface, IamInterface, KmsInterface, KmsRuntimeInterface, LocalNodeNameInterface, LockClientInterface,
OutboundTlsRuntimeInterface, RegionInterface, ReplicationStatsInterface, RuntimePortInterface, ScannerMetricsInterface,
ServerConfigInterface, StorageClassInterface, TierConfigInterface, TierStatsInterface,
EndpointsInterface, IamInterface, InternodeMetricsInterface, KmsInterface, KmsRuntimeInterface, LocalNodeNameInterface,
LockClientInterface, LockClientsInterface, OutboundTlsRuntimeInterface, PerformanceMetricsInterface, RegionInterface,
ReplicationStatsInterface, RuntimePortInterface, S3SelectDbInterface, ScannerMetricsInterface, ServerConfigInterface,
StorageClassInterface, TierConfigInterface, TierStatsInterface,
};
use crate::config::{RustFSBufferConfig, WorkloadProfile};
use async_trait::async_trait;
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics};
use rustfs_lock::{LocalClient, LockClient};
use rustfs_s3select_api::{
QueryResult,
query::{Query, execution::QueryStateMachineRef, logical_planner::Plan},
server::dbms::{DatabaseManagerSystem, QueryHandle},
};
use std::path::PathBuf;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, SystemTime};
@@ -573,6 +667,76 @@ mod tests {
}
}
struct TestLockClientsInterface {
clients: Option<HashMap<String, Arc<dyn LockClient>>>,
}
impl LockClientsInterface for TestLockClientsInterface {
fn handle(&self) -> Option<HashMap<String, Arc<dyn LockClient>>> {
self.clients.clone()
}
}
struct TestPerformanceMetricsInterface {
metrics: Arc<PerformanceMetrics>,
}
impl PerformanceMetricsInterface for TestPerformanceMetricsInterface {
fn handle(&self) -> Arc<PerformanceMetrics> {
self.metrics.clone()
}
}
struct TestInternodeMetricsInterface {
metrics: Arc<InternodeMetrics>,
}
impl InternodeMetricsInterface for TestInternodeMetricsInterface {
fn handle(&self) -> Arc<InternodeMetrics> {
self.metrics.clone()
}
}
struct TestS3SelectDbInterface {
db: Arc<dyn DatabaseManagerSystem + Send + Sync>,
}
#[async_trait]
impl S3SelectDbInterface for TestS3SelectDbInterface {
async fn get(
&self,
_input: SelectObjectContentInput,
_enable_debug: bool,
) -> QueryResult<Arc<dyn DatabaseManagerSystem + Send + Sync>> {
Ok(self.db.clone())
}
}
struct TestS3SelectDb;
#[async_trait]
impl DatabaseManagerSystem for TestS3SelectDb {
async fn execute(&self, _query: &Query) -> QueryResult<QueryHandle> {
unreachable!("resolver tests only compare database handles")
}
async fn build_query_state_machine(&self, _query: Query) -> QueryResult<QueryStateMachineRef> {
unreachable!("resolver tests only compare database handles")
}
async fn build_logical_plan(&self, _query_state_machine: QueryStateMachineRef) -> QueryResult<Option<Plan>> {
unreachable!("resolver tests only compare database handles")
}
async fn execute_logical_plan(
&self,
_logical_plan: Plan,
_query_state_machine: QueryStateMachineRef,
) -> QueryResult<QueryHandle> {
unreachable!("resolver tests only compare database handles")
}
}
struct TestLocalNodeNameInterface {
name: String,
}
@@ -699,6 +863,25 @@ mod tests {
(temp_dir, store, endpoint_pools)
}
fn test_select_input() -> SelectObjectContentInput {
SelectObjectContentInput {
bucket: "test-bucket".to_string(),
expected_bucket_owner: None,
key: "test.csv".to_string(),
sse_customer_algorithm: None,
sse_customer_key: None,
sse_customer_key_md5: None,
request: s3s::dto::SelectObjectContentRequest {
expression: "SELECT * FROM S3Object".to_string(),
expression_type: s3s::dto::ExpressionType::from_static("SQL"),
input_serialization: s3s::dto::InputSerialization::default(),
output_serialization: s3s::dto::OutputSerialization::default(),
request_progress: None,
scan_range: None,
},
}
}
#[tokio::test]
async fn resolver_helpers_are_context_first_and_fallback_when_context_is_absent() {
let (_temp_dir, object_store, endpoints) = test_store().await;
@@ -736,6 +919,16 @@ mod tests {
let fallback_deployment_id = "fallback-deployment".to_string();
let context_runtime_port = 19000;
let fallback_runtime_port = 29000;
let mut context_lock_clients = HashMap::new();
context_lock_clients.insert("context-node:9000".to_string(), context_lock_client.clone());
let mut fallback_lock_clients = HashMap::new();
fallback_lock_clients.insert("fallback-node:9000".to_string(), fallback_lock_client.clone());
let context_performance_metrics = Arc::new(PerformanceMetrics::new());
let fallback_performance_metrics = Arc::new(PerformanceMetrics::new());
let context_internode_metrics = Arc::new(InternodeMetrics::default());
let fallback_internode_metrics = Arc::new(InternodeMetrics::default());
let context_s3select_db: Arc<dyn DatabaseManagerSystem + Send + Sync> = Arc::new(TestS3SelectDb);
let fallback_s3select_db: Arc<dyn DatabaseManagerSystem + Send + Sync> = Arc::new(TestS3SelectDb);
let context_outbound_tls_state = GlobalPublishedOutboundTlsState {
generation: TlsGeneration(41),
root_ca_pem: Some(b"context-root-ca".to_vec()),
@@ -796,6 +989,18 @@ mod tests {
lock_client: Arc::new(TestLockClientInterface {
client: Some(context_lock_client.clone()),
}),
lock_clients: Arc::new(TestLockClientsInterface {
clients: Some(context_lock_clients.clone()),
}),
performance_metrics: Arc::new(TestPerformanceMetricsInterface {
metrics: context_performance_metrics.clone(),
}),
internode_metrics: Arc::new(TestInternodeMetricsInterface {
metrics: context_internode_metrics.clone(),
}),
s3select_db: Arc::new(TestS3SelectDbInterface {
db: context_s3select_db.clone(),
}),
local_node_name: Arc::new(TestLocalNodeNameInterface {
name: context_node_name.clone(),
}),
@@ -882,6 +1087,29 @@ mod tests {
&resolve_lock_client_with(Some(context.clone()), || None).expect("context lock client"),
&context_lock_client
));
assert!(Arc::ptr_eq(
resolve_lock_clients_handle_with(Some(context.clone()), || None)
.expect("context lock clients")
.get("context-node:9000")
.expect("context lock client entry"),
&context_lock_client
));
assert!(Arc::ptr_eq(
&resolve_performance_metrics_with(Some(context.clone()), || fallback_performance_metrics.clone()),
&context_performance_metrics
));
assert!(Arc::ptr_eq(
&resolve_internode_metrics_with(Some(context.clone()), || fallback_internode_metrics.clone()),
&context_internode_metrics
));
assert!(Arc::ptr_eq(
&resolve_s3select_db_with(Some(context.clone()), test_select_input(), false, |_input, _enable_debug| async {
Ok(fallback_s3select_db.clone())
})
.await
.expect("context S3 Select DB"),
&context_s3select_db
));
assert_eq!(
resolve_local_node_name_with(Some(context.clone()), || async { fallback_node_name.clone() }).await,
context_node_name
@@ -972,6 +1200,29 @@ mod tests {
&resolve_lock_client_with(None, || Some(fallback_lock_client.clone())).expect("fallback lock client"),
&fallback_lock_client
));
assert!(Arc::ptr_eq(
resolve_lock_clients_handle_with(None, || Some(fallback_lock_clients.clone()))
.expect("fallback lock clients")
.get("fallback-node:9000")
.expect("fallback lock client entry"),
&fallback_lock_client
));
assert!(Arc::ptr_eq(
&resolve_performance_metrics_with(None, || fallback_performance_metrics.clone()),
&fallback_performance_metrics
));
assert!(Arc::ptr_eq(
&resolve_internode_metrics_with(None, || fallback_internode_metrics.clone()),
&fallback_internode_metrics
));
assert!(Arc::ptr_eq(
&resolve_s3select_db_with(None, test_select_input(), false, |_input, _enable_debug| async {
Ok(fallback_s3select_db.clone())
})
.await
.expect("fallback S3 Select DB"),
&fallback_s3select_db
));
assert_eq!(
resolve_local_node_name_with(None, || async { fallback_node_name.clone() }).await,
fallback_node_name
+42 -7
View File
@@ -16,17 +16,20 @@ use super::super::{ECStore, set_object_store_resolver};
use super::handles::{
IamHandle, KmsHandle, default_action_credential_interface, default_boot_time_interface, default_bucket_metadata_interface,
default_bucket_monitor_interface, default_buffer_config_interface, default_deployment_id_interface,
default_endpoints_interface, default_kms_runtime_interface, default_local_node_name_interface, default_lock_client_interface,
default_endpoints_interface, default_internode_metrics_interface, default_kms_runtime_interface,
default_local_node_name_interface, default_lock_client_interface, default_lock_clients_interface,
default_notification_system_interface, default_notify_interface, default_outbound_tls_runtime_interface,
default_region_interface, default_replication_pool_interface, default_replication_stats_interface,
default_runtime_port_interface, default_scanner_metrics_interface, default_server_config_interface,
default_storage_class_interface, default_tier_config_interface, default_tier_stats_interface,
default_performance_metrics_interface, default_region_interface, default_replication_pool_interface,
default_replication_stats_interface, default_runtime_port_interface, default_s3select_db_interface,
default_scanner_metrics_interface, default_server_config_interface, default_storage_class_interface,
default_tier_config_interface, default_tier_stats_interface,
};
use super::interfaces::{
ActionCredentialInterface, BootTimeInterface, BucketMetadataInterface, BucketMonitorInterface, BufferConfigInterface,
DeploymentIdInterface, EndpointsInterface, IamInterface, KmsInterface, KmsRuntimeInterface, LocalNodeNameInterface,
LockClientInterface, NotificationSystemInterface, NotifyInterface, OutboundTlsRuntimeInterface, RegionInterface,
ReplicationPoolInterface, ReplicationStatsInterface, RuntimePortInterface, ScannerMetricsInterface, ServerConfigInterface,
DeploymentIdInterface, EndpointsInterface, IamInterface, InternodeMetricsInterface, KmsInterface, KmsRuntimeInterface,
LocalNodeNameInterface, LockClientInterface, LockClientsInterface, NotificationSystemInterface, NotifyInterface,
OutboundTlsRuntimeInterface, PerformanceMetricsInterface, RegionInterface, ReplicationPoolInterface,
ReplicationStatsInterface, RuntimePortInterface, S3SelectDbInterface, ScannerMetricsInterface, ServerConfigInterface,
StorageClassInterface, TierConfigInterface, TierStatsInterface,
};
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
@@ -55,6 +58,10 @@ pub struct AppContext {
deployment_id: Arc<dyn DeploymentIdInterface>,
runtime_port: Arc<dyn RuntimePortInterface>,
lock_client: Arc<dyn LockClientInterface>,
lock_clients: Arc<dyn LockClientsInterface>,
performance_metrics: Arc<dyn PerformanceMetricsInterface>,
internode_metrics: Arc<dyn InternodeMetricsInterface>,
s3select_db: Arc<dyn S3SelectDbInterface>,
local_node_name: Arc<dyn LocalNodeNameInterface>,
action_credentials: Arc<dyn ActionCredentialInterface>,
region: Arc<dyn RegionInterface>,
@@ -85,6 +92,10 @@ impl AppContext {
deployment_id: default_deployment_id_interface(),
runtime_port: default_runtime_port_interface(),
lock_client: default_lock_client_interface(),
lock_clients: default_lock_clients_interface(),
performance_metrics: default_performance_metrics_interface(),
internode_metrics: default_internode_metrics_interface(),
s3select_db: default_s3select_db_interface(),
local_node_name: default_local_node_name_interface(),
action_credentials: default_action_credential_interface(),
region: default_region_interface(),
@@ -176,6 +187,22 @@ impl AppContext {
self.lock_client.clone()
}
pub fn lock_clients(&self) -> Arc<dyn LockClientsInterface> {
self.lock_clients.clone()
}
pub fn performance_metrics(&self) -> Arc<dyn PerformanceMetricsInterface> {
self.performance_metrics.clone()
}
pub fn internode_metrics(&self) -> Arc<dyn InternodeMetricsInterface> {
self.internode_metrics.clone()
}
pub fn s3select_db(&self) -> Arc<dyn S3SelectDbInterface> {
self.s3select_db.clone()
}
pub fn local_node_name(&self) -> Arc<dyn LocalNodeNameInterface> {
self.local_node_name.clone()
}
@@ -224,6 +251,10 @@ pub(super) struct AppContextTestInterfaces {
pub(super) deployment_id: Arc<dyn DeploymentIdInterface>,
pub(super) runtime_port: Arc<dyn RuntimePortInterface>,
pub(super) lock_client: Arc<dyn LockClientInterface>,
pub(super) lock_clients: Arc<dyn LockClientsInterface>,
pub(super) performance_metrics: Arc<dyn PerformanceMetricsInterface>,
pub(super) internode_metrics: Arc<dyn InternodeMetricsInterface>,
pub(super) s3select_db: Arc<dyn S3SelectDbInterface>,
pub(super) local_node_name: Arc<dyn LocalNodeNameInterface>,
pub(super) action_credentials: Arc<dyn ActionCredentialInterface>,
pub(super) region: Arc<dyn RegionInterface>,
@@ -255,6 +286,10 @@ impl AppContext {
deployment_id: interfaces.deployment_id,
runtime_port: interfaces.runtime_port,
lock_client: interfaces.lock_client,
lock_clients: interfaces.lock_clients,
performance_metrics: interfaces.performance_metrics,
internode_metrics: interfaces.internode_metrics,
s3select_db: interfaces.s3select_db,
local_node_name: interfaces.local_node_name,
action_credentials: interfaces.action_credentials,
region: interfaces.region,
+76 -7
View File
@@ -18,15 +18,16 @@ use super::super::TierConfigMgr;
use super::super::metadata_sys::{BucketMetadataSys, get_global_bucket_metadata_sys};
use super::super::{
collect_scanner_metrics_report, get_daily_all_tier_stats, get_global_boot_time, get_global_bucket_monitor,
get_global_deployment_id, get_global_endpoints_opt, get_global_lock_client, get_global_notification_sys, get_global_region,
get_global_replication_pool, get_global_replication_stats, get_global_tier_config_mgr, global_rustfs_port,
set_global_storage_class,
get_global_deployment_id, get_global_endpoints_opt, get_global_lock_client, get_global_lock_clients,
get_global_notification_sys, get_global_region, get_global_replication_pool, get_global_replication_stats,
get_global_tier_config_mgr, global_rustfs_port, set_global_storage_class,
};
use super::interfaces::{
ActionCredentialInterface, BootTimeInterface, BucketMetadataInterface, BucketMonitorInterface, BufferConfigInterface,
DeploymentIdInterface, EndpointsInterface, IamInterface, KmsInterface, KmsRuntimeInterface, LocalNodeNameInterface,
LockClientInterface, NotificationSystemInterface, NotifyInterface, OutboundTlsRuntimeInterface, RegionInterface,
ReplicationPoolInterface, ReplicationStatsInterface, RuntimePortInterface, ScannerMetricsInterface, ServerConfigInterface,
DeploymentIdInterface, EndpointsInterface, IamInterface, InternodeMetricsInterface, KmsInterface, KmsRuntimeInterface,
LocalNodeNameInterface, LockClientInterface, LockClientsInterface, NotificationSystemInterface, NotifyInterface,
OutboundTlsRuntimeInterface, PerformanceMetricsInterface, RegionInterface, ReplicationPoolInterface,
ReplicationStatsInterface, RuntimePortInterface, S3SelectDbInterface, ScannerMetricsInterface, ServerConfigInterface,
StorageClassInterface, TierConfigInterface, TierStatsInterface,
};
use crate::config::{RustFSBufferConfig, get_global_buffer_config};
@@ -36,14 +37,21 @@ use rustfs_config::server_config::Config;
use rustfs_config::server_config::{get_global_server_config, set_global_server_config};
use rustfs_credentials::{Credentials, get_global_action_cred};
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
use rustfs_io_metrics::{
PerformanceMetrics,
global_metrics::get_global_metrics,
internode_metrics::{InternodeMetrics, global_internode_metrics},
};
use rustfs_kms::{KmsServiceManager, get_global_kms_service_manager};
use rustfs_lock::LockClient;
use rustfs_notify::{EventArgs, NotificationError, notifier_global};
use rustfs_s3select_api::{QueryResult, server::dbms::DatabaseManagerSystem};
use rustfs_targets::{EventName, arn::TargetID};
use rustfs_tls_runtime::{
GlobalPublishedOutboundTlsState, TlsGeneration, load_global_outbound_tls_generation, load_global_outbound_tls_state,
};
use std::{sync::Arc, time::SystemTime};
use s3s::dto::SelectObjectContentInput;
use std::{collections::HashMap, sync::Arc, time::SystemTime};
use tokio::sync::RwLock;
/// Default IAM interface adapter.
@@ -256,6 +264,51 @@ impl LockClientInterface for LockClientHandle {
}
}
/// Default lock clients interface adapter.
#[derive(Default)]
pub struct LockClientsHandle;
impl LockClientsInterface for LockClientsHandle {
fn handle(&self) -> Option<HashMap<String, Arc<dyn LockClient>>> {
get_global_lock_clients().cloned()
}
}
/// Default performance metrics interface adapter.
#[derive(Default)]
pub struct PerformanceMetricsHandle;
impl PerformanceMetricsInterface for PerformanceMetricsHandle {
fn handle(&self) -> Arc<PerformanceMetrics> {
get_global_metrics()
}
}
/// Default internode metrics interface adapter.
#[derive(Default)]
pub struct InternodeMetricsHandle;
impl InternodeMetricsInterface for InternodeMetricsHandle {
fn handle(&self) -> Arc<InternodeMetrics> {
global_internode_metrics().clone()
}
}
/// Default S3 Select database interface adapter.
#[derive(Default)]
pub struct S3SelectDbHandle;
#[async_trait]
impl S3SelectDbInterface for S3SelectDbHandle {
async fn get(
&self,
input: SelectObjectContentInput,
enable_debug: bool,
) -> QueryResult<Arc<dyn DatabaseManagerSystem + Send + Sync>> {
rustfs_s3select_query::get_global_db(input, enable_debug).await
}
}
/// Default local node name interface adapter.
#[derive(Default)]
pub struct LocalNodeNameHandle;
@@ -391,6 +444,22 @@ pub fn default_lock_client_interface() -> Arc<dyn LockClientInterface> {
Arc::new(LockClientHandle)
}
pub fn default_lock_clients_interface() -> Arc<dyn LockClientsInterface> {
Arc::new(LockClientsHandle)
}
pub fn default_performance_metrics_interface() -> Arc<dyn PerformanceMetricsInterface> {
Arc::new(PerformanceMetricsHandle)
}
pub fn default_internode_metrics_interface() -> Arc<dyn InternodeMetricsInterface> {
Arc::new(InternodeMetricsHandle)
}
pub fn default_s3select_db_interface() -> Arc<dyn S3SelectDbInterface> {
Arc::new(S3SelectDbHandle)
}
pub fn default_local_node_name_interface() -> Arc<dyn LocalNodeNameInterface> {
Arc::new(LocalNodeNameHandle)
}
+29 -1
View File
@@ -24,12 +24,15 @@ use async_trait::async_trait;
use rustfs_config::server_config::Config;
use rustfs_credentials::Credentials;
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics};
use rustfs_kms::KmsServiceManager;
use rustfs_lock::LockClient;
use rustfs_notify::{EventArgs, NotificationError};
use rustfs_s3select_api::{QueryResult, server::dbms::DatabaseManagerSystem};
use rustfs_targets::{EventName, arn::TargetID};
use rustfs_tls_runtime::{GlobalPublishedOutboundTlsState, TlsGeneration};
use std::{sync::Arc, time::SystemTime};
use s3s::dto::SelectObjectContentInput;
use std::{collections::HashMap, sync::Arc, time::SystemTime};
use tokio::sync::RwLock;
/// IAM interface for application-layer use-cases.
@@ -133,6 +136,31 @@ pub trait LockClientInterface: Send + Sync {
fn handle(&self) -> Option<Arc<dyn LockClient>>;
}
/// Lock clients interface for runtime readiness integration.
pub trait LockClientsInterface: Send + Sync {
fn handle(&self) -> Option<HashMap<String, Arc<dyn LockClient>>>;
}
/// Performance metrics interface for storage runtime integration.
pub trait PerformanceMetricsInterface: Send + Sync {
fn handle(&self) -> Arc<PerformanceMetrics>;
}
/// Internode metrics interface for RPC runtime integration.
pub trait InternodeMetricsInterface: Send + Sync {
fn handle(&self) -> Arc<InternodeMetrics>;
}
/// S3 Select database interface for object query execution.
#[async_trait]
pub trait S3SelectDbInterface: Send + Sync {
async fn get(
&self,
input: SelectObjectContentInput,
enable_debug: bool,
) -> QueryResult<Arc<dyn DatabaseManagerSystem + Send + Sync>>;
}
/// Local node name interface for application-layer use-cases.
#[async_trait]
pub trait LocalNodeNameInterface: Send + Sync {
+5 -1
View File
@@ -41,7 +41,7 @@ mod lifecycle_transition_api_test;
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::Arc;
use std::{collections::HashMap, sync::Arc};
mod ecstore_admin {
pub(crate) use crate::storage::ecstore_admin::get_server_info;
@@ -646,6 +646,10 @@ pub(crate) fn get_global_lock_client() -> Option<Arc<dyn rustfs_lock::client::Lo
crate::storage::get_global_lock_client()
}
pub(crate) fn get_global_lock_clients() -> Option<&'static HashMap<String, Arc<dyn rustfs_lock::client::LockClient>>> {
crate::storage::get_global_lock_clients()
}
pub(crate) fn get_global_region() -> Option<s3s::region::Region> {
crate::storage::get_global_region()
}
+4 -2
View File
@@ -1,3 +1,4 @@
use crate::app::context::resolve_s3select_db;
use crate::error::ApiError;
use crate::storage::options::get_opts;
use crate::storage::request_context::spawn_traced;
@@ -15,7 +16,6 @@ use rustfs_s3select_api::{
object_store::{INVALID_SCAN_RANGE_MESSAGE, validate_scan_range_bounds},
query::{Context, Query},
};
use rustfs_s3select_query::get_global_db;
use rustfs_storage_api::ObjectOperations as _;
use s3s::dto::*;
use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
@@ -55,7 +55,9 @@ pub async fn execute_select_object_content(
validate_scan_range_for_object_size(&input.request, metadata.size)?;
let input = Arc::new(input);
let db = get_global_db((*input).clone(), false).await.map_err(map_query_error_to_s3)?;
let db = resolve_s3select_db((*input).clone(), false)
.await
.map_err(map_query_error_to_s3)?;
let query = Query::new(Context { input: input.clone() }, input.request.expression.clone());
let output = db
.execute(&query)
+5 -4
View File
@@ -19,6 +19,7 @@
//! configuration, and dependencies.
use super::{InfoOpts, InfoType};
use crate::app::context::resolve_buffer_config;
use crate::version::build;
use rustfs_credentials::Masked;
use serde::Serialize;
@@ -543,9 +544,9 @@ fn collect_config_info_json() -> ConfigInfoJson {
// Get workload profile info
let workload_profile = {
use super::workload_profiles::{get_global_buffer_config, is_buffer_profile_enabled};
use super::workload_profiles::is_buffer_profile_enabled;
if is_buffer_profile_enabled() {
let config = get_global_buffer_config();
let config = resolve_buffer_config();
let profile = config.workload_profile();
let buffer_config = profile.config();
Some(WorkloadProfileJson {
@@ -863,13 +864,13 @@ fn format_protocol_config_info() -> String {
/// Get workload profile information from global buffer config
fn get_workload_profile_info() -> String {
use super::workload_profiles::{get_global_buffer_config, is_buffer_profile_enabled};
use super::workload_profiles::is_buffer_profile_enabled;
if !is_buffer_profile_enabled() {
return "| Workload Profile | (disabled) |".to_string();
}
let config = get_global_buffer_config();
let config = resolve_buffer_config();
let profile = config.workload_profile();
let name = config.workload_name();
let buffer_config = profile.config();
+3 -3
View File
@@ -12,10 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::app::context::resolve_region;
use crate::server::ShutdownHandle;
use crate::storage::{
get_bucket_notification_config, get_global_region, process_lambda_configurations, process_queue_configurations,
process_topic_configurations,
get_bucket_notification_config, process_lambda_configurations, process_queue_configurations, process_topic_configurations,
};
use crate::{admin, config, version};
use rustfs_config::{
@@ -159,7 +159,7 @@ fn arn_to_target_id(arn_str: &str) -> Result<rustfs_targets::arn::TargetID, Targ
/// * `buckets` - A vector of bucket names to process
#[instrument(skip_all)]
pub async fn add_bucket_notification_configuration(buckets: Vec<String>) {
let global_region = get_global_region();
let global_region = resolve_region();
let region = global_region
.as_ref()
.filter(|r| !r.as_str().is_empty())
+3 -3
View File
@@ -12,10 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::app::context::{resolve_endpoints_handle, resolve_iam_ready};
use crate::app::context::{resolve_endpoints_handle, resolve_iam_ready, resolve_lock_clients_handle};
use crate::server::{ServiceState, ServiceStateManager};
use crate::server::{has_path_prefix, is_table_catalog_path};
use crate::storage::{Endpoint, EndpointServerPools, get_global_lock_clients, is_dist_erasure, resolve_object_store_handle};
use crate::storage::{Endpoint, EndpointServerPools, is_dist_erasure, resolve_object_store_handle};
#[cfg(test)]
use crate::storage::{Endpoints, PoolEndpoints};
use bytes::Bytes;
@@ -584,7 +584,7 @@ async fn collect_lock_quorum_status_uncached() -> LockQuorumStatus {
let Some(pool_endpoints) = resolve_endpoints_handle() else {
return LockQuorumStatus::default();
};
let Some(lock_clients) = get_global_lock_clients() else {
let Some(lock_clients) = resolve_lock_clients_handle() else {
return LockQuorumStatus::default();
};
+3 -2
View File
@@ -20,7 +20,8 @@
//! (system roots, leaf-as-CA, mTLS env-var path overrides) and builds the server
//! TLS acceptor directly from the pre-loaded server material — no double reads.
use rustfs_common::{MtlsIdentityPem, get_global_outbound_tls_generation};
use crate::app::context::resolve_outbound_tls_generation;
use rustfs_common::MtlsIdentityPem;
use rustfs_config::{
DEFAULT_SERVER_MTLS_ENABLE, DEFAULT_TLS_KEYLOG, DEFAULT_TLS_RELOAD_ENABLE, DEFAULT_TLS_RELOAD_INTERVAL,
DEFAULT_TRUST_LEAF_CERT_AS_CA, DEFAULT_TRUST_SYSTEM_CA, ENV_MTLS_CLIENT_CERT, ENV_MTLS_CLIENT_KEY, ENV_SERVER_MTLS_ENABLE,
@@ -552,7 +553,7 @@ pub(crate) fn spawn_reload_loop(tls_path: String, holder: Arc<TlsAcceptorHolder>
continue;
}
let generation = get_global_outbound_tls_generation().saturating_add(1);
let generation = resolve_outbound_tls_generation().0.saturating_add(1);
publish_global_outbound_tls_state(TlsGeneration(generation), &snapshot.outbound).await;
record_tls_generation("rustfs_server_reload_loop", generation);
if !snapshot.outbound.root_ca_pem.is_empty() {
+3 -4
View File
@@ -12,9 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage::{
ECStore, get_global_replication_pool, init_bucket_metadata_sys, try_migrate_bucket_metadata, try_migrate_iam_config,
};
use crate::app::context::resolve_replication_pool_handle;
use crate::storage::{ECStore, init_bucket_metadata_sys, try_migrate_bucket_metadata, try_migrate_iam_config};
use rustfs_storage_api::{BucketOperations, BucketOptions};
use std::{
io::{Error, Result},
@@ -53,7 +52,7 @@ pub(crate) async fn init_bucket_metadata_runtime(store: Arc<ECStore>, ctx: Cance
try_migrate_bucket_metadata(store.clone()).await;
if let Some(pool) = get_global_replication_pool() {
if let Some(pool) = resolve_replication_pool_handle() {
pool.init_resync(ctx, buckets.clone()).await?;
}
+2 -1
View File
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::app::context::resolve_outbound_tls_generation;
use crate::config::Config;
use std::io::{Error, Result};
use tracing::{error, info};
@@ -28,7 +29,7 @@ pub(crate) async fn init_outbound_tls_material(config: &Config) -> Result<()> {
Ok(snapshot) => {
use rustfs_tls_runtime::{publish_global_outbound_tls_state, record_tls_generation};
let generation = next_tls_generation(rustfs_common::get_global_outbound_tls_generation());
let generation = next_tls_generation(resolve_outbound_tls_generation().0);
publish_global_outbound_tls_state(generation, &snapshot.outbound).await;
record_tls_generation(TLS_STARTUP_GENERATION_CONSUMER, generation.0);
info!(
+2 -1
View File
@@ -18,6 +18,7 @@ use super::resolve_object_store_handle;
use super::{
PolicySys, StorageError, get_bucket_metadata, get_bucket_policy_raw, get_public_access_block_config, is_err_bucket_not_found,
};
use crate::app::context::resolve_region;
use crate::auth::{check_key_valid, get_condition_values_with_query_and_client_info, get_session_token};
use crate::error::ApiError;
use crate::license::license_check;
@@ -930,7 +931,7 @@ impl S3Access for FS {
let req_info = ReqInfo {
cred,
is_owner,
region: super::get_global_region(),
region: resolve_region(),
request_context,
..Default::default()
};
+3 -3
View File
@@ -19,6 +19,7 @@ use super::io_schedule::{
get_advanced_buffer_size,
};
use super::request_guard::GetObjectGuard;
use crate::app::context::resolve_performance_metrics;
use rustfs_concurrency::{
AdmissionState, GetObjectQueueSnapshot, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot,
WorkloadAdmissionSnapshotProvider, WorkloadClass,
@@ -27,7 +28,6 @@ use rustfs_config::{KI_B, MI_B};
use rustfs_io_core::BytesPool;
use rustfs_io_core::io_profile::{AccessPattern, IoPatternDetector, StorageMedia, detect_storage_media};
use rustfs_io_metrics::bandwidth::{BandwidthMonitor, BandwidthSnapshot};
use rustfs_io_metrics::global_metrics::get_global_metrics;
use rustfs_io_metrics::{MetricsCollector, PerformanceMetrics};
use std::sync::{Arc, LazyLock, Mutex};
use std::time::Duration;
@@ -118,7 +118,7 @@ impl ConcurrencyManager {
// Use global performance metrics instance for consistent metrics tracking
// This allows AutoTuner and other components to access the same metrics data
let performance_metrics = get_global_metrics();
let performance_metrics = resolve_performance_metrics();
// Initialize metrics collector for I/O latency tracking
// Keep 1000 samples for P95/P99 calculation
@@ -231,7 +231,7 @@ impl ConcurrencyManager {
///
/// Arc-wrapped PerformanceMetrics instance
pub fn performance_metrics(&self) -> Arc<PerformanceMetrics> {
get_global_metrics()
resolve_performance_metrics()
}
/// Calculate an adaptive I/O strategy based on disk permit wait time.
+11 -11
View File
@@ -13,9 +13,9 @@
// limitations under the License.
use super::*;
use crate::app::context::resolve_internode_metrics;
use rustfs_io_metrics::internode_metrics::{
INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC,
global_internode_metrics,
};
use serde::de::DeserializeOwned;
use std::io::Cursor;
@@ -934,11 +934,12 @@ impl NodeService {
pub(super) async fn handle_write_all(&self, request: Request<WriteAllRequest>) -> Result<Response<WriteAllResponse>, Status> {
let request = request.into_inner();
let data_len = request.data.len();
global_internode_metrics().record_incoming_request_for_operation_and_backend(
let metrics = resolve_internode_metrics();
metrics.record_incoming_request_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_WRITE_ALL,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
global_internode_metrics().record_recv_bytes_for_operation_and_backend(
metrics.record_recv_bytes_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_WRITE_ALL,
INTERNODE_TRANSPORT_BACKEND_GRPC,
data_len,
@@ -950,7 +951,7 @@ impl NodeService {
error: None,
})),
Err(err) => {
global_internode_metrics().record_error_for_operation_and_backend(
metrics.record_error_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_WRITE_ALL,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
@@ -961,8 +962,7 @@ impl NodeService {
}
}
} else {
global_internode_metrics()
.record_error_for_operation_and_backend(INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC);
metrics.record_error_for_operation_and_backend(INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC);
Ok(Response::new(WriteAllResponse {
success: false,
error: Some(DiskError::other("can not find disk".to_string()).into()),
@@ -974,14 +974,15 @@ impl NodeService {
debug!("read all");
let request = request.into_inner();
global_internode_metrics().record_incoming_request_for_operation_and_backend(
let metrics = resolve_internode_metrics();
metrics.record_incoming_request_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_ALL,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
if let Some(disk) = self.find_disk(&request.disk).await {
match disk.read_all(&request.volume, &request.path).await {
Ok(data) => {
global_internode_metrics().record_sent_bytes_for_operation_and_backend(
metrics.record_sent_bytes_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_ALL,
INTERNODE_TRANSPORT_BACKEND_GRPC,
data.len(),
@@ -993,7 +994,7 @@ impl NodeService {
}))
}
Err(err) => {
global_internode_metrics().record_error_for_operation_and_backend(
metrics.record_error_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_ALL,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
@@ -1005,8 +1006,7 @@ impl NodeService {
}
}
} else {
global_internode_metrics()
.record_error_for_operation_and_backend(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC);
metrics.record_error_for_operation_and_backend(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC);
Ok(Response::new(ReadAllResponse {
success: false,
data: Bytes::new(),
+14 -13
View File
@@ -17,6 +17,7 @@ use super::super::StorageDiskRpcExt as _;
use super::super::WalkDirOptions;
use super::super::find_local_disk_by_ref;
use super::super::verify_rpc_signature;
use crate::app::context::resolve_internode_metrics;
use crate::server::RPC_PREFIX;
use crate::storage::request_context::spawn_traced;
use bytes::{Bytes, BytesMut};
@@ -27,7 +28,7 @@ use hyper::body::Incoming;
use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
use rustfs_io_metrics::internode_metrics::{
INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_OPERATION_WALK_DIR,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
};
use rustfs_utils::net::bytes_stream;
use s3s::Body;
@@ -311,11 +312,10 @@ fn internode_http_operation(path: &str) -> Option<&'static str> {
}
fn record_internode_rpc_error(operation: Option<&'static str>) {
let metrics = resolve_internode_metrics();
match operation {
Some(operation) => {
global_internode_metrics().record_error_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP)
}
None => global_internode_metrics().record_error(),
Some(operation) => metrics.record_error_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP),
None => metrics.record_error(),
}
}
@@ -400,7 +400,7 @@ async fn handle_read_file(req: Request<Incoming>) -> Response<Body> {
}
};
global_internode_metrics().record_incoming_request_for_operation_and_backend(
resolve_internode_metrics().record_incoming_request_for_operation_and_backend(
INTERNODE_OPERATION_READ_FILE_STREAM,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
);
@@ -420,7 +420,7 @@ fn read_file_body_stream<R>(
where
R: tokio::io::AsyncRead + Unpin + Send + Sync + 'static,
{
let metrics = global_internode_metrics().clone();
let metrics = resolve_internode_metrics();
let stream = ReaderStream::with_capacity(reader, DEFAULT_READ_BUFFER_SIZE).map_ok(move |bytes| {
metrics.record_sent_bytes_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, bytes.len());
bytes
@@ -535,9 +535,9 @@ async fn handle_walk_dir(req: Request<Incoming>) -> Response<Body> {
}
});
global_internode_metrics()
resolve_internode_metrics()
.record_incoming_request_for_operation_and_backend(INTERNODE_OPERATION_WALK_DIR, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP);
let metrics = global_internode_metrics().clone();
let metrics = resolve_internode_metrics();
let stream = ReaderStream::with_capacity(rd, DEFAULT_READ_BUFFER_SIZE).map_ok(move |bytes| {
metrics.record_sent_bytes_for_operation_and_backend(
INTERNODE_OPERATION_WALK_DIR,
@@ -604,14 +604,15 @@ async fn handle_put_file(req: Request<Incoming>) -> Response<Body> {
}
};
global_internode_metrics().record_incoming_request_for_operation_and_backend(
let metrics = resolve_internode_metrics();
metrics.record_incoming_request_for_operation_and_backend(
INTERNODE_OPERATION_PUT_FILE_STREAM,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
);
global_internode_metrics().record_recv_bytes_for_operation_and_backend(
metrics.record_recv_bytes_for_operation_and_backend(
INTERNODE_OPERATION_PUT_FILE_STREAM,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
copied as usize,
usize::try_from(copied).unwrap_or(usize::MAX),
);
if let Err(e) = file.flush().await {
@@ -634,7 +635,7 @@ where
let mut pending = BytesMut::with_capacity(DEFAULT_READ_BUFFER_SIZE);
while let Some(bytes) = body.try_next().await.map_err(io::Error::other)? {
copied += bytes.len() as u64;
copied = copied.saturating_add(u64::try_from(bytes.len()).unwrap_or(u64::MAX));
pending.extend_from_slice(&bytes);
if pending.len() >= DEFAULT_READ_BUFFER_SIZE {
+7 -6
View File
@@ -70,6 +70,7 @@
//! ```
use super::StorageError;
use crate::app::context::resolve_encryption_service;
#[cfg(feature = "rio-v2")]
use aes_gcm::aead::Payload;
use aes_gcm::{
@@ -86,7 +87,7 @@ use http::{HeaderMap, HeaderValue};
use rand::Rng;
#[cfg(feature = "rio-v2")]
use rand::RngExt;
use rustfs_kms::{DataKey, service_manager::get_global_encryption_service, types::ObjectEncryptionContext};
use rustfs_kms::{DataKey, types::ObjectEncryptionContext};
use rustfs_utils::get_env_opt_str;
use s3s::S3ErrorCode;
use s3s::dto::ServerSideEncryption;
@@ -1532,7 +1533,7 @@ async fn apply_managed_encryption_material(
let mut kms_key_candidate = kms_key_id.clone();
if kms_key_candidate.is_none() {
// Try to get default key from KMS service (if available)
if let Some(service) = get_global_encryption_service().await {
if let Some(service) = resolve_encryption_service().await {
kms_key_candidate = service.get_default_key_id().cloned();
}
}
@@ -1625,7 +1626,7 @@ async fn apply_managed_decryption_material(
.cloned()
.unwrap_or_else(|| "AES256".to_string()),
)
} else if let Some(service) = get_global_encryption_service().await {
} else if let Some(service) = resolve_encryption_service().await {
// Production mode: use service for metadata parsing
let parsed = service
.headers_to_metadata(&normalized_metadata)
@@ -1767,7 +1768,7 @@ pub trait SseDekProvider: Send + Sync {
// ============================================================================
/// Production KMS-backed DEK provider
/// Resolves the latest global ObjectEncryptionService on each call.
/// Resolves the latest ObjectEncryptionService on each call.
struct KmsSseDekProvider;
impl KmsSseDekProvider {
@@ -1780,7 +1781,7 @@ impl KmsSseDekProvider {
}
async fn current_service() -> Option<Arc<rustfs_kms::service::ObjectEncryptionService>> {
get_global_encryption_service().await
resolve_encryption_service().await
}
}
@@ -2040,7 +2041,7 @@ static GLOBAL_SSE_DEK_PROVIDER: LazyLock<RwLock<Option<Arc<dyn SseDekProvider>>>
/// .await?;
/// ```
pub async fn get_sse_dek_provider() -> Result<Arc<dyn SseDekProvider>, ApiError> {
if get_global_encryption_service().await.is_some() {
if resolve_encryption_service().await.is_some() {
debug!("Using KmsSseDekProvider (KMS configured)");
return Ok(Arc::new(KmsSseDekProvider::new().await?));
}
+3 -2
View File
@@ -12,7 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage::{bucket_metadata_runtime_initialized, get_global_replication_pool, replication_queue_current_count};
use crate::app::context::resolve_replication_pool_handle;
use crate::storage::{bucket_metadata_runtime_initialized, replication_queue_current_count};
use rustfs_concurrency::{
AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider,
WorkloadClass,
@@ -151,7 +152,7 @@ fn repair_workload_admission_snapshot_from_counts(
}
pub fn replication_workload_admission_snapshot() -> WorkloadAdmissionSnapshot {
let Some(pool) = get_global_replication_pool() else {
let Some(pool) = resolve_replication_pool_handle() else {
return replication_workload_admission_snapshot_from_counts(false, None, None);
};
+12
View File
@@ -40,6 +40,8 @@ dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::m
dep|rustfs/src/app/object_usecase.rs|app->interface|crate::storage::ecfs
dep|rustfs/src/app/object_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_list_parts_params
dep|rustfs/src/auth.rs|infra->app|crate::app::context::resolve_action_credentials
dep|rustfs/src/config/info.rs|infra->app|crate::app::context::resolve_buffer_config
dep|rustfs/src/init.rs|infra->app|crate::app::context::resolve_region
dep|rustfs/src/init.rs|infra->interface|crate::admin
dep|rustfs/src/protocols/client.rs|infra->app|crate::app::context::resolve_action_credentials
dep|rustfs/src/protocols/client.rs|infra->interface|crate::storage::ecfs::FS
@@ -52,6 +54,10 @@ dep|rustfs/src/server/layer.rs|infra->interface|crate::admin::handlers::health::
dep|rustfs/src/server/layer.rs|infra->interface|crate::admin::handlers::health::build_health_response_parts
dep|rustfs/src/server/readiness.rs|infra->app|crate::app::context::resolve_endpoints_handle
dep|rustfs/src/server/readiness.rs|infra->app|crate::app::context::resolve_iam_ready
dep|rustfs/src/server/readiness.rs|infra->app|crate::app::context::resolve_lock_clients_handle
dep|rustfs/src/server/tls_material.rs|infra->app|crate::app::context::resolve_outbound_tls_generation
dep|rustfs/src/startup_bucket_metadata.rs|infra->app|crate::app::context::resolve_replication_pool_handle
dep|rustfs/src/startup_tls_material.rs|infra->app|crate::app::context::resolve_outbound_tls_generation
dep|rustfs/src/startup_iam.rs|infra->app|crate::app::context::AppContext
dep|rustfs/src/storage/ecfs_extend.rs|infra->app|crate::app::context::resolve_buffer_config
dep|rustfs/src/storage/ecfs_extend.rs|infra->interface|crate::storage::ecfs::ListObjectUnorderedQuery
@@ -59,10 +65,16 @@ dep|rustfs/src/storage/ecfs_test.rs|infra->interface|crate::storage::ecfs::FS
dep|rustfs/src/storage/ecfs_test.rs|infra->interface|crate::storage::ecfs::validate_object_lock_configuration_input
dep|rustfs/src/storage/ecfs_test.rs|infra->interface|crate::storage::s3_api::common::rustfs_initiator
dep|rustfs/src/storage/ecfs_test.rs|infra->interface|crate::storage::s3_api::common::rustfs_owner
dep|rustfs/src/storage/access.rs|infra->app|crate::app::context::resolve_region
dep|rustfs/src/storage/concurrency/manager.rs|infra->app|crate::app::context::resolve_performance_metrics
dep|rustfs/src/storage/sse.rs|infra->app|crate::app::context::resolve_encryption_service
dep|rustfs/src/storage/helper.rs|infra->app|crate::app::context::resolve_action_credentials
dep|rustfs/src/storage/rpc/disk.rs|infra->app|crate::app::context::resolve_internode_metrics
dep|rustfs/src/storage/rpc/health.rs|infra->app|crate::app::context::resolve_local_node_name
dep|rustfs/src/storage/rpc/http_service.rs|infra->app|crate::app::context::resolve_internode_metrics
dep|rustfs/src/storage/rpc/node_service.rs|infra->app|crate::app::context::resolve_iam_handle
dep|rustfs/src/storage/rpc/node_service.rs|infra->app|crate::app::context::resolve_lock_client
dep|rustfs/src/storage/rpc/node_service.rs|infra->interface|crate::admin::service::config::reload_dynamic_config_runtime_state
dep|rustfs/src/storage/rpc/node_service.rs|infra->interface|crate::admin::service::config::reload_runtime_config_snapshot
dep|rustfs/src/storage/rpc/node_service.rs|infra->interface|crate::admin::service::site_replication::reload_site_replication_runtime_state
dep|rustfs/src/workload_admission.rs|infra->app|crate::app::context::resolve_replication_pool_handle