refactor: route server storage lookups through resolver (#3439)

docs/architecture/migration-progress.md

@@ -5,19 +5,18 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 ## Current Context
 
 - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
-- Branch: `overtrue/arch-object-store-resolver-crates`
+- Branch: `overtrue/arch-server-storage-object-store-resolver`
-- Baseline: `upstream/main` at `59f99a30e2dee0794658cb1a2793cf48ed62e5ef`
+- Baseline: `upstream/main` at `9372ee70329d08ea7aafae8df84f6b1ecb5bd686`
 - PR type for this branch: `consumer-migration`
-- Runtime behavior changes: no external behavior change expected; standalone
+- Runtime behavior changes: no external behavior change expected; server and
-  protocol, observability, scanner, notify, and S3 Select object-store lookups
+  storage infra object-store lookups prefer the AppContext-owned object store
-  prefer the AppContext-owned object store when the application context is
+  through the ECStore-owned resolver and keep the existing global fallback.
-  installed and keep the existing global fallback.
+- Rust code changes: migrate server readiness/module-switch and storage access,
-- Rust code changes: add an ECStore-owned object-store resolver hook, install
+  ecfs extension, and node RPC object-store lookups to the ECStore-owned
-  it from AppContext initialization, and migrate the standalone crate consumer
+  resolver without touching app compatibility fallbacks or ECStore internal
-  group without touching server/storage infra modules or ECStore internal
   background consumers.
 - CI/script changes: none.
-- Docs changes: record `CTX-008` consumer migration scope and verification.
+- Docs changes: record `CTX-009` consumer migration scope and verification.
 
 ## Phase 0 Tasks
 
@@ -206,6 +205,18 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
     collection, and existing storage error paths.
   - Verification: formatting, compile checks, migration guards, diff hygiene,
     Rust risk scan, and full `make pre-commit`.
+- [x] `CTX-009` Migrate server/storage infra object-store consumers.
+  - Do: migrate server readiness/module-switch and storage access, ecfs
+    extension, and node RPC object-store lookups to the ECStore-owned resolver.
+  - Acceptance: server/storage infra consumers prefer the AppContext-owned
+    object store after context initialization and preserve the existing global
+    object-layer fallback.
+  - Must preserve: readiness reporting, module-switch config persistence,
+    storage access authorization checks, ecfs extension validation, node RPC
+    metadata/storage-info/rebalance/tier reload behavior, and existing storage
+    error paths.
+  - Verification: formatting, compile checks, migration guards, diff hygiene,
+    Rust risk scan, and full `make pre-commit`.
 
 ## Phase 1 Security Governance Tasks
 
@@ -512,8 +523,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 
 ## Next PRs
 
-1. `consumer-migration`: migrate the remaining server/storage infra object-store
+1. `consumer-migration`: review app compatibility fallback call sites and
-   consumers after confirming the ECStore-owned resolver is the right boundary.
+   ECStore internal background consumers before the final global-accessor
+   cleanup phase.
 2. `pure-move`: start `R-009` boot wrapper with the IAM degraded readiness
    contract covered.
 
@@ -521,44 +533,40 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 
 | Expert | Status | Notes |
 |---|---|---|
-| Quality/architecture | passed | Single consumer-migration slice; ECStore owns the resolver hook, standalone crates do not depend on `rustfs`, and the layer guard reports no new reverse dependencies. |
+| Quality/architecture | passed | Single consumer-migration slice; server/storage infra consumers use the ECStore-owned resolver without widening application-layer dependencies or changing ECStore internals. |
-| Migration preservation | passed | AppContext object-store resolution is preferred after context initialization, while the old global object-layer fallback and existing storage error paths are preserved. |
+| Migration preservation | passed | Readiness, module-switch, storage access, ecfs extension, and node RPC call sites keep the same error paths while preferring the AppContext-owned store when installed. |
-| Testing/verification | passed | Formatting, compile checks, Swift feature check, diff hygiene, migration/layer guards, added-line Rust risk scan, and full `make pre-commit` passed. |
+| Testing/verification | passed | Formatting, compile check, diff hygiene, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. |
 
 ## Verification Notes
 
-Passed on `59f99a30e2dee0794658cb1a2793cf48ed62e5ef`:
+Passed on `9372ee70329d08ea7aafae8df84f6b1ecb5bd686`:
 
-- `cargo fmt --all`.
 - `cargo fmt --all --check`.
-- `cargo check -p rustfs-ecstore -p rustfs-s3select-api -p rustfs-scanner -p rustfs-notify -p rustfs-obs`.
-- `cargo check -p rustfs-protocols --features swift`.
 - `cargo check -p rustfs --lib`.
 - `git diff --check`.
 - `./scripts/check_architecture_migration_rules.sh`.
 - `./scripts/check_layer_dependencies.sh`.
+- `git rev-list --left-right --count HEAD...origin/main` returned `0 0`.
 - Rust risk scan for changed Rust files; full-file matches were existing tests,
-  numeric casts, and relaxed counters, and the added-line scan returned no
+  existing relaxed counters, existing numeric casts, and existing string error
-  `unwrap`/`expect`, numeric cast, string error, boxed error, print macro, or
+  signatures, and the added-line scan returned no `unwrap`/`expect`, numeric
-  relaxed-ordering match.
+  cast, string error, boxed error, print macro, or relaxed-ordering match.
-- `make pre-commit`; all checks passed, including nextest 5941 passed and 111
+- `make pre-commit`; all checks passed, including nextest 5954 passed and 111
   skipped, plus doctests.
 
 Notes:
 
-- This slice adds an ECStore-owned object-store resolver hook for standalone
+- This slice migrates server readiness/module-switch and storage access, ecfs
-  crates.
+  extension, and node RPC object-store lookups to the ECStore-owned resolver.
-- Swift, S3 Select, scanner, notify, and observability consumers now use the
+- App compatibility fallback call sites remain on `new_object_layer_fn` as an
-  resolver without depending on the `rustfs` application crate.
+  explicit fallback path.
-- Server/storage infra consumers and ECStore internal background consumers
+- ECStore internal background consumers remain on the old global accessor for a
-  remain on the old global accessor for a follow-up slice.
+  separate review.
-- Global object-layer fallback remains available until the planned cleanup
-  phase.
 
 ## Handoff Notes
 
-- CTX-008 is complete.
+- CTX-009 is complete.
-- Remaining server/storage infra consumers can migrate to the ECStore-owned
+- App compatibility fallback call sites remain on `new_object_layer_fn` as an
-  resolver in the next consumer-migration slice if the layer guard still passes.
+  explicit fallback path.
 - ECStore internal background consumers should be reviewed separately before
   changing their global accessor behavior.

rustfs/src/server/module_switch.rs

@@ -15,7 +15,7 @@
 use rustfs_ecstore::{
     config::com::{read_config, save_config},
     error::Error as StorageError,
-    new_object_layer_fn,
+    resolve_object_store_handle,
 };
 use serde::{Deserialize, Serialize};
 use std::sync::atomic::{AtomicBool, Ordering};
@@ -162,7 +162,7 @@ pub(crate) fn validate_module_switch_update(requested: PersistedModuleSwitches)
 }
 
 pub(crate) async fn refresh_persisted_module_switches_from_store() -> Result<PersistedModuleSwitches, String> {
-    let Some(store) = new_object_layer_fn() else {
+    let Some(store) = resolve_object_store_handle() else {
         return Err("storage layer not initialized".to_string());
     };
 
@@ -183,7 +183,7 @@ pub(crate) async fn refresh_persisted_module_switches_from_store() -> Result<Per
 }
 
 pub(crate) async fn save_persisted_module_switches_to_store(config: PersistedModuleSwitches) -> Result<(), String> {
-    let Some(store) = new_object_layer_fn() else {
+    let Some(store) = resolve_object_store_handle() else {
         return Err("storage layer not initialized".to_string());
     };
 

rustfs/src/server/readiness.rs

@@ -23,7 +23,7 @@ use metrics::{counter, gauge};
 use rustfs_common::GlobalReadiness;
 use rustfs_ecstore::global::is_dist_erasure;
 use rustfs_ecstore::global::{get_global_endpoints_opt, get_global_lock_clients};
-use rustfs_ecstore::new_object_layer_fn;
+use rustfs_ecstore::resolve_object_store_handle;
 use rustfs_iam::get_global_iam_sys;
 use rustfs_madmin::{Disk, StorageInfo};
 use rustfs_storage_api::StorageAdminApi;
@@ -488,7 +488,7 @@ async fn collect_dependency_readiness_uncached() -> DependencyReadiness {
 }
 
 async fn collect_storage_readiness_uncached() -> bool {
-    if let Some(store) = new_object_layer_fn() {
+    if let Some(store) = resolve_object_store_handle() {
         let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
         storage_ready_from_runtime_state(&storage_info)
     } else {

rustfs/src/storage/access.rs

@@ -22,7 +22,7 @@ use metrics::counter;
 use rustfs_ecstore::bucket::metadata_sys;
 use rustfs_ecstore::bucket::policy_sys::PolicySys;
 use rustfs_ecstore::error::{StorageError, is_err_bucket_not_found};
-use rustfs_ecstore::new_object_layer_fn;
+use rustfs_ecstore::resolve_object_store_handle;
 use rustfs_ecstore::store::ECStore;
 use rustfs_ecstore::store_api::BucketOperations;
 use rustfs_iam::error::Error as IamError;
@@ -784,7 +784,7 @@ fn table_data_plane_admin_action(action: Action) -> Option<AdminAction> {
 }
 
 fn table_catalog_store_for_data_plane() -> S3Result<crate::table_catalog::EcStoreTableCatalogStore<ECStore>> {
-    let store = new_object_layer_fn().ok_or_else(|| s3_error!(InternalError, "object store not initialized"))?;
+    let store = resolve_object_store_handle().ok_or_else(|| s3_error!(InternalError, "object store not initialized"))?;
     let backend = crate::table_catalog::EcStoreTableCatalogObjectBackend::new(store);
     Ok(crate::table_catalog::ObjectTableCatalogStore::new(backend))
 }
@@ -1183,7 +1183,7 @@ impl S3Access for FS {
     ///
     /// This method returns `Ok(())` by default.
     async fn delete_object(&self, req: &mut S3Request<DeleteObjectInput>) -> S3Result<()> {
-        if let Some(store) = new_object_layer_fn()
+        if let Some(store) = resolve_object_store_handle()
             && let Err(err) = store.get_bucket_info(&req.input.bucket, &Default::default()).await
             && is_err_bucket_not_found(&err)
         {

rustfs/src/storage/ecfs_extend.rs

@@ -24,7 +24,7 @@ use rustfs_ecstore::bucket::metadata_sys::get_replication_config;
 use rustfs_ecstore::bucket::object_lock::objectlock_sys;
 use rustfs_ecstore::bucket::replication::ReplicationConfigurationExt;
 use rustfs_ecstore::error::StorageError;
-use rustfs_ecstore::new_object_layer_fn;
+use rustfs_ecstore::resolve_object_store_handle;
 use rustfs_ecstore::store_api::{BucketOperations, ObjectInfo, ObjectToDelete};
 use rustfs_storage_api::BucketOptions;
 use rustfs_targets::EventName;
@@ -739,7 +739,7 @@ pub(crate) async fn has_replication_rules(bucket: &str, objects: &[ObjectToDelet
 
 /// Helper function to get store and validate bucket exists
 pub(crate) async fn get_validated_store(bucket: &str) -> S3Result<Arc<rustfs_ecstore::store::ECStore>> {
-    let Some(store) = new_object_layer_fn() else {
+    let Some(store) = resolve_object_store_handle() else {
         return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
     };
 

rustfs/src/storage/rpc/bucket.rs

@@ -42,7 +42,7 @@ impl NodeService {
             }));
         }
 
-        let Some(store) = new_object_layer_fn() else {
+        let Some(store) = resolve_object_store_handle() else {
             return Ok(Response::new(LoadBucketMetadataResponse {
                 success: false,
                 error_info: Some("errServerNotInitialized".to_string()),

rustfs/src/storage/rpc/health.rs

@@ -214,7 +214,7 @@ impl NodeService {
         &self,
         _request: Request<LocalStorageInfoRequest>,
     ) -> Result<Response<LocalStorageInfoResponse>, Status> {
-        let Some(store) = new_object_layer_fn() else {
+        let Some(store) = resolve_object_store_handle() else {
             return Ok(Response::new(LocalStorageInfoResponse {
                 success: false,
                 storage_info: Bytes::new(),

rustfs/src/storage/rpc/node_service.rs

@@ -31,7 +31,7 @@ use rustfs_ecstore::{
     get_global_lock_client,
     global::GLOBAL_TierConfigMgr,
     metrics_realtime::{CollectMetricsOpts, MetricType, collect_local_metrics},
-    new_object_layer_fn,
+    resolve_object_store_handle,
     rpc::{
         LocalPeerS3Client, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerS3Client, SERVICE_SIGNAL_REFRESH_CONFIG,
         SERVICE_SIGNAL_RELOAD_DYNAMIC,
@@ -813,7 +813,7 @@ impl Node for NodeService {
         &self,
         _request: Request<ReloadSiteReplicationConfigRequest>,
     ) -> Result<Response<ReloadSiteReplicationConfigResponse>, Status> {
-        let Some(_store) = new_object_layer_fn() else {
+        let Some(_store) = resolve_object_store_handle() else {
             return Ok(Response::new(ReloadSiteReplicationConfigResponse {
                 success: false,
                 error_info: Some("errServerNotInitialized".to_string()),
@@ -902,7 +902,7 @@ impl Node for NodeService {
         &self,
         _request: Request<ReloadPoolMetaRequest>,
     ) -> Result<Response<ReloadPoolMetaResponse>, Status> {
-        let Some(store) = new_object_layer_fn() else {
+        let Some(store) = resolve_object_store_handle() else {
             return Ok(Response::new(ReloadPoolMetaResponse {
                 success: false,
                 error_info: Some("errServerNotInitialized".to_string()),
@@ -921,7 +921,7 @@ impl Node for NodeService {
     }
 
     async fn stop_rebalance(&self, _request: Request<StopRebalanceRequest>) -> Result<Response<StopRebalanceResponse>, Status> {
-        let Some(store) = new_object_layer_fn() else {
+        let Some(store) = resolve_object_store_handle() else {
             return Ok(Response::new(StopRebalanceResponse {
                 success: false,
                 error_info: Some("errServerNotInitialized".to_string()),
@@ -940,7 +940,7 @@ impl Node for NodeService {
         &self,
         request: Request<LoadRebalanceMetaRequest>,
     ) -> Result<Response<LoadRebalanceMetaResponse>, Status> {
-        let Some(store) = new_object_layer_fn() else {
+        let Some(store) = resolve_object_store_handle() else {
             return Ok(Response::new(LoadRebalanceMetaResponse {
                 success: false,
                 error_info: Some("errServerNotInitialized".to_string()),
@@ -978,7 +978,7 @@ impl Node for NodeService {
         &self,
         _request: Request<LoadTransitionTierConfigRequest>,
     ) -> Result<Response<LoadTransitionTierConfigResponse>, Status> {
-        let Some(store) = new_object_layer_fn() else {
+        let Some(store) = resolve_object_store_handle() else {
             return Ok(Response::new(LoadTransitionTierConfigResponse {
                 success: false,
                 error_info: Some("errServerNotInitialized".to_string()),