refactor: centralize startup storage runtime (#3465)

docs/architecture/migration-progress.md

@@ -5,18 +5,17 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 ## Current Context
 
 - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
-- Branch: `overtrue/arch-startup-storage-bootstrap`
-- Baseline: `origin/main` at `6508f88d3a5edb428a5d623f927ce384691f0cd4`
+- Branch: `overtrue/arch-startup-storage-runtime`
+- Baseline: `origin/main` at `a19560da419c6d354a805fc127951cc9b56ad887`
 - PR type for this branch: `pure-move`
-- Runtime behavior changes: no external behavior change expected; endpoint
-  parsing, unsupported filesystem policy enforcement, global endpoint/erasure
-  type publication, local disk/prewarm, lock clients, and storage pool logging
-  still run in the same relative order between listen context and HTTP server
-  startup.
-- Rust code changes: add `startup_storage::init_startup_storage_foundation`
-  and use it from binary startup.
+- Runtime behavior changes: no external behavior change expected; ECStore
+  creation, ECStore config initialization, global config retry policy,
+  `StorageReady` publication, and background replication startup still run in
+  the same relative order after HTTP server startup and before KMS startup.
+- Rust code changes: add `startup_storage::init_startup_storage_runtime` and
+  use it from binary startup.
 - CI/script changes: none.
-- Docs changes: record `R-015` startup storage foundation bootstrap progress and
+- Docs changes: record `R-016` startup storage runtime bootstrap progress and
   verification.
 
 ## Phase 0 Tasks
@@ -662,6 +661,25 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
     formatting, migration guards, Rust risk scan, branch freshness check, and
     pre-commit quality gate.
 
+- [x] `R-016` Centralize startup storage runtime bootstrap.
+  - Do: move runtime cancellation token creation, ECStore initialization,
+    ECStore config initialization, server-config migration attempt, global
+    config retry loop, `StorageReady` stage publication, and background
+    replication startup behind the `startup_storage` boundary.
+  - Acceptance: storage runtime still starts after HTTP server startup and
+    before KMS startup; ECStore init failure keeps the same structured error log
+    and propagated error; global config init still logs every failed attempt,
+    sleeps between attempts, and becomes fatal after the 16th failed attempt;
+    `StorageReady` is still marked after global config init succeeds and before
+    background replication startup.
+  - Must preserve: cancellation token ownership for later shutdown, endpoint
+    pool clone ownership for ECStore startup, ECStore config init/migration
+    order, retry count/log fields, fatal error string, readiness stage timing,
+    and non-fatal background replication startup behavior.
+  - Verification: focused startup storage tests, binary/lib compile checks,
+    formatting, migration guards, Rust risk scan, branch freshness check, and
+    pre-commit quality gate.
+
 ## Next PRs
 
 1. `pure-move`: continue extracting startup boot wrappers in larger slices while
@@ -673,13 +691,13 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 
 | Expert | Status | Notes |
 |---|---|---|
-| Quality/architecture | passed | Pure-move slice removes storage foundation details from binary startup behind the existing startup module pattern. |
-| Migration preservation | passed | Endpoint parsing, filesystem policy, global endpoint/erasure publication, local disk/prewarm, lock clients, and pool logging order are preserved. |
+| Quality/architecture | passed | Pure-move slice removes storage runtime details from binary startup behind the existing startup storage boundary. |
+| Migration preservation | passed | HTTP server startup, ECStore creation, config init/migration, global config retry, `StorageReady`, replication startup, and KMS startup order are preserved. |
 | Testing/verification | passed | Focused startup storage tests, compile checks, formatting, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. |
 
 ## Verification Notes
 
-Passed on `6508f88d3a5edb428a5d623f927ce384691f0cd4`:
+Passed on `a19560da419c6d354a805fc127951cc9b56ad887`:
 
 - `cargo test -p rustfs startup_storage --no-fail-fast`: passed.
 - `cargo check -p rustfs --lib`: passed.
@@ -688,24 +706,23 @@ Passed on `6508f88d3a5edb428a5d623f927ce384691f0cd4`:
 - `git diff --check`: passed.
 - `./scripts/check_architecture_migration_rules.sh`: passed.
 - `./scripts/check_layer_dependencies.sh`: passed.
-- `git rev-list --left-right --count HEAD...origin/main` returned `0 0`
-  before commit.
+- `git rev-list --left-right --count HEAD...origin/main` returned `1 0`
+  after commit and rebase.
 - Added-line Rust risk scan for changed Rust files: no matches.
-- Full-file risk scan for changed Rust files: matches are existing docs example
-  output and existing binary startup alias/stderr/expect usage.
-- `make pre-commit`: all checks passed, including nextest with 6009 passed
+- Full-file risk scan for changed Rust files: matches are existing binary
+  startup stderr/expect usage.
+- `make pre-commit`: all checks passed, including nextest with 6016 passed
   and 111 skipped, plus doctests.
 
 Notes:
 
-- This slice centralizes startup storage foundation without changing startup
-  ordering or endpoint pool ownership.
-- Storage foundation remains after listen context setup and before HTTP server
-  startup.
+- This slice centralizes startup storage runtime without changing startup
+  ordering, shutdown token ownership, or endpoint pool ownership.
+- Storage runtime remains after HTTP server startup and before KMS startup.
 
 ## Handoff Notes
 
-- R-015 is complete.
+- R-016 is complete.
 - Next startup slices can keep using larger pure moves, but must keep startup
   ordering, fatal/non-fatal boundaries, shutdown ownership, and readiness
   ownership explicit in tests.

docs/architecture/startup-timeline.md

@@ -33,9 +33,9 @@ new startup semantics.
 | `RUN-008` | `rustfs/src/main.rs:311` | Initialize local disks, prewarm local disk id map, and initialize lock clients. | Opens local disk state, primes disk id lookup, and creates global lock clients. | Local disk init is fatal; prewarm and lock-client setup are non-fatal in this path. | None |
 | `RUN-009` | `rustfs/src/main.rs:350` | Initialize capacity management and service state manager. | Starts capacity management and moves service state to `Starting`. | Non-fatal in this path. | None |
 | `RUN-010` | `rustfs/src/main.rs:356` | Start S3 HTTP listener and optional console listener before storage is ready. | Starts HTTP servers with readiness gates; console listener starts only when enabled and configured. | Fatal if a configured listener cannot start. | Requests remain gated until full readiness except probe/admin/console/rpc/tonic/table-catalog exempt paths |
-| `RUN-011` | `rustfs/src/main.rs:372` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None |
-| `RUN-012` | `rustfs/src/main.rs:382` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, and lock quorum before `FullReady` |
-| `RUN-013` | `rustfs/src/main.rs:397` | Start replication and KMS systems. | Starts background replication pool and initializes KMS. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady` |
+| `RUN-011` | `rustfs/src/startup_storage.rs` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None |
+| `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, and lock quorum before `FullReady` |
+| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/main.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from main startup. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady` |
 | `RUN-014` | `rustfs/src/main.rs:402` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None |
 | `RUN-015` | `rustfs/src/main.rs:482` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
 | `RUN-016` | `rustfs/src/main.rs:503` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |

rustfs/src/main.rs

@@ -26,15 +26,12 @@ use rustfs::startup_iam::{bootstrap_or_defer_iam_init, publish_ready_for_iam_boo
 use rustfs::startup_preflight::{StartupServerPreflightError, bootstrap_external_prefix_compat, init_startup_server_preflight};
 use rustfs::startup_protocols::{ProtocolShutdownSenders, init_protocol_shutdown_senders};
 use rustfs::startup_server::{StartupHttpServers, StartupListenContext, init_startup_http_servers, init_startup_listen_context};
-use rustfs::startup_storage::init_startup_storage_foundation;
-use rustfs_common::SystemStage;
+use rustfs::startup_storage::{StartupStorageRuntime, init_startup_storage_foundation, init_startup_storage_runtime};
 use rustfs_ecstore::{
     bucket::metadata_sys::init_bucket_metadata_sys,
     bucket::migration::{try_migrate_bucket_metadata, try_migrate_iam_config},
-    bucket::replication::{get_global_replication_pool, init_background_replication},
-    config as ecconfig,
+    bucket::replication::get_global_replication_pool,
     global::shutdown_background_services,
-    store::ECStore,
     store_api::BucketOperations,
 };
 use rustfs_heal::{
@@ -57,9 +54,7 @@ const ENV_HEAL_ENABLED_DEPRECATED: &str = "RUSTFS_ENABLE_HEAL";
 const LOG_COMPONENT_MAIN: &str = "main";
 const LOG_SUBSYSTEM_STARTUP: &str = "startup";
 const LOG_SUBSYSTEM_AUTH: &str = "auth";
-const LOG_SUBSYSTEM_STORAGE: &str = "storage";
 const EVENT_SERVER_RUNTIME_FAILED: &str = "server_runtime_failed";
-const EVENT_STARTUP_STORAGE_STAGE: &str = "startup_storage_stage";
 const EVENT_PROTOCOL_SYSTEM_STATE: &str = "protocol_system_state";
 const EVENT_AUDIT_SYSTEM_STATE: &str = "audit_system_state";
 const EVENT_DEADLOCK_DETECTOR_STATE: &str = "deadlock_detector_state";
@@ -169,61 +164,11 @@ async fn run(config: rustfs::config::Config) -> Result<()> {
         console_shutdown_tx,
     } = init_startup_http_servers(&config, readiness.clone()).await?;
 
-    let ctx = CancellationToken::new();
+    let StartupStorageRuntime {
+        store,
+        shutdown_token: ctx,
+    } = init_startup_storage_runtime(server_addr, &endpoint_pools, readiness.clone()).await?;
 
-    // init store
-    // 2. Start Storage Engine (ECStore)
-    debug!(
-        target: "rustfs::main::run",
-        event = EVENT_STARTUP_STORAGE_STAGE,
-        component = LOG_COMPONENT_MAIN,
-        subsystem = LOG_SUBSYSTEM_STORAGE,
-        stage = "ecstore_initialization",
-        state = "starting",
-        "starting ECStore initialization"
-    );
-    let store = ECStore::new(server_addr, endpoint_pools.clone(), ctx.clone())
-        .await
-        .inspect_err(|err| {
-            error!(
-                target: "rustfs::main::run",
-                event = EVENT_STARTUP_STORAGE_STAGE,
-                component = LOG_COMPONENT_MAIN,
-                subsystem = LOG_SUBSYSTEM_STORAGE,
-                stage = "ecstore_initialization",
-                state = "failed",
-                error = ?err,
-                "ECStore initialization failed"
-            );
-        })?;
-
-    ecconfig::init();
-    ecconfig::try_migrate_server_config(store.clone()).await;
-
-    // // Initialize global configuration system
-    let mut retry_count = 0;
-    while let Err(e) = ecconfig::init_global_config_sys(store.clone()).await {
-        error!(
-            target: "rustfs::main::run",
-            event = EVENT_STARTUP_STORAGE_STAGE,
-            component = LOG_COMPONENT_MAIN,
-            subsystem = LOG_SUBSYSTEM_STORAGE,
-            stage = "global_config_initialization",
-            state = "retrying",
-            retry_count = retry_count + 1,
-            error = ?e,
-            "Global config initialization retry failed"
-        );
-        // TODO: check error type
-        retry_count += 1;
-        if retry_count > 15 {
-            return Err(Error::other("ecconfig::init_global_config_sys failed"));
-        }
-        tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
-    }
-    readiness.mark_stage(SystemStage::StorageReady);
-    // init replication_pool
-    init_background_replication(store.clone()).await;
     // Initialize KMS system if enabled
     init_kms_system(&config).await?;
 

rustfs/src/startup_storage.rs

@@ -13,22 +13,36 @@
 // limitations under the License.
 
 use crate::startup_fs_guard::enforce_unsupported_fs_policy;
+use rustfs_common::{GlobalReadiness, SystemStage};
 use rustfs_ecstore::{
+    bucket::replication::init_background_replication,
+    config as ecconfig,
     endpoints::EndpointServerPools,
     set_global_endpoints,
-    store::{init_local_disks, init_lock_clients, prewarm_local_disk_id_map},
+    store::{ECStore, init_local_disks, init_lock_clients, prewarm_local_disk_id_map},
     update_erasure_type,
 };
-use std::io::{Error, Result};
+use std::{
+    io::{Error, Result},
+    net::SocketAddr,
+    sync::Arc,
+};
+use tokio_util::sync::CancellationToken;
 use tracing::{debug, error, info, warn};
 
 const LOG_COMPONENT_MAIN: &str = "main";
 const LOG_SUBSYSTEM_STORAGE: &str = "storage";
+const GLOBAL_CONFIG_INIT_MAX_RETRIES: usize = 15;
 const EVENT_ENDPOINT_PARSING_STARTED: &str = "endpoint_parsing_started";
 const EVENT_STARTUP_STORAGE_STAGE: &str = "startup_storage_stage";
 const EVENT_STORAGE_POOL_FORMATTING: &str = "storage_pool_formatting";
 const EVENT_STORAGE_POOL_HOST_RISK: &str = "storage_pool_host_risk";
 
+pub struct StartupStorageRuntime {
+    pub store: Arc<ECStore>,
+    pub shutdown_token: CancellationToken,
+}
+
 pub async fn init_startup_storage_foundation(server_address: &str, volumes: &[String]) -> Result<EndpointServerPools> {
     info!(
         target: "rustfs::main::run",
@@ -91,6 +105,76 @@ pub async fn init_startup_storage_foundation(server_address: &str, volumes: &[St
     Ok(endpoint_pools)
 }
 
+pub async fn init_startup_storage_runtime(
+    server_addr: SocketAddr,
+    endpoint_pools: &EndpointServerPools,
+    readiness: Arc<GlobalReadiness>,
+) -> Result<StartupStorageRuntime> {
+    let ctx = CancellationToken::new();
+
+    debug!(
+        target: "rustfs::main::run",
+        event = EVENT_STARTUP_STORAGE_STAGE,
+        component = LOG_COMPONENT_MAIN,
+        subsystem = LOG_SUBSYSTEM_STORAGE,
+        stage = "ecstore_initialization",
+        state = "starting",
+        "starting ECStore initialization"
+    );
+    let store = ECStore::new(server_addr, endpoint_pools.clone(), ctx.clone())
+        .await
+        .inspect_err(|err| {
+            error!(
+                target: "rustfs::main::run",
+                event = EVENT_STARTUP_STORAGE_STAGE,
+                component = LOG_COMPONENT_MAIN,
+                subsystem = LOG_SUBSYSTEM_STORAGE,
+                stage = "ecstore_initialization",
+                state = "failed",
+                error = ?err,
+                "ECStore initialization failed"
+            );
+        })?;
+
+    init_startup_storage_global_config(store.clone()).await?;
+    readiness.mark_stage(SystemStage::StorageReady);
+    init_background_replication(store.clone()).await;
+
+    Ok(StartupStorageRuntime {
+        store,
+        shutdown_token: ctx,
+    })
+}
+
+async fn init_startup_storage_global_config(store: Arc<ECStore>) -> Result<()> {
+    ecconfig::init();
+    ecconfig::try_migrate_server_config(store.clone()).await;
+
+    let mut retry_count = 0;
+    while let Err(e) = ecconfig::init_global_config_sys(store.clone()).await {
+        let next_retry_count = retry_count + 1;
+        error!(
+            target: "rustfs::main::run",
+            event = EVENT_STARTUP_STORAGE_STAGE,
+            component = LOG_COMPONENT_MAIN,
+            subsystem = LOG_SUBSYSTEM_STORAGE,
+            stage = "global_config_initialization",
+            state = "retrying",
+            retry_count = next_retry_count,
+            error = ?e,
+            "Global config initialization retry failed"
+        );
+        // TODO: check error type
+        retry_count = next_retry_count;
+        if global_config_retry_exhausted(retry_count) {
+            return Err(Error::other("ecconfig::init_global_config_sys failed"));
+        }
+        tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
+    }
+
+    Ok(())
+}
+
 fn log_storage_pool_layout(endpoint_pools: &EndpointServerPools) {
     for (i, eps) in endpoint_pools.as_ref().iter().enumerate() {
         info!(
@@ -142,9 +226,13 @@ fn storage_pool_has_host_failure_risk(drives_per_set: usize) -> bool {
     drives_per_set > 1
 }
 
+fn global_config_retry_exhausted(retry_count: usize) -> bool {
+    retry_count > GLOBAL_CONFIG_INIT_MAX_RETRIES
+}
+
 #[cfg(test)]
 mod tests {
-    use super::storage_pool_has_host_failure_risk;
+    use super::{global_config_retry_exhausted, storage_pool_has_host_failure_risk};
 
     #[test]
     fn reports_host_failure_risk_only_for_multi_drive_sets() {
@@ -152,4 +240,10 @@ mod tests {
         assert!(!storage_pool_has_host_failure_risk(1));
         assert!(storage_pool_has_host_failure_risk(2));
     }
+
+    #[test]
+    fn global_config_retry_limit_matches_startup_policy() {
+        assert!(!global_config_retry_exhausted(15));
+        assert!(global_config_retry_exhausted(16));
+    }
 }