refactor: centralize startup runtime bootstrap (#3457)

docs/architecture/migration-progress.md

@@ -5,17 +5,18 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 ## Current Context
 
 - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
-- Branch: `overtrue/arch-startup-subsystems-bootstrap`
-- Baseline: `origin/main` at `949939137066435e590c8bd0f029ed1500e0e1cd`
+- Branch: `overtrue/arch-startup-core-runtime-bootstrap`
+- Baseline: `origin/main` at `73e534682e93d1ed2c7b0e752ad550d969ca96b2`
 - PR type for this branch: `pure-move`
-- Runtime behavior changes: no external behavior change expected; FTP, FTPS,
-  WebDAV, and SFTP startup still honors compile features and env-driven
-  enable/disable behavior, startup failure mapping, and shutdown handle flow.
-- Rust code changes: centralize protocol sidecar bootstrap in
-  `startup_protocols::init_protocol_shutdown_senders` and use it from binary
+- Runtime behavior changes: no external behavior change expected; startup
+  runtime foundation still logs dial9/license state, prints the logo, initializes
+  profiling and trusted proxies, installs the rustls provider, and publishes
+  outbound TLS material in the same order and with the same fatal boundary.
+- Rust code changes: centralize startup runtime foundation bootstrap in
+  `startup_runtime::init_startup_runtime_foundation` and use it from binary
   startup.
 - CI/script changes: none.
-- Docs changes: record `R-011` startup protocol bootstrap wrapper progress and
+- Docs changes: record `R-012` startup runtime foundation bootstrap progress and
   verification.
 
 ## Phase 0 Tasks
@@ -590,6 +591,24 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
   - Verification: focused startup protocol tests, binary/lib compile checks,
     formatting, migration guards, Rust risk scan, and pre-commit quality gate.
 
+- [x] `R-012` Centralize startup runtime foundation bootstrap.
+  - Do: move dial9 runtime status logging, runtime license status logging,
+    startup logo logging, profiling setup, trusted-proxy setup, rustls provider
+    setup, and outbound TLS material publication behind
+    `startup_runtime::init_startup_runtime_foundation`.
+  - Acceptance: BOOT-006 order is unchanged, configured TLS material load
+    remains fatal with the same `Error::other(err.to_string())` mapping, TLS
+    generation remains saturating, TLS metrics still initialize only when
+    metrics are enabled and TLS is configured, and profiling/proxy/provider
+    setup remains non-fatal.
+  - Must preserve: dial9/license log event names and fields, startup logo
+    logging, profiling init timing, trusted-proxy init timing, crypto provider
+    already-installed handling, outbound TLS publication, generation metric
+    consumer, TLS metric init condition, and fatal boundaries.
+  - Verification: focused startup runtime tests, binary/lib compile checks,
+    formatting, migration guards, Rust risk scan, branch freshness check, and
+    pre-commit quality gate.
+
 ## Next PRs
 
 1. `pure-move`: continue extracting startup boot wrappers in larger slices while
@@ -601,39 +620,38 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 
 | Expert | Status | Notes |
 |---|---|---|
-| Quality/architecture | passed | Pure-move slice centralizes protocol sidecar startup while keeping the binary boundary in charge of shutdown wiring. |
-| Migration preservation | passed | Protocol feature gates, env-driven disabled handling, startup log event/state/protocol fields, and `Error::other` failure mapping are preserved. |
-| Testing/verification | passed | Focused startup protocol tests, compile checks, formatting, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. |
+| Quality/architecture | passed | Pure-move slice removes startup runtime foundation details from binary startup without changing ownership or adding abstraction beyond the existing startup module pattern. |
+| Migration preservation | passed | BOOT-006 order, dial9/license log fields, profiling/proxy/provider non-fatal setup, TLS fatal boundary, generation saturation, and TLS metric condition are preserved. |
+| Testing/verification | passed | Focused startup runtime tests, compile checks, formatting, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. |
 
 ## Verification Notes
 
-Passed on `949939137066435e590c8bd0f029ed1500e0e1cd`:
+Passed on `73e534682e93d1ed2c7b0e752ad550d969ca96b2`:
 
+- `cargo test -p rustfs startup_runtime --no-fail-fast`.
 - `cargo check -p rustfs --lib`.
 - `cargo check -p rustfs --bin rustfs`.
-- `cargo test -p rustfs startup_protocols --no-fail-fast`.
 - `cargo fmt --all --check`.
 - `git diff --check`.
 - `./scripts/check_architecture_migration_rules.sh`.
 - `./scripts/check_layer_dependencies.sh`.
 - `git rev-list --left-right --count HEAD...origin/main` returned `0 0`.
-- Rust risk scan for changed Rust files: the only added-line match is the
-  `Box<dyn Error + Send + Sync>` type required by existing protocol init
-  signatures; full-file matches were existing docs examples and existing binary
-  startup error output.
-- `make pre-commit`: all checks passed, including nextest with 5971 passed and
+- Added-line Rust risk scan for changed Rust files: no matches.
+- Full-file risk scan for changed Rust files: matches are existing docs example
+  output and existing binary startup stderr/expect usage.
+- `make pre-commit`: all checks passed, including nextest with 6002 passed and
   111 skipped, plus doctests.
 
 Notes:
 
-- This slice centralizes startup protocol sidecar bootstrap without changing
-  startup ordering or shutdown handle ownership.
-- Protocol logging keeps the existing event, subsystem, protocol, and state
-  values.
+- This slice centralizes startup runtime foundation bootstrap without changing
+  startup ordering or fatal boundaries.
+- Observability initialization remains in binary startup so early fatal stderr
+  handling stays unchanged.
 
 ## Handoff Notes
 
-- R-011 is complete.
-- Next startup slices can be larger pure moves, but must keep startup ordering,
-  fatal/non-fatal boundaries, shutdown ownership, and readiness ownership
-  explicit in tests.
+- R-012 is complete.
+- Next startup slices can keep using larger pure moves, but must keep startup
+  ordering, fatal/non-fatal boundaries, shutdown ownership, and readiness
+  ownership explicit in tests.

rustfs/src/lib.rs

@@ -70,6 +70,7 @@ pub mod server;
 pub mod startup_fs_guard;
 pub mod startup_iam;
 pub mod startup_protocols;
+pub mod startup_runtime;
 pub mod startup_services;
 pub mod storage;
 pub(crate) mod table_catalog;

rustfs/src/main.rs

@@ -19,7 +19,7 @@ use rustfs::init::{
 
 use futures_util::future::join_all;
 use rustfs::capacity::capacity_integration::init_capacity_management;
-use rustfs::license::{init_license, license_status};
+use rustfs::license::init_license;
 use rustfs::server::{
     ServiceState, ServiceStateManager, ShutdownHandle, ShutdownSignal, shutdown_event_notifier, start_http_server,
     stop_audit_system, wait_for_shutdown,
@@ -27,6 +27,7 @@ use rustfs::server::{
 use rustfs::startup_fs_guard::enforce_unsupported_fs_policy;
 use rustfs::startup_iam::{bootstrap_or_defer_iam_init, publish_ready_for_iam_bootstrap};
 use rustfs::startup_protocols::{ProtocolShutdownSenders, init_protocol_shutdown_senders};
+use rustfs::startup_runtime::init_startup_runtime_foundation;
 use rustfs_common::{GlobalReadiness, SystemStage, set_global_addr};
 use rustfs_credentials::init_global_action_credentials;
 use rustfs_ecstore::store::init_lock_clients;
@@ -54,7 +55,6 @@ use rustfs_storage_api::BucketOptions;
 use rustfs_utils::{
     ExternalEnvCompatReport, apply_external_env_compat, get_env_bool_with_aliases, net::parse_and_resolve_address,
 };
-use rustls::crypto::aws_lc_rs::default_provider;
 use std::io::{Error, Result};
 use std::sync::Arc;
 use tokio_util::sync::CancellationToken;
@@ -66,16 +66,11 @@ const ENV_HEAL_ENABLED: &str = "RUSTFS_HEAL_ENABLED";
 const ENV_HEAL_ENABLED_DEPRECATED: &str = "RUSTFS_ENABLE_HEAL";
 const LOG_COMPONENT_MAIN: &str = "main";
 const LOG_SUBSYSTEM_STARTUP: &str = "startup";
-const LOG_SUBSYSTEM_LICENSE: &str = "license";
 const LOG_SUBSYSTEM_AUTH: &str = "auth";
 const LOG_SUBSYSTEM_STORAGE: &str = "storage";
 const EVENT_EXTERNAL_ENV_COMPAT_CONFLICT: &str = "external_env_compat_conflict";
 const EVENT_EXTERNAL_ENV_COMPAT_APPLIED: &str = "external_env_compat_applied";
-const EVENT_DIAL9_RUNTIME_STATUS: &str = "dial9_runtime_status";
-const EVENT_RUNTIME_LICENSE_STATUS: &str = "runtime_license_status";
 const EVENT_OBSERVABILITY_GUARD_SET_FAILED: &str = "observability_guard_set_failed";
-const EVENT_TLS_OUTBOUND_INITIALIZED: &str = "tls_outbound_initialized";
-const EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED: &str = "tls_outbound_initialization_failed";
 const EVENT_DEFAULT_CREDENTIALS_DETECTED: &str = "default_credentials_detected";
 const EVENT_SERVER_CONFIG_SANITIZED: &str = "server_config_sanitized";
 const EVENT_SERVER_STARTING: &str = "server_starting";
@@ -83,7 +78,6 @@ const EVENT_SERVER_RUNTIME_FAILED: &str = "server_runtime_failed";
 const EVENT_ACTION_CREDENTIALS_INITIALIZED: &str = "action_credentials_initialized";
 const EVENT_ACTION_CREDENTIALS_INITIALIZATION_FAILED: &str = "action_credentials_initialization_failed";
 const EVENT_OBSERVABILITY_GUARD_SET: &str = "observability_guard_set";
-const EVENT_CRYPTO_PROVIDER_STATE: &str = "crypto_provider_state";
 const EVENT_ENDPOINT_PARSING_STARTED: &str = "endpoint_parsing_started";
 const EVENT_STARTUP_STORAGE_STAGE: &str = "startup_storage_stage";
 const EVENT_STORAGE_POOL_FORMATTING: &str = "storage_pool_formatting";
@@ -217,99 +211,7 @@ async fn async_main() -> Result<()> {
     }
 
     log_external_prefix_compat_report(&env_compat_report);
-
-    // Check dial9 Tokio runtime telemetry status
-    // Note: The actual telemetry session is created in build_tokio_runtime()
-    // which stores the TelemetryGuard globally for the program duration.
-    if rustfs_obs::dial9::is_enabled() {
-        info!(
-            target: "rustfs::main",
-            event = EVENT_DIAL9_RUNTIME_STATUS,
-            component = LOG_COMPONENT_MAIN,
-            subsystem = LOG_SUBSYSTEM_STARTUP,
-            enabled = true,
-            "Dial9 Tokio runtime telemetry is enabled"
-        );
-    } else {
-        debug!(
-            target: "rustfs::main",
-            event = EVENT_DIAL9_RUNTIME_STATUS,
-            component = LOG_COMPONENT_MAIN,
-            subsystem = LOG_SUBSYSTEM_STARTUP,
-            enabled = false,
-            "Dial9 Tokio runtime telemetry is disabled"
-        );
-    }
-
-    info!(
-        target: "rustfs::main",
-        event = EVENT_RUNTIME_LICENSE_STATUS,
-        component = LOG_COMPONENT_MAIN,
-        subsystem = LOG_SUBSYSTEM_LICENSE,
-        license_status = %license_status(),
-        "Initialized runtime license state"
-    );
-
-    // print startup logo
-    debug!("{}", rustfs::server::LOGO);
-
-    // Initialize performance profiling if enabled
-    rustfs::profiling::init_from_env().await;
-
-    // Initialize trusted proxies system
-    rustfs_trusted_proxies::init();
-
-    // Make sure to use a modern encryption suite
-    if default_provider().install_default().is_err() {
-        // A crypto provider is already installed (e.g. by the host process); this is fine.
-        debug!(
-            target: "rustfs::main",
-            event = EVENT_CRYPTO_PROVIDER_STATE,
-            component = LOG_COMPONENT_MAIN,
-            subsystem = LOG_SUBSYSTEM_STARTUP,
-            provider = "aws_lc_rs",
-            state = "already_installed",
-            "Rustls crypto provider state checked"
-        );
-    }
-    // Initialize TLS outbound material (root CAs, mTLS identity) if configured.
-    // Server-side TLS acceptor is built separately inside start_http_server().
-    // Single load via tls-runtime; outbound is enriched with platform CAs and
-    // published to the global state before any HTTP listener starts.
-    if let Some(tls_path) = config.tls_path.as_deref().map(str::trim).filter(|path| !path.is_empty()) {
-        match rustfs::server::tls_material::load_tls_material(tls_path).await {
-            Ok(snapshot) => {
-                use rustfs_tls_runtime::{TlsGeneration, publish_global_outbound_tls_state, record_tls_generation};
-                let generation = TlsGeneration(rustfs_common::get_global_outbound_tls_generation().saturating_add(1));
-                publish_global_outbound_tls_state(generation, &snapshot.outbound).await;
-                record_tls_generation("rustfs_server_startup", generation.0);
-                info!(
-                    target: "rustfs::main",
-                    event = EVENT_TLS_OUTBOUND_INITIALIZED,
-                    component = LOG_COMPONENT_MAIN,
-                    subsystem = LOG_SUBSYSTEM_STARTUP,
-                    tls_path,
-                    generation = generation.0,
-                    "Initialized TLS outbound material"
-                );
-            }
-            Err(e) => {
-                error!(
-                    target: "rustfs::main",
-                    event = EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED,
-                    component = LOG_COMPONENT_MAIN,
-                    subsystem = LOG_SUBSYSTEM_STARTUP,
-                    tls_path,
-                    error = %e,
-                    "Failed to initialize TLS outbound material"
-                );
-                return Err(Error::other(e.to_string()));
-            }
-        }
-        if rustfs_obs::observability_metric_enabled() {
-            rustfs_tls_runtime::init_tls_metrics();
-        }
-    }
+    init_startup_runtime_foundation(&config).await?;
 
     // Run parameters
     match run(*config).await {

rustfs/src/startup_runtime.rs

@@ -0,0 +1,158 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::{config::Config, license::license_status};
+use rustls::crypto::aws_lc_rs::default_provider;
+use std::io::{Error, Result};
+use tracing::{debug, error, info};
+
+const LOG_COMPONENT_MAIN: &str = "main";
+const LOG_SUBSYSTEM_LICENSE: &str = "license";
+const LOG_SUBSYSTEM_STARTUP: &str = "startup";
+const EVENT_CRYPTO_PROVIDER_STATE: &str = "crypto_provider_state";
+const EVENT_DIAL9_RUNTIME_STATUS: &str = "dial9_runtime_status";
+const EVENT_RUNTIME_LICENSE_STATUS: &str = "runtime_license_status";
+const EVENT_TLS_OUTBOUND_INITIALIZED: &str = "tls_outbound_initialized";
+const EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED: &str = "tls_outbound_initialization_failed";
+const TLS_STARTUP_GENERATION_CONSUMER: &str = "rustfs_server_startup";
+
+pub async fn init_startup_runtime_foundation(config: &Config) -> Result<()> {
+    log_dial9_runtime_status();
+    log_runtime_license_status();
+
+    debug!("{}", crate::server::LOGO);
+
+    crate::profiling::init_from_env().await;
+    rustfs_trusted_proxies::init();
+    install_default_crypto_provider();
+    init_outbound_tls_material(config).await
+}
+
+fn log_dial9_runtime_status() {
+    if rustfs_obs::dial9::is_enabled() {
+        info!(
+            target: "rustfs::main",
+            event = EVENT_DIAL9_RUNTIME_STATUS,
+            component = LOG_COMPONENT_MAIN,
+            subsystem = LOG_SUBSYSTEM_STARTUP,
+            enabled = true,
+            "Dial9 Tokio runtime telemetry is enabled"
+        );
+    } else {
+        debug!(
+            target: "rustfs::main",
+            event = EVENT_DIAL9_RUNTIME_STATUS,
+            component = LOG_COMPONENT_MAIN,
+            subsystem = LOG_SUBSYSTEM_STARTUP,
+            enabled = false,
+            "Dial9 Tokio runtime telemetry is disabled"
+        );
+    }
+}
+
+fn log_runtime_license_status() {
+    info!(
+        target: "rustfs::main",
+        event = EVENT_RUNTIME_LICENSE_STATUS,
+        component = LOG_COMPONENT_MAIN,
+        subsystem = LOG_SUBSYSTEM_LICENSE,
+        license_status = %license_status(),
+        "Initialized runtime license state"
+    );
+}
+
+fn install_default_crypto_provider() {
+    if default_provider().install_default().is_err() {
+        debug!(
+            target: "rustfs::main",
+            event = EVENT_CRYPTO_PROVIDER_STATE,
+            component = LOG_COMPONENT_MAIN,
+            subsystem = LOG_SUBSYSTEM_STARTUP,
+            provider = "aws_lc_rs",
+            state = "already_installed",
+            "Rustls crypto provider state checked"
+        );
+    }
+}
+
+async fn init_outbound_tls_material(config: &Config) -> Result<()> {
+    if let Some(tls_path) = normalized_tls_path(config.tls_path.as_deref()) {
+        match crate::server::tls_material::load_tls_material(tls_path).await {
+            Ok(snapshot) => {
+                use rustfs_tls_runtime::{publish_global_outbound_tls_state, record_tls_generation};
+
+                let generation = next_tls_generation(rustfs_common::get_global_outbound_tls_generation());
+                publish_global_outbound_tls_state(generation, &snapshot.outbound).await;
+                record_tls_generation(TLS_STARTUP_GENERATION_CONSUMER, generation.0);
+                info!(
+                    target: "rustfs::main",
+                    event = EVENT_TLS_OUTBOUND_INITIALIZED,
+                    component = LOG_COMPONENT_MAIN,
+                    subsystem = LOG_SUBSYSTEM_STARTUP,
+                    tls_path,
+                    generation = generation.0,
+                    "Initialized TLS outbound material"
+                );
+            }
+            Err(err) => {
+                error!(
+                    target: "rustfs::main",
+                    event = EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED,
+                    component = LOG_COMPONENT_MAIN,
+                    subsystem = LOG_SUBSYSTEM_STARTUP,
+                    tls_path,
+                    error = %err,
+                    "Failed to initialize TLS outbound material"
+                );
+                return Err(Error::other(err.to_string()));
+            }
+        }
+        if rustfs_obs::observability_metric_enabled() {
+            rustfs_tls_runtime::init_tls_metrics();
+        }
+    }
+
+    Ok(())
+}
+
+fn normalized_tls_path(path: Option<&str>) -> Option<&str> {
+    path.map(str::trim).filter(|value| !value.is_empty())
+}
+
+fn next_tls_generation(current: u64) -> rustfs_tls_runtime::TlsGeneration {
+    rustfs_tls_runtime::TlsGeneration(current.saturating_add(1))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn normalized_tls_path_ignores_empty_values() {
+        assert_eq!(normalized_tls_path(None), None);
+        assert_eq!(normalized_tls_path(Some("")), None);
+        assert_eq!(normalized_tls_path(Some("   ")), None);
+    }
+
+    #[test]
+    fn normalized_tls_path_trims_configured_path() {
+        assert_eq!(normalized_tls_path(Some(" /tmp/rustfs-tls ")), Some("/tmp/rustfs-tls"));
+    }
+
+    #[test]
+    fn next_tls_generation_saturates() {
+        assert_eq!(next_tls_generation(0).0, 1);
+        assert_eq!(next_tls_generation(u64::MAX).0, u64::MAX);
+    }
+}