diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index 5a10bc508..09e99e824 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,17 +5,18 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-startup-subsystems-bootstrap` -- Baseline: `origin/main` at `949939137066435e590c8bd0f029ed1500e0e1cd` +- Branch: `overtrue/arch-startup-core-runtime-bootstrap` +- Baseline: `origin/main` at `73e534682e93d1ed2c7b0e752ad550d969ca96b2` - PR type for this branch: `pure-move` -- Runtime behavior changes: no external behavior change expected; FTP, FTPS, - WebDAV, and SFTP startup still honors compile features and env-driven - enable/disable behavior, startup failure mapping, and shutdown handle flow. -- Rust code changes: centralize protocol sidecar bootstrap in - `startup_protocols::init_protocol_shutdown_senders` and use it from binary +- Runtime behavior changes: no external behavior change expected; startup + runtime foundation still logs dial9/license state, prints the logo, initializes + profiling and trusted proxies, installs the rustls provider, and publishes + outbound TLS material in the same order and with the same fatal boundary. +- Rust code changes: centralize startup runtime foundation bootstrap in + `startup_runtime::init_startup_runtime_foundation` and use it from binary startup. - CI/script changes: none. -- Docs changes: record `R-011` startup protocol bootstrap wrapper progress and +- Docs changes: record `R-012` startup runtime foundation bootstrap progress and verification. ## Phase 0 Tasks @@ -590,6 +591,24 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block - Verification: focused startup protocol tests, binary/lib compile checks, formatting, migration guards, Rust risk scan, and pre-commit quality gate. +- [x] `R-012` Centralize startup runtime foundation bootstrap. + - Do: move dial9 runtime status logging, runtime license status logging, + startup logo logging, profiling setup, trusted-proxy setup, rustls provider + setup, and outbound TLS material publication behind + `startup_runtime::init_startup_runtime_foundation`. + - Acceptance: BOOT-006 order is unchanged, configured TLS material load + remains fatal with the same `Error::other(err.to_string())` mapping, TLS + generation remains saturating, TLS metrics still initialize only when + metrics are enabled and TLS is configured, and profiling/proxy/provider + setup remains non-fatal. + - Must preserve: dial9/license log event names and fields, startup logo + logging, profiling init timing, trusted-proxy init timing, crypto provider + already-installed handling, outbound TLS publication, generation metric + consumer, TLS metric init condition, and fatal boundaries. + - Verification: focused startup runtime tests, binary/lib compile checks, + formatting, migration guards, Rust risk scan, branch freshness check, and + pre-commit quality gate. + ## Next PRs 1. `pure-move`: continue extracting startup boot wrappers in larger slices while @@ -601,39 +620,38 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block | Expert | Status | Notes | |---|---|---| -| Quality/architecture | passed | Pure-move slice centralizes protocol sidecar startup while keeping the binary boundary in charge of shutdown wiring. | -| Migration preservation | passed | Protocol feature gates, env-driven disabled handling, startup log event/state/protocol fields, and `Error::other` failure mapping are preserved. | -| Testing/verification | passed | Focused startup protocol tests, compile checks, formatting, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. | +| Quality/architecture | passed | Pure-move slice removes startup runtime foundation details from binary startup without changing ownership or adding abstraction beyond the existing startup module pattern. | +| Migration preservation | passed | BOOT-006 order, dial9/license log fields, profiling/proxy/provider non-fatal setup, TLS fatal boundary, generation saturation, and TLS metric condition are preserved. | +| Testing/verification | passed | Focused startup runtime tests, compile checks, formatting, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. | ## Verification Notes -Passed on `949939137066435e590c8bd0f029ed1500e0e1cd`: +Passed on `73e534682e93d1ed2c7b0e752ad550d969ca96b2`: +- `cargo test -p rustfs startup_runtime --no-fail-fast`. - `cargo check -p rustfs --lib`. - `cargo check -p rustfs --bin rustfs`. -- `cargo test -p rustfs startup_protocols --no-fail-fast`. - `cargo fmt --all --check`. - `git diff --check`. - `./scripts/check_architecture_migration_rules.sh`. - `./scripts/check_layer_dependencies.sh`. - `git rev-list --left-right --count HEAD...origin/main` returned `0 0`. -- Rust risk scan for changed Rust files: the only added-line match is the - `Box` type required by existing protocol init - signatures; full-file matches were existing docs examples and existing binary - startup error output. -- `make pre-commit`: all checks passed, including nextest with 5971 passed and +- Added-line Rust risk scan for changed Rust files: no matches. +- Full-file risk scan for changed Rust files: matches are existing docs example + output and existing binary startup stderr/expect usage. +- `make pre-commit`: all checks passed, including nextest with 6002 passed and 111 skipped, plus doctests. Notes: -- This slice centralizes startup protocol sidecar bootstrap without changing - startup ordering or shutdown handle ownership. -- Protocol logging keeps the existing event, subsystem, protocol, and state - values. +- This slice centralizes startup runtime foundation bootstrap without changing + startup ordering or fatal boundaries. +- Observability initialization remains in binary startup so early fatal stderr + handling stays unchanged. ## Handoff Notes -- R-011 is complete. -- Next startup slices can be larger pure moves, but must keep startup ordering, - fatal/non-fatal boundaries, shutdown ownership, and readiness ownership - explicit in tests. +- R-012 is complete. +- Next startup slices can keep using larger pure moves, but must keep startup + ordering, fatal/non-fatal boundaries, shutdown ownership, and readiness + ownership explicit in tests. diff --git a/rustfs/src/lib.rs b/rustfs/src/lib.rs index a6e41bef9..ea253bfc9 100644 --- a/rustfs/src/lib.rs +++ b/rustfs/src/lib.rs @@ -70,6 +70,7 @@ pub mod server; pub mod startup_fs_guard; pub mod startup_iam; pub mod startup_protocols; +pub mod startup_runtime; pub mod startup_services; pub mod storage; pub(crate) mod table_catalog; diff --git a/rustfs/src/main.rs b/rustfs/src/main.rs index 0b5ca339f..a7660fbaa 100644 --- a/rustfs/src/main.rs +++ b/rustfs/src/main.rs @@ -19,7 +19,7 @@ use rustfs::init::{ use futures_util::future::join_all; use rustfs::capacity::capacity_integration::init_capacity_management; -use rustfs::license::{init_license, license_status}; +use rustfs::license::init_license; use rustfs::server::{ ServiceState, ServiceStateManager, ShutdownHandle, ShutdownSignal, shutdown_event_notifier, start_http_server, stop_audit_system, wait_for_shutdown, @@ -27,6 +27,7 @@ use rustfs::server::{ use rustfs::startup_fs_guard::enforce_unsupported_fs_policy; use rustfs::startup_iam::{bootstrap_or_defer_iam_init, publish_ready_for_iam_bootstrap}; use rustfs::startup_protocols::{ProtocolShutdownSenders, init_protocol_shutdown_senders}; +use rustfs::startup_runtime::init_startup_runtime_foundation; use rustfs_common::{GlobalReadiness, SystemStage, set_global_addr}; use rustfs_credentials::init_global_action_credentials; use rustfs_ecstore::store::init_lock_clients; @@ -54,7 +55,6 @@ use rustfs_storage_api::BucketOptions; use rustfs_utils::{ ExternalEnvCompatReport, apply_external_env_compat, get_env_bool_with_aliases, net::parse_and_resolve_address, }; -use rustls::crypto::aws_lc_rs::default_provider; use std::io::{Error, Result}; use std::sync::Arc; use tokio_util::sync::CancellationToken; @@ -66,16 +66,11 @@ const ENV_HEAL_ENABLED: &str = "RUSTFS_HEAL_ENABLED"; const ENV_HEAL_ENABLED_DEPRECATED: &str = "RUSTFS_ENABLE_HEAL"; const LOG_COMPONENT_MAIN: &str = "main"; const LOG_SUBSYSTEM_STARTUP: &str = "startup"; -const LOG_SUBSYSTEM_LICENSE: &str = "license"; const LOG_SUBSYSTEM_AUTH: &str = "auth"; const LOG_SUBSYSTEM_STORAGE: &str = "storage"; const EVENT_EXTERNAL_ENV_COMPAT_CONFLICT: &str = "external_env_compat_conflict"; const EVENT_EXTERNAL_ENV_COMPAT_APPLIED: &str = "external_env_compat_applied"; -const EVENT_DIAL9_RUNTIME_STATUS: &str = "dial9_runtime_status"; -const EVENT_RUNTIME_LICENSE_STATUS: &str = "runtime_license_status"; const EVENT_OBSERVABILITY_GUARD_SET_FAILED: &str = "observability_guard_set_failed"; -const EVENT_TLS_OUTBOUND_INITIALIZED: &str = "tls_outbound_initialized"; -const EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED: &str = "tls_outbound_initialization_failed"; const EVENT_DEFAULT_CREDENTIALS_DETECTED: &str = "default_credentials_detected"; const EVENT_SERVER_CONFIG_SANITIZED: &str = "server_config_sanitized"; const EVENT_SERVER_STARTING: &str = "server_starting"; @@ -83,7 +78,6 @@ const EVENT_SERVER_RUNTIME_FAILED: &str = "server_runtime_failed"; const EVENT_ACTION_CREDENTIALS_INITIALIZED: &str = "action_credentials_initialized"; const EVENT_ACTION_CREDENTIALS_INITIALIZATION_FAILED: &str = "action_credentials_initialization_failed"; const EVENT_OBSERVABILITY_GUARD_SET: &str = "observability_guard_set"; -const EVENT_CRYPTO_PROVIDER_STATE: &str = "crypto_provider_state"; const EVENT_ENDPOINT_PARSING_STARTED: &str = "endpoint_parsing_started"; const EVENT_STARTUP_STORAGE_STAGE: &str = "startup_storage_stage"; const EVENT_STORAGE_POOL_FORMATTING: &str = "storage_pool_formatting"; @@ -217,99 +211,7 @@ async fn async_main() -> Result<()> { } log_external_prefix_compat_report(&env_compat_report); - - // Check dial9 Tokio runtime telemetry status - // Note: The actual telemetry session is created in build_tokio_runtime() - // which stores the TelemetryGuard globally for the program duration. - if rustfs_obs::dial9::is_enabled() { - info!( - target: "rustfs::main", - event = EVENT_DIAL9_RUNTIME_STATUS, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STARTUP, - enabled = true, - "Dial9 Tokio runtime telemetry is enabled" - ); - } else { - debug!( - target: "rustfs::main", - event = EVENT_DIAL9_RUNTIME_STATUS, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STARTUP, - enabled = false, - "Dial9 Tokio runtime telemetry is disabled" - ); - } - - info!( - target: "rustfs::main", - event = EVENT_RUNTIME_LICENSE_STATUS, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_LICENSE, - license_status = %license_status(), - "Initialized runtime license state" - ); - - // print startup logo - debug!("{}", rustfs::server::LOGO); - - // Initialize performance profiling if enabled - rustfs::profiling::init_from_env().await; - - // Initialize trusted proxies system - rustfs_trusted_proxies::init(); - - // Make sure to use a modern encryption suite - if default_provider().install_default().is_err() { - // A crypto provider is already installed (e.g. by the host process); this is fine. - debug!( - target: "rustfs::main", - event = EVENT_CRYPTO_PROVIDER_STATE, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STARTUP, - provider = "aws_lc_rs", - state = "already_installed", - "Rustls crypto provider state checked" - ); - } - // Initialize TLS outbound material (root CAs, mTLS identity) if configured. - // Server-side TLS acceptor is built separately inside start_http_server(). - // Single load via tls-runtime; outbound is enriched with platform CAs and - // published to the global state before any HTTP listener starts. - if let Some(tls_path) = config.tls_path.as_deref().map(str::trim).filter(|path| !path.is_empty()) { - match rustfs::server::tls_material::load_tls_material(tls_path).await { - Ok(snapshot) => { - use rustfs_tls_runtime::{TlsGeneration, publish_global_outbound_tls_state, record_tls_generation}; - let generation = TlsGeneration(rustfs_common::get_global_outbound_tls_generation().saturating_add(1)); - publish_global_outbound_tls_state(generation, &snapshot.outbound).await; - record_tls_generation("rustfs_server_startup", generation.0); - info!( - target: "rustfs::main", - event = EVENT_TLS_OUTBOUND_INITIALIZED, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STARTUP, - tls_path, - generation = generation.0, - "Initialized TLS outbound material" - ); - } - Err(e) => { - error!( - target: "rustfs::main", - event = EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STARTUP, - tls_path, - error = %e, - "Failed to initialize TLS outbound material" - ); - return Err(Error::other(e.to_string())); - } - } - if rustfs_obs::observability_metric_enabled() { - rustfs_tls_runtime::init_tls_metrics(); - } - } + init_startup_runtime_foundation(&config).await?; // Run parameters match run(*config).await { diff --git a/rustfs/src/startup_runtime.rs b/rustfs/src/startup_runtime.rs new file mode 100644 index 000000000..2003a9bc0 --- /dev/null +++ b/rustfs/src/startup_runtime.rs @@ -0,0 +1,158 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use crate::{config::Config, license::license_status}; +use rustls::crypto::aws_lc_rs::default_provider; +use std::io::{Error, Result}; +use tracing::{debug, error, info}; + +const LOG_COMPONENT_MAIN: &str = "main"; +const LOG_SUBSYSTEM_LICENSE: &str = "license"; +const LOG_SUBSYSTEM_STARTUP: &str = "startup"; +const EVENT_CRYPTO_PROVIDER_STATE: &str = "crypto_provider_state"; +const EVENT_DIAL9_RUNTIME_STATUS: &str = "dial9_runtime_status"; +const EVENT_RUNTIME_LICENSE_STATUS: &str = "runtime_license_status"; +const EVENT_TLS_OUTBOUND_INITIALIZED: &str = "tls_outbound_initialized"; +const EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED: &str = "tls_outbound_initialization_failed"; +const TLS_STARTUP_GENERATION_CONSUMER: &str = "rustfs_server_startup"; + +pub async fn init_startup_runtime_foundation(config: &Config) -> Result<()> { + log_dial9_runtime_status(); + log_runtime_license_status(); + + debug!("{}", crate::server::LOGO); + + crate::profiling::init_from_env().await; + rustfs_trusted_proxies::init(); + install_default_crypto_provider(); + init_outbound_tls_material(config).await +} + +fn log_dial9_runtime_status() { + if rustfs_obs::dial9::is_enabled() { + info!( + target: "rustfs::main", + event = EVENT_DIAL9_RUNTIME_STATUS, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + enabled = true, + "Dial9 Tokio runtime telemetry is enabled" + ); + } else { + debug!( + target: "rustfs::main", + event = EVENT_DIAL9_RUNTIME_STATUS, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + enabled = false, + "Dial9 Tokio runtime telemetry is disabled" + ); + } +} + +fn log_runtime_license_status() { + info!( + target: "rustfs::main", + event = EVENT_RUNTIME_LICENSE_STATUS, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_LICENSE, + license_status = %license_status(), + "Initialized runtime license state" + ); +} + +fn install_default_crypto_provider() { + if default_provider().install_default().is_err() { + debug!( + target: "rustfs::main", + event = EVENT_CRYPTO_PROVIDER_STATE, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + provider = "aws_lc_rs", + state = "already_installed", + "Rustls crypto provider state checked" + ); + } +} + +async fn init_outbound_tls_material(config: &Config) -> Result<()> { + if let Some(tls_path) = normalized_tls_path(config.tls_path.as_deref()) { + match crate::server::tls_material::load_tls_material(tls_path).await { + Ok(snapshot) => { + use rustfs_tls_runtime::{publish_global_outbound_tls_state, record_tls_generation}; + + let generation = next_tls_generation(rustfs_common::get_global_outbound_tls_generation()); + publish_global_outbound_tls_state(generation, &snapshot.outbound).await; + record_tls_generation(TLS_STARTUP_GENERATION_CONSUMER, generation.0); + info!( + target: "rustfs::main", + event = EVENT_TLS_OUTBOUND_INITIALIZED, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + tls_path, + generation = generation.0, + "Initialized TLS outbound material" + ); + } + Err(err) => { + error!( + target: "rustfs::main", + event = EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + tls_path, + error = %err, + "Failed to initialize TLS outbound material" + ); + return Err(Error::other(err.to_string())); + } + } + if rustfs_obs::observability_metric_enabled() { + rustfs_tls_runtime::init_tls_metrics(); + } + } + + Ok(()) +} + +fn normalized_tls_path(path: Option<&str>) -> Option<&str> { + path.map(str::trim).filter(|value| !value.is_empty()) +} + +fn next_tls_generation(current: u64) -> rustfs_tls_runtime::TlsGeneration { + rustfs_tls_runtime::TlsGeneration(current.saturating_add(1)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn normalized_tls_path_ignores_empty_values() { + assert_eq!(normalized_tls_path(None), None); + assert_eq!(normalized_tls_path(Some("")), None); + assert_eq!(normalized_tls_path(Some(" ")), None); + } + + #[test] + fn normalized_tls_path_trims_configured_path() { + assert_eq!(normalized_tls_path(Some(" /tmp/rustfs-tls ")), Some("/tmp/rustfs-tls")); + } + + #[test] + fn next_tls_generation_saturates() { + assert_eq!(next_tls_generation(0).0, 1); + assert_eq!(next_tls_generation(u64::MAX).0, u64::MAX); + } +}