diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index cc3c3ec93..5cad85b42 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,17 +5,17 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-protocol-runtime-sidecar-boundary` +- Branch: `overtrue/arch-startup-runtime-hooks-boundary` - Baseline: `origin/main` - (`aa5de1c908d6ffb778533c895c200f3c60ca1ed5`). -- Stacked on: rustfs/rustfs#3636. + (`5e96151d9c57cc8764f187cc0838dc111ddf0636`). +- Stacked on: rustfs/rustfs#3637. - PR type for this branch: `pure-move` - Runtime behavior changes: none. -- Rust code changes: move optional runtime sidecar ownership and protocol - shutdown planning from `startup_optional_runtimes` into - `startup_optional_runtime_sidecars`. +- Rust code changes: add `startup_runtime_hooks` and move runtime diagnostics, + profiling hook dispatch, and crypto provider installation out of + `startup_runtime` and `startup_profiling`. - CI/script changes: none. -- Docs changes: record the R-026 optional runtime sidecar boundary slice. +- Docs changes: record the R-027 startup runtime hook boundary slice. ## Phase 0 Tasks @@ -2110,6 +2110,20 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block risk scan, branch freshness check, pre-commit quality gate, and three-expert review. +- [x] `R-027` Extract startup runtime hook boundary. + - Do: add `startup_runtime_hooks` and move startup runtime diagnostics, + profiling hook dispatch, shutdown profiling dispatch, and default crypto + provider installation out of `startup_runtime` and `startup_profiling`. + - Acceptance: BOOT-006 keeps diagnostics, profiling init, trusted proxy init, + provider install, and outbound TLS material load in the same order, while + STOP-004 still stops profiling through the existing compatibility path. + - Must preserve: startup logo and telemetry/license log behavior, profiling + hook dispatch behavior, rustls provider install behavior, trusted proxy init + order, outbound TLS fatal boundary, and profiling shutdown call path. + - Verification: focused runtime hook/profiling/runtime/shutdown tests, RustFS + lib check, migration/layer guards, formatting, diff hygiene, Rust risk scan, + branch freshness check, pre-commit quality gate, and three-expert review. + ## Next PRs 1. `pure-move`: continue larger lifecycle hook slices for startup profiling and @@ -2121,9 +2135,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block | Expert | Status | Notes | |---|---|---| -| Quality/architecture | passed | R-026 moves optional runtime sidecar ownership into `startup_optional_runtime_sidecars` while keeping the old module path as a compatibility handoff. | -| Migration preservation | passed | Protocol startup order, shutdown planning order, and HTTP-before-optional-runtime shutdown ordering remain unchanged. | -| Testing/verification | passed | Focused optional runtime sidecar/runtime/shutdown checks, guards, final hygiene, and full pre-commit passed. | +| Quality/architecture | passed | R-027 moves runtime hook side effects into `startup_runtime_hooks` while keeping old profiling entrypoints as compatibility handoffs. | +| Migration preservation | passed | BOOT-006 and STOP-004 ordering remain unchanged, including trusted proxy and outbound TLS ordering. | +| Testing/verification | passed | Focused runtime hook/profiling/runtime/shutdown checks, guards, final hygiene, and full pre-commit passed. | ## Verification Notes @@ -2234,6 +2248,21 @@ Passed before push: - `make pre-commit`: passed. - Three-expert review: passed. +- Issue #660 R-027 current slice: + - `cargo test -p rustfs --lib startup_runtime_hooks -- --nocapture`: + passed. + - `cargo test -p rustfs --lib startup_profiling -- --nocapture`: passed. + - `cargo test -p rustfs --lib startup_runtime -- --nocapture`: passed. + - `cargo test -p rustfs --lib startup_shutdown -- --nocapture`: passed. + - `cargo check -p rustfs --lib`: passed. + - `./scripts/check_architecture_migration_rules.sh`: passed. + - `./scripts/check_layer_dependencies.sh`: passed. + - `cargo fmt --all --check`: passed. + - `git diff --check`: passed. + - Rust risk scan on changed Rust files: passed. + - `make pre-commit`: passed. + - Three-expert review: passed. + - Issue #660 R-020 current slice: - `cargo test -p rustfs --lib startup_profiling -- --nocapture`: passed. - `cargo check -p rustfs --lib`: passed. diff --git a/docs/architecture/runtime-lifecycle.md b/docs/architecture/runtime-lifecycle.md index 4d4d44d07..150e0649c 100644 --- a/docs/architecture/runtime-lifecycle.md +++ b/docs/architecture/runtime-lifecycle.md @@ -65,6 +65,15 @@ sidecars must enter this boundary with explicit shutdown handles and status snapshots instead of adding ad hoc startup or shutdown work to `startup_services`. +## Startup Runtime Hook Boundary + +`startup_runtime_hooks` owns runtime hook side effects that wrap the startup +foundation but are not TLS material loading: startup diagnostics, profiling +hook dispatch, and default crypto provider installation. `startup_runtime` +preserves BOOT-006 orchestration and outbound TLS fatal behavior, while +`startup_profiling` remains a compatibility handoff for the old profiling hook +path. + ## AppContext Foundation Early AppContext work should split resolver files and add compatibility tests before diff --git a/docs/architecture/startup-timeline.md b/docs/architecture/startup-timeline.md index ed082063e..76edf0901 100644 --- a/docs/architecture/startup-timeline.md +++ b/docs/architecture/startup-timeline.md @@ -23,7 +23,7 @@ new startup semantics. | `BOOT-003` | `rustfs/src/startup_entrypoint.rs` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None | | `BOOT-004` | `rustfs/src/startup_preflight.rs` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None | | `BOOT-005` | `rustfs/src/startup_preflight.rs` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None | -| `BOOT-006` | `rustfs/src/startup_runtime.rs` and `rustfs/src/startup_profiling.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None | +| `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, and `rustfs/src/startup_profiling.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None | | `RUN-001` | `rustfs/src/startup_server.rs` | Enter startup run orchestration and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready | | `RUN-002` | `rustfs/src/startup_server.rs` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None | | `RUN-003` | `rustfs/src/startup_server.rs` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None | @@ -72,7 +72,7 @@ new startup semantics. | `STOP-001` | `rustfs/src/startup_shutdown.rs` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here | | `STOP-002` | `rustfs/src/startup_shutdown.rs` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change | | `STOP-003` | `rustfs/src/startup_optional_runtime_sidecars.rs` | Plan optional runtime shutdown and log stopping state for FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown handles. | Non-fatal in this path. | No readiness-stage change | -| `STOP-004` | `rustfs/src/startup_shutdown.rs` and `rustfs/src/startup_profiling.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change | +| `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs`, and `rustfs/src/startup_profiling.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change | | `STOP-005` | `rustfs/src/startup_shutdown.rs` and `rustfs/src/startup_optional_runtime_sidecars.rs` | Stop S3 and console HTTP servers, signal and wait for optional protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here | ## Migration Rules diff --git a/rustfs/src/lib.rs b/rustfs/src/lib.rs index e80ffe702..7b74eaf7f 100644 --- a/rustfs/src/lib.rs +++ b/rustfs/src/lib.rs @@ -78,6 +78,7 @@ pub mod startup_preflight; pub mod startup_profiling; pub mod startup_protocols; pub mod startup_runtime; +pub mod startup_runtime_hooks; pub mod startup_server; pub mod startup_service_components; pub mod startup_services; diff --git a/rustfs/src/startup_profiling.rs b/rustfs/src/startup_profiling.rs index 1e9267619..546dfb3a0 100644 --- a/rustfs/src/startup_profiling.rs +++ b/rustfs/src/startup_profiling.rs @@ -12,60 +12,4 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::future::Future; - -pub async fn init_profiling_runtime() { - init_profiling_runtime_with(crate::profiling::init_from_env).await; -} - -async fn init_profiling_runtime_with(init: InitFn) -where - InitFn: FnOnce() -> InitFuture, - InitFuture: Future, -{ - init().await; -} - -pub fn shutdown_profiling_runtime() { - shutdown_profiling_runtime_with(crate::profiling::shutdown_profiling); -} - -fn shutdown_profiling_runtime_with(shutdown: ShutdownFn) -where - ShutdownFn: FnOnce(), -{ - shutdown(); -} - -#[cfg(test)] -mod tests { - use super::{init_profiling_runtime_with, shutdown_profiling_runtime_with}; - use std::sync::{ - Arc, - atomic::{AtomicBool, Ordering}, - }; - - #[tokio::test] - async fn init_profiling_runtime_invokes_registered_hook() { - let called = Arc::new(AtomicBool::new(false)); - let hook_called = called.clone(); - - init_profiling_runtime_with(move || async move { - hook_called.store(true, Ordering::SeqCst); - }) - .await; - - assert!(called.load(Ordering::SeqCst)); - } - - #[test] - fn shutdown_profiling_runtime_invokes_registered_hook() { - let called = AtomicBool::new(false); - - shutdown_profiling_runtime_with(|| { - called.store(true, Ordering::SeqCst); - }); - - assert!(called.load(Ordering::SeqCst)); - } -} +pub use crate::startup_runtime_hooks::{init_profiling_runtime, shutdown_profiling_runtime}; diff --git a/rustfs/src/startup_runtime.rs b/rustfs/src/startup_runtime.rs index bf5abfd45..d1432caf8 100644 --- a/rustfs/src/startup_runtime.rs +++ b/rustfs/src/startup_runtime.rs @@ -12,80 +12,27 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::{config::Config, license::license_status}; -use rustls::crypto::aws_lc_rs::default_provider; +use crate::{ + config::Config, + startup_runtime_hooks::{init_profiling_runtime, install_default_crypto_provider, log_startup_runtime_diagnostics}, +}; use std::io::{Error, Result}; -use tracing::{debug, error, info}; +use tracing::{error, info}; const LOG_COMPONENT_MAIN: &str = "main"; -const LOG_SUBSYSTEM_LICENSE: &str = "license"; const LOG_SUBSYSTEM_STARTUP: &str = "startup"; -const EVENT_CRYPTO_PROVIDER_STATE: &str = "crypto_provider_state"; -const EVENT_DIAL9_RUNTIME_STATUS: &str = "dial9_runtime_status"; -const EVENT_RUNTIME_LICENSE_STATUS: &str = "runtime_license_status"; const EVENT_TLS_OUTBOUND_INITIALIZED: &str = "tls_outbound_initialized"; const EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED: &str = "tls_outbound_initialization_failed"; const TLS_STARTUP_GENERATION_CONSUMER: &str = "rustfs_server_startup"; pub async fn init_startup_runtime_foundation(config: &Config) -> Result<()> { - log_dial9_runtime_status(); - log_runtime_license_status(); - - debug!("{}", crate::server::LOGO); - - crate::startup_profiling::init_profiling_runtime().await; + log_startup_runtime_diagnostics(); + init_profiling_runtime().await; rustfs_trusted_proxies::init(); install_default_crypto_provider(); init_outbound_tls_material(config).await } -fn log_dial9_runtime_status() { - if rustfs_obs::dial9::is_enabled() { - info!( - target: "rustfs::main", - event = EVENT_DIAL9_RUNTIME_STATUS, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STARTUP, - enabled = true, - "Dial9 Tokio runtime telemetry is enabled" - ); - } else { - debug!( - target: "rustfs::main", - event = EVENT_DIAL9_RUNTIME_STATUS, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STARTUP, - enabled = false, - "Dial9 Tokio runtime telemetry is disabled" - ); - } -} - -fn log_runtime_license_status() { - info!( - target: "rustfs::main", - event = EVENT_RUNTIME_LICENSE_STATUS, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_LICENSE, - license_status = %license_status(), - "Initialized runtime license state" - ); -} - -fn install_default_crypto_provider() { - if default_provider().install_default().is_err() { - debug!( - target: "rustfs::main", - event = EVENT_CRYPTO_PROVIDER_STATE, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STARTUP, - provider = "aws_lc_rs", - state = "already_installed", - "Rustls crypto provider state checked" - ); - } -} - async fn init_outbound_tls_material(config: &Config) -> Result<()> { if let Some(tls_path) = normalized_tls_path(config.tls_path.as_deref()) { match crate::server::tls_material::load_tls_material(tls_path).await { diff --git a/rustfs/src/startup_runtime_hooks.rs b/rustfs/src/startup_runtime_hooks.rs new file mode 100644 index 000000000..d6b075bfc --- /dev/null +++ b/rustfs/src/startup_runtime_hooks.rs @@ -0,0 +1,134 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use crate::license::license_status; +use rustls::crypto::aws_lc_rs::default_provider; +use std::future::Future; +use tracing::{debug, info}; + +const LOG_COMPONENT_MAIN: &str = "main"; +const LOG_SUBSYSTEM_LICENSE: &str = "license"; +const LOG_SUBSYSTEM_STARTUP: &str = "startup"; +const EVENT_CRYPTO_PROVIDER_STATE: &str = "crypto_provider_state"; +const EVENT_DIAL9_RUNTIME_STATUS: &str = "dial9_runtime_status"; +const EVENT_RUNTIME_LICENSE_STATUS: &str = "runtime_license_status"; + +pub fn log_startup_runtime_diagnostics() { + log_dial9_runtime_status(); + log_runtime_license_status(); + debug!("{}", crate::server::LOGO); +} + +fn log_dial9_runtime_status() { + if rustfs_obs::dial9::is_enabled() { + info!( + target: "rustfs::main", + event = EVENT_DIAL9_RUNTIME_STATUS, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + enabled = true, + "Dial9 Tokio runtime telemetry is enabled" + ); + } else { + debug!( + target: "rustfs::main", + event = EVENT_DIAL9_RUNTIME_STATUS, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + enabled = false, + "Dial9 Tokio runtime telemetry is disabled" + ); + } +} + +fn log_runtime_license_status() { + info!( + target: "rustfs::main", + event = EVENT_RUNTIME_LICENSE_STATUS, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_LICENSE, + license_status = %license_status(), + "Initialized runtime license state" + ); +} + +pub async fn init_profiling_runtime() { + init_profiling_runtime_with(crate::profiling::init_from_env).await; +} + +async fn init_profiling_runtime_with(init: InitFn) +where + InitFn: FnOnce() -> InitFuture, + InitFuture: Future, +{ + init().await; +} + +pub fn shutdown_profiling_runtime() { + shutdown_profiling_runtime_with(crate::profiling::shutdown_profiling); +} + +fn shutdown_profiling_runtime_with(shutdown: ShutdownFn) +where + ShutdownFn: FnOnce(), +{ + shutdown(); +} + +pub fn install_default_crypto_provider() { + if default_provider().install_default().is_err() { + debug!( + target: "rustfs::main", + event = EVENT_CRYPTO_PROVIDER_STATE, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + provider = "aws_lc_rs", + state = "already_installed", + "Rustls crypto provider state checked" + ); + } +} + +#[cfg(test)] +mod tests { + use super::{init_profiling_runtime_with, shutdown_profiling_runtime_with}; + use std::sync::{ + Arc, + atomic::{AtomicBool, Ordering}, + }; + + #[tokio::test] + async fn init_profiling_runtime_invokes_registered_hook() { + let called = Arc::new(AtomicBool::new(false)); + let hook_called = called.clone(); + + init_profiling_runtime_with(move || async move { + hook_called.store(true, Ordering::SeqCst); + }) + .await; + + assert!(called.load(Ordering::SeqCst)); + } + + #[test] + fn shutdown_profiling_runtime_invokes_registered_hook() { + let called = AtomicBool::new(false); + + shutdown_profiling_runtime_with(|| { + called.store(true, Ordering::SeqCst); + }); + + assert!(called.load(Ordering::SeqCst)); + } +}