refactor: extract startup ready lifecycle boundary (#3635)

docs/architecture/migration-progress.md

@@ -5,15 +5,15 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 ## Current Context
 
 - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
-- Branch: `overtrue/arch-startup-shutdown-lifecycle-boundary`
-- Baseline: `origin/main`
-  (`eec1792c7bd1bf650dbb3c46d7873db6fed6150f`).
+- Branch: `overtrue/arch-startup-ready-lifecycle-boundary`
+- Baseline: `origin/main` (`e9037f9eb09fcbb31d980efac8b8e6b8f12bcaef`).
 - PR type for this branch: `pure-move`
 - Runtime behavior changes: none.
-- Rust code changes: move startup shutdown sequencing from `startup_services`
-  into a dedicated shutdown lifecycle boundary.
+- Rust code changes: move ready publication, scanner startup, shutdown-signal
+  wait, and stopped-state final logging from `startup_services` into a
+  dedicated startup lifecycle boundary.
 - CI/script changes: none.
-- Docs changes: record the R-023 startup shutdown lifecycle boundary slice.
+- Docs changes: record the R-024 startup lifecycle boundary slice.
 
 ## Phase 0 Tasks
 
@@ -2063,6 +2063,20 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
     check, migration/layer guards, formatting, diff hygiene, Rust risk scan,
     branch freshness check, pre-commit quality gate, and three-expert review.
 
+- [x] `R-024` Extract startup ready lifecycle boundary.
+  - Do: add `startup_lifecycle` and move ready publication, global init time,
+    scanner startup, shutdown-signal wait, shutdown delegation, and final
+    stopped-state logging out of `startup_services`.
+  - Acceptance: lifecycle order stays server-ready log, IAM readiness
+    publication, global init time, optional scanner startup, shutdown wait,
+    shutdown sequence delegation, and final stopped log.
+  - Must preserve: inline/deferred IAM readiness behavior, scanner start timing,
+    global init-time timing, shutdown signal wait semantics, shutdown ordering,
+    service state reporting, and fatal boundary on readiness publication.
+  - Verification: focused lifecycle/service/shutdown tests, RustFS lib check,
+    migration/layer guards, formatting, diff hygiene, Rust risk scan, branch
+    freshness check, pre-commit quality gate, and three-expert review.
+
 ## Next PRs
 
 1. `pure-move`: continue larger lifecycle hook slices for optional runtime
@@ -2074,9 +2088,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 
 | Expert | Status | Notes |
 |---|---|---|
-| Quality/architecture | passed | R-023 keeps startup orchestration in `startup_services` and moves shutdown sequencing into `startup_shutdown`. |
-| Migration preservation | passed | Runtime token cancellation, background shutdown, optional runtime planning, notifier/audit/profiling shutdown, HTTP shutdown, optional runtime waits, and service state transitions remain ordered. |
-| Testing/verification | passed | Focused shutdown, optional runtime, and startup service tests plus final checks are tracked below before push. |
+| Quality/architecture | passed | R-024 keeps runtime service initialization in `startup_services` and moves ready/scanner/shutdown-wait orchestration into `startup_lifecycle`. |
+| Migration preservation | passed | Ready publication, global init time, scanner startup, shutdown wait/delegation, and final stopped logging remain ordered. |
+| Testing/verification | passed | Focused lifecycle, shutdown, and startup service tests plus final checks are tracked below before push. |
 
 ## Verification Notes
 
@@ -2145,6 +2159,19 @@ Passed before push:
   - `make pre-commit`: passed.
   - Three-expert review: passed.
 
+- Issue #660 R-024 current slice:
+  - `cargo test -p rustfs --lib startup_lifecycle -- --nocapture`: passed.
+  - `cargo test -p rustfs --lib startup_services -- --nocapture`: passed.
+  - `cargo test -p rustfs --lib startup_shutdown -- --nocapture`: passed.
+  - `cargo check -p rustfs --lib`: passed.
+  - `./scripts/check_architecture_migration_rules.sh`: passed.
+  - `./scripts/check_layer_dependencies.sh`: passed.
+  - `cargo fmt --all --check`: passed.
+  - `git diff --check`: passed.
+  - Rust risk scan on changed Rust files: passed.
+  - `make pre-commit`: passed.
+  - Three-expert review: passed.
+
 - Issue #660 R-020 current slice:
   - `cargo test -p rustfs --lib startup_profiling -- --nocapture`: passed.
   - `cargo check -p rustfs --lib`: passed.

docs/architecture/runtime-lifecycle.md

@@ -39,6 +39,14 @@ must not reorder runtime-token cancellation, background service shutdown,
 optional runtime shutdown planning, notifier/audit/profiling shutdown, HTTP
 shutdown, optional runtime waits, or final service-state publication.
 
+## Startup Lifecycle Boundary
+
+`startup_lifecycle` owns the ready-to-shutdown orchestration after runtime
+services initialize. Service modules may return initialized handles into this
+boundary, but they must not reorder ready publication, global init-time
+publication, scanner startup, shutdown-signal wait, shutdown delegation, or the
+final stopped-state log.
+
 ## Optional Runtime Boundary
 
 `startup_optional_runtimes` owns startup and shutdown handoff for optional

docs/architecture/startup-timeline.md

@@ -45,9 +45,9 @@ new startup semantics.
 | `RUN-019` | `rustfs/src/startup_services.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None |
 | `RUN-020` | `rustfs/src/startup_services.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None |
 | `RUN-021` | `rustfs/src/startup_services.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None |
-| `RUN-022` | `rustfs/src/startup_services.rs` and `rustfs/src/startup_iam.rs` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here |
-| `RUN-023` | `rustfs/src/startup_services.rs` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery |
-| `RUN-024` | `rustfs/src/startup_services.rs` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state |
+| `RUN-022` | `rustfs/src/startup_lifecycle.rs` and `rustfs/src/startup_iam.rs` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here |
+| `RUN-023` | `rustfs/src/startup_lifecycle.rs` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery |
+| `RUN-024` | `rustfs/src/startup_lifecycle.rs` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state |
 
 ## Deferred IAM Readiness
 

rustfs/src/lib.rs

@@ -71,6 +71,7 @@ pub mod server;
 pub mod startup_entrypoint;
 pub mod startup_fs_guard;
 pub mod startup_iam;
+pub mod startup_lifecycle;
 pub mod startup_optional_runtimes;
 pub mod startup_preflight;
 pub mod startup_profiling;

rustfs/src/startup_entrypoint.rs

@@ -14,9 +14,10 @@
 
 use crate::{
     config::{CommandResult, Config, Opt},
+    startup_lifecycle::{StartupRuntimeLifecycle, run_startup_runtime_lifecycle},
     startup_preflight::{StartupServerPreflightError, bootstrap_external_prefix_compat, init_startup_server_preflight},
     startup_server::{StartupHttpServers, StartupListenContext, init_startup_http_servers, init_startup_listen_context},
-    startup_services::{StartupRuntimeLifecycle, init_startup_runtime_services, run_startup_runtime_lifecycle},
+    startup_services::init_startup_runtime_services,
     startup_storage::{StartupStorageRuntime, init_startup_storage_foundation, init_startup_storage_runtime},
 };
 use std::io::{Error, Result};

rustfs/src/startup_lifecycle.rs

@@ -0,0 +1,100 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::{
+    server::{ServiceStateManager, ShutdownHandle, wait_for_shutdown},
+    startup_iam::publish_ready_for_iam_bootstrap,
+    startup_services::StartupServiceRuntime,
+    startup_shutdown::run_startup_shutdown_sequence,
+    storage_compat::ECStore,
+};
+use rustfs_common::GlobalReadiness;
+use rustfs_scanner::init_data_scanner;
+use std::{io::Result, sync::Arc};
+use tokio_util::sync::CancellationToken;
+use tracing::info;
+
+const LOG_COMPONENT_MAIN: &str = "main";
+const LOG_SUBSYSTEM_STARTUP: &str = "startup";
+const EVENT_SERVER_READY: &str = "server_ready";
+const EVENT_SERVER_SHUTDOWN_STATE: &str = "server_shutdown_state";
+
+pub struct StartupRuntimeLifecycle {
+    pub server_address: String,
+    pub state_manager: Arc<ServiceStateManager>,
+    pub s3_shutdown_tx: Option<ShutdownHandle>,
+    pub console_shutdown_tx: Option<ShutdownHandle>,
+    pub service_runtime: StartupServiceRuntime,
+    pub store: Arc<ECStore>,
+    pub shutdown_token: CancellationToken,
+    pub readiness: Arc<GlobalReadiness>,
+}
+
+pub async fn run_startup_runtime_lifecycle(lifecycle: StartupRuntimeLifecycle) -> Result<()> {
+    let StartupRuntimeLifecycle {
+        server_address,
+        state_manager,
+        s3_shutdown_tx,
+        console_shutdown_tx,
+        service_runtime,
+        store,
+        shutdown_token,
+        readiness,
+    } = lifecycle;
+    let StartupServiceRuntime {
+        optional_runtimes,
+        iam_bootstrap,
+        enable_scanner,
+    } = service_runtime;
+
+    info!(
+        target: "rustfs::main::run",
+        event = EVENT_SERVER_READY,
+        component = LOG_COMPONENT_MAIN,
+        subsystem = LOG_SUBSYSTEM_STARTUP,
+        version = %crate::version::get_version(),
+        server_address = %server_address,
+        started_at = %jiff::Zoned::now(),
+        iam_bootstrap = ?iam_bootstrap,
+        "RustFS server ready"
+    );
+    publish_ready_for_iam_bootstrap(iam_bootstrap, readiness.as_ref(), Some(state_manager.as_ref())).await?;
+    rustfs_common::set_global_init_time_now().await;
+
+    if enable_scanner {
+        init_data_scanner(shutdown_token.clone(), store).await;
+    }
+
+    let shutdown_signal = wait_for_shutdown().await;
+    run_startup_shutdown_sequence(
+        &state_manager,
+        shutdown_signal,
+        s3_shutdown_tx,
+        console_shutdown_tx,
+        optional_runtimes,
+        shutdown_token,
+    )
+    .await;
+
+    info!(
+        target: "rustfs::main::run",
+        event = EVENT_SERVER_SHUTDOWN_STATE,
+        component = LOG_COMPONENT_MAIN,
+        subsystem = LOG_SUBSYSTEM_STARTUP,
+        state = ?state_manager.current_state(),
+        result = "stopped",
+        "RustFS server stopped"
+    );
+    Ok(())
+}

rustfs/src/startup_services.rs

@@ -22,17 +22,15 @@ use crate::{
         add_bucket_notification_configuration, init_auto_tuner, init_buffer_profile_system, init_kms_system, init_update_check,
         print_server_info,
     },
-    server::{ServiceStateManager, ShutdownHandle, init_event_notifier, start_audit_system, wait_for_shutdown},
-    startup_iam::{IamBootstrapDisposition, bootstrap_or_defer_iam_init, publish_ready_for_iam_bootstrap},
+    server::{ServiceStateManager, init_event_notifier, start_audit_system},
+    startup_iam::{IamBootstrapDisposition, bootstrap_or_defer_iam_init},
     startup_optional_runtimes::{OptionalRuntimeServices, init_optional_runtime_services},
-    startup_shutdown::run_startup_shutdown_sequence,
 };
 use rustfs_audit::AuditResult;
 use rustfs_common::GlobalReadiness;
 use rustfs_heal::{create_ahm_services_cancel_token, heal::storage::ECStoreHealStorage, init_heal_manager};
 use rustfs_iam::init_oidc_sys;
 use rustfs_obs::init_metrics_runtime;
-use rustfs_scanner::init_data_scanner;
 use rustfs_storage_api::{BucketOperations, BucketOptions};
 use rustfs_utils::get_env_bool_with_aliases;
 use std::{
@@ -57,8 +55,6 @@ const EVENT_KEYSTONE_AUTH_INITIALIZATION_FAILED: &str = "keystone_auth_initializ
 const EVENT_OIDC_INITIALIZATION_FAILED: &str = "oidc_initialization_failed";
 const EVENT_NOTIFICATION_SYSTEM_INITIALIZATION_FAILED: &str = "notification_system_initialization_failed";
 const EVENT_BACKGROUND_SERVICES_CONFIGURED: &str = "background_services_configured";
-const EVENT_SERVER_READY: &str = "server_ready";
-const EVENT_SERVER_SHUTDOWN_STATE: &str = "server_shutdown_state";
 
 pub struct StartupServiceRuntime {
     pub optional_runtimes: OptionalRuntimeServices,
@@ -66,17 +62,6 @@ pub struct StartupServiceRuntime {
     pub enable_scanner: bool,
 }
 
-pub struct StartupRuntimeLifecycle {
-    pub server_address: String,
-    pub state_manager: Arc<ServiceStateManager>,
-    pub s3_shutdown_tx: Option<ShutdownHandle>,
-    pub console_shutdown_tx: Option<ShutdownHandle>,
-    pub service_runtime: StartupServiceRuntime,
-    pub store: Arc<ECStore>,
-    pub shutdown_token: CancellationToken,
-    pub readiness: Arc<GlobalReadiness>,
-}
-
 pub async fn init_startup_runtime_services(
     config: &Config,
     endpoint_pools: EndpointServerPools,
@@ -107,64 +92,6 @@ pub async fn init_startup_runtime_services(
     })
 }
 
-pub async fn run_startup_runtime_lifecycle(lifecycle: StartupRuntimeLifecycle) -> Result<()> {
-    let StartupRuntimeLifecycle {
-        server_address,
-        state_manager,
-        s3_shutdown_tx,
-        console_shutdown_tx,
-        service_runtime,
-        store,
-        shutdown_token,
-        readiness,
-    } = lifecycle;
-    let StartupServiceRuntime {
-        optional_runtimes,
-        iam_bootstrap,
-        enable_scanner,
-    } = service_runtime;
-
-    info!(
-        target: "rustfs::main::run",
-        event = EVENT_SERVER_READY,
-        component = LOG_COMPONENT_MAIN,
-        subsystem = LOG_SUBSYSTEM_STARTUP,
-        version = %crate::version::get_version(),
-        server_address = %server_address,
-        started_at = %jiff::Zoned::now(),
-        iam_bootstrap = ?iam_bootstrap,
-        "RustFS server ready"
-    );
-    publish_ready_for_iam_bootstrap(iam_bootstrap, readiness.as_ref(), Some(state_manager.as_ref())).await?;
-    rustfs_common::set_global_init_time_now().await;
-
-    if enable_scanner {
-        init_data_scanner(shutdown_token.clone(), store).await;
-    }
-
-    let shutdown_signal = wait_for_shutdown().await;
-    run_startup_shutdown_sequence(
-        &state_manager,
-        shutdown_signal,
-        s3_shutdown_tx,
-        console_shutdown_tx,
-        optional_runtimes,
-        shutdown_token,
-    )
-    .await;
-
-    info!(
-        target: "rustfs::main::run",
-        event = EVENT_SERVER_SHUTDOWN_STATE,
-        component = LOG_COMPONENT_MAIN,
-        subsystem = LOG_SUBSYSTEM_STARTUP,
-        state = ?state_manager.current_state(),
-        result = "stopped",
-        "RustFS server stopped"
-    );
-    Ok(())
-}
-
 async fn init_audit_runtime() {
     match init_event_notifier_and_audit().await {
         Ok(()) => info!(