refactor: extract embedded runtime orchestration hooks (#3659)

docs/architecture/migration-progress.md

@@ -2217,6 +2217,34 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
     check, migration/layer guards, formatting, diff hygiene, Rust risk scan,
     branch freshness check, pre-commit quality gate, and three-expert review.
 
+- [x] `R-034` Extract embedded runtime hook boundary.
+  - Do: move embedded observability guard setup, default crypto provider
+    installation, and trusted proxy initialization behind startup runtime hooks.
+  - Acceptance: embedded startup keeps observability initialization before the
+    global startup guard/listen/storage phases while sharing the runtime hook
+    owner used by normal startup.
+  - Must preserve: `init_obs` and `set_global_guard` error prefixes, embedded
+    crypto provider already-installed debug fields, trusted proxy init timing,
+    and no added embedded server runtime behavior.
+  - Verification: focused embedded/runtime hook checks, RustFS lib check,
+    migration/layer guards, formatting, diff hygiene, Rust risk scan, branch
+    freshness check, pre-commit quality gate, and three-expert review.
+
+- [x] `R-035` Extract embedded shutdown glue boundary.
+  - Do: move embedded async shutdown logging, cancellation, event/audit cleanup,
+    HTTP shutdown, and temporary directory cleanup behind startup shutdown
+    helpers.
+  - Acceptance: embedded server shutdown preserves the same stopping/stopped
+    logs, cancellation timing, best-effort audit cleanup, HTTP shutdown, and
+    temp-dir cleanup behavior while leaving `Drop` as a synchronous best-effort
+    fallback.
+  - Must preserve: event notifier shutdown before audit stop, audit stop
+    warning-only behavior, HTTP shutdown after background cancellation, temp
+    directory cleanup warning fields, and final stopped log.
+  - Verification: focused embedded/shutdown checks, RustFS lib check,
+    migration/layer guards, formatting, diff hygiene, Rust risk scan, branch
+    freshness check, pre-commit quality gate, and three-expert review.
+
 - [x] `E-001/E-SET-001` Add ECStore layout skeleton and set-layout boundary.
   - Do: create the ECStore internal layout ownership buckets and pin static set
     layout versus runtime `Sets`/`SetDisks` orchestration boundaries before any
@@ -2460,20 +2488,34 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
 ## Next PRs
 
 1. `pure-move`: continue pruning residual embedded startup-only orchestration
-   once the lifecycle helpers are merged.
+   after the runtime hook and shutdown glue boundaries land.
 
 ## Pre-Push Review Log
 
 | Expert | Status | Notes |
 |---|---|---|
-| Quality/architecture | passed | E-012 through E-016 move control/runtime/entry flow, tests, and type contracts into rebalance submodules while preserving root exports. |
-| Migration preservation | passed | Lifecycle filtering, delete-marker skips, deferred transient failures, source cleanup warnings, cancellation, bucket outcome precedence, serde contracts, and test coverage remain preserved. |
-| Testing/verification | passed | Focused rebalance tests, compile checks, guards, formatting, diff hygiene, Rust risk scan, and pre-commit gate passed. |
+| Quality/architecture | passed | R-034 and R-035 move embedded runtime hooks and async shutdown glue into startup owners while keeping the embedded builder focused on startup orchestration. |
+| Migration preservation | passed | Observability error prefixes, embedded crypto log fields, trusted-proxy timing, cancellation, audit cleanup, HTTP shutdown, temp-dir cleanup, and stopped logs stay in the same order. |
+| Testing/verification | passed | Focused runtime/embedded checks, RustFS lib check, architecture/layer guards, formatting, diff hygiene, Rust risk scan, and pre-commit gate passed. |
 
 ## Verification Notes
 
 Passed before push:
 
+- Issue #660 R-034/R-035 current slice:
+  - `cargo test -p rustfs --lib startup_runtime_hooks -- --nocapture`:
+    passed.
+  - `cargo test -p rustfs --lib embedded -- --nocapture`: passed; no
+    matching unit tests currently exist.
+  - `cargo check -p rustfs --lib`: passed.
+  - `./scripts/check_architecture_migration_rules.sh`: passed.
+  - `./scripts/check_layer_dependencies.sh`: passed.
+  - `cargo fmt --all --check`: passed.
+  - `git diff --check`: passed.
+  - Rust risk scan on changed Rust files: passed; only existing default
+    credential fields and moved temp-dir cleanup paths were present.
+  - `make pre-commit`: passed.
+
 - Issue #660 R-031 current slice:
   - `cargo test -p rustfs --lib startup_lifecycle -- --nocapture`: passed;
     no matching unit tests currently exist.

rustfs/src/embedded.rs

@@ -49,20 +49,15 @@
 use crate::config::Config;
 use crate::server::{ShutdownHandle, start_http_server};
 use crate::startup_lifecycle::{log_embedded_server_ready, publish_embedded_startup_ready};
+use crate::startup_runtime_hooks::init_embedded_runtime_hooks;
 use crate::startup_server::init_embedded_startup_listen_context;
 use crate::startup_services::init_embedded_startup_runtime_services;
-use crate::startup_shutdown::run_embedded_shutdown_cleanup;
+use crate::startup_shutdown::run_embedded_server_shutdown;
 use crate::startup_storage::{init_embedded_startup_storage_foundation, init_embedded_startup_storage_runtime};
-use rustfs_obs::{init_obs, set_global_guard};
-use rustls::crypto::aws_lc_rs::default_provider;
 use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
 use std::path::{Path, PathBuf};
 use std::sync::atomic::{AtomicBool, Ordering};
 use tokio_util::sync::CancellationToken;
-use tracing::{debug, info, warn};
-
-const LOG_COMPONENT_EMBEDDED: &str = "embedded";
-const LOG_SUBSYSTEM_EMBEDDED: &str = "embedded";
 
 /// Tracks whether a server has been started in this process.
 static SERVER_STARTED: AtomicBool = AtomicBool::new(false);
@@ -270,26 +265,9 @@ impl RustFSServerBuilder {
 
         // --- Initialization sequence (mirrors main.rs::run) ---
 
-        // Observability (minimal / no-op endpoint for embedded use).
-        let guard = init_obs(Some(config.obs_endpoint.clone()))
+        init_embedded_runtime_hooks(config.obs_endpoint.clone())
             .await
-            .map_err(|e| ServerError::Init(format!("init_obs: {e}")))?;
-        set_global_guard(guard).map_err(|e| ServerError::Init(format!("set_global_guard: {e}")))?;
-
-        // Crypto provider.
-        if let Err(err) = default_provider().install_default() {
-            debug!(
-                component = LOG_COMPONENT_EMBEDDED,
-                subsystem = LOG_SUBSYSTEM_EMBEDDED,
-                event = "crypto_provider_state",
-                state = "already_installed",
-                error = ?err,
-                "Embedded crypto provider state changed"
-            );
-        }
-
-        // Trusted proxies.
-        rustfs_trusted_proxies::init();
+            .map_err(|e| ServerError::Init(e.to_string()))?;
 
         let listen_context = init_embedded_startup_listen_context(&config)
             .await
@@ -420,48 +398,7 @@ impl RustFSServer {
     }
 
     async fn do_shutdown(&mut self) {
-        info!(
-            target: "rustfs::embedded",
-            component = LOG_COMPONENT_EMBEDDED,
-            subsystem = LOG_SUBSYSTEM_EMBEDDED,
-            event = "embedded_server_state",
-            state = "stopping",
-            "Embedded server state changed"
-        );
-
-        // Cancel background services.
-        self.cancel_token.cancel();
-
-        run_embedded_shutdown_cleanup().await;
-
-        // Signal HTTP server to stop.
-        if let Some(shutdown_handle) = self.shutdown_handle.take() {
-            shutdown_handle.shutdown().await;
-        }
-
-        // Clean up temp directory if we created it.
-        if let Some(ref dir) = self.temp_dir
-            && let Err(e) = tokio::fs::remove_dir_all(dir).await
-        {
-            warn!(
-                component = LOG_COMPONENT_EMBEDDED,
-                subsystem = LOG_SUBSYSTEM_EMBEDDED,
-                event = "embedded_shutdown_cleanup_failed",
-                service = "temp_dir",
-                path = %dir.display(),
-                error = %e,
-                "Embedded shutdown cleanup failed"
-            );
-        }
-
-        info!(
-            target: "rustfs::embedded",
-            component = LOG_COMPONENT_EMBEDDED,
-            subsystem = LOG_SUBSYSTEM_EMBEDDED,
-            event = "embedded_server_state",
-            state = "stopped",
-            "Embedded server state changed"
-        );
+        run_embedded_server_shutdown(&self.cancel_token, &mut self.shutdown_handle, self.temp_dir.as_deref()).await;
     }
 }
 

rustfs/src/startup_runtime_hooks.rs

@@ -15,9 +15,12 @@
 use crate::license::license_status;
 use rustls::crypto::aws_lc_rs::default_provider;
 use std::future::Future;
+use std::io::{Error, Result};
 use tracing::{debug, info};
 
+const LOG_COMPONENT_EMBEDDED: &str = "embedded";
 const LOG_COMPONENT_MAIN: &str = "main";
+const LOG_SUBSYSTEM_EMBEDDED: &str = "embedded";
 const LOG_SUBSYSTEM_LICENSE: &str = "license";
 const LOG_SUBSYSTEM_STARTUP: &str = "startup";
 const EVENT_CRYPTO_PROVIDER_STATE: &str = "crypto_provider_state";
@@ -100,6 +103,31 @@ pub fn install_default_crypto_provider() {
     }
 }
 
+pub async fn init_embedded_runtime_hooks(obs_endpoint: String) -> Result<()> {
+    let guard = rustfs_obs::init_obs(Some(obs_endpoint))
+        .await
+        .map_err(|err| Error::other(format!("init_obs: {err}")))?;
+    rustfs_obs::set_global_guard(guard).map_err(|err| Error::other(format!("set_global_guard: {err}")))?;
+
+    install_embedded_default_crypto_provider();
+    rustfs_trusted_proxies::init();
+
+    Ok(())
+}
+
+fn install_embedded_default_crypto_provider() {
+    if let Err(err) = default_provider().install_default() {
+        debug!(
+            component = LOG_COMPONENT_EMBEDDED,
+            subsystem = LOG_SUBSYSTEM_EMBEDDED,
+            event = EVENT_CRYPTO_PROVIDER_STATE,
+            state = "already_installed",
+            error = ?err,
+            "Embedded crypto provider state changed"
+        );
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::{init_profiling_runtime_with, shutdown_profiling_runtime_with};

rustfs/src/startup_shutdown.rs

@@ -21,6 +21,7 @@ use crate::{
 };
 use rustfs_heal::shutdown_ahm_services;
 use rustfs_utils::get_env_bool_with_aliases;
+use std::path::Path;
 use tokio_util::sync::CancellationToken;
 use tracing::{error, info, warn};
 
@@ -33,6 +34,7 @@ const LOG_COMPONENT_EMBEDDED: &str = "embedded";
 const LOG_SUBSYSTEM_STARTUP: &str = "startup";
 const LOG_SUBSYSTEM_EMBEDDED: &str = "embedded";
 const EVENT_AUDIT_SYSTEM_STATE: &str = "audit_system_state";
+const EVENT_EMBEDDED_SERVER_STATE: &str = "embedded_server_state";
 const EVENT_EMBEDDED_SHUTDOWN_CLEANUP_FAILED: &str = "embedded_shutdown_cleanup_failed";
 const EVENT_SHUTDOWN_SIGNAL_RECEIVED: &str = "shutdown_signal_received";
 const EVENT_BACKGROUND_SERVICE_SHUTDOWN: &str = "background_service_shutdown";
@@ -214,6 +216,52 @@ pub async fn run_embedded_shutdown_cleanup() {
     }
 }
 
+pub async fn run_embedded_server_shutdown(
+    ctx: &CancellationToken,
+    shutdown_handle: &mut Option<ShutdownHandle>,
+    temp_dir: Option<&Path>,
+) {
+    info!(
+        target: "rustfs::embedded",
+        component = LOG_COMPONENT_EMBEDDED,
+        subsystem = LOG_SUBSYSTEM_EMBEDDED,
+        event = EVENT_EMBEDDED_SERVER_STATE,
+        state = "stopping",
+        "Embedded server state changed"
+    );
+
+    ctx.cancel();
+
+    run_embedded_shutdown_cleanup().await;
+
+    if let Some(shutdown_handle) = shutdown_handle.take() {
+        shutdown_handle.shutdown().await;
+    }
+
+    if let Some(dir) = temp_dir
+        && let Err(err) = tokio::fs::remove_dir_all(dir).await
+    {
+        warn!(
+            component = LOG_COMPONENT_EMBEDDED,
+            subsystem = LOG_SUBSYSTEM_EMBEDDED,
+            event = EVENT_EMBEDDED_SHUTDOWN_CLEANUP_FAILED,
+            service = "temp_dir",
+            path = %dir.display(),
+            error = %err,
+            "Embedded shutdown cleanup failed"
+        );
+    }
+
+    info!(
+        target: "rustfs::embedded",
+        component = LOG_COMPONENT_EMBEDDED,
+        subsystem = LOG_SUBSYSTEM_EMBEDDED,
+        event = EVENT_EMBEDDED_SERVER_STATE,
+        state = "stopped",
+        "Embedded server state changed"
+    );
+}
+
 #[cfg(test)]
 mod tests {
     use super::{BackgroundShutdownStep, background_shutdown_steps};