diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index 076005401..b9621966c 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,18 +5,17 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-startup-storage-bootstrap` -- Baseline: `origin/main` at `6508f88d3a5edb428a5d623f927ce384691f0cd4` +- Branch: `overtrue/arch-startup-storage-runtime` +- Baseline: `origin/main` at `a19560da419c6d354a805fc127951cc9b56ad887` - PR type for this branch: `pure-move` -- Runtime behavior changes: no external behavior change expected; endpoint - parsing, unsupported filesystem policy enforcement, global endpoint/erasure - type publication, local disk/prewarm, lock clients, and storage pool logging - still run in the same relative order between listen context and HTTP server - startup. -- Rust code changes: add `startup_storage::init_startup_storage_foundation` - and use it from binary startup. +- Runtime behavior changes: no external behavior change expected; ECStore + creation, ECStore config initialization, global config retry policy, + `StorageReady` publication, and background replication startup still run in + the same relative order after HTTP server startup and before KMS startup. +- Rust code changes: add `startup_storage::init_startup_storage_runtime` and + use it from binary startup. - CI/script changes: none. -- Docs changes: record `R-015` startup storage foundation bootstrap progress and +- Docs changes: record `R-016` startup storage runtime bootstrap progress and verification. ## Phase 0 Tasks @@ -662,6 +661,25 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block formatting, migration guards, Rust risk scan, branch freshness check, and pre-commit quality gate. +- [x] `R-016` Centralize startup storage runtime bootstrap. + - Do: move runtime cancellation token creation, ECStore initialization, + ECStore config initialization, server-config migration attempt, global + config retry loop, `StorageReady` stage publication, and background + replication startup behind the `startup_storage` boundary. + - Acceptance: storage runtime still starts after HTTP server startup and + before KMS startup; ECStore init failure keeps the same structured error log + and propagated error; global config init still logs every failed attempt, + sleeps between attempts, and becomes fatal after the 16th failed attempt; + `StorageReady` is still marked after global config init succeeds and before + background replication startup. + - Must preserve: cancellation token ownership for later shutdown, endpoint + pool clone ownership for ECStore startup, ECStore config init/migration + order, retry count/log fields, fatal error string, readiness stage timing, + and non-fatal background replication startup behavior. + - Verification: focused startup storage tests, binary/lib compile checks, + formatting, migration guards, Rust risk scan, branch freshness check, and + pre-commit quality gate. + ## Next PRs 1. `pure-move`: continue extracting startup boot wrappers in larger slices while @@ -673,13 +691,13 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block | Expert | Status | Notes | |---|---|---| -| Quality/architecture | passed | Pure-move slice removes storage foundation details from binary startup behind the existing startup module pattern. | -| Migration preservation | passed | Endpoint parsing, filesystem policy, global endpoint/erasure publication, local disk/prewarm, lock clients, and pool logging order are preserved. | +| Quality/architecture | passed | Pure-move slice removes storage runtime details from binary startup behind the existing startup storage boundary. | +| Migration preservation | passed | HTTP server startup, ECStore creation, config init/migration, global config retry, `StorageReady`, replication startup, and KMS startup order are preserved. | | Testing/verification | passed | Focused startup storage tests, compile checks, formatting, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. | ## Verification Notes -Passed on `6508f88d3a5edb428a5d623f927ce384691f0cd4`: +Passed on `a19560da419c6d354a805fc127951cc9b56ad887`: - `cargo test -p rustfs startup_storage --no-fail-fast`: passed. - `cargo check -p rustfs --lib`: passed. @@ -688,24 +706,23 @@ Passed on `6508f88d3a5edb428a5d623f927ce384691f0cd4`: - `git diff --check`: passed. - `./scripts/check_architecture_migration_rules.sh`: passed. - `./scripts/check_layer_dependencies.sh`: passed. -- `git rev-list --left-right --count HEAD...origin/main` returned `0 0` - before commit. +- `git rev-list --left-right --count HEAD...origin/main` returned `1 0` + after commit and rebase. - Added-line Rust risk scan for changed Rust files: no matches. -- Full-file risk scan for changed Rust files: matches are existing docs example - output and existing binary startup alias/stderr/expect usage. -- `make pre-commit`: all checks passed, including nextest with 6009 passed +- Full-file risk scan for changed Rust files: matches are existing binary + startup stderr/expect usage. +- `make pre-commit`: all checks passed, including nextest with 6016 passed and 111 skipped, plus doctests. Notes: -- This slice centralizes startup storage foundation without changing startup - ordering or endpoint pool ownership. -- Storage foundation remains after listen context setup and before HTTP server - startup. +- This slice centralizes startup storage runtime without changing startup + ordering, shutdown token ownership, or endpoint pool ownership. +- Storage runtime remains after HTTP server startup and before KMS startup. ## Handoff Notes -- R-015 is complete. +- R-016 is complete. - Next startup slices can keep using larger pure moves, but must keep startup ordering, fatal/non-fatal boundaries, shutdown ownership, and readiness ownership explicit in tests. diff --git a/docs/architecture/startup-timeline.md b/docs/architecture/startup-timeline.md index 4523c8fe5..64937d30b 100644 --- a/docs/architecture/startup-timeline.md +++ b/docs/architecture/startup-timeline.md @@ -33,9 +33,9 @@ new startup semantics. | `RUN-008` | `rustfs/src/main.rs:311` | Initialize local disks, prewarm local disk id map, and initialize lock clients. | Opens local disk state, primes disk id lookup, and creates global lock clients. | Local disk init is fatal; prewarm and lock-client setup are non-fatal in this path. | None | | `RUN-009` | `rustfs/src/main.rs:350` | Initialize capacity management and service state manager. | Starts capacity management and moves service state to `Starting`. | Non-fatal in this path. | None | | `RUN-010` | `rustfs/src/main.rs:356` | Start S3 HTTP listener and optional console listener before storage is ready. | Starts HTTP servers with readiness gates; console listener starts only when enabled and configured. | Fatal if a configured listener cannot start. | Requests remain gated until full readiness except probe/admin/console/rpc/tonic/table-catalog exempt paths | -| `RUN-011` | `rustfs/src/main.rs:372` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None | -| `RUN-012` | `rustfs/src/main.rs:382` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, and lock quorum before `FullReady` | -| `RUN-013` | `rustfs/src/main.rs:397` | Start replication and KMS systems. | Starts background replication pool and initializes KMS. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady` | +| `RUN-011` | `rustfs/src/startup_storage.rs` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None | +| `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, and lock quorum before `FullReady` | +| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/main.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from main startup. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady` | | `RUN-014` | `rustfs/src/main.rs:402` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None | | `RUN-015` | `rustfs/src/main.rs:482` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None | | `RUN-016` | `rustfs/src/main.rs:503` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready | diff --git a/rustfs/src/main.rs b/rustfs/src/main.rs index 38285dfff..c78d0ce79 100644 --- a/rustfs/src/main.rs +++ b/rustfs/src/main.rs @@ -26,15 +26,12 @@ use rustfs::startup_iam::{bootstrap_or_defer_iam_init, publish_ready_for_iam_boo use rustfs::startup_preflight::{StartupServerPreflightError, bootstrap_external_prefix_compat, init_startup_server_preflight}; use rustfs::startup_protocols::{ProtocolShutdownSenders, init_protocol_shutdown_senders}; use rustfs::startup_server::{StartupHttpServers, StartupListenContext, init_startup_http_servers, init_startup_listen_context}; -use rustfs::startup_storage::init_startup_storage_foundation; -use rustfs_common::SystemStage; +use rustfs::startup_storage::{StartupStorageRuntime, init_startup_storage_foundation, init_startup_storage_runtime}; use rustfs_ecstore::{ bucket::metadata_sys::init_bucket_metadata_sys, bucket::migration::{try_migrate_bucket_metadata, try_migrate_iam_config}, - bucket::replication::{get_global_replication_pool, init_background_replication}, - config as ecconfig, + bucket::replication::get_global_replication_pool, global::shutdown_background_services, - store::ECStore, store_api::BucketOperations, }; use rustfs_heal::{ @@ -57,9 +54,7 @@ const ENV_HEAL_ENABLED_DEPRECATED: &str = "RUSTFS_ENABLE_HEAL"; const LOG_COMPONENT_MAIN: &str = "main"; const LOG_SUBSYSTEM_STARTUP: &str = "startup"; const LOG_SUBSYSTEM_AUTH: &str = "auth"; -const LOG_SUBSYSTEM_STORAGE: &str = "storage"; const EVENT_SERVER_RUNTIME_FAILED: &str = "server_runtime_failed"; -const EVENT_STARTUP_STORAGE_STAGE: &str = "startup_storage_stage"; const EVENT_PROTOCOL_SYSTEM_STATE: &str = "protocol_system_state"; const EVENT_AUDIT_SYSTEM_STATE: &str = "audit_system_state"; const EVENT_DEADLOCK_DETECTOR_STATE: &str = "deadlock_detector_state"; @@ -169,61 +164,11 @@ async fn run(config: rustfs::config::Config) -> Result<()> { console_shutdown_tx, } = init_startup_http_servers(&config, readiness.clone()).await?; - let ctx = CancellationToken::new(); + let StartupStorageRuntime { + store, + shutdown_token: ctx, + } = init_startup_storage_runtime(server_addr, &endpoint_pools, readiness.clone()).await?; - // init store - // 2. Start Storage Engine (ECStore) - debug!( - target: "rustfs::main::run", - event = EVENT_STARTUP_STORAGE_STAGE, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STORAGE, - stage = "ecstore_initialization", - state = "starting", - "starting ECStore initialization" - ); - let store = ECStore::new(server_addr, endpoint_pools.clone(), ctx.clone()) - .await - .inspect_err(|err| { - error!( - target: "rustfs::main::run", - event = EVENT_STARTUP_STORAGE_STAGE, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STORAGE, - stage = "ecstore_initialization", - state = "failed", - error = ?err, - "ECStore initialization failed" - ); - })?; - - ecconfig::init(); - ecconfig::try_migrate_server_config(store.clone()).await; - - // // Initialize global configuration system - let mut retry_count = 0; - while let Err(e) = ecconfig::init_global_config_sys(store.clone()).await { - error!( - target: "rustfs::main::run", - event = EVENT_STARTUP_STORAGE_STAGE, - component = LOG_COMPONENT_MAIN, - subsystem = LOG_SUBSYSTEM_STORAGE, - stage = "global_config_initialization", - state = "retrying", - retry_count = retry_count + 1, - error = ?e, - "Global config initialization retry failed" - ); - // TODO: check error type - retry_count += 1; - if retry_count > 15 { - return Err(Error::other("ecconfig::init_global_config_sys failed")); - } - tokio::time::sleep(tokio::time::Duration::from_secs(1)).await; - } - readiness.mark_stage(SystemStage::StorageReady); - // init replication_pool - init_background_replication(store.clone()).await; // Initialize KMS system if enabled init_kms_system(&config).await?; diff --git a/rustfs/src/startup_storage.rs b/rustfs/src/startup_storage.rs index 2ccb1dd64..c114b4ccf 100644 --- a/rustfs/src/startup_storage.rs +++ b/rustfs/src/startup_storage.rs @@ -13,22 +13,36 @@ // limitations under the License. use crate::startup_fs_guard::enforce_unsupported_fs_policy; +use rustfs_common::{GlobalReadiness, SystemStage}; use rustfs_ecstore::{ + bucket::replication::init_background_replication, + config as ecconfig, endpoints::EndpointServerPools, set_global_endpoints, - store::{init_local_disks, init_lock_clients, prewarm_local_disk_id_map}, + store::{ECStore, init_local_disks, init_lock_clients, prewarm_local_disk_id_map}, update_erasure_type, }; -use std::io::{Error, Result}; +use std::{ + io::{Error, Result}, + net::SocketAddr, + sync::Arc, +}; +use tokio_util::sync::CancellationToken; use tracing::{debug, error, info, warn}; const LOG_COMPONENT_MAIN: &str = "main"; const LOG_SUBSYSTEM_STORAGE: &str = "storage"; +const GLOBAL_CONFIG_INIT_MAX_RETRIES: usize = 15; const EVENT_ENDPOINT_PARSING_STARTED: &str = "endpoint_parsing_started"; const EVENT_STARTUP_STORAGE_STAGE: &str = "startup_storage_stage"; const EVENT_STORAGE_POOL_FORMATTING: &str = "storage_pool_formatting"; const EVENT_STORAGE_POOL_HOST_RISK: &str = "storage_pool_host_risk"; +pub struct StartupStorageRuntime { + pub store: Arc, + pub shutdown_token: CancellationToken, +} + pub async fn init_startup_storage_foundation(server_address: &str, volumes: &[String]) -> Result { info!( target: "rustfs::main::run", @@ -91,6 +105,76 @@ pub async fn init_startup_storage_foundation(server_address: &str, volumes: &[St Ok(endpoint_pools) } +pub async fn init_startup_storage_runtime( + server_addr: SocketAddr, + endpoint_pools: &EndpointServerPools, + readiness: Arc, +) -> Result { + let ctx = CancellationToken::new(); + + debug!( + target: "rustfs::main::run", + event = EVENT_STARTUP_STORAGE_STAGE, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STORAGE, + stage = "ecstore_initialization", + state = "starting", + "starting ECStore initialization" + ); + let store = ECStore::new(server_addr, endpoint_pools.clone(), ctx.clone()) + .await + .inspect_err(|err| { + error!( + target: "rustfs::main::run", + event = EVENT_STARTUP_STORAGE_STAGE, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STORAGE, + stage = "ecstore_initialization", + state = "failed", + error = ?err, + "ECStore initialization failed" + ); + })?; + + init_startup_storage_global_config(store.clone()).await?; + readiness.mark_stage(SystemStage::StorageReady); + init_background_replication(store.clone()).await; + + Ok(StartupStorageRuntime { + store, + shutdown_token: ctx, + }) +} + +async fn init_startup_storage_global_config(store: Arc) -> Result<()> { + ecconfig::init(); + ecconfig::try_migrate_server_config(store.clone()).await; + + let mut retry_count = 0; + while let Err(e) = ecconfig::init_global_config_sys(store.clone()).await { + let next_retry_count = retry_count + 1; + error!( + target: "rustfs::main::run", + event = EVENT_STARTUP_STORAGE_STAGE, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STORAGE, + stage = "global_config_initialization", + state = "retrying", + retry_count = next_retry_count, + error = ?e, + "Global config initialization retry failed" + ); + // TODO: check error type + retry_count = next_retry_count; + if global_config_retry_exhausted(retry_count) { + return Err(Error::other("ecconfig::init_global_config_sys failed")); + } + tokio::time::sleep(tokio::time::Duration::from_secs(1)).await; + } + + Ok(()) +} + fn log_storage_pool_layout(endpoint_pools: &EndpointServerPools) { for (i, eps) in endpoint_pools.as_ref().iter().enumerate() { info!( @@ -142,9 +226,13 @@ fn storage_pool_has_host_failure_risk(drives_per_set: usize) -> bool { drives_per_set > 1 } +fn global_config_retry_exhausted(retry_count: usize) -> bool { + retry_count > GLOBAL_CONFIG_INIT_MAX_RETRIES +} + #[cfg(test)] mod tests { - use super::storage_pool_has_host_failure_risk; + use super::{global_config_retry_exhausted, storage_pool_has_host_failure_risk}; #[test] fn reports_host_failure_risk_only_for_multi_drive_sets() { @@ -152,4 +240,10 @@ mod tests { assert!(!storage_pool_has_host_failure_risk(1)); assert!(storage_pool_has_host_failure_risk(2)); } + + #[test] + fn global_config_retry_limit_matches_startup_policy() { + assert!(!global_config_retry_exhausted(15)); + assert!(global_config_retry_exhausted(16)); + } }