From 0f9584c8d9351c757437405ee69a4e64bbcd94b5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=AE=89=E6=AD=A3=E8=B6=85?= Date: Sun, 7 Jun 2026 20:18:30 +0800 Subject: [PATCH] docs: add startup timeline baseline (#3256) --- docs/architecture/migration-progress.md | 47 +++++--------- docs/architecture/startup-timeline.md | 85 +++++++++++++++++++++++++ 2 files changed, 102 insertions(+), 30 deletions(-) create mode 100644 docs/architecture/startup-timeline.md diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index e5403bf83..4ba3222c2 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,13 +5,13 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-migration-ci-gates` -- Baseline: `upstream/main` at `f00898d0703ac94a4a215a54b6666c747ebe6622` -- PR type for this branch: `ci-gate` +- Branch: `overtrue/arch-startup-timeline` +- Baseline: `upstream/main` at `ae9d25879d72bc8977f08e61062c022e2142483b` +- PR type for this branch: `docs-only` - Runtime behavior changes: none - Rust code changes: none -- Script changes: stabilize `scripts/check_layer_dependencies.sh` against the - current accepted baseline. +- Docs changes: add the binary startup timeline baseline for later + runtime/lifecycle migration work. ## Phase 0 Tasks @@ -29,9 +29,12 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block - Acceptance: `./scripts/check_layer_dependencies.sh` passes on current `upstream/main` while still rejecting new unaccepted layer dependencies. - [~] `G-006` Create migration loss-prevention checks. - - Current branch: keeps the existing layer guard stable. + - Current branch: not in scope. - Next PR: add checks for public re-export, route matrix, and storage trait coverage before pure moves. +- [x] `G-007` Create startup timeline table. + - Acceptance: [`startup-timeline.md`](startup-timeline.md) records current + binary startup order, side effects, fatal boundaries, and readiness stages. - [x] `G-009` Enforce pre-push three-expert review. - Acceptance: [`crate-boundaries.md`](crate-boundaries.md) requires quality/architecture, migration-preservation, and testing/verification review @@ -43,8 +46,7 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Next PRs -1. `docs-only` or `test-only`: capture startup timeline and admin route-action - snapshot. +1. `docs-only` or `test-only`: capture admin route-action snapshot. 2. `docs-only`: inventory `ecstore::config::{Config, KV, KVS}` consumers. 3. `ci-gate`: add focused checks for PR type vocabulary and temporary compatibility marker/register consistency. @@ -53,9 +55,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block | Expert | Status | Notes | |---|---|---| -| Quality/architecture | pass | Final review confirmed the guard scope, baseline logic, and storage target classification; the segment-level `storage::ecfs` / `storage::s3_api` follow-up passed | -| Migration preservation | pass | Final review confirmed no Rust, runtime, storage hot-path, or global-state logic changes and no missing `RUSTFS_COMPAT_TODO` marker | -| Testing/verification | pass | Final review accepted the full gate, focused guard checks, stale-baseline check, and negative baseline-removal check as sufficient for this `ci-gate` PR | +| Quality/architecture | pass | Final review confirmed the startup, deferred IAM, readiness, and shutdown tables match current source behavior after blocker fixes | +| Migration preservation | pass | Final review confirmed this branch is docs-only and does not touch runtime logic, storage hot paths, global-state migration, or compatibility code | +| Testing/verification | pass | Final review accepted docs-only verification with layer guard, metrics reference guard, diff checks, and staged whitespace check | ## Verification Notes @@ -63,30 +65,15 @@ Passed: - `./scripts/check_layer_dependencies.sh` - `./scripts/check_metrics_migration_refs.sh` -- `bash -n scripts/check_layer_dependencies.sh` -- `make pre-commit` -- temporary negative check that adds an unaccepted reverse dependency to a copied - fixture and confirms the guard fails -- temporary negative check that adds single-line root grouped - `use crate::{admin::bad};` and confirms the guard fails -- temporary negative check that adds multiline root grouped - `use crate::{ ... }` and confirms the guard fails -- temporary negative check that adds root grouped alias - `use crate::{admin as admin_alias};` and confirms the guard fails -- temporary negative check that adds grouped `self` imports and confirms the - guard fails -- temporary negative check that adds a stale baseline row and confirms the guard - fails -- temporary negative check that removes the accepted - `crate::storage::ecfs::FS` baseline row for `rustfs/src/protocols/client.rs` - and confirms the guard fails - `git diff --check` -- focused shell review of changed scripts +- `git diff --cached --check` +- focused source review of `rustfs/src/main.rs`, `rustfs/src/startup_iam.rs`, + and `rustfs/src/server/readiness.rs` ## Handoff Notes - Keep Phase 0 PRs small. Do not start Config, Storage API, Runtime, or ECStore - movement inside this `ci-gate` branch. + movement inside this `docs-only` branch. - Keep CI checks in a separate `ci-gate` PR so the PR type rule remains enforceable. - Do not add temporary compatibility code without a matching `RUSTFS_COMPAT_TODO()` marker and cleanup-register entry. diff --git a/docs/architecture/startup-timeline.md b/docs/architecture/startup-timeline.md new file mode 100644 index 000000000..4523c8fe5 --- /dev/null +++ b/docs/architecture/startup-timeline.md @@ -0,0 +1,85 @@ +# Startup Timeline Baseline + +This document records the current binary startup order before runtime/lifecycle +migration work. It is a behavior-preservation baseline only; it does not define +new startup semantics. + +## Scope + +- Baseline commit: `ae9d25879d72bc8977f08e61062c022e2142483b` +- Entry points covered: `rustfs/src/main.rs::main`, `async_main`, `run`, and + `handle_shutdown` +- Related migration task: `G-007` +- Out of scope for this baseline: embedded startup, admin route-action matrix, + and any runtime/lifecycle code movement + +## Startup Stages + +| Step | Source | Current action | Side effects | Fatal boundary | Ready stage | +|---|---|---|---|---|---| +| `BOOT-001` | `rustfs/src/main.rs:84` | Apply external-prefix environment compatibility before the Tokio runtime is created. | Copies supported external env aliases into canonical `RUSTFS_*` process env keys and prints warnings or info to stderr. | Non-fatal; failure is logged to stderr and startup continues. | None | +| `BOOT-002` | `rustfs/src/main.rs:89` | Build the Tokio runtime. | Installs runtime configuration and any runtime telemetry guard created by the runtime builder. | Fatal through `expect`; process exits if the runtime cannot be built. | None | +| `BOOT-003` | `rustfs/src/main.rs:139` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None | +| `BOOT-004` | `rustfs/src/main.rs:167` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None | +| `BOOT-005` | `rustfs/src/main.rs:173` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None | +| `BOOT-006` | `rustfs/src/main.rs:208` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None | +| `RUN-001` | `rustfs/src/main.rs:256` | Enter `run` and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready | +| `RUN-002` | `rustfs/src/main.rs:261` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None | +| `RUN-003` | `rustfs/src/main.rs:268` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None | +| `RUN-004` | `rustfs/src/main.rs:286` | Initialize global action credentials. | Publishes root/action credentials used by auth paths. | Fatal if global credentials cannot be initialized. | None | +| `RUN-005` | `rustfs/src/main.rs:298` | Publish server port and address. | Updates global RustFS port and global address. | Non-fatal in this path. | None | +| `RUN-006` | `rustfs/src/main.rs:302` | Build endpoint pools and enforce unsupported filesystem policy. | Derives pool/set/disk layout from configured volumes and validates unsupported filesystem policy. | Fatal on endpoint build or unsupported filesystem policy error. | None | +| `RUN-007` | `rustfs/src/main.rs:308` | Publish endpoints and erasure type. | Updates global endpoints and erasure type. | Non-fatal in this path. | None | +| `RUN-008` | `rustfs/src/main.rs:311` | Initialize local disks, prewarm local disk id map, and initialize lock clients. | Opens local disk state, primes disk id lookup, and creates global lock clients. | Local disk init is fatal; prewarm and lock-client setup are non-fatal in this path. | None | +| `RUN-009` | `rustfs/src/main.rs:350` | Initialize capacity management and service state manager. | Starts capacity management and moves service state to `Starting`. | Non-fatal in this path. | None | +| `RUN-010` | `rustfs/src/main.rs:356` | Start S3 HTTP listener and optional console listener before storage is ready. | Starts HTTP servers with readiness gates; console listener starts only when enabled and configured. | Fatal if a configured listener cannot start. | Requests remain gated until full readiness except probe/admin/console/rpc/tonic/table-catalog exempt paths | +| `RUN-011` | `rustfs/src/main.rs:372` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None | +| `RUN-012` | `rustfs/src/main.rs:382` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, and lock quorum before `FullReady` | +| `RUN-013` | `rustfs/src/main.rs:397` | Start replication and KMS systems. | Starts background replication pool and initializes KMS. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady` | +| `RUN-014` | `rustfs/src/main.rs:402` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None | +| `RUN-015` | `rustfs/src/main.rs:482` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None | +| `RUN-016` | `rustfs/src/main.rs:503` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready | +| `RUN-017` | `rustfs/src/main.rs:523` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task | +| `RUN-018` | `rustfs/src/main.rs:535` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None | +| `RUN-019` | `rustfs/src/main.rs:552` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None | +| `RUN-020` | `rustfs/src/main.rs:560` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None | +| `RUN-021` | `rustfs/src/main.rs:584` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None | +| `RUN-022` | `rustfs/src/main.rs:599` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here | +| `RUN-023` | `rustfs/src/main.rs:609` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery | +| `RUN-024` | `rustfs/src/main.rs:616` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state | + +## Deferred IAM Readiness + +| Step | Source | Current action | Side effects | Fatal boundary | Ready stage | +|---|---|---|---|---|---| +| `IAM-001` | `rustfs/src/startup_iam.rs:256` | Attempt `init_iam_sys` during bootstrap. | Initializes IAM against the ECStore object layer when possible. | Recoverable failures can enter deferred mode; unrecoverable errors propagate from bootstrap. | None | +| `IAM-002` | `rustfs/src/startup_iam.rs:73` | Spawn IAM recovery loop when bootstrap is deferred. | Retries IAM initialization with exponential backoff until shutdown or success. | Retry failures are logged; the service remains degraded. | None | +| `IAM-003` | `rustfs/src/startup_iam.rs:52` | Finalize IAM recovery after init succeeds. | Initializes `AppContext` if needed, marks `IamReady`, and calls runtime readiness publication. | Finalize failures are retried by the recovery loop. | Marks `IamReady`, then `FullReady` when runtime readiness succeeds | + +## Readiness Gate + +| Step | Source | Current action | Side effects | Fatal boundary | Ready stage | +|---|---|---|---|---|---| +| `READY-001` | `rustfs/src/server/readiness.rs:130` | Treat exact probe paths and admin/console/rpc/tonic/table-catalog prefixes as readiness-gate bypass paths. | Bypass paths continue to the inner service while the global readiness gate is not ready. | Non-fatal. | Does not change readiness stages | +| `READY-002` | `rustfs/src/server/readiness.rs:171` | Reject non-probe requests while `GlobalReadiness` is not ready. | Returns `503 Service Unavailable`, `Retry-After: 5`, `Content-Type: text/plain; charset=utf-8`, and `Cache-Control: no-store`. | Non-fatal. | Does not change readiness stages | +| `READY-003` | `rustfs/src/server/readiness.rs:202` | Wait for runtime storage, IAM, and lock quorum readiness before publishing ready state. | Marks `FullReady` and updates `ServiceState` to `Ready` only when a state manager is provided. | Returns an error on timeout; inline startup treats that as fatal, while deferred IAM recovery retries finalization. | Marks `FullReady` | + +## Shutdown Order + +| Step | Source | Current action | Side effects | Fatal boundary | Ready stage | +|---|---|---|---|---|---| +| `STOP-001` | `rustfs/src/main.rs:647` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here | +| `STOP-002` | `rustfs/src/main.rs:675` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change | +| `STOP-003` | `rustfs/src/main.rs:699` | Signal optional FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown futures. | Non-fatal in this path. | No readiness-stage change | +| `STOP-004` | `rustfs/src/main.rs:735` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change | +| `STOP-005` | `rustfs/src/main.rs:763` | Stop S3 and console HTTP servers, wait for protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here | + +## Migration Rules + +- Runtime/lifecycle PRs must map each moved startup line back to one of the + `BOOT-*`, `RUN-*`, `IAM-*`, `READY-*`, or `STOP-*` rows. +- A `pure-move` PR must keep the fatal boundary and ready-stage column unchanged. +- Any intentional change to this table is a separate `behavior-change` PR with + focused negative tests. +- Do not use this document to justify changing readiness, IAM recovery, HTTP + listener timing, lock quorum, or shutdown order in a docs-only PR.