mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-04 20:37:43 +00:00
refactor: centralize startup runtime services (#3467)
This commit is contained in:
安正超
@@ -5,17 +5,18 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
## Current Context
|
||||
|
||||
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
|
||||
- Branch: `overtrue/arch-startup-storage-runtime`
|
||||
- Baseline: `origin/main` at `a19560da419c6d354a805fc127951cc9b56ad887`
|
||||
- Branch: `overtrue/arch-startup-runtime-services`
|
||||
- Baseline: `origin/main` at `3e723a2476692051cac1eab2cc31984c62063130`
|
||||
- PR type for this branch: `pure-move`
|
||||
- Runtime behavior changes: no external behavior change expected; ECStore
|
||||
creation, ECStore config initialization, global config retry policy,
|
||||
`StorageReady` publication, and background replication startup still run in
|
||||
the same relative order after HTTP server startup and before KMS startup.
|
||||
- Rust code changes: add `startup_storage::init_startup_storage_runtime` and
|
||||
- Runtime behavior changes: no external behavior change expected; KMS, optional
|
||||
protocols, diagnostics services, bucket metadata migration, IAM bootstrap,
|
||||
auth integrations, notification runtime, heal/scanner setup, and metrics
|
||||
initialization still run in the same relative order after storage runtime and
|
||||
before the server-ready log.
|
||||
- Rust code changes: add `startup_services::init_startup_runtime_services` and
|
||||
use it from binary startup.
|
||||
- CI/script changes: none.
|
||||
- Docs changes: record `R-016` startup storage runtime bootstrap progress and
|
||||
- Docs changes: record `R-017` startup runtime service bootstrap progress and
|
||||
verification.
|
||||
|
||||
## Phase 0 Tasks
|
||||
@@ -680,10 +681,33 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
formatting, migration guards, Rust risk scan, branch freshness check, and
|
||||
pre-commit quality gate.
|
||||
|
||||
- [x] `R-017` Centralize startup runtime service bootstrap.
|
||||
- Do: move KMS startup, optional protocol shutdown collection, buffer
|
||||
profiling, event notifier/audit startup, deadlock detector startup, bucket
|
||||
metadata migration, replication resync, IAM bootstrap, Keystone/OIDC auth
|
||||
integration startup, notification runtime setup, AHM/heal setup, server info,
|
||||
update check, allocator reclaim, metrics runtime, memory observability, and
|
||||
auto-tuner startup behind the `startup_services` boundary.
|
||||
- Acceptance: startup service initialization still runs after storage runtime
|
||||
initialization and before the server-ready log; `main.rs` keeps ownership of
|
||||
shutdown handling, server-ready publication, global init time, and scanner
|
||||
start; `startup_services` returns protocol shutdown handles, IAM bootstrap
|
||||
disposition, and scanner enablement.
|
||||
- Must preserve: KMS fatal behavior, protocol fatal/disabled behavior, audit
|
||||
non-fatal behavior, deadlock detector logging, bucket list and replication
|
||||
resync fatal behavior, bucket/IAM metadata migration non-fatal behavior, IAM
|
||||
deferred recovery semantics, Keystone parse fatal and runtime non-fatal
|
||||
behavior, OIDC non-fatal behavior, notification init fatal behavior,
|
||||
scanner-implies-heal behavior, metric-enabled guard, and shutdown token
|
||||
ownership.
|
||||
- Verification: focused startup services tests, binary/lib compile checks,
|
||||
formatting, migration guards, Rust risk scan, branch freshness check, and
|
||||
pre-commit quality gate.
|
||||
|
||||
## Next PRs
|
||||
|
||||
1. `pure-move`: continue extracting startup boot wrappers in larger slices while
|
||||
preserving startup order and readiness ownership.
|
||||
1. `pure-move`: continue extracting startup ready/scanner/shutdown wrappers while
|
||||
preserving startup order, readiness ownership, and shutdown ownership.
|
||||
2. `ci-gate`: finish `G-006` public re-export and storage trait coverage checks
|
||||
before the remaining cleanup slices.
|
||||
|
||||
@@ -691,38 +715,39 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
|
||||
| Expert | Status | Notes |
|
||||
|---|---|---|
|
||||
| Quality/architecture | passed | Pure-move slice removes storage runtime details from binary startup behind the existing startup storage boundary. |
|
||||
| Migration preservation | passed | HTTP server startup, ECStore creation, config init/migration, global config retry, `StorageReady`, replication startup, and KMS startup order are preserved. |
|
||||
| Testing/verification | passed | Focused startup storage tests, compile checks, formatting, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. |
|
||||
| Quality/architecture | passed | Pure-move slice removes runtime service details from binary startup behind the existing startup services boundary. |
|
||||
| Migration preservation | passed | Storage runtime, KMS, optional protocols, diagnostics, bucket metadata, IAM, auth, notification, heal/scanner, metrics, and server-ready order are preserved. |
|
||||
| Testing/verification | passed | Focused startup services tests, compile checks, formatting, migration/layer guards, Rust risk scan, branch freshness check, and full `make pre-commit` passed. |
|
||||
|
||||
## Verification Notes
|
||||
|
||||
Passed on `a19560da419c6d354a805fc127951cc9b56ad887`:
|
||||
Passed on `3e723a2476692051cac1eab2cc31984c62063130`:
|
||||
|
||||
- `cargo test -p rustfs startup_storage --no-fail-fast`: passed.
|
||||
- `cargo test -p rustfs startup_services --no-fail-fast`: passed.
|
||||
- `cargo check -p rustfs --lib`: passed.
|
||||
- `cargo check -p rustfs --bin rustfs`: passed.
|
||||
- `cargo fmt --all --check`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `./scripts/check_layer_dependencies.sh`: passed.
|
||||
- `git rev-list --left-right --count HEAD...origin/main` returned `1 0`
|
||||
after commit and rebase.
|
||||
- Added-line Rust risk scan for changed Rust files: no matches.
|
||||
- Full-file risk scan for changed Rust files: matches are existing binary
|
||||
startup stderr/expect usage.
|
||||
- `make pre-commit`: all checks passed, including nextest with 6016 passed
|
||||
and 111 skipped, plus doctests.
|
||||
- Added-line Rust risk scan for changed Rust files: passed.
|
||||
- Full-file risk scan for changed Rust files: existing `main.rs` process setup
|
||||
entries only.
|
||||
- `make pre-commit`: passed, including nextest `6027` passed / `111` skipped
|
||||
and doctests.
|
||||
- `git rev-list --left-right --count HEAD...origin/main`: returned `1 0`
|
||||
after commit.
|
||||
|
||||
Notes:
|
||||
|
||||
- This slice centralizes startup storage runtime without changing startup
|
||||
ordering, shutdown token ownership, or endpoint pool ownership.
|
||||
- Storage runtime remains after HTTP server startup and before KMS startup.
|
||||
- This slice centralizes startup runtime services without changing startup
|
||||
ordering, shutdown token ownership, readiness ownership, or endpoint pool
|
||||
ownership.
|
||||
- Runtime services remain after storage runtime and before the server-ready log.
|
||||
|
||||
## Handoff Notes
|
||||
|
||||
- R-016 is complete.
|
||||
- R-017 is implemented, locally verified, and current with `origin/main`.
|
||||
- Next startup slices can keep using larger pure moves, but must keep startup
|
||||
ordering, fatal/non-fatal boundaries, shutdown ownership, and readiness
|
||||
ownership explicit in tests.
|
||||
ownership explicit in tests and review notes.
|
||||
|
||||
@@ -35,15 +35,15 @@ new startup semantics.
|
||||
| `RUN-010` | `rustfs/src/main.rs:356` | Start S3 HTTP listener and optional console listener before storage is ready. | Starts HTTP servers with readiness gates; console listener starts only when enabled and configured. | Fatal if a configured listener cannot start. | Requests remain gated until full readiness except probe/admin/console/rpc/tonic/table-catalog exempt paths |
|
||||
| `RUN-011` | `rustfs/src/startup_storage.rs` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None |
|
||||
| `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, and lock quorum before `FullReady` |
|
||||
| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/main.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from main startup. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady` |
|
||||
| `RUN-014` | `rustfs/src/main.rs:402` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None |
|
||||
| `RUN-015` | `rustfs/src/main.rs:482` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
|
||||
| `RUN-016` | `rustfs/src/main.rs:503` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |
|
||||
| `RUN-017` | `rustfs/src/main.rs:523` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task |
|
||||
| `RUN-018` | `rustfs/src/main.rs:535` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None |
|
||||
| `RUN-019` | `rustfs/src/main.rs:552` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None |
|
||||
| `RUN-020` | `rustfs/src/main.rs:560` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None |
|
||||
| `RUN-021` | `rustfs/src/main.rs:584` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None |
|
||||
| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady` |
|
||||
| `RUN-014` | `rustfs/src/startup_services.rs` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None |
|
||||
| `RUN-015` | `rustfs/src/startup_services.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
|
||||
| `RUN-016` | `rustfs/src/startup_services.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |
|
||||
| `RUN-017` | `rustfs/src/startup_services.rs` and `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task |
|
||||
| `RUN-018` | `rustfs/src/startup_services.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None |
|
||||
| `RUN-019` | `rustfs/src/startup_services.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None |
|
||||
| `RUN-020` | `rustfs/src/startup_services.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None |
|
||||
| `RUN-021` | `rustfs/src/startup_services.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None |
|
||||
| `RUN-022` | `rustfs/src/main.rs:599` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here |
|
||||
| `RUN-023` | `rustfs/src/main.rs:609` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery |
|
||||
| `RUN-024` | `rustfs/src/main.rs:616` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state |
|
||||
|
||||
Reference in New Issue
Block a user