mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
docs: add architecture readiness matrix (#3938)
This commit is contained in:
Zhengchao An
@@ -36,6 +36,21 @@ Existing migration checks live in:
|
||||
|
||||
Extend these guardrails instead of adding a parallel system.
|
||||
|
||||
## Required Architecture Documents
|
||||
|
||||
The migration guard must keep these baseline documents present and anchored to
|
||||
their required sections:
|
||||
|
||||
- `docs/architecture/overview.md`: Baseline, Core Principle, Phase Order.
|
||||
- `docs/architecture/runtime-lifecycle.md`: Startup And Readiness, Shutdown
|
||||
Lifecycle Boundary, AppContext Foundation.
|
||||
- `docs/architecture/storage-control-data-plane.md`: Storage API Contracts,
|
||||
Cluster Control Plane, Background Controllers.
|
||||
- `docs/architecture/crate-boundaries.md`: PR Types, Dependency Direction,
|
||||
Required Architecture Documents.
|
||||
- `docs/architecture/readiness-matrix.md`: Request Behavior Matrix, Runtime
|
||||
Dependency Matrix, Probe Semantics.
|
||||
|
||||
## Pre-Push Expert Review
|
||||
|
||||
Before pushing any PR branch, record three expert reviews in the task notes:
|
||||
|
||||
@@ -5,98 +5,22 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
## Current Context
|
||||
|
||||
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
|
||||
- Branch: `overtrue/arch-cluster-lock-health-phase`
|
||||
- Branch: `overtrue/arch-docs-phase`
|
||||
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213/API-214/API-215/API-216/API-217/API-218/API-219/API-220/API-221/API-222/API-223/API-224/API-225/API-226/API-227/API-228/API-229/API-230/API-231/API-232/API-233/API-234/API-235/API-236/API-237/API-238/API-239/API-240/API-241/API-242/API-243/API-244/API-245/API-246/API-247/API-248/API-249/API-250/API-251/API-252/API-253/API-254/CTX-002`.
|
||||
- Current baseline also includes API-255 from PR #3923, API-256 from PR
|
||||
#3925, CFG-009 from PR #3927, and C-007/C-009 from PR #3935.
|
||||
- Current phase PR: C-008/C-010 cluster lock-registry consumption and gated
|
||||
peer-health readiness impact batch.
|
||||
- Based on: `origin/main` after PR #3935 merged.
|
||||
- PR type for this branch: `contract`.
|
||||
- Runtime behavior changes: lock client selection now consumes a runtime
|
||||
`LockRegistry` snapshot instead of directly scanning global clients inside
|
||||
each set; peer-health readiness impact is gated by
|
||||
`RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` and remains disabled by default.
|
||||
- Rust code changes: add the ECStore lock-registry runtime source, route
|
||||
`Sets::new` through endpoint-ordered registry selection, add a disabled-by
|
||||
default peer-health readiness gate, carry the peer-health readiness bit
|
||||
through health and cluster runtime status views, and cover default-disabled
|
||||
and enabled-degraded readiness behavior.
|
||||
- CI/script changes: guard that the ECStore RPC boundary snapshot and
|
||||
internode data-transport control-plane separation note remain present; reject
|
||||
restoring ECStore root `store.rs`, `set_disk.rs`,
|
||||
`store_list_objects.rs`, `store_utils.rs`, `store_init.rs`,
|
||||
`disks_layout.rs`, `endpoints.rs`, `storage_api_contracts.rs`,
|
||||
`batch_processor.rs`, `event_notification.rs`, `metrics_realtime.rs`, or
|
||||
`notification_sys.rs`, `data_movement.rs`, or
|
||||
`data_movement_backpressure.rs`, `admin_server_info.rs`, `data_usage.rs`,
|
||||
`get_diagnostics.rs`, `global.rs`, `runtime_sources.rs`, `bitrot.rs`,
|
||||
`compress.rs`, `rio.rs`, `erasure_codec`, `erasure_coding`, `error.rs`,
|
||||
`rebalance.rs`, `pools.rs`, `sets.rs`, `pools_test.rs`, or `store_test.rs`,
|
||||
keep the ECStore cluster root module as a re-export-only owner facade, reject
|
||||
restoring ECStore root RPC modules or root `lib.rs` declaration, reject
|
||||
ECStore-internal `crate::rpc` consumers, reject restoring old bucket/set-disk
|
||||
metadata owner paths, and reject
|
||||
restoring ECStore root `lib.rs` owner path shims for core, store, layout, data
|
||||
movement, diagnostics, services, I/O support, and runtime modules; reject
|
||||
restoring ECStore root rebalance or tier service-domain modules or
|
||||
ECStore-internal `crate::rebalance`/`crate::tier` consumers;
|
||||
lock completed config model ownership against restoring
|
||||
old `rustfs_ecstore::config` model/accessor compatibility paths, and lock
|
||||
ECStore public facades against re-exporting the moved server-config symbols;
|
||||
retain the existing completed owner and test/fuzz boundaries against
|
||||
bare/glob imports, scattered raw ECStore facade subpaths, and startup
|
||||
runtime/root-server/table/S3/app shared/app bucket/app ECStore/admin facade
|
||||
regressions, plus external runtime, test, fuzz, and storage-owner module
|
||||
ECStore compatibility bypasses, plus runtime crate, owner crate, test/fuzz,
|
||||
and storage owner thin bridge regressions, plus app context and notify
|
||||
event-bridge thin module regressions, plus IAM runtime-source bypasses;
|
||||
accept the reviewed AppContext resolver reverse dependencies in the layer
|
||||
baseline, and block direct admin AppContext resolver consumers outside the
|
||||
admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, narrow the object-usecase ECFS layer baseline entry to `FS`, reject direct storage S3 API helper imports from app usecase files, reject direct storage helper imports from app select/usecase files, reject completed app/admin storage helper bypasses, reject app usecase bypasses for migrated storage IO/compression/set-disk helpers, reject app usecase/test bypasses for migrated storage error, ETag, and storage-class helpers, reject app root bucket owner facade bypasses from migrated app consumers, reject app/admin runtime/data-usage root facade regressions, reject admin root storage facade regressions from migrated admin consumers, reject root/server/startup direct storage facade regressions from migrated outer consumers, reject root/server/startup direct storage contract imports from migrated outer consumers, reject app/admin direct storage contract imports from migrated owner consumers, keep app S3 helper imports routed through `app::storage_api`, reject scanner/heal direct ECStore or storage contract imports outside their local `storage_api` boundaries, reject external runtime/test/fuzz ECStore or storage contract imports outside their local `storage_api` boundaries, reject storage owner direct ECStore/storage-api imports outside the owner-local `storage_api` boundary, reject ECStore internal direct storage-api imports outside the owner-local `storage_api_contracts` boundary, reject direct storage ECFS app usecase construction outside the storage S3 API boundary, reject flat root `storage_api` imports outside the new root-local domain modules, reject flat admin `storage_api` imports outside the new admin domain modules, reject flat app `storage_api` imports outside the new app consumer-domain modules, reject flat external crate-local `storage_api` imports outside the new consumer-domain modules, reject residual flat local `storage_api` imports from notify, ECStore tests/benches, and remaining RustFS app storage API type references, reject external/test local storage API root contract re-exports, reject RustFS local storage API root contract re-exports, reject RustFS app/admin storage helper root re-exports, and reject ECStore storage_api_contracts root re-exports or root consumers.
|
||||
- Docs changes: record the API-136 through API-242 owner facade,
|
||||
runtime-source, ECFS usecase, root storage API domain-boundary, and admin
|
||||
storage API domain-boundary cleanup, and app storage API consumer-domain
|
||||
cleanup, plus external crate-local and residual local storage API
|
||||
consumer-domain cleanup, external/test storage contract root re-export
|
||||
cleanup, RustFS local storage contract root re-export cleanup, RustFS
|
||||
app/admin storage helper root re-export cleanup, ECStore
|
||||
storage_api_contracts domain segmentation, root storage contract facade
|
||||
domain segmentation, root runtime facade consumer-domain segmentation, and
|
||||
storage-owner internal consumer-domain cleanup; reject storage-owner internal
|
||||
root contract/object helper facade consumers after API-250, reject
|
||||
storage-owner runtime source, object-lock helper, RPC relative root, and ECFS
|
||||
test root consumers after API-251, reject restored RPC wildcard imports
|
||||
after API-252, reject restored parent wildcard imports anywhere under
|
||||
`rustfs/src/storage` after API-253, reject restoring storage owner root
|
||||
wildcard re-exports after API-254, reject direct storage owner paths from the
|
||||
root/app/admin storage facades after API-255, reject restoring storage
|
||||
root SSE re-exports after API-255, reject direct external
|
||||
`rustfs_ecstore::api` facade imports outside local `storage_api` boundary
|
||||
files after API-256, and reject restoring old ECStore server-config model or
|
||||
global accessor compatibility paths after CFG-009, reject restoring ECStore
|
||||
root `store.rs` or `set_disk.rs` after E-017, reject restoring ECStore root
|
||||
store support modules after E-018, reject restoring ECStore root
|
||||
`store_init.rs` after E-019, reject restoring ECStore root layout facade
|
||||
and storage API contract support modules after E-020, and reject restoring
|
||||
ECStore root service runtime modules after E-021, and reject restoring
|
||||
ECStore root data movement modules after E-022, and reject restoring ECStore
|
||||
root usage and diagnostics modules after E-023, and reject restoring ECStore
|
||||
root runtime global modules after E-024, and reject restoring ECStore root
|
||||
I/O support modules after E-025, and reject restoring ECStore root error and
|
||||
rebalance facade modules after E-026, and reject restoring ECStore root core
|
||||
runtime/test modules after E-027, keep the ECStore cluster root module as a
|
||||
re-export-only owner facade after E-028, reject restoring ECStore root RPC
|
||||
support files after E-029, reject restoring old bucket/set-disk metadata owner
|
||||
paths after E-030, reject restoring ECStore root owner path shims for
|
||||
diagnostics, services, I/O support, and runtime modules after E-031, and
|
||||
reject restoring ECStore root owner path shims for core, store, layout, and
|
||||
data movement modules after E-032, reject restoring ECStore root erasure
|
||||
codec or coding modules after E-033/E-004, and reject restoring ECStore root
|
||||
RPC facade modules or ECStore-internal `crate::rpc` consumers after E-034,
|
||||
and reject restoring ECStore root rebalance/tier service-domain modules or
|
||||
ECStore-internal `crate::rebalance`/`crate::tier` consumers after E-035/E-008,
|
||||
and reject losing the C-009 cluster RPC boundary model.
|
||||
#3925, CFG-009 from PR #3927, C-007/C-009 from PR #3935, and C-008/C-010
|
||||
from PR #3936.
|
||||
- Current phase PR: DOC-001/DOC-002/DOC-003/DOC-004/DOC-005/TEST-DOC-001
|
||||
architecture documentation guard batch.
|
||||
- Based on: `origin/main` after PR #3936 merged.
|
||||
- PR type for this branch: `ci-gate`.
|
||||
- Runtime behavior changes: none.
|
||||
- Rust code changes: none.
|
||||
- CI/script changes: require the core architecture docs and readiness matrix
|
||||
sections in `scripts/check_architecture_migration_rules.sh`.
|
||||
- Docs changes: add the readiness matrix, link it from the architecture overview,
|
||||
align runtime/startup/control-plane docs with the gated peer-health readiness
|
||||
contract, and record required architecture docs in crate boundaries.
|
||||
|
||||
## Phase 0 Tasks
|
||||
|
||||
@@ -1653,6 +1577,48 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
- Verification: formatting, compile checks, migration guards, diff hygiene,
|
||||
Rust risk scan, and full `make pre-commit`.
|
||||
|
||||
## Phase 1c Architecture Documentation Tasks
|
||||
|
||||
- [x] `DOC-001` Add architecture overview.
|
||||
- Completed slice: keep the architecture document index current and link the
|
||||
readiness matrix from [`overview.md`](overview.md).
|
||||
- Acceptance: overview records baseline, core principle, phase order, and the
|
||||
required architecture document set.
|
||||
- Verification: architecture documentation guard and diff hygiene.
|
||||
- [x] `DOC-002` Add runtime lifecycle architecture document.
|
||||
- Completed slice: align the runtime lifecycle contract with effective
|
||||
`FullReady` and the disabled-by-default peer-health readiness gate.
|
||||
- Acceptance: startup/readiness, shutdown, AppContext, and peer-health gate
|
||||
contracts remain documented without changing runtime behavior.
|
||||
- Verification: architecture documentation guard and diff hygiene.
|
||||
- [x] `DOC-003` Add storage control/data plane document.
|
||||
- Completed slice: link storage, lock quorum, peer health, probes, admin,
|
||||
RPC, and S3 data-plane readiness behavior from the control-plane baseline.
|
||||
- Acceptance: StorageCore, ClusterControlPlane, data-plane, control-plane, and
|
||||
background-controller boundaries remain explicit.
|
||||
- Verification: architecture documentation guard and diff hygiene.
|
||||
- [x] `DOC-004` Add crate boundaries document.
|
||||
- Completed slice: record the required architecture document set in
|
||||
[`crate-boundaries.md`](crate-boundaries.md).
|
||||
- Acceptance: storage-api, config, security-governance, extension-plane, and
|
||||
ECStore boundary docs remain guarded by existing migration rules.
|
||||
- Verification: architecture documentation guard and diff hygiene.
|
||||
- [x] `DOC-005` Add readiness matrix.
|
||||
- Completed slice: add [`readiness-matrix.md`](readiness-matrix.md) for S3,
|
||||
admin/console, internode RPC/gRPC, table catalog, probes, optional
|
||||
sidecars, runtime dependencies, and probe semantics.
|
||||
- Acceptance: S3 data requests remain gated until `FullReady`; probe/admin/RPC
|
||||
bypass behavior, `StorageReady`, `IamReady`, lock quorum, peer-health gate,
|
||||
and KMS compatibility readiness semantics are documented.
|
||||
- Verification: architecture documentation guard and diff hygiene.
|
||||
- [x] `TEST-DOC-001` Guard architecture documentation coverage.
|
||||
- Completed slice: extend `scripts/check_architecture_migration_rules.sh` to
|
||||
require the core architecture docs and readiness matrix anchors.
|
||||
- Acceptance: the migration guard fails if required architecture docs or
|
||||
required readiness-matrix sections are removed.
|
||||
- Verification: `bash -n scripts/check_architecture_migration_rules.sh`;
|
||||
`./scripts/check_architecture_migration_rules.sh`; `git diff --check`.
|
||||
|
||||
## Phase 1 Security Governance Tasks
|
||||
|
||||
- [x] `S-001` Add `crates/security-governance`.
|
||||
@@ -6035,6 +6001,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
|
||||
| Expert | Status | Notes |
|
||||
|---|---|---|
|
||||
| Quality/architecture | pass | DOC-001 through DOC-005 add one phase-level architecture documentation batch and keep the guard anchored to required sections instead of duplicating a parallel docs system. |
|
||||
| Migration preservation | pass | The readiness matrix and lifecycle docs only record existing S3/admin/RPC/probe behavior plus the disabled-by-default peer-health readiness gate; no runtime code or behavior changes are included. |
|
||||
| Testing/verification | pass | Shell syntax, architecture migration guard, diff hygiene, and full `make pre-pr` passed after rebasing onto current `origin/main`. |
|
||||
| Quality/architecture | pass | E-028 moves the ECStore cluster control-plane implementation into an owner submodule and leaves the root module as a facade. |
|
||||
| Migration preservation | pass | `crate::cluster::*` and `rustfs_ecstore::api::cluster::*` stay stable through root re-exports; the moved implementation is a 100% rename. |
|
||||
| Testing/verification | pass | ECStore cluster tests, ECStore full tests, architecture/layer guards, formatting, diff hygiene, diff-added Rust risk scan, and pre-commit passed. |
|
||||
@@ -6383,6 +6352,16 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
|
||||
Passed before push:
|
||||
|
||||
- Issue #660 DOC-001/DOC-002/DOC-003/DOC-004/DOC-005/TEST-DOC-001 current
|
||||
slice:
|
||||
- Branch freshness check: rebased onto `origin/main` after PR #3936 merged.
|
||||
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- Rust source risk scan: passed; no Rust source changed.
|
||||
- Three-expert review: passed.
|
||||
- `make pre-pr`: passed.
|
||||
|
||||
- Issue #660 API-256 current slice:
|
||||
- Branch freshness check: based on current `origin/main` after PR #3923
|
||||
merged.
|
||||
|
||||
@@ -20,6 +20,8 @@ hot-path behavior must not drift during this migration.
|
||||
|
||||
- [`runtime-lifecycle.md`](runtime-lifecycle.md): runtime, AppContext,
|
||||
startup/readiness, and shutdown contracts.
|
||||
- [`readiness-matrix.md`](readiness-matrix.md): request-surface behavior,
|
||||
runtime dependency readiness, probe semantics, and preservation rules.
|
||||
- [`s3-tables-support-matrix.md`](s3-tables-support-matrix.md): supported,
|
||||
preview, reference-only, and not-claimed S3 Tables and Iceberg REST Catalog
|
||||
surfaces.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# Readiness Matrix
|
||||
|
||||
This document records the current request and dependency behavior around
|
||||
startup readiness. It is a behavior-preservation baseline for architecture
|
||||
migration work, not a new readiness policy.
|
||||
|
||||
## Request Behavior Matrix
|
||||
|
||||
| Surface | Path examples | Before `FullReady` | After `FullReady` | Dependency notes |
|
||||
|---|---|---|---|---|
|
||||
| Health probe | `/health`, `/health/live`, `/health/ready`, `/minio/health/*` | Bypasses the HTTP readiness gate and returns probe-specific liveness or readiness state. | Same path-specific probe behavior. | Probe handlers compute health independently of the outer request gate. |
|
||||
| Admin and console | `/admin/*`, `/console/*`, `/rustfs/admin/*`, `/minio/admin/*` | Bypasses the outer readiness gate; route auth, handler setup, and handler-specific dependencies still apply. | Same handler behavior without outer gate rejection. | Do not use admin bypasses as proof that storage, IAM, or lock quorum is ready. |
|
||||
| Internode RPC and gRPC | `/rustfs/rpc/*`, `/minio/rpc/*`, tonic routes | Bypasses the HTTP readiness gate so control-plane peers can communicate during startup. | Same RPC routing behavior. | RPC signature, auth, transport, and handler failures remain authoritative. |
|
||||
| Table catalog | `/iceberg/*`, `/v1/*`, table-catalog routes | Bypasses the outer readiness gate when the route is registered. | Same registered route behavior. | Catalog handlers keep their own dependency checks. |
|
||||
| S3 data plane | Bucket/object S3 API routes | Receives `503 Service Unavailable` from the readiness gate with `Retry-After: 5`. | Routed to S3 handlers. | This is the main data-plane behavior protected by `FullReady`. |
|
||||
| Optional sidecars | FTP, FTPS, SFTP, WebDAV | Governed by protocol-specific startup and shutdown handling. | Governed by protocol-specific runtime handling. | These are not HTTP readiness-gate surfaces. |
|
||||
|
||||
## Runtime Dependency Matrix
|
||||
|
||||
| Dependency | Ready signal | Blocks `FullReady` | Notes |
|
||||
|---|---|---|---|
|
||||
| StorageReady | Storage/global config readiness publication plus runtime storage checks. | Yes. | Startup can mark the stage before later runtime readiness rechecks storage. |
|
||||
| IamReady | Inline IAM bootstrap or deferred IAM recovery publication. | Yes. | Deferred recovery can publish IAM readiness after HTTP has already started. |
|
||||
| Lock quorum | Per-set write quorum readiness. | Yes. | Do not replace the distributed lock quorum check with node count or endpoint count. |
|
||||
| Peer health | `peer_health_ready` runtime status. | Only when `RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` is enabled. | The gate is disabled by default; unknown peer health degrades readiness only when enabled. |
|
||||
| KMS compatibility | KMS health compatibility readiness. | Only when the KMS compatibility readiness check is enabled. | KMS startup fatality and health reporting remain separate from pure docs work. |
|
||||
|
||||
Effective `FullReady` is:
|
||||
|
||||
```text
|
||||
storage_ready && iam_ready && lock_quorum_ready && peer_health_ready
|
||||
```
|
||||
|
||||
`peer_health_ready` is true by default unless
|
||||
`RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` is enabled. When that flag is enabled,
|
||||
an unknown or unsupported peer-health snapshot degrades readiness with
|
||||
`peer_health_unavailable`.
|
||||
|
||||
## Probe Semantics
|
||||
|
||||
- Liveness reports process availability and must not depend on storage, IAM,
|
||||
lock quorum, or peer health.
|
||||
- Node readiness reports local dependency readiness.
|
||||
- Cluster write readiness requires write quorum and the runtime dependency
|
||||
readiness used by `FullReady`.
|
||||
- Cluster read readiness may use the read-quorum path and cluster-health
|
||||
timeout behavior.
|
||||
- `HEAD` health probes keep header/status semantics and do not require response
|
||||
bodies.
|
||||
|
||||
## Preservation Rules
|
||||
|
||||
- Do not move peer-health checks into the S3 data hot path.
|
||||
- Do not make peer health affect readiness unless
|
||||
`RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` is enabled.
|
||||
- Do not use this matrix to change the early HTTP listener plus readiness-gate
|
||||
split.
|
||||
- Do not simplify distributed lock quorum, IAM deferred recovery, or KMS fatal
|
||||
boundaries in a documentation or guardrail PR.
|
||||
@@ -6,13 +6,17 @@ shutdown semantics.
|
||||
## Startup And Readiness
|
||||
|
||||
- HTTP can listen early, but normal requests must remain behind readiness gates.
|
||||
- `FullReady = storage_ready && iam_ready && lock_quorum_ready`.
|
||||
- Effective `FullReady = storage_ready && iam_ready && lock_quorum_ready &&
|
||||
peer_health_ready`; `peer_health_ready` is true by default unless
|
||||
`RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` is enabled.
|
||||
- Boot phases must keep the old fatal and non-fatal boundaries.
|
||||
- AppContext migration keeps context-first lookup with global fallback until the
|
||||
global path is proven unused.
|
||||
- Notify and audit lifecycle behavior must not drift during lifecycle movement.
|
||||
- IAM and KMS startup, deferred recovery, and fatal boundary behavior must not be
|
||||
changed by pure movement PRs.
|
||||
- Request-surface and dependency details are tracked in
|
||||
[`readiness-matrix.md`](readiness-matrix.md).
|
||||
|
||||
## Service Registry Scope
|
||||
|
||||
|
||||
@@ -33,10 +33,10 @@ new startup semantics.
|
||||
| `RUN-007` | `rustfs/src/startup_storage.rs` | Publish endpoints and erasure type. | Updates global endpoints and erasure type. | Non-fatal in this path. | None |
|
||||
| `RUN-008` | `rustfs/src/startup_storage.rs` | Initialize local disks, prewarm local disk id map, and initialize lock clients. | Opens local disk state, primes disk id lookup, and creates global lock clients. | Local disk init is fatal; prewarm and lock-client setup are non-fatal in this path. | None |
|
||||
| `RUN-009` | `rustfs/src/startup_server.rs` | Initialize capacity management and service state manager. | Starts capacity management and moves service state to `Starting`. | Non-fatal in this path. | None |
|
||||
| `RUN-010` | `rustfs/src/startup_server.rs` | Start S3 HTTP listener and optional console listener before storage is ready. | Starts HTTP servers with readiness gates; console listener starts only when enabled and configured. | Fatal if a configured listener cannot start. | Requests remain gated until full readiness except probe/admin/console/rpc/tonic/table-catalog exempt paths |
|
||||
| `RUN-010` | `rustfs/src/startup_server.rs` | Start S3 HTTP listener and optional console listener before storage is ready. | Starts HTTP servers with readiness gates; console listener starts only when enabled and configured. | Fatal if a configured listener cannot start. | Requests remain gated until full readiness except probe/admin/console/rpc/tonic/table-catalog exempt paths; see [`readiness-matrix.md`](readiness-matrix.md) |
|
||||
| `RUN-011` | `rustfs/src/startup_storage.rs` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None |
|
||||
| `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, and lock quorum before `FullReady` |
|
||||
| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady` |
|
||||
| `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, lock quorum, and gated peer health before `FullReady` |
|
||||
| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady`; KMS compatibility readiness remains feature-gated health behavior |
|
||||
| `RUN-014` | `rustfs/src/startup_optional_runtime_sidecars.rs` and `rustfs/src/startup_protocols.rs` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None |
|
||||
| `RUN-015` | `rustfs/src/startup_services.rs` and `rustfs/src/startup_service_components.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
|
||||
| `RUN-016` | `rustfs/src/startup_service_components.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |
|
||||
@@ -63,7 +63,7 @@ new startup semantics.
|
||||
|---|---|---|---|---|---|
|
||||
| `READY-001` | `rustfs/src/server/readiness.rs:130` | Treat exact probe paths and admin/console/rpc/tonic/table-catalog prefixes as readiness-gate bypass paths. | Bypass paths continue to the inner service while the global readiness gate is not ready. | Non-fatal. | Does not change readiness stages |
|
||||
| `READY-002` | `rustfs/src/server/readiness.rs:171` | Reject non-probe requests while `GlobalReadiness` is not ready. | Returns `503 Service Unavailable`, `Retry-After: 5`, `Content-Type: text/plain; charset=utf-8`, and `Cache-Control: no-store`. | Non-fatal. | Does not change readiness stages |
|
||||
| `READY-003` | `rustfs/src/server/readiness.rs:202` | Wait for runtime storage, IAM, and lock quorum readiness before publishing ready state. | Marks `FullReady` and updates `ServiceState` to `Ready` only when a state manager is provided. | Returns an error on timeout; inline startup treats that as fatal, while deferred IAM recovery retries finalization. | Marks `FullReady` |
|
||||
| `READY-003` | `rustfs/src/server/readiness.rs:202` | Wait for runtime storage, IAM, lock quorum, and gated peer-health readiness before publishing ready state. | Marks `FullReady` and updates `ServiceState` to `Ready` only when a state manager is provided. | Returns an error on timeout; inline startup treats that as fatal, while deferred IAM recovery retries finalization. | Marks `FullReady` |
|
||||
|
||||
## Shutdown Order
|
||||
|
||||
|
||||
@@ -45,6 +45,9 @@ The same facade also owns static pool-state, local-node storage, and peer-health
|
||||
status projections. Peer health remains explicitly unknown until a later slice
|
||||
wires real health signals; this document does not authorize background probes or
|
||||
RPC-based health checks.
|
||||
Readiness impact for storage, lock quorum, peer health, probes, admin routes,
|
||||
RPC, and the S3 data plane is recorded in
|
||||
[`readiness-matrix.md`](readiness-matrix.md).
|
||||
|
||||
Risk controls:
|
||||
|
||||
|
||||
@@ -41,6 +41,28 @@ require_source_contains() {
|
||||
fi
|
||||
}
|
||||
|
||||
require_source_contains "docs/architecture/overview.md" "## Baseline" "architecture overview baseline section"
|
||||
require_source_contains "docs/architecture/overview.md" "## Core Principle" "architecture overview core principle section"
|
||||
require_source_contains "docs/architecture/overview.md" "## Phase Order" "architecture overview phase order section"
|
||||
require_source_contains "docs/architecture/runtime-lifecycle.md" "## Startup And Readiness" "runtime lifecycle startup readiness section"
|
||||
require_source_contains "docs/architecture/runtime-lifecycle.md" "## Shutdown Lifecycle Boundary" "runtime lifecycle shutdown boundary section"
|
||||
require_source_contains "docs/architecture/runtime-lifecycle.md" "## AppContext Foundation" "runtime lifecycle AppContext section"
|
||||
require_source_contains "docs/architecture/storage-control-data-plane.md" "## Storage API Contracts" "storage control/data plane contracts section"
|
||||
require_source_contains "docs/architecture/storage-control-data-plane.md" "## Cluster Control Plane" "storage control/data plane cluster section"
|
||||
require_source_contains "docs/architecture/storage-control-data-plane.md" "## Background Controllers" "storage control/data plane background controllers section"
|
||||
require_source_contains "docs/architecture/crate-boundaries.md" "## Dependency Direction" "crate boundaries dependency direction section"
|
||||
require_source_contains "docs/architecture/crate-boundaries.md" "storage-api -> ecstore" "crate boundaries storage dependency rule"
|
||||
require_source_contains "docs/architecture/crate-boundaries.md" "extension-schema -> rustfs" "crate boundaries extension dependency rule"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "## Request Behavior Matrix" "readiness matrix request behavior section"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "## Runtime Dependency Matrix" "readiness matrix runtime dependency section"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "## Probe Semantics" "readiness matrix probe semantics section"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "S3 data plane" "readiness matrix S3 data plane behavior"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "Internode RPC and gRPC" "readiness matrix internode RPC behavior"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "StorageReady" "readiness matrix storage dependency"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "IamReady" "readiness matrix IAM dependency"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "FullReady" "readiness matrix full readiness dependency"
|
||||
require_source_contains "docs/architecture/readiness-matrix.md" "RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE" "readiness matrix peer health gate"
|
||||
|
||||
TMP_DIR="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP_DIR"' EXIT
|
||||
|
||||
|
||||
Reference in New Issue
Block a user