gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-09 14:49:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: retire completed-migration scaffolding, wire orphaned boundary check (#4719)

Browse Source 
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all
closed). Review of every migration-era test/gate measure found three things
actually retirable or broken — everything else is a live anti-regression
guard and stays.

Remove:
- scripts/check_metrics_migration_refs.sh — guards a migration that
  finished: rustfs_metrics:: has zero hits, the metrics crate no longer
  exists, and the script was never wired into CI or make (only reference
  was one line in config-model-boundary-adr.md, also removed).
- crates/obs init_metrics_collectors — the "backward-compatible alias kept
  during migration" the removed script was guarding. Zero callers; pure
  delegate to init_metrics_runtime.

Archive (docs/superpowers/plans/, continuing the 2026-07 convention,
with the standard archived banner):
- startup-timeline.md, scheduler-baseline.md,
  profiling-numa-capability-inventory.md,
  kms-development-defaults-inventory.md — one-shot snapshots whose only
  consumer is the already-archived migration-progress ledger (their
  same-dir links there start resolving again after the move); zero script
  pins; fed the closed backlog#660/#665 architecture-review ledger.
  Fixed the one outbound link (startup-timeline -> readiness-matrix) that
  the move would have broken — check_doc_paths.sh deliberately does not
  scan plans/, so nothing else would have caught it.

Wire (found orphaned by the same review):
- scripts/check_extension_schema_boundaries.sh guards a live contract
  crate but was never invoked anywhere. Add lint-fmt.mak target, include
  in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job
  already installs ripgrep), sync the CONTRIBUTING.md enumerated list,
  and harden the script against a silently-passing rg probe when src/
  is missing.

Keep (verified live, documented so the next cleanup pass does not repeat
this analysis):
- scripts/check_architecture_migration_rules.sh — added a header stating
  it is a permanent boundary guard, not retirable migration scaffolding;
  'migration' in the name is historical.
- check_migration_gate_count.sh + floor, delete-marker e2e proof, all
  pinned docs, compat-cleanup-register sync, remaining inventories
  (referenced by live docs).

Verification: all 7 guard scripts pass, actionlint clean,
cargo check --workspace (excl e2e) clean, cargo fmt --check clean.
Adversarially reviewed by two independent skeptic passes; their 7
findings (alias left behind, broken outbound link, missing banners,
wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing
header rationale) are all folded in.
This commit is contained in:
Zhengchao An
2026-07-11 13:42:56 +08:00
committed by GitHub
parent a97f3a9c52
commit f63af3df63
16 changed files with 52 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/kms-development-defaults-inventory.md
+105
View File
@@ -0,0 +1,105 @@
> **Archived migration snapshot** — moved from `docs/architecture/` (2026-07)
> when the architecture-review ledger it fed closed out. Kept for history; not
> maintained.
# KMS Development Defaults Inventory
This inventory tracks `KMSD-001` for
[`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660). It records
current KMS development defaults before any production default hardening.
## Scope
- Source files reviewed: `crates/kms/src/config.rs` and
`crates/kms/src/api_types.rs`.
- This is a `docs-only` task.
- No runtime behavior, config serialization, admin authorization, startup order,
global state, storage path, or crate boundary changes are included.
- Follow-up hardening must be done in separate `security-change` PRs with focused
tests.
## Current Defaults
| Source | Default | Current behavior | Classification |
|---|---|---|---|
| `KmsConfig::default()` | Local backend | Uses `LocalConfig::default()` and validates successfully. | dev-only |
| `LocalConfig::default().key_dir` | OS temp dir plus `rustfs_kms_keys` | Keys are stored under the process temp directory. | dev-only |
| `LocalConfig::default().master_key` | `None` | Local key files are stored in plaintext when no master key is configured. | invalid for production |
| `LocalConfig::default().file_permissions` | `0o600` | Owner read/write only for key files. | production-safe as a permission default, but not sufficient without encrypted key storage |
| `KmsConfig::local(key_dir)` | caller-provided key dir, default local fields | Keeps `master_key = None` unless the caller supplies one later. | dev-only unless explicit encryption material is configured |
| `KmsConfig::from_env()` local key dir | `./kms_keys` | The env loader builds a relative path, then existing validation rejects it because local key dirs must be absolute. | invalid as a standalone default |
| `KmsConfig::from_env()` local master key | absent | Leaves `master_key = None`. | invalid for production |
| `VaultConfig::default().address` | `http://localhost:8200` | HTTP is accepted by validation. | dev-only |
| `VaultTransitConfig::default().address` | `http://localhost:8200` | HTTP is accepted by validation. | dev-only |
| `VaultConfig::default().auth_method` | token dev-token | The default token is accepted if used as-is. | invalid for production |
| `VaultTransitConfig::default().auth_method` | token dev-token | The default token is accepted if used as-is. | invalid for production |
| `VaultConfig::default().tls` | `None` | No custom TLS settings. HTTPS without custom TLS relies on system CA when no skip flag is set. | production-safe only when HTTPS and system trust are intended |
| `VaultTransitConfig::default().tls` | `None` | Same TLS behavior as Vault KV2. | production-safe only when HTTPS and system trust are intended |
| `ConfigureVaultKmsRequest.skip_tls_verify` | omitted means false | `to_kms_config()` leaves TLS config as `None` unless the request explicitly sets true. | production-safe when omitted |
| `ConfigureVaultTransitKmsRequest.skip_tls_verify` | omitted means false | Same behavior as Vault KV2 configure requests. | production-safe when omitted |
| `skip_tls_verify = true` in configure requests | explicit insecure opt-in | Creates a TLS config with `skip_verify = true`. | invalid for production |
## Existing Validation Boundary
- Local key directories must be absolute.
- Timeout and retry attempts must be greater than zero.
- Vault addresses must use HTTP or HTTPS.
- Vault mount paths must be non-empty.
- HTTPS with custom TLS config and verification enabled warns when relying on
system CA instead of custom CA/client certificates.
- Existing validation does not fail closed for HTTP Vault addresses, dev-token,
missing local master key, temp key dirs, or explicit `skip_tls_verify = true`.
## Hardening Behavior
`KMSD-002` makes Local KMS unsafe defaults explicit development opt-ins or
production failures:
- no local master key fails validation unless
`allow_insecure_dev_defaults = true`;
- local key directories under the process temp directory fail validation unless
`allow_insecure_dev_defaults = true`;
- `RUSTFS_KMS_LOCAL_MASTER_KEY` is the production-safe local CLI/env path for
encrypted local key files;
- `RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true` is the development-only escape
hatch for local plaintext or temp-dir setups.
`KMSD-003` makes Vault unsafe defaults explicit development opt-ins or
production failures:
- HTTP Vault addresses fail validation unless explicit development opt-in is set;
- default `dev-token` credentials fail validation unless explicit development
opt-in is set;
- explicit `skip_tls_verify = true` fails validation unless explicit development
opt-in is set;
- `RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true` applies to `KmsConfig::from_env`;
- admin configure requests can set `allow_insecure_dev_defaults = true` for the
same development-only behavior.
These checks run in `KmsConfig::validate()` so CLI startup, persisted dynamic
configuration, service-manager start/reconfigure, and direct backend
construction use the same fail-closed behavior.
## Compatibility Notes
- Production Local KMS deployments should configure an absolute key directory
outside the process temp directory and set `RUSTFS_KMS_LOCAL_MASTER_KEY`.
- Local development setups that intentionally store plaintext key files or use
temp directories must set `RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true` or the
admin request field `allow_insecure_dev_defaults = true`.
- Production Vault deployments should use HTTPS, non-default credentials, and
TLS verification.
- Local Vault development setups that intentionally use HTTP, `dev-token`, or
skip TLS verification must set the same explicit development opt-in.
- Legacy persisted KMS config JSON remains deserializable; old unsafe persisted
values default to production mode and fail validation until secured or
explicitly marked development-only.
## Test Coverage
- Local production fail-closed and development opt-in validation.
- Vault HTTP, default token, and skip-TLS fail-closed validation plus explicit
development opt-in.
- `KmsConfig::from_env()` development default rejection and opt-in behavior.
- Admin configure request conversion to the same validation behavior.
- KMS service manager rejects unsafe configs before moving to `Configured`.
docs/superpowers/plans/profiling-numa-capability-inventory.md
+99
View File
@@ -0,0 +1,99 @@
> **Archived migration snapshot** — moved from `docs/architecture/` (2026-07)
> when the architecture-review ledger it fed closed out. Kept for history; not
> maintained.
# Profiling And NUMA Capability Inventory
This inventory covers `G-013` for `rustfs/backlog#667`. It records the current
profiling, memory sampling, allocator, and NUMA baseline before optional runtime
sidecars are designed.
## Platform Support Matrix
| Capability | Current support | Current owner | Baseline recommendation |
|---|---|---|---|
| CPU pprof dump | Linux and macOS builds use `pprof`; other targets return an unsupported-platform error. | `rustfs/src/profiling.rs` | Keep CPU profiling opt-in through existing env flags and cancellation token. |
| Continuous CPU profiling | Linux and macOS builds can hold a continuous `ProfilerGuard` when enabled. | `rustfs/src/profiling.rs` | Preserve single-guard ownership and avoid starting multiple continuous guards. |
| Periodic CPU profiling | Linux and macOS builds can spawn a periodic sampling loop. | `rustfs/src/profiling.rs` | Keep the loop cancellation-driven and non-fatal. |
| Jemalloc memory pprof | Only `linux` + `gnu` + `x86_64` exposes jemalloc pprof dumping. Other supported builds return an unsupported-target error. | `rustfs/src/profiling.rs` | Treat memory pprof as optional and target-gated. |
| Periodic memory pprof | Only runs where jemalloc profiling control is available and active. | `rustfs/src/profiling.rs` | Keep inactive jemalloc as a skipped dump, not a startup failure. |
| Process/system memory sampling | Uses `rustfs_io_metrics::snapshot_process_resource_and_system` plus `sysinfo` total memory. | `rustfs/src/memory_observability.rs` | Keep sampling portable and metric-gated. |
| cgroup memory sampling | Reads Linux cgroup v2 or v1 memory files when present. Missing files produce no cgroup split. | `rustfs/src/memory_observability.rs` | Keep cgroup data opportunistic and absent-safe. |
| Allocator reclaim | Uses jemalloc backend on `linux` + `gnu` + `x86_64`; otherwise mimalloc variants. | `rustfs/src/allocator_reclaim.rs` | Keep backend detection read-only and preserve effective-force behavior. |
| eBPF | No runtime eBPF sidecar is currently wired into startup. | N/A | Treat eBPF as future optional Linux-only inventory, never as a required baseline. |
| NUMA | No NUMA placement or topology controller is currently wired into startup. | N/A | Treat NUMA as future optional capability with no-op fallback. |
## Cross-Platform Baseline
The current safe baseline is:
- Profiling is opt-in through env flags and must not make startup fatal.
- Startup and shutdown call profiling through `startup_profiling` lifecycle
hooks; `profiling.rs` remains the CPU/memory profiling implementation and
admin dump API owner.
- Unsupported profiling targets return structured unsupported errors or skip
startup tasks.
- Memory observability records process/system metrics and adds cgroup split
only when cgroup files exist.
- Allocator reclaim observes active HTTP, delete-tail, scanner, heal, erasure,
and GET-buffer activity before reclaiming.
- Runtime thread sizing remains owned by the Tokio runtime builder and sysinfo
core detection, not NUMA topology.
## Optional Sidecar Invariants
Future sidecars for profiling, eBPF, or NUMA must preserve these invariants:
- Sidecars must be disabled by default or target-gated until explicitly enabled.
- Unsupported targets must degrade to no-op status, not panic or fail startup.
- Sidecars must use the runtime cancellation token or an equivalent explicit
shutdown handle.
- Sidecars must not mutate Tokio worker counts after runtime creation.
- Profiling output directory fallback must stay local to profiling and must not
affect object storage paths.
- NUMA fallback must preserve current runtime thread defaults, storage set
placement, and request admission behavior.
## First Implementation Candidates
`API-013`:
- Define a read-only capability contract for profiling, cgroup memory, eBPF,
allocator backend, and NUMA availability.
- Keep the contract in a low-dependency crate and report unsupported states
explicitly.
`R-016`:
- Wire storage runtime startup to consume capability snapshots read-only.
- Do not start sidecars or mutate runtime worker ownership in the same PR.
`X-012`:
- Define the `ops.profiler.v1` extension schema for profiling capability
reporting, backend status, redaction requirements, and provenance.
- Keep the schema capability-only; it must not request profiler execution,
start sidecars, or change profile export behavior.
- Keep unsupported targets, disabled sidecars, and unknown future backends
representable as no-op capability states.
`X-013`:
- Add the extension capability snapshot contract for disabled, unsupported, and
enabled profiler backends.
- Verify optional profiler sidecar and Wasm runtimes stay disabled by default
and cannot declare a startup fatal boundary.
`R-021`:
- If a runtime service sidecar is added later, enter it through the optional
runtime boundary with explicit shutdown ownership.
- Preserve current service order, KMS/audit/notification fatal boundaries, and
scanner/heal startup semantics.
`R-022`:
- Keep optional runtime startup handoff in `startup_optional_runtimes` while
leaving concrete protocol adapters in `startup_protocols`.
- Preserve KMS-before-protocol startup ordering and disabled protocol no-op
behavior.
docs/superpowers/plans/scheduler-baseline.md
+81
View File
@@ -0,0 +1,81 @@
> **Archived migration snapshot** — moved from `docs/architecture/` (2026-07)
> when the architecture-review ledger it fed closed out. Kept for history; not
> maintained.
# Scheduler Baseline Inventory
This inventory covers `G-011` for `rustfs/backlog#675`. It is a docs-only
snapshot of the current scheduling, backpressure, worker, scanner, heal, and
runtime-builder ownership. It does not define new behavior.
## Current Owners
| Surface | Current owner | Current responsibility | Migration boundary |
|---|---|---|---|
| `ConcurrencyManager` | `rustfs/src/storage/concurrency/manager.rs` | Owns the RustFS S3 read-path disk-read semaphore, I/O metrics, priority queue, storage media detection, access-pattern detection, and buffer strategy. | Keep request admission and I/O metrics behavior stable until a controller can consume the same state explicitly. |
| I/O scheduler core | `crates/io-core/src/scheduler.rs` | Owns the reusable buffer-size and priority algorithms consumed by the RustFS S3 read path (`rustfs/src/storage/concurrency/io_schedule.rs`). The former `SchedulerManager` facade in `rustfs-concurrency` was removed as zero-caller dead code (backlog#1025). | Treat `rustfs-io-core` as the reusable algorithm surface; the RustFS S3 read path owns its own scheduling wiring. |
| RustFS backpressure monitor | `rustfs/src/storage/backpressure.rs` | Tracks object-pipe watermark state used by RustFS storage backpressure tests and helpers, using the shared `PipeBackpressurePolicy` from `crates/concurrency/src/backpressure.rs`. The former `BackpressureManager`/`BackpressurePipe` facade in `rustfs-concurrency` was removed as zero-caller dead code (backlog#1025). | Preserve current state labels and watermark semantics; keep pipe sizing and watermark policy separate from object-read disk semaphore admission. |
| `Workers` | `crates/concurrency/src/workers.rs` | Provides cooperative worker-slot admission with `take`, `give`, and `wait`; current background workflows use it for bounded set workers. | Preserve blocking/wakeup semantics and over-release clamping. |
| Scanner cycle budget | `crates/scanner/src/scanner_budget.rs` | Cancels a child token when runtime, object-count, or directory-count budget is reached. | Preserve partial-cycle reason mapping and checkpoint accounting. |
| Heal admission | `crates/heal/src/heal/manager.rs`, `crates/heal/src/heal/channel.rs`, `rustfs_common::heal_channel` | Owns priority queue admission, duplicate merge/drop/full results, active-task tracking, retry admission, and channel responses. | Preserve low-priority scanner behavior and high-priority escalation gates. |
| Tokio runtime builder | `rustfs/src/server/runtime.rs` | Builds the multi-thread runtime from env/defaults, sets thread counts, stack, queue/event intervals, I/O event cap, thread name, and optional dial9 tracing. | Keep runtime defaults and env names stable when later startup phases move ownership. |
## Current Flow
```mermaid
flowchart TD
http["HTTP/S3 request"] --> app["app object usecase"]
app --> guard["ConcurrencyManager::track_request"]
app --> permit["disk-read semaphore permit"]
permit --> strategy["I/O queue status and buffer strategy"]
strategy --> ecstore["ECStore object/read path"]
ecstore --> setdisks["hashed set disks"]
scanner["scanner cycle"] --> budget["ScannerCycleBudget"]
budget --> folder["folder/object scan"]
folder --> healreq["heal channel request"]
healreq --> admission["HealManager admission queue"]
admission --> healworker["heal workers and retries"]
startup["startup entrypoint"] --> runtime["Tokio runtime builder"]
runtime --> services["background services"]
```
## Missing State For Later Work
`R-015` storage foundation:
- Needs a stable inventory of endpoint publication, local disk prewarm, lock
client setup, and per-set readiness state before any scheduler/controller
consumes storage topology.
- Must not infer set availability only from request-path I/O metrics.
`E-011` extension/runtime consumers:
- Need explicit ownership for runtime admission snapshots before extensions can
observe scheduler or backpressure state.
- Must not receive mutable handles to `ConcurrencyManager`, heal queues, or
scanner budget tokens.
`C-011` controller work:
- Needs desired/current/status snapshots for request admission, scanner budget,
and heal queue pressure before any controller can reconcile them.
- Must keep worker mutation explicit. Read-only status should report `None` or
no-op mutation until a reviewed worker lifecycle PR exists.
## Preservation Invariants
- Request reads must keep the same disk-read semaphore admission and active GET
accounting.
- I/O queue status and congestion metrics must remain derived from the same
permit counts.
- Scanner budget cancellation must keep its reason as runtime, objects, or
directories.
- Scanner inline-heal compatibility must continue to use asynchronous heal
admission.
- Heal duplicate admission must prefer merge semantics before full-queue
rejection.
- High-priority heal admission must still be able to displace lower-priority
queued work where the current manager allows it.
- Tokio runtime env names and fallback defaults must remain unchanged.
docs/superpowers/plans/startup-timeline.md
+90
View File
@@ -0,0 +1,90 @@
> **Archived migration snapshot** — moved from `docs/architecture/` (2026-07)
> when the architecture-review ledger it fed closed out. Kept for history; not
> maintained.
# Startup Timeline Baseline
This document records the current binary startup order before runtime/lifecycle
migration work. It is a behavior-preservation baseline only; it does not define
new startup semantics.
## Scope
- Baseline commit: `ae9d25879d72bc8977f08e61062c022e2142483b`
- Entry points covered: `rustfs/src/main.rs::main`,
`rustfs/src/startup_entrypoint.rs::{run_process, async_main, run}`, and
startup lifecycle helpers
- Related migration task: `G-007`
- Out of scope for this baseline: embedded startup, admin route-action matrix,
and any runtime/lifecycle code movement
## Startup Stages
| Step | Source | Current action | Side effects | Fatal boundary | Ready stage |
|---|---|---|---|---|---|
| `BOOT-001` | `rustfs/src/startup_entrypoint.rs` | Apply external-prefix environment compatibility during async startup before command parsing. | Copies supported external env aliases into canonical `RUSTFS_*` process env keys and prints warnings or info to stderr. | Non-fatal; failure is logged to stderr and startup continues. | None |
| `BOOT-002` | `rustfs/src/main.rs` and `rustfs/src/startup_entrypoint.rs` | Call the startup entrypoint and build the Tokio runtime. | Installs runtime configuration and any runtime telemetry guard created by the runtime builder. | Fatal through `expect`; process exits if the runtime cannot be built. | None |
| `BOOT-003` | `rustfs/src/startup_entrypoint.rs` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None |
| `BOOT-004` | `rustfs/src/startup_preflight.rs` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None |
| `BOOT-005` | `rustfs/src/startup_preflight.rs` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None |
| `BOOT-006` | `rustfs/src/startup_runtime.rs`, `rustfs/src/startup_runtime_hooks.rs`, `rustfs/src/startup_tls_material.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None |
| `RUN-001` | `rustfs/src/startup_server.rs` | Enter startup run orchestration and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready |
| `RUN-002` | `rustfs/src/startup_server.rs` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None |
| `RUN-003` | `rustfs/src/startup_server.rs` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None |
| `RUN-004` | `rustfs/src/startup_server.rs` | Initialize global action credentials. | Publishes root/action credentials used by auth paths. | Fatal if global credentials cannot be initialized. | None |
| `RUN-005` | `rustfs/src/startup_server.rs` | Publish server port and address. | Updates global RustFS port and global address. | Non-fatal in this path. | None |
| `RUN-006` | `rustfs/src/startup_storage.rs` | Build endpoint pools and enforce unsupported filesystem policy. | Derives pool/set/disk layout from configured volumes and validates unsupported filesystem policy. | Fatal on endpoint build or unsupported filesystem policy error. | None |
| `RUN-007` | `rustfs/src/startup_storage.rs` | Publish endpoints and erasure type. | Updates global endpoints and erasure type. | Non-fatal in this path. | None |
| `RUN-008` | `rustfs/src/startup_storage.rs` | Initialize local disks, prewarm local disk id map, and initialize lock clients. | Opens local disk state, primes disk id lookup, and creates global lock clients. | Local disk init is fatal; prewarm and lock-client setup are non-fatal in this path. | None |
| `RUN-009` | `rustfs/src/startup_server.rs` | Initialize capacity management and service state manager. | Starts capacity management and moves service state to `Starting`. | Non-fatal in this path. | None |
| `RUN-010` | `rustfs/src/startup_server.rs` | Start S3 HTTP listener and optional console listener before storage is ready. | Starts HTTP servers with readiness gates; console listener starts only when enabled and configured. | Fatal if a configured listener cannot start. | Requests remain gated until full readiness except probe/admin/console/rpc/tonic/table-catalog exempt paths; see [`readiness-matrix.md`](../../architecture/readiness-matrix.md) |
| `RUN-011` | `rustfs/src/startup_storage.rs` | Create cancellation token and initialize `ECStore`. | Creates the runtime cancellation token and storage engine. | Fatal if `ECStore::new` fails. | None |
| `RUN-012` | `rustfs/src/startup_storage.rs` | Initialize ECStore config and global config system. | Initializes ECStore config, attempts server-config migration, then retries global config init up to 15 times. | Migration attempt is non-fatal in this path; global config init becomes fatal after retries. | Marks the `GlobalReadiness` `StorageReady` stage after global config init succeeds; later runtime readiness still rechecks storage, IAM, lock quorum, and gated peer health before `FullReady` |
| `RUN-013` | `rustfs/src/startup_storage.rs` and `rustfs/src/startup_services.rs` | Start replication and KMS systems. | Starts background replication pool, then initializes KMS from startup services. | Replication init is non-fatal in this path; KMS init is fatal on error. | `StorageReady` stage is already marked; dynamic runtime storage readiness is still checked before `FullReady`; KMS compatibility readiness remains feature-gated health behavior |
| `RUN-014` | `rustfs/src/startup_optional_runtime_sidecars.rs` and `rustfs/src/startup_protocols.rs` | Initialize optional protocol servers. | Starts FTP/FTPS/WebDAV/SFTP when feature-enabled and configured, collecting shutdown handles. | Feature-enabled protocol init is fatal on error; disabled protocols are non-fatal. | None |
| `RUN-015` | `rustfs/src/startup_services.rs`, `rustfs/src/startup_audit.rs`, and `rustfs/src/startup_deadlock.rs` | Initialize buffer profiling, event notifier, audit, and deadlock detector. | Starts buffer profile system, event notifier, audit system, and optional deadlock detector. | Audit startup failure is logged and non-fatal; the others are non-fatal in this path. | None |
| `RUN-016` | `rustfs/src/startup_bucket_metadata.rs` | List buckets and run bucket/replication/IAM metadata migrations. | Reads bucket names, migrates bucket metadata, initializes replication resync, migrates IAM config, and initializes bucket metadata system. | Bucket list and replication resync are fatal on error; metadata migration calls are non-fatal in this path. | Storage remains ready; IAM not yet ready |
| `RUN-017` | `rustfs/src/startup_iam.rs` | Bootstrap IAM inline or defer recovery. | Initializes IAM when possible; otherwise starts the deferred IAM recovery path through `startup_iam`. | Fatal only when `bootstrap_or_defer_iam_init` returns an unrecoverable error. | Inline success marks `IamReady`; deferred mode publishes `IamReady` later from the recovery task |
| `RUN-018` | `rustfs/src/startup_auth.rs` | Initialize Keystone and OIDC auth integrations. | Loads Keystone env config and initializes OIDC providers. | Keystone config parse is fatal; Keystone runtime init failure is non-fatal; OIDC init failure is non-fatal. | None |
| `RUN-019` | `rustfs/src/startup_notification.rs` | Add bucket notification config and initialize notification system. | Adds bucket notification configuration and publishes the global notification system. | Notification config add is non-fatal in this path; global notification init is fatal on error. | None |
| `RUN-020` | `rustfs/src/startup_background.rs` | Create AHM cancellation token and initialize heal manager when scanner or heal is enabled. | Creates AHM cancellation token and starts heal manager for heal/scanner workflows. | Heal manager init is fatal when enabled. | None |
| `RUN-021` | `rustfs/src/startup_observability.rs` | Print server info, init update check, allocator reclaim, metrics, memory observability, and auto-tuner. | Starts informational/update/memory/metrics background tasks when enabled. | Non-fatal in this path. | None |
| `RUN-022` | `rustfs/src/startup_lifecycle.rs` and `rustfs/src/startup_iam.rs` | Log successful startup and publish full readiness for inline IAM. | Logs version/address, checks runtime readiness, marks `FullReady`, and sets service state to `Ready` when IAM was ready inline. | Fatal if runtime readiness is not reached within the startup wait. | Marks `FullReady` only for inline IAM here |
| `RUN-023` | `rustfs/src/startup_lifecycle.rs` | Publish global init time and start data scanner when enabled. | Sets global init time and starts scanner after the successful-startup log. | Scanner start is non-fatal in this path. | Full readiness may already be published or may await deferred IAM recovery |
| `RUN-024` | `rustfs/src/startup_lifecycle.rs` | Wait for shutdown signal. | Blocks the main task until a shutdown signal is received. | Non-fatal. | Runtime remains in its current readiness state |
## Deferred IAM Readiness
| Step | Source | Current action | Side effects | Fatal boundary | Ready stage |
|---|---|---|---|---|---|
| `IAM-001` | `rustfs/src/startup_iam.rs:256` | Attempt `init_iam_sys` during bootstrap. | Initializes IAM against the ECStore object layer when possible. | Recoverable failures can enter deferred mode; unrecoverable errors propagate from bootstrap. | None |
| `IAM-002` | `rustfs/src/startup_iam.rs:73` | Spawn IAM recovery loop when bootstrap is deferred. | Retries IAM initialization with exponential backoff until shutdown or success. | Retry failures are logged; the service remains degraded. | None |
| `IAM-003` | `rustfs/src/startup_iam.rs:52` | Finalize IAM recovery after init succeeds. | Initializes `AppContext` if needed, marks `IamReady`, and calls runtime readiness publication. | Finalize failures are retried by the recovery loop. | Marks `IamReady`, then `FullReady` when runtime readiness succeeds |
## Readiness Gate
| Step | Source | Current action | Side effects | Fatal boundary | Ready stage |
|---|---|---|---|---|---|
| `READY-001` | `rustfs/src/server/readiness.rs:130` | Treat exact probe paths and admin/console/rpc/tonic/table-catalog prefixes as readiness-gate bypass paths. | Bypass paths continue to the inner service while the global readiness gate is not ready. | Non-fatal. | Does not change readiness stages |
| `READY-002` | `rustfs/src/server/readiness.rs:171` | Reject non-probe requests while `GlobalReadiness` is not ready. | Returns `503 Service Unavailable`, `Retry-After: 5`, `Content-Type: text/plain; charset=utf-8`, and `Cache-Control: no-store`. | Non-fatal. | Does not change readiness stages |
| `READY-003` | `rustfs/src/server/readiness.rs:202` | Wait for runtime storage, IAM, lock quorum, and gated peer-health readiness before publishing ready state. | Marks `FullReady` and updates `ServiceState` to `Ready` only when a state manager is provided. | Returns an error on timeout; inline startup treats that as fatal, while deferred IAM recovery retries finalization. | Marks `FullReady` |
## Shutdown Order
| Step | Source | Current action | Side effects | Fatal boundary | Ready stage |
|---|---|---|---|---|---|
| `STOP-001` | `rustfs/src/startup_shutdown.rs` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here |
| `STOP-002` | `rustfs/src/startup_shutdown.rs` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change |
| `STOP-003` | `rustfs/src/startup_optional_runtime_sidecars.rs` | Plan optional runtime shutdown and log stopping state for FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown handles. | Non-fatal in this path. | No readiness-stage change |
| `STOP-004` | `rustfs/src/startup_shutdown.rs`, `rustfs/src/startup_runtime_hooks.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change |
| `STOP-005` | `rustfs/src/startup_shutdown.rs` and `rustfs/src/startup_optional_runtime_sidecars.rs` | Stop S3 and console HTTP servers, signal and wait for optional protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here |
## Migration Rules
- Runtime/lifecycle PRs must map each moved startup line back to one of the
`BOOT-*`, `RUN-*`, `IAM-*`, `READY-*`, or `STOP-*` rows.
- A `pure-move` PR must keep the fatal boundary and ready-stage column unchanged.
- Any intentional change to this table is a separate `behavior-change` PR with
focused negative tests.
- Do not use this document to justify changing readiness, IAM recovery, HTTP
listener timing, lock quorum, or shutdown order in a docs-only PR.