diff --git a/docs/architecture/background-services-inventory.md b/docs/architecture/background-services-inventory.md new file mode 100644 index 000000000..b1464c653 --- /dev/null +++ b/docs/architecture/background-services-inventory.md @@ -0,0 +1,86 @@ +# Background Services Inventory + +This document records the current background service surface before +BackgroundController work. It is a behavior-preservation inventory only; it does +not define a new scheduler, controller framework, or shutdown contract. + +## Scope + +- Related migration task: `BGC-001`. +- PR type: `docs-only`. +- Baseline: `upstream/main` at + `03eb10b07f5f968c531151ae667dfe218050493d`. +- Out of scope: changing startup order, shutdown order, readiness, storage + writes, heal admission, scanner scheduling, replication queues, config reload + behavior, metrics intervals, or worker counts. + +## Startup And Shutdown Owners + +| Area | Startup owner | Shutdown owner | Current cancellation source | +|---|---|---|---| +| Main runtime token | `rustfs/src/main.rs::run` creates `ctx` after HTTP listeners start and before ECStore creation. | `rustfs/src/main.rs::handle_shutdown` calls `ctx.cancel()` before service-specific shutdown. | Shared `tokio_util::sync::CancellationToken`. | +| Scanner | `rustfs/src/main.rs::run` calls `init_data_scanner(ctx.clone(), store.clone())` after successful startup log and global init time. | Main shutdown calls `ctx.cancel()`; if scanner was enabled it also calls `shutdown_background_services()`. | Scanner loop receives the main runtime token. | +| Heal/AHM | Main creates `create_ahm_services_cancel_token()` before scanner/heal feature checks and calls `init_heal_manager(...)` when heal or scanner is enabled. | Main shutdown calls `shutdown_ahm_services()` when heal or scanner was enabled. | Global AHM token plus channel/worker-local state. | +| Replication pool | Main calls `init_background_replication(store.clone())` after global config init, then `pool.init_resync(ctx.clone(), buckets.clone())` after bucket listing. | No direct main shutdown call for the replication pool; resync receives the main runtime token. | Resync routine uses the main runtime token; per-bucket resync uses registered cancel tokens. | +| Lifecycle expiry/transition | `ECStore::init` calls `init_background_expiry(self.clone())` and `init_background_stale_multipart_upload_cleanup(self.clone())`. | Expiry workers read `get_background_services_cancel_token()` and fall back to a private token if none exists. Stale multipart cleanup exits when the weak ECStore reference cannot upgrade. | Inventory search found no current startup caller for `create_background_services_cancel_token()`. | +| Notification runtime | Main calls `init_event_notifier()` after buffer profile init. | Main shutdown calls `shutdown_event_notifier().await`. | Notification runtime owns target/replay shutdown internally. | +| Audit runtime | Main calls `start_audit_system().await`. | Main shutdown calls `stop_audit_system().await`. | Audit runtime owns target/replay shutdown internally. | +| Metrics and memory loops | Main calls `init_metrics_runtime(ctx.clone())`, `init_memory_observability(ctx.clone())`, and `init_auto_tuner(ctx.clone())` when observability metrics are enabled. | Main shutdown only cancels the shared runtime token. | Shared runtime token. | +| Allocator reclaim | Main calls `init_allocator_reclaim(ctx.clone())` unconditionally. | Main shutdown cancels the shared runtime token. | Shared runtime token. | +| Capacity manager | Main calls `init_capacity_management().await` before HTTP listener startup and ECStore creation. | No direct main shutdown call. | Current scheduled capacity and metrics loops do not receive a shutdown token. | +| Optional protocol servers | Main calls feature-gated FTP, FTPS, WebDAV, and SFTP init functions. | Main shutdown calls each stored `ShutdownHandle` and waits for all protocol shutdown futures. | Per-protocol broadcast shutdown handles. | +| Deferred IAM recovery | `bootstrap_or_defer_iam_init(...)` may spawn a deferred recovery loop. | Main shutdown cancels the shared runtime token. | Shared runtime token. | + +## Service Inventory + +| Service | Trigger and workers | Side effects | Status and metrics | Migration notes | +|---|---|---|---|---| +| Capacity background refresh | `rustfs/src/capacity/capacity_integration.rs::init_capacity_management` delegates to `init_capacity_management_for_local_disks`, then `crates/object-capacity/src/capacity_manager.rs::start_background_task` spawns a scheduled refresh loop and a runtime summary loop. | Refreshes global capacity cache from local disks and logs runtime summaries. | Uses the object-capacity manager state and log summaries; no explicit shutdown status surface is exposed here. | Add read-only status before any controller migration. A future controller must not change scheduled interval defaults or singleflight refresh behavior. | +| ECStore endpoint monitor | `crates/ecstore/src/sets.rs::new` spawns `monitor_and_connect_endpoints`. | Monitors endpoint connectivity and reconnect behavior for erasure sets. | Logs monitor start, cancellation, and exit. | This is storage-adjacent and must stay outside broad controller movement until storage shutdown semantics are explicitly covered. | +| Local disk health monitor | `crates/ecstore/src/store/init.rs::init` enables disk health checks after store initialization; `crates/ecstore/src/disk/disk_store.rs::enable_health_check` spawns writable and recovery monitors. | Periodically probes disk writability, can create test objects named `health-check-*`, and updates disk runtime health state. | Disk info includes runtime health metrics and waiting counts. | Do not merge this with scanner/heal controller work; probes affect disk health semantics. | +| Data scanner | `crates/scanner/src/scanner.rs::init_data_scanner` configures scanner defaults, applies runtime config, waits the initial scanner delay, then loops `run_data_scanner`. | Updates data usage cache, scans buckets/sets, evaluates lifecycle rules, queues replication heal, queues scanner heal, and emits scanner alerts. | Scanner runtime config/status is exposed through admin scanner status; scanner metrics record ILM, replication admission, heal admission, checkpoints, yields, and alerts. | Scanner implies heal because scanner can enqueue heal requests. Future controller status must separate scheduler state from scanner work-source accounting. | +| Heal/AHM | `crates/heal/src/lib.rs::init_heal_manager` starts `HealManager`, initializes the shared heal channel, and spawns `HealChannelProcessor`. | Consumes heal requests from the global heal channel and drives heal work through the configured heal storage API. | Global active-task and queue-length atomics track current heal pressure. | Keep heal admission and channel semantics intact. Controller work should first expose queue/active status and shutdown state. | +| Bucket replication pool | `crates/ecstore/src/bucket/replication/replication_pool.rs::init_background_replication` creates global replication stats and the global pool; pool resizing spawns regular, large-object, and failed-object workers. | Replicates object and delete operations, updates queue stats, and maintains replication worker pools. | Replication stats expose active worker counts and queue accounting. | Worker resize behavior currently closes channels to stop workers. Do not replace this with a generic controller until queue close semantics are captured by tests. | +| Bucket replication resync | Main calls `get_global_replication_pool().init_resync(ctx.clone(), buckets.clone())`; the pool spawns `start_resync_routine`. Admin site-replication handlers can start or cancel per-bucket resync with dedicated tokens. | Loads persisted resync state, starts bucket resync, persists status, and can cancel per-target resync. | Admin site-replication status surfaces resync state. | Preserve the split between startup resync and admin-triggered resync operations. | +| Lifecycle expiry and transition | `ECStore::init` calls `init_background_expiry(self.clone())`. Scanner evaluates lifecycle events and queues expiry/transition work through `apply_expiry_rule` and `apply_transition_rule`. | Deletes expired objects, queues transitions, updates lifecycle stats, and accounts scanner ILM actions only when work is queued. | Lifecycle state tracks worker counts, active tasks, queue send timeouts, compensation tasks, and transition stats. | This is not a separate periodic controller today; scanner is the main event source for object lifecycle evaluation. | +| Stale multipart cleanup | `ECStore::init` calls `init_background_stale_multipart_upload_cleanup(self.clone())`. | Periodically deletes stale multipart upload data. | Logs cleanup passes when objects are deleted. | Current loop has no explicit cancellation token and exits when ECStore is dropped. Future controller work needs an explicit lifecycle decision before changing it. | +| Notification runtime | `rustfs/src/server/event.rs::init_event_notifier` initializes live event stream support even when notification targets are disabled; when enabled, it loads server config and activates targets. Config reload uses `NotificationConfigManager::reload_config`. | Installs ECStore event dispatch hook, activates notification targets, manages replay/runtime target state, and supports live event streams. | Notification module state is refreshed from persisted module switches; target health is available through runtime target status. | Keep live event stream support separate from target delivery enablement. Reload must remain admin-triggered and peer-signaled. | +| Audit runtime | `rustfs/src/server/audit.rs::start_audit_system` starts audit only when module switches and configured targets allow it. `AuditSystem::reload_config` replaces runtime targets. | Dispatches audit events to configured targets and manages replay workers. | Audit observability records config reloads and target delivery metrics. | Do not couple audit lifecycle to notification lifecycle even though the runtime patterns are similar. | +| Dynamic config reload | Admin config handlers call `apply_dynamic_config_for_subsystem`, then `signal_dynamic_config_reload` or `signal_config_snapshot_reload` through the global notification system. | Applies scanner/heal runtime config, audit reloads, notification reloads, and peer reload signals. | Logs local and peer reload failures. Audit reload increments audit config reload metrics. | This is admin-triggered fanout, not a background scheduler. Controller work should preserve per-subsystem validation and error boundaries. | +| Metrics runtime | `crates/obs/src/metrics/scheduler.rs::init_metrics_runtime` spawns multiple interval loops for cluster, bucket, node, resource, audit, notification, and replication bandwidth metrics. | Periodically collects and reports metrics. | Reports through the metrics runtime and logs cancellation warnings. | Keep intervals and collector grouping stable while adding status snapshots. | +| Memory observability | `rustfs/src/memory_observability.rs::init_memory_observability` spawns a token-cancelled sampler. | Periodically records memory snapshots. | Emits memory observability metrics. | This can be a low-risk pilot for read-only controller status because it already has a simple token loop. | +| Allocator reclaim | `rustfs/src/allocator_reclaim.rs::init_allocator_reclaim` spawns a token-cancelled reclaim loop when enabled. | Observes reclaimable work and may run allocator reclaim after idle intervals. | Emits reclaim enabled/backend counters, active-request gauges, scanner/heal activity gauges, and reclaim result counters. | A controller must preserve idle-streak logic and backend-specific force behavior. | +| Auto-tuner | `rustfs/src/init.rs::init_auto_tuner` optionally spawns a 60-second loop when `RUSTFS_AUTOTUNER_ENABLED` is true. | Tunes concurrency manager settings from performance metrics. | Logs iteration success/failure. | Treat as behavior-sensitive; a future controller needs explicit rollback because it can change runtime concurrency. | +| Update check | `rustfs/src/init.rs::init_update_check` spawns one async task with a 30-second timeout when update checks are enabled. | Performs version check network I/O and logs available updates. | Logs result only. | This is a one-shot task, not a controller candidate for the first BGC PRs. | +| Deferred IAM recovery | `rustfs/src/startup_iam.rs::spawn_iam_recovery_task` retries IAM init with backoff and finalizes readiness when successful. | Can initialize IAM later, initialize AppContext if needed, mark `IamReady`, and publish `FullReady`. | Readiness state reflects deferred recovery progress. | Keep this lifecycle-critical path separate from generic background controllers. | +| Optional protocol servers | `rustfs/src/init.rs` starts FTP, FTPS, WebDAV, and SFTP with per-protocol `ShutdownHandle`s when features and config enable them. | Serve protocol traffic in background tasks. | Shutdown logs per protocol. | Protocol servers already have explicit handles; do not fold them into BGC until the service registry owns shutdown ordering. | + +## Current Gaps To Preserve Before Controller Work + +- ECStore background-service cancellation has a public global token API, but this + inventory found no current startup call to create that token. Lifecycle expiry + workers therefore use their fallback token when no global token exists. +- Capacity manager loops do not receive the main runtime cancellation token. +- Replication worker pools stop some workers by closing channels, while resync + uses cancellation tokens. These are different shutdown contracts. +- Scanner, lifecycle, replication, and heal are coupled by work queues and + metrics. Moving one without status snapshots for the others risks hiding work + admission failures. +- Dynamic config reload is admin-triggered and peer-signaled, not a periodic + background loop. + +## BGC-002 Contract Inputs + +Future controller contract work should start with a read-only shape: + +- `desired`: enabled/disabled plus static config source. +- `current`: started, stopped, running, degraded, or disabled. +- `status`: worker counts, queue lengths, last cycle/reload time, and last error. +- `shutdown`: cancellation source and whether the service has an explicit stop + handle. +- `side_effects`: storage writes, target activation, external I/O, metrics, and + readiness changes. + +The first pilot should use a service with an existing simple cancellation loop +and no storage writes, such as memory observability. Scanner, heal, replication, +lifecycle, and disk health must wait for focused preservation tests. diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index 239df75cc..d45861440 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,20 +5,14 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-kms-redaction` -- Baseline: `upstream/main` at `0cdcd1eb7bfd5fc229eb45f851c624084b072365` -- PR type for this branch: `security-change` -- Runtime behavior changes: KMS secret-bearing `Debug` output and admin status - summary views no longer expose local master keys, Vault tokens, or AppRole - secret IDs. KMS backend behavior, authorization, production defaults, and - config persistence are unchanged. -- Rust code changes: add KMS redaction rules, safe `Debug` implementations for - secret-bearing KMS config and configure request types, and focused tests that - prove secrets are absent from debug/admin views while serde persistence keeps - the original values. +- Branch: `overtrue/arch-background-services-inventory` +- Baseline: `upstream/main` at `03eb10b07f5f968c531151ae667dfe218050493d` +- PR type for this branch: `docs-only` +- Runtime behavior changes: none. +- Rust code changes: none. - CI/script changes: none -- Docs changes: record S-013 redaction status and the no-behavior-drift - migration boundary. +- Docs changes: add BGC-001 background service inventory and index it from the + architecture overview. ## Phase 0 Tasks @@ -120,55 +114,53 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block - Verification: focused KMS redaction/status tests, full KMS tests, migration guards, Rust quality scan, clippy, and `make pre-commit` passed. +## Phase 8 Background Controller Tasks + +- [x] `BGC-001` Inventory background services. + - Acceptance: + [`background-services-inventory.md`](background-services-inventory.md) + records scanner, heal, lifecycle, replication, config reload, metrics, + shutdown, cancellation, and side-effect surfaces before controller work. + - Must preserve: no code behavior change and no new controller contract in + this PR. + - Verification: docs-only architecture checks and diff hygiene. + ## Next PRs -1. `security-change`: inventory KMS development defaults before any production - default hardening. -2. `security-change`: apply IAM and plugin redaction in a separate S-014 PR. +1. `contract`: define the minimal BackgroundController status vocabulary after + this inventory is reviewed. +2. `test-only`: add focused preservation tests before moving scanner, heal, + replication, lifecycle, or disk health workers. ## Pre-Push Review Log | Expert | Status | Notes | |---|---|---| -| Quality/architecture | pass | Single `security-change` PR; redaction rules use the security-governance crate, custom `Debug` stays local to secret-bearing KMS types, and no startup/storage/global-state path is touched. | -| Migration preservation | pass | Runtime secret access and persisted config serialization are explicitly preserved by tests; no temporary compatibility path is introduced. | -| Testing/verification | pass | Focused redaction/status tests, full KMS tests, admin KMS handler tests, governance tests, clippy, migration guards, Rust quality scan, nextest, doctests, and `make pre-commit` passed. | +| Quality/architecture | pass | Single `docs-only` BGC-001 inventory; it records current owners, cancellation, side effects, and follow-up inputs without adding a controller abstraction. | +| Migration preservation | pass | No Rust source, Cargo manifest, workflow, script, or runtime config diff; storage hot path and shutdown behavior are untouched. | +| Testing/verification | pass | Architecture migration rules, layer dependency guard, metrics reference guard, docs diff hygiene, and no-code-diff check passed. | ## Verification Notes Passed: -- `cargo test -p rustfs-kms redaction -- --nocapture` -- `cargo test -p rustfs-kms status_response -- --nocapture` -- `cargo test -p rustfs-kms --no-fail-fast` -- `cargo test -p rustfs admin::handlers::kms --no-fail-fast` -- `cargo test -p rustfs-security-governance --no-fail-fast` -- `cargo clippy -p rustfs-kms --all-targets --all-features -- -D warnings` -- Rust code quality scan on changed KMS source files -- `cargo fmt --all --check` -- `./scripts/check_layer_dependencies.sh` - `./scripts/check_architecture_migration_rules.sh` +- `./scripts/check_layer_dependencies.sh` - `./scripts/check_metrics_migration_refs.sh` - `git diff --check` -- `make pre-commit` +- `git diff --name-only -- '*.rs' 'Cargo.toml' 'Cargo.lock' '.github/**' 'Makefile' 'Justfile'` Notes: -- This branch changes only KMS redaction for debug/admin view surfaces. It does - not change KMS authorization, production defaults, startup order, global - state, storage paths, route registration, or crate boundaries. -- Config serialization still preserves secret values for persisted cluster - config; this is tested explicitly to avoid runtime data loss. -- `make pre-commit` passed all checks, including 5691 nextest tests, 111 - skipped tests, and workspace doctests. +- This branch changes architecture documentation only. +- No Rust source, Cargo manifest, workflow, script, or runtime configuration is + changed. +- `make pre-commit` is intentionally not required for this docs-only PR. ## Handoff Notes -- Keep this S-013 branch as a focused `security-change` PR. Do not change KMS - defaults, admin authorization, admin route registration shape, Config moves, - Storage API moves, Runtime moves, or ECStore moves. -- `rustfs` may depend on `rustfs-security-governance` for contract metadata; - the security-governance crate must stay independent from implementation - crates and runtime state. +- Keep this BGC-001 branch as a focused `docs-only` PR. +- Do not add controller traits, status structs, service registry code, shutdown + wiring, worker tests, or runtime behavior changes in this PR. +- Follow-up BGC-002 may define a minimal read-only controller status vocabulary + after this inventory is reviewed. - Do not add temporary compatibility code without a matching `RUSTFS_COMPAT_TODO()` marker and cleanup-register entry. -- KMS production default hardening remains a separate task group; do not bundle - it with this redaction PR. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 8402cefc9..676073375 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -22,6 +22,9 @@ hot-path behavior must not drift during this migration. startup/readiness, and shutdown contracts. - [`storage-control-data-plane.md`](storage-control-data-plane.md): boundaries between StorageCore, ECStore, ClusterControlPlane, and BackgroundControllers. +- [`background-services-inventory.md`](background-services-inventory.md): current + scanner, heal, lifecycle, replication, config reload, metrics, and shutdown + surface before BackgroundController work. - [`crate-boundaries.md`](crate-boundaries.md): PR types, crate direction, compatibility rules, and migration guardrails. - [`ecstore-config-consumer-inventory.md`](ecstore-config-consumer-inventory.md):