diff --git a/docs/architecture/background-controller-contract.md b/docs/architecture/background-controller-contract.md new file mode 100644 index 000000000..bd5062720 --- /dev/null +++ b/docs/architecture/background-controller-contract.md @@ -0,0 +1,138 @@ +# Background Controller Contract + +This document defines `BGC-002` for +[`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660). It turns +the background service inventory into a shared vocabulary for future read-only +status work. It does not add a Rust trait, a scheduler, a service registry, or +any worker start/stop behavior. + +## Scope + +- PR type: `docs-only`. +- Baseline: `upstream/main` at + `f9a5e6d7e67322ac6f626b6f437a5e722fbe22e2`. +- Applies to future controller work for scanner, heal, lifecycle, replication, + dynamic config reload, capacity, metrics, memory observability, allocator + reclaim, and auto-tuning. +- Out of scope: worker creation, worker shutdown, queue resizing, storage + writes, readiness changes, peer signaling changes, scheduler replacement, and + crate splitting. + +## Contract Vocabulary + +| Term | Meaning | BGC-002 boundary | +|---|---|---| +| Desired | Static intent from env, persisted config, module switches, feature flags, bucket config, or admin configuration. | Read only. Do not normalize or mutate config while collecting desired state. | +| Current | Observed local runtime state such as configured, disabled, running, degraded, stopping, or unknown. | Read only. Do not infer state by starting probes that create storage or network side effects. | +| Status | Human-readable and machine-checkable snapshot of runtime counters, worker counts, queue pressure, last successful cycle, last error, cancellation source, and shutdown handle shape. | Side-effect-free. Missing status surfaces must be reported as `unknown`, not guessed. | +| Reconcile | Future comparison between desired, current, and status that can produce a recommendation. | No action in `BGC-002`; future reconcile must not start or stop workers until a tested pilot PR allows it. | +| Side effects | Writes, deletes, queue admission, target activation, external I/O, metrics emission, readiness publication, peer signal, or config reload fanout. | Must be declared before any controller migration touches that service. | + +## State Model + +Future status snapshots should use the narrowest state that the current code can +prove: + +| State | Meaning | Notes | +|---|---|---| +| NotConfigured | No valid desired source exists for this service. | Use when config/module switches/features make the service absent. | +| Disabled | Desired source exists and explicitly disables the service. | Do not use for missing config. | +| Starting | Startup was requested and has not reached steady state. | Only expose when current code has a start boundary. | +| Running | The service is active according to existing runtime state. | Do not use merely because config is enabled. | +| Degraded | The service is active but current status exposes known error, partial, or stalled state. | Do not introduce new failure classification in docs-only work. | +| Stopping | Shutdown was requested and the service has not fully exited. | Only expose where shutdown can be observed. | +| Stopped | The service was started before and is now fully stopped. | Do not confuse with `Disabled` or `NotConfigured`. | +| Unknown | Current code lacks a safe status surface. | Preferred over speculative status. | + +## Lifecycle Boundary + +```mermaid +flowchart LR + D["Desired source"] + C["Current runtime state"] + S["Read-only status snapshot"] + R["Future reconcile recommendation"] + W["Workers and side effects"] + + D --> S + C --> S + S --> R + R -. "future tested pilot only" .-> W +``` + +`BGC-002` stops at the read-only contract. The arrow from reconcile to workers is +intentionally dotted because this PR does not allow any implementation to start, +stop, resize, or reconfigure workers. + +## Service Boundaries + +| Service area | Desired source | Current/status inputs | Side effects to preserve | +|---|---|---|---| +| Data scanner | Scanner env and runtime scanner config. | Admin scanner status, scanner metrics, scanner cancellation token, checkpoint/yield/alert counters. | Data usage cache updates, lifecycle evaluation, replication heal admission, scanner heal admission, alerts, and scanner metrics. | +| Heal/AHM | Heal enablement and scanner-driven heal admission. | Heal manager global channel, active task atomics, queue length atomics, AHM cancellation token. | Heal queue consumption, heal storage writes, and channel close semantics. | +| Lifecycle expiry/transition | Bucket lifecycle config and scanner event source. | Lifecycle worker counts, active tasks, queue send timeouts, transition stats, expiry/transition queues. | Object deletes, transition queueing, stale multipart cleanup, and lifecycle metrics. | +| Replication pool | Bucket/site replication config and resync admin requests. | Global replication stats, worker pool sizes, queue counters, persisted resync state, per-bucket cancel tokens. | Object replication, delete replication, queue resizing by channel close, persisted resync metadata, and admin-triggered cancel paths. | +| Dynamic config reload | Persisted server config, admin config calls, and peer snapshot signals. | Last local reload result, per-subsystem reload errors, peer reload signal result. | Scanner/heal runtime config updates, audit reload, notification reload, peer signaling, and config snapshot fanout. | +| Capacity manager | Local disk inventory and capacity feature state. | Capacity manager cache age, scheduled refresh state, last refresh result, runtime summary loop. | Global capacity cache refresh and runtime summary metrics/logging. | +| Metrics runtime | Observability metrics feature state and collector configuration. | Collector intervals, last collection result, cancellation token state, collector grouping. | Metrics collection and emission only. | +| Memory observability | Observability feature state and memory sampling config. | Sampler loop state, last sample time, last sample error, runtime cancellation token. | Memory metric emission. This is the preferred first BGC-003 status candidate. | +| Allocator reclaim | Allocator reclaim env/config and backend support. | Enabled flag, idle streak, active request gauge, scanner/heal activity gauges, last reclaim result. | Backend-specific allocator reclaim and metrics. | +| Auto-tuner | `RUSTFS_AUTOTUNER_ENABLED` and tuning inputs. | Last tuning attempt, last tuning error, 60-second loop state. | Runtime concurrency tuning. Treat as behavior-sensitive. | + +The following areas stay outside the first controller migrations: + +- deferred IAM recovery, because it can publish readiness; +- optional protocol servers, because they already have protocol shutdown handles; +- ECStore endpoint monitor and disk health monitor, because they are storage- + adjacent and can affect disk state; +- notification and audit runtime coupling, because live streams, replay, target + activation, and reload behavior need dedicated preservation tests. + +## Read-Only Snapshot Requirements + +Any future `BGC-003` status implementation must satisfy all of these: + +- status collection must not start, stop, resize, or wake a worker; +- status collection must not write storage data, object metadata, target state, + queue entries, persisted config, or resync metadata; +- status collection must not publish readiness or peer reload signals; +- missing fields must be represented as `unknown` or omitted with a documented + reason; +- cancellation source and shutdown handle shape must be reported separately from + desired enabled/disabled state; +- scanner, heal, lifecycle, and replication status must not hide their queue and + admission coupling. + +## Future Reconcile Rules + +Future reconcile work is allowed only after a read-only status snapshot exists. +The first reconcile pilot must: + +- choose one low-risk service; +- compare desired/current/status without side effects; +- prove idempotence under repeated calls; +- prove no duplicate workers are created; +- preserve existing shutdown order and cancellation source; +- include rollback guidance that removes the pilot without changing existing + worker behavior. + +Memory observability is the recommended first candidate because it already has a +simple runtime cancellation loop and no storage writes. Scanner, heal, +replication, lifecycle, disk health, deferred IAM recovery, and auto-tuning must +wait for focused preservation tests. + +## Verification Expectations + +For this docs-only contract: + +- architecture migration guard scripts must pass; +- layer dependency and metrics reference guards must pass; +- no Rust source, Cargo metadata, CI workflow, Makefile, or runtime config file + may change. + +For the next implementation PRs: + +- add focused tests before changing behavior; +- do not modify production logic only to make tests pass; +- keep compatibility comments searchable with `RUSTFS_COMPAT_TODO()` + whenever temporary old paths are retained for later deletion. diff --git a/docs/architecture/background-services-inventory.md b/docs/architecture/background-services-inventory.md index b1464c653..26e5432c9 100644 --- a/docs/architecture/background-services-inventory.md +++ b/docs/architecture/background-services-inventory.md @@ -71,6 +71,9 @@ not define a new scheduler, controller framework, or shutdown contract. ## BGC-002 Contract Inputs +These inputs are formalized in +[`background-controller-contract.md`](background-controller-contract.md). + Future controller contract work should start with a read-only shape: - `desired`: enabled/disabled plus static config source. diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index d45861440..cc7be8ba5 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,14 +5,14 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-background-services-inventory` -- Baseline: `upstream/main` at `03eb10b07f5f968c531151ae667dfe218050493d` +- Branch: `overtrue/arch-background-controller-contract` +- Baseline: `upstream/main` at `f9a5e6d7e67322ac6f626b6f437a5e722fbe22e2` - PR type for this branch: `docs-only` - Runtime behavior changes: none. - Rust code changes: none. - CI/script changes: none -- Docs changes: add BGC-001 background service inventory and index it from the - architecture overview. +- Docs changes: add BGC-002 background controller contract vocabulary and index + it from the background service inventory and architecture overview. ## Phase 0 Tasks @@ -124,20 +124,32 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block - Must preserve: no code behavior change and no new controller contract in this PR. - Verification: docs-only architecture checks and diff hygiene. +- [x] `BGC-002` Define minimal controller contract. + - Acceptance: + [`background-controller-contract.md`](background-controller-contract.md) + defines desired/current/status/reconcile vocabulary, status state + semantics, service boundaries, and side-effect rules without starting + workers or changing scheduling. + - Must preserve: no Rust trait, scheduler, service registry, worker + start/stop path, storage write, readiness change, peer signal, or runtime + behavior change. + - Verification: docs-only architecture checks and diff hygiene. ## Next PRs -1. `contract`: define the minimal BackgroundController status vocabulary after - this inventory is reviewed. -2. `test-only`: add focused preservation tests before moving scanner, heal, +1. `test-only`: add focused preservation tests before moving scanner, heal, replication, lifecycle, or disk health workers. +2. `contract`: add a side-effect-free BGC-003 status snapshot for a low-risk + service such as memory observability. +3. `behavior-change`: migrate one low-risk controller behind idempotence and + shutdown preservation tests. ## Pre-Push Review Log | Expert | Status | Notes | |---|---|---| -| Quality/architecture | pass | Single `docs-only` BGC-001 inventory; it records current owners, cancellation, side effects, and follow-up inputs without adding a controller abstraction. | -| Migration preservation | pass | No Rust source, Cargo manifest, workflow, script, or runtime config diff; storage hot path and shutdown behavior are untouched. | +| Quality/architecture | pass | Single `docs-only` BGC-002 contract; it defines vocabulary and boundaries without adding a Rust trait, scheduler, or service registry. | +| Migration preservation | pass | No Rust source, Cargo manifest, workflow, script, runtime config, worker start/stop path, readiness path, or storage hot path diff. | | Testing/verification | pass | Architecture migration rules, layer dependency guard, metrics reference guard, docs diff hygiene, and no-code-diff check passed. | ## Verification Notes @@ -157,10 +169,11 @@ Notes: ## Handoff Notes -- Keep this BGC-001 branch as a focused `docs-only` PR. -- Do not add controller traits, status structs, service registry code, shutdown - wiring, worker tests, or runtime behavior changes in this PR. -- Follow-up BGC-002 may define a minimal read-only controller status vocabulary - after this inventory is reviewed. +- Keep this BGC-002 branch as a focused `docs-only` PR. +- Do not add controller traits, status structs, service registry code, + scheduling, shutdown wiring, worker tests, or runtime behavior changes in this + PR. +- Follow-up BGC-003 should add a side-effect-free status snapshot for one + low-risk service first, preferably memory observability. - Do not add temporary compatibility code without a matching `RUSTFS_COMPAT_TODO()` marker and cleanup-register entry. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 676073375..c59d31149 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -25,6 +25,9 @@ hot-path behavior must not drift during this migration. - [`background-services-inventory.md`](background-services-inventory.md): current scanner, heal, lifecycle, replication, config reload, metrics, and shutdown surface before BackgroundController work. +- [`background-controller-contract.md`](background-controller-contract.md): + desired/current/status/reconcile vocabulary and lifecycle boundaries for + future read-only BackgroundController work. - [`crate-boundaries.md`](crate-boundaries.md): PR types, crate direction, compatibility rules, and migration guardrails. - [`ecstore-config-consumer-inventory.md`](ecstore-config-consumer-inventory.md):