gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-07 04:25:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(knowledge-base): prune stale content and add agent-facing index (#7035)

Browse Source
This commit is contained in:
Zhengchao An
2026-09-02 08:26:59 +08:00
committed by GitHub
parent ceeff52229
commit 0a975f2fe2
99 changed files with 3312 additions and 10590 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/background-controller-contract.md
+37 -169
View File
@@ -1,187 +1,55 @@
# Background Controller Contract
This document defines `BGC-002` for
[`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660). It turns
the background service inventory into a shared vocabulary for future read-only
status work. It does not add a Rust trait, a scheduler, a service registry, or
any worker start/stop behavior.
**Use this when:** you add a status snapshot or reconcile surface for a background service (scanner, heal, lifecycle, replication, config reload, capacity, metrics, memory observability, allocator reclaim, auto-tuner), or you are tempted to fold several of them into a generic controller.
**Source of truth:** the shipped reference surfaces — `MemoryObservabilityReconcilePlan` and `reconcile()` in `rustfs/src/memory_observability.rs`, `AllocatorReclaimControllerSnapshot` and `AllocatorReclaimReconcilePlan` in `rustfs/src/allocator_reclaim.rs`, `MetricsRuntimeReconcilePlan` in `crates/obs/src/metrics/scheduler.rs`. Startup and shutdown ordering is owned by [runtime-lifecycle.md](runtime-lifecycle.md); the plane-level overview is in [storage-control-data-plane.md](storage-control-data-plane.md).
## Scope
There is no `BackgroundController` trait, scheduler, or service registry. Each service exposes its own typed snapshot and reconcile plan; this page fixes the vocabulary and the rules those surfaces follow.
- PR type: `docs-only`.
- Baseline: `upstream/main` at
`f9a5e6d7e67322ac6f626b6f437a5e722fbe22e2`.
- Applies to future controller work for scanner, heal, lifecycle, replication,
dynamic config reload, capacity, metrics, memory observability, allocator
reclaim, and auto-tuning.
- Out of scope: worker creation, worker shutdown, queue resizing, storage
writes, readiness changes, peer signaling changes, scheduler replacement, and
crate splitting.
## Vocabulary
## Contract Vocabulary
| Term | Meaning | BGC-002 boundary |
| Term | Meaning | Boundary |
|---|---|---|
| Desired | Static intent from env, persisted config, module switches, feature flags, bucket config, or admin configuration. | Read only. Do not normalize or mutate config while collecting desired state. |
| Current | Observed local runtime state such as configured, disabled, running, degraded, stopping, or unknown. | Read only. Do not infer state by starting probes that create storage or network side effects. |
| Status | Human-readable and machine-checkable snapshot of runtime counters, worker counts, queue pressure, last successful cycle, last error, cancellation source, and shutdown handle shape. | Side-effect-free. Missing status surfaces must be reported as `unknown`, not guessed. |
| Reconcile | Future comparison between desired, current, and status that can produce a recommendation. | No action in `BGC-002`; future reconcile must not start or stop workers until a tested pilot PR allows it. |
| Side effects | Writes, deletes, queue admission, target activation, external I/O, metrics emission, readiness publication, peer signal, or config reload fanout. | Must be declared before any controller migration touches that service. |
| Desired | Static intent from env, persisted config, module switches, feature flags, bucket config, or admin configuration. | Read only; collecting desired state never normalizes or mutates config. |
| Current | Observed local runtime state: configured, disabled, running, degraded, stopping, or unknown. | Read only; never inferred by probes that create storage or network side effects. |
| Status | Machine-checkable snapshot of counters, worker counts, queue pressure, last cycle, last error, cancellation source, and shutdown-handle shape. | Side-effect-free; a missing surface is reported as `unknown`, never guessed. |
| Reconcile | Comparison of desired, current, and status that yields a plan. | Shipped plans only report; the only worker mutation they may request is `none`. |
| Side effects | Writes, deletes, queue admission, target activation, external I/O, metrics emission, readiness publication, peer signals, config reload fanout. | Declared per service before any controller touches it. |
## State Model
Future status snapshots should use the narrowest state that the current code can
prove:
Snapshots use the narrowest state the code can prove:
| State | Meaning | Notes |
|---|---|---|
| NotConfigured | No valid desired source exists for this service. | Use when config/module switches/features make the service absent. |
| Disabled | Desired source exists and explicitly disables the service. | Do not use for missing config. |
| Starting | Startup was requested and has not reached steady state. | Only expose when current code has a start boundary. |
| Running | The service is active according to existing runtime state. | Do not use merely because config is enabled. |
| Degraded | The service is active but current status exposes known error, partial, or stalled state. | Do not introduce new failure classification in docs-only work. |
| Stopping | Shutdown was requested and the service has not fully exited. | Only expose where shutdown can be observed. |
| Stopped | The service was started before and is now fully stopped. | Do not confuse with `Disabled` or `NotConfigured`. |
| Unknown | Current code lacks a safe status surface. | Preferred over speculative status. |
## Lifecycle Boundary
```mermaid
flowchart LR
D["Desired source"]
C["Current runtime state"]
S["Read-only status snapshot"]
R["Future reconcile recommendation"]
W["Workers and side effects"]
D --> S
C --> S
S --> R
R -. "future tested pilot only" .-> W
```
`BGC-002` stops at the read-only contract. The arrow from reconcile to workers is
intentionally dotted because this PR does not allow any implementation to start,
stop, resize, or reconfigure workers.
## Service Boundaries
| Service area | Desired source | Current/status inputs | Side effects to preserve |
|---|---|---|---|
| Data scanner | Scanner env and runtime scanner config. | Admin scanner status, scanner metrics, scanner cancellation token, checkpoint/yield/alert counters. | Data usage cache updates, lifecycle evaluation, replication heal admission, scanner heal admission, alerts, and scanner metrics. |
| Heal/AHM | Heal enablement and scanner-driven heal admission. | Heal manager global channel, active task atomics, queue length atomics, AHM cancellation token. | Heal queue consumption, heal storage writes, and channel close semantics. |
| Lifecycle expiry/transition | Bucket lifecycle config and scanner event source. | Lifecycle worker counts, active tasks, queue send timeouts, transition stats, expiry/transition queues. | Object deletes, transition queueing, stale multipart cleanup, and lifecycle metrics. |
| Replication pool | Bucket/site replication config and resync admin requests. | Global replication stats, worker pool sizes, queue counters, persisted resync state, per-bucket cancel tokens. | Object replication, delete replication, queue resizing by channel close, persisted resync metadata, and admin-triggered cancel paths. |
| Dynamic config reload | Persisted server config, admin config calls, and peer snapshot signals. | Last local reload result, per-subsystem reload errors, peer reload signal result. | Scanner/heal runtime config updates, audit reload, notification reload, peer signaling, and config snapshot fanout. |
| Capacity manager | Local disk inventory and capacity feature state. | Capacity manager cache age, scheduled refresh state, last refresh result, runtime summary loop. | Global capacity cache refresh and runtime summary metrics/logging. |
| Metrics runtime | Observability metrics feature state and collector configuration. | Collector intervals, last collection result, cancellation token state, collector grouping. | Metrics collection and emission only. |
| Memory observability | Observability feature state and memory sampling config. | Sampler loop state, last sample time, last sample error, runtime cancellation token. | Memory metric emission. This is the preferred first BGC-003 status candidate. |
| Allocator reclaim | Allocator reclaim env/config and backend support. | Enabled flag, idle streak, active request gauge, scanner/heal activity gauges, last reclaim result. | Backend-specific allocator reclaim and metrics. |
| Auto-tuner | `RUSTFS_AUTOTUNER_ENABLED` and tuning inputs. | Last tuning attempt, last tuning error, 60-second loop state. | Runtime concurrency tuning. Treat as behavior-sensitive. |
The following areas stay outside the first controller migrations:
- deferred IAM recovery, because it can publish readiness;
- optional protocol servers, because they already have protocol shutdown handles;
- ECStore endpoint monitor and disk health monitor, because they are storage-
adjacent and can affect disk state;
- notification and audit runtime coupling, because live streams, replay, target
activation, and reload behavior need dedicated preservation tests.
| NotConfigured | No valid desired source exists. | Config, module switches, or features make the service absent. |
| Disabled | A desired source exists and explicitly disables the service. | Not for missing config. |
| Starting | Start requested, steady state not reached. | Only where a start boundary exists. |
| Running | Active according to existing runtime state. | Not merely because config is enabled. |
| Degraded | Active with known error, partial, or stalled status. | No new failure classification is invented for a snapshot. |
| Stopping | Shutdown requested, not fully exited. | Only where shutdown is observable. |
| Stopped | Started earlier, now fully stopped. | Distinct from `Disabled` and `NotConfigured`. |
| Unknown | No safe status surface exists. | Preferred over speculation. |
## Read-Only Snapshot Requirements
Any future `BGC-003` status implementation must satisfy all of these:
- Status collection never starts, stops, resizes, or wakes a worker.
- Status collection never writes storage data, object metadata, target state, queue entries, persisted config, or resync metadata.
- Status collection never publishes readiness or peer reload signals.
- Missing fields are `unknown` or omitted with a documented reason.
- Cancellation source and shutdown-handle shape are reported separately from desired enabled/disabled state.
- Repeated `reconcile` calls over the same snapshot return the same plan.
- Scanner, heal, lifecycle, and replication status must not hide their queue and admission coupling.
- status collection must not start, stop, resize, or wake a worker;
- status collection must not write storage data, object metadata, target state,
queue entries, persisted config, or resync metadata;
- status collection must not publish readiness or peer reload signals;
- missing fields must be represented as `unknown` or omitted with a documented
reason;
- cancellation source and shutdown handle shape must be reported separately from
desired enabled/disabled state;
- scanner, heal, lifecycle, and replication status must not hide their queue and
admission coupling.
## Coupling Notes
## BGC-003 Snapshot Pilot
The services below share state or shutdown contracts and must not be folded into a generic controller without service-specific preservation tests:
The first read-only snapshot is memory observability status. It reports the
service name, whether observability metrics currently enable the sampler, the
configured sampler interval, runtime-token cancellation state, and the absence
of a dedicated shutdown handle.
This snapshot intentionally does not define an admin route, scheduler, service
registry, worker start/stop path, readiness signal, peer signal, storage write,
or metrics emission change.
## BGC-004 Controller Pilot
The first controller pilot is also memory observability. It converts the
existing desired inputs and status snapshot into a typed reconcile plan. The
pilot reports desired state, current state, and worker mutation intent.
The only allowed worker mutation for this pilot is `none`. Repeated reconcile
calls must return the same plan for the same snapshot and must not request a
worker start, stop, resize, wakeup, storage write, readiness signal, peer
signal, or metrics emission.
## BGC-005 Allocator Reclaim Status And Controller Surface
The second low-risk controller/status surface is allocator reclaim. It reports
the service name, desired enablement, configured force flag, backend-specific
effective force, idle interval settings, runtime-token cancellation state, and
the absence of a dedicated shutdown handle.
The only allowed worker mutation for this surface is `none`. Reconcile output is
read-only and must not start, stop, resize, wake, or otherwise drive the
allocator reclaim loop. Existing backend-specific force handling, idle-streak
logic, metrics emission, and runtime-token shutdown behavior remain owned by the
current loop.
## BGC-006 Metrics Runtime Status And Controller Surface
The third low-risk controller/status surface is metrics runtime. It reports the
service name, observability metrics enablement, collector task count, configured
collector intervals, replication bandwidth zero-tombstone cycle count,
runtime-token cancellation state, and the absence of a dedicated shutdown
handle.
The only allowed worker mutation for this surface is `none`. Reconcile output is
read-only and must not start, stop, resize, wake, or otherwise drive metrics
collector tasks. Existing collector grouping, interval parsing, metrics
emission, replication bandwidth tombstone handling, and runtime-token shutdown
behavior remain owned by the current loops.
## Future Reconcile Rules
Future reconcile work is allowed only after a read-only status snapshot exists.
The first reconcile pilot must:
- choose one low-risk service;
- compare desired/current/status without side effects;
- prove idempotence under repeated calls;
- prove no duplicate workers are created;
- preserve existing shutdown order and cancellation source;
- include rollback guidance that removes the pilot without changing existing
worker behavior.
Memory observability is the recommended first candidate because it already has a
simple runtime cancellation loop and no storage writes. Scanner, heal,
replication, lifecycle, disk health, deferred IAM recovery, and auto-tuning must
wait for focused preservation tests.
## Verification Expectations
For this docs-only contract:
- architecture migration guard scripts must pass;
- layer dependency and metrics reference guards must pass;
- no Rust source, Cargo metadata, CI workflow, Makefile, or runtime config file
may change.
For the next implementation PRs:
- add focused tests before changing behavior;
- do not modify production logic only to make tests pass;
- keep compatibility comments searchable with `RUSTFS_COMPAT_TODO(<task-id>)`
whenever temporary old paths are retained for later deletion.
- Scanner implies heal: the loop started by `init_data_scanner` (`rustfs/src/startup_lifecycle.rs`) enqueues heal work, so scanner status must separate scheduler state from work-source accounting.
- Heal/AHM owns its own token: `create_ahm_services_cancel_token` and `init_heal_manager` run in `rustfs/src/startup_background.rs`; `shutdown_ahm_services` runs in `rustfs/src/startup_shutdown.rs`. Heal admission and channel-close semantics stay intact.
- Replication has two shutdown contracts: the pool started by `init_background_replication` (`rustfs/src/startup_storage.rs`) stops workers by closing channels, while resync started by `init_resync` (`rustfs/src/startup_bucket_metadata.rs`) uses cancellation tokens, and admin-triggered resync uses per-bucket tokens.
- Lifecycle expiry, transition, and stale-multipart cleanup are started by `ECStore::init` (`init_background_expiry`, `init_background_stale_multipart_upload_cleanup` in `crates/ecstore/src/store/init.rs`), which binds the runtime token through `bind_background_cancel_token`; the scanner is their event source, so they are not a separate periodic controller.
- Notification and audit share a runtime pattern but not a lifecycle: `init_event_notifier` and `start_audit_system` (`rustfs/src/startup_audit.rs`), `shutdown_event_notifier` and `stop_audit_system` (`rustfs/src/startup_shutdown.rs`). Live event streams stay separate from target-delivery enablement.
- Dynamic config reload is admin-triggered fanout (`apply_dynamic_config_for_subsystem`, `signal_dynamic_config_reload`, `signal_config_snapshot_reload` in `rustfs/src/admin/service/config.rs`), not a loop; per-subsystem validation and error boundaries are preserved.
- Capacity refresh tasks are owned through `CapacityBackgroundTasks` returned by `init_capacity_management_managed` (`rustfs/src/capacity/capacity_integration.rs`, called from `rustfs/src/startup_entrypoint.rs`); scheduled interval defaults and singleflight refresh stay unchanged.
- Storage-adjacent monitors (`monitor_and_connect_endpoints` in `crates/ecstore/src/core/sets.rs`, `enable_health_check` in `crates/ecstore/src/disk/disk_store.rs`) change disk state and stay outside controller work.
- Deferred IAM recovery (`spawn_iam_recovery_task`, `rustfs/src/startup_iam.rs`) publishes readiness; optional protocol servers already own `ShutdownHandle`s; the auto-tuner (`init_auto_tuner` in `rustfs/src/init.rs`) changes runtime concurrency. All three stay outside generic controllers.