From 3c71d5ef1c63bec63780d48caa2a9fe880ea0c64 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=AE=89=E6=AD=A3=E8=B6=85?= Date: Sun, 7 Jun 2026 21:44:55 +0800 Subject: [PATCH] docs: inventory ecstore config consumers (#3259) --- .../ecstore-config-consumer-inventory.md | 195 ++++++++++++++++++ docs/architecture/migration-progress.md | 46 +++-- docs/architecture/overview.md | 5 +- 3 files changed, 229 insertions(+), 17 deletions(-) create mode 100644 docs/architecture/ecstore-config-consumer-inventory.md diff --git a/docs/architecture/ecstore-config-consumer-inventory.md b/docs/architecture/ecstore-config-consumer-inventory.md new file mode 100644 index 000000000..aa7485593 --- /dev/null +++ b/docs/architecture/ecstore-config-consumer-inventory.md @@ -0,0 +1,195 @@ +# ECStore Config Consumer Inventory + +This inventory is the Phase 0 baseline for moving +`rustfs_ecstore::config::{Config, KV, KVS}` safely. It records the current +definitions, persistence helpers, global accessors, and direct consumers before +any contract extraction, global-state migration, or crate split. + +Related issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) + +## Scope + +In scope: + +- `rustfs_ecstore::config::KV` +- `rustfs_ecstore::config::KVS` +- `rustfs_ecstore::config::Config` +- `rustfs_ecstore::config::DEFAULT_KVS` +- `rustfs_ecstore::config::{get_global_server_config, set_global_server_config}` +- `rustfs_ecstore::config::com::{read_config_without_migrate, save_server_config}` +- Consumers that persist, clone, inspect, mutate, or pass these types across + runtime boundaries. +- Adjacent users of `rustfs_ecstore::config::com::{read_config, save_config, + delete_config}` are listed separately because moving `com.rs` would affect + them even when they do not consume `Config`, `KV`, or `KVS`. + +Out of scope: + +- Unrelated `Config` types from `rustfs::config`, SDKs, TLS, SSH, KMS, OIDC + client libraries, or local module-specific config structs. +- Storage-class-only imports are not treated as `Config`, `KV`, or `KVS` + consumers unless they also use the server-config model. +- Pure route/action snapshot work already covered by + [`admin-route-action-snapshot.md`](admin-route-action-snapshot.md). + +## Current Shape + +Arrows show current source dependency or call direction: the left node imports +or calls the right node. + +```mermaid +flowchart TB + EC["crates/ecstore/src/config"] + Store["crates/ecstore/src/store.rs"] + AppCtx["rustfs/src/app/context.rs"] + Server["rustfs/src/server/{event,audit}.rs"] + Admin["rustfs/src/admin"] + Notify["crates/notify"] + Audit["crates/audit"] + Targets["crates/targets"] + IAM["crates/iam/src/oidc.rs"] + Scanner["crates/scanner/src/{runtime_config,scanner}.rs"] + + Store --> EC + AppCtx --> EC + Server --> AppCtx + Admin --> EC + Notify --> EC + Audit --> EC + Targets --> EC + Notify --> Targets + Audit --> Targets + IAM --> EC + Scanner --> EC +``` + +The config model is currently both a persisted server-config representation and +the runtime carrier for notify, audit, target-plugin, scanner, and OIDC +settings. Any move must preserve that dual role until consumers are migrated +behind narrower contracts. + +## Core Model And Global State + +| Item | Current owner | Current role | Migration note | +|---|---|---|---| +| `KV` | `crates/ecstore/src/config/mod.rs` | Key/value entry with `hidden_if_empty` metadata and serde compatibility. | Preserve field names, aliases, defaults, and redaction behavior before any model move. | +| `KVS(Vec)` | `crates/ecstore/src/config/mod.rs` | Ordered key/value set used by server config, target factories, admin rendering, tests, and examples. | Preserve tuple shape and methods: `new`, `get`, `lookup`, `is_empty`, `keys`, `insert`, `extend`. | +| `Config(HashMap>)` | `crates/ecstore/src/config/mod.rs` | Server config map by subsystem and target. | Direct `.0` access is widespread; add wrappers only after preserving the current public shape. | +| `DEFAULT_KVS` | `crates/ecstore/src/config/mod.rs` | Registry for defaults across storage class, scanner, notify, audit, and OIDC. | Move defaults only after an explicit registration contract exists. | +| `GLOBAL_SERVER_CONFIG` | `crates/ecstore/src/config/mod.rs` | Process-wide mutable server config snapshot. | Migrate readers behind `AppContext` or a server-config provider before changing storage. | +| `ConfigSys::init` | `crates/ecstore/src/config/mod.rs` | Reads persisted config, looks up derived config, and stores the global snapshot. | Startup order must remain unchanged until the lifecycle contract owns this dependency. | +| `read_config_without_migrate` | `crates/ecstore/src/config/com.rs` | Loads persisted server config through `StorageAPI`. | Persistence stays in `ecstore` until pure model and persistence are separated. | +| `save_server_config` | `crates/ecstore/src/config/com.rs` | Persists the canonical server config object. | Preserve external object shape and config-history behavior. | +| `get_global_server_config` / `set_global_server_config` | `crates/ecstore/src/config/mod.rs` | Clone/read and replace the global server-config snapshot. | Do not remove until all runtime readers have an injected provider path. | + +## Consumer Map + +### ECStore Ownership, Persistence, And Defaults + +| Files | Current usage | +|---|---| +| `crates/ecstore/src/config/mod.rs` | Defines `KV`, `KVS`, `Config`, defaults, global snapshot, initialization, and tests. | +| `crates/ecstore/src/config/com.rs` | Encodes, decodes, reads, writes, creates, and normalizes server config objects through `StorageAPI`. | +| `crates/ecstore/src/config/{notify,audit,oidc,scanner,storageclass}.rs` | Register default `KVS` values and subsystem-specific parsing helpers. | +| `crates/ecstore/src/store.rs` | Exposes store-level server-config accessors that delegate to the global config snapshot. | + +### App Context And Server Startup Consumers + +| Files | Current usage | +|---|---| +| `rustfs/src/app/context.rs` | Defines `ServerConfigInterface`, keeps an `AppContext` server-config handle, and still falls back to `get_global_server_config`. | +| `rustfs/src/server/event.rs` | Resolves server config through app context/global fallback before starting the notification runtime. | +| `rustfs/src/server/audit.rs` | Resolves server config through app context/global fallback before starting the audit runtime. | + +### Admin Control-Plane Readers And Writers + +| Files | Current usage | +|---|---| +| `rustfs/src/admin/handlers/config_admin.rs` | Reads active/persisted server config, validates against `DEFAULT_KVS`, mutates `KVS`, saves config history, saves server config, and updates the global snapshot. | +| `rustfs/src/admin/handlers/oidc.rs` | Reads and writes OIDC provider `KVS`, saves server config, and compares persisted config against the global snapshot for restart signaling. | +| `rustfs/src/admin/handlers/audit_runtime_config.rs` | Reads persisted config, applies audit runtime target changes, saves server config, and reloads audit runtime state. | +| `rustfs/src/admin/handlers/notify_runtime_access.rs` | Reads notification runtime config snapshots and passes `KVS` target changes into the notification system. | +| `rustfs/src/admin/handlers/{event,audit}.rs` | Lists and validates notification/audit targets from `Config`; tests build `KV` and `KVS` fixtures. | +| `rustfs/src/admin/handlers/plugins_instances.rs` | Maps target plugin `KVS` to response payloads and applies runtime target edits. | +| `rustfs/src/admin/handlers/target_descriptor.rs` | Converts descriptor payloads into `KVS` for target plugin instances. | +| `rustfs/src/admin/handlers/site_replication.rs` | Reads global server config for LDAP settings and parses LDAP `KVS` fixtures. | +| `rustfs/src/admin/service/config.rs` | Reads persisted server config, validates storage-class `KVS`, derives target state, and updates global config/storage-class state. | +| `rustfs/src/admin/router.rs` | Reads persisted/global server config for admin route behavior; route tests construct `Config`, `KV`, and `KVS`. | + +### Adjacent ECStore Config-Object Helper Users + +| Files | Current usage | +|---|---| +| `rustfs/src/admin/handlers/kms_dynamic.rs` | Uses generic `read_config` and `save_config` for dynamic KMS config objects. | +| `rustfs/src/admin/handlers/site_replication.rs` | Uses generic `read_config`, `save_config`, and `delete_config` for site-replication state objects. | +| `rustfs/src/admin/service/site_replication.rs` | Uses generic `read_config` and `save_config` for site-replication state normalization. | +| `crates/scanner/src/{scanner,data_usage_define}.rs` | Uses generic `read_config` and `save_config` for scanner metadata and cache persistence paths. | + +### Runtime Target, Notify, And Audit Crates + +| Files | Current usage | +|---|---| +| `crates/notify/src/{global,integration,services,registry}.rs` | Carries `Config` into notification runtime startup/reload and target creation. | +| `crates/notify/src/config_manager.rs` | Mutates `Config`, reads persisted server config with `read_config_without_migrate`, persists changes with `save_server_config`, and applies per-target `KVS` updates. | +| `crates/notify/src/factory.rs` | Builds notification target arguments from `KVS`. | +| `crates/notify/examples/{full_demo,full_demo_one}.rs` | Constructs `Config`, `KV`, and `KVS` directly for examples. | +| `crates/audit/src/{global,system,registry}.rs` | Carries `Config` into audit runtime startup/reload and target creation. | +| `crates/audit/src/factory.rs` | Builds audit target arguments from `KVS`. | +| `crates/audit/tests/*.rs` | Constructs `Config` and `KVS` directly for runtime and parsing tests. | +| `crates/audit/README.md` | Documents current direct `Config` usage. | +| `crates/targets/src/plugin.rs` | Creates plugin targets from `Config` and merged `KVS`. | +| `crates/targets/src/catalog/builtin.rs` | Declares builtin target descriptors and default `KVS` fields. | +| `crates/targets/src/config/{common,target_args,loader,instance}.rs` | Collects, normalizes, redacts, and materializes target configs from `Config` and `KVS`, including environment overrides. | + +### Identity, Scanner, Tests, And Fixtures + +| Files | Current usage | +|---|---| +| `crates/iam/src/oidc.rs` | Reads global server config and parses OIDC provider `KVS`. | +| `crates/scanner/src/{runtime_config,scanner}.rs` | Reads the global server-config snapshot and resolves scanner runtime config from `Config` and `KVS`. | +| `rustfs/src/admin` handler/router tests, `crates/audit/tests/*.rs`, and selected in-crate tests in `crates/{targets,scanner}/src` | Build direct tuple-struct fixtures; use them as candidate regression guards during a pure model move. | + +## Dependency Risk Classification + +| Risk | Why it matters | Guardrail | +|---|---|---| +| `Config` is both persistence model and runtime input | A move can accidentally change persisted JSON/object shape or runtime target behavior. | Separate pure model contract from persistence helpers before moving `com.rs`. | +| Direct `.0` map access is common | Replacing the tuple struct too early would create broad churn and likely behavior drift. | Preserve tuple shape in the first move, then add typed readers in later PRs. | +| `KVS` is the effective target config carrier | Notify, audit, and target factories consume `KVS` after file/env merge. | Keep `KVS` API stable until target descriptor and runtime crates are behind a shared contract. | +| `DEFAULT_KVS` registration is global | Defaults are initialized centrally and used by admin validation/rendering. | Add a registration contract before changing initialization order. | +| Global snapshot readers still exist | Server, admin, IAM, scanner, and site-replication paths can still read global config. | Migrate readers through `AppContext`/provider paths in small steps after the model contract is stable. | +| Persistence helpers depend on `StorageAPI` | Moving them with the pure model would pull storage implementation dependencies upward. | Keep read/write helpers in `ecstore` until a storage-facing persistence contract is explicit. | + +## Recommended Migration Order + +1. Keep this inventory current while Phase 0 guardrails land. +2. Add a focused contract surface for `KV`, `KVS`, and `Config` without changing + serialization, tuple-struct shape, or method names. +3. Add compile-time or scripted checks for temporary compatibility markers and + config-model re-export coverage. +4. Move only the pure model and defaults registration surface after targeted + regression checks cover unchanged persisted object shape, target `KVS` merge + behavior, and representative admin config rendering paths. +5. Migrate global `Config` readers behind `ServerConfigInterface` or a narrower + provider in small PRs. +6. Move persistence helpers only after `StorageAPI` dependencies can stay below + the model contract. +7. Evaluate crate split only after consumers no longer need old paths except + explicit `RUSTFS_COMPAT_TODO()` compatibility shims. + +## Do-Not-Change Contract + +The first migration steps must preserve: + +- `KV { key, value, hidden_if_empty }` serde behavior and redaction semantics. +- `KVS(Vec)` tuple shape and public methods. +- `Config(HashMap>)` tuple shape and public methods. +- `Config::set_defaults`, `Config::unmarshal`, `Config::marshal`, and + `Config::merge` behavior. +- `read_config_without_migrate` fallback/creation behavior for missing server + config objects. +- `save_server_config` external object shape and config-history compatibility. +- Existing notify, audit, scanner, OIDC, and target-plugin enable/disable + interpretation from `Config`/`KVS` inputs; business-rule changes stay out of + migration PRs. +- AppContext/global fallback behavior until all readers are explicitly migrated. diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index 1d88b080a..222786821 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,13 +5,13 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-admin-route-snapshot` -- Baseline: `upstream/main` at `0f9584c8d9351c757437405ee69a4e64bbcd94b5` +- Branch: `overtrue/arch-ecstore-config-inventory` +- Baseline: `upstream/main` at `241e45d3b040684e5d217d2dd95f84c72f1de316` - PR type for this branch: `docs-only` - Runtime behavior changes: none - Rust code changes: none -- Docs changes: add the admin route/action snapshot baseline for later - admin module movement and route-matrix guard work. +- Docs changes: add the ECStore config consumer inventory for later config-model + contract, pure-move, and global-state migration work. ## Phase 0 Tasks @@ -44,6 +44,11 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block - Acceptance: [`crate-boundaries.md`](crate-boundaries.md) requires quality/architecture, migration-preservation, and testing/verification review before push. +- [x] `G-010` Inventory `ecstore::config::{Config, KV, KVS}` consumers. + - Acceptance: + [`ecstore-config-consumer-inventory.md`](ecstore-config-consumer-inventory.md) + records the current model definitions, global accessors, persistence helpers, + consumer groups, migration risks, and do-not-change contract. - [~] `TEST-PRTYPE-001` Check PR type enum consistency. - Current branch: not in scope. - Next PR: add a mechanical check that all migration docs use the same PR type @@ -51,36 +56,45 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Next PRs -1. `docs-only`: inventory `ecstore::config::{Config, KV, KVS}` consumers. -2. `ci-gate`: add focused checks for PR type vocabulary and temporary +1. `ci-gate`: add focused checks for PR type vocabulary and temporary compatibility marker/register consistency. -3. `test-only`: add a mechanical admin route matrix guard from the current +2. `test-only`: add a mechanical admin route matrix guard from the current snapshot and `route_registration_test.rs`. +3. `contract`: define the config-model contract surface while preserving the + existing `Config`, `KV`, and `KVS` behavior. ## Pre-Push Review Log | Expert | Status | Notes | |---|---|---| -| Quality/architecture | pass | Final review confirmed route/action/public-exception rows are source-backed, including OIDC path-based bypass, console bypass, credential-only metrics/list-remote-targets, notification targets, table-catalog prefix, and site-replication edit | -| Migration preservation | pass | Final review confirmed this branch is docs-only, aligned with Phase 0, and does not touch runtime logic, storage hot paths, global state, compatibility implementation, or crate boundaries | -| Testing/verification | pass | Final review accepted docs-only verification with layer guard, metrics reference guard, diff checks, staged diff coverage, and future route-matrix handoff | +| Quality/architecture | pass | Re-review confirmed dependency/call direction arrows, scanner global readers, adjacent scanner persistence helpers, notify config-manager persistence, narrowed test wording, and `Config model contract` wording are source-backed | +| Migration preservation | pass | Confirmed this branch is docs-only, aligned with `rustfs/backlog#660`, and does not touch runtime logic, storage hot paths, global state implementation, compatibility code, scripts, or crate boundaries | +| Testing/verification | pass | Confirmed docs-only verification is sufficient after wording was narrowed and the final staged diff check covers all docs | ## Verification Notes -Passed: +Passed locally (docs-only): - `./scripts/check_layer_dependencies.sh` - `./scripts/check_metrics_migration_refs.sh` - `git diff --check` + +Final pre-push after staging all docs: + - `git diff --cached --check` -- focused source review of `rustfs/src/admin/mod.rs`, - `rustfs/src/admin/router.rs`, `rustfs/src/admin/route_registration_test.rs`, - and `rustfs/src/admin/handlers/*.rs` route/action declarations +- focused source review of `crates/ecstore/src/config/mod.rs`, + `crates/ecstore/src/config/com.rs`, `rustfs/src/app/context.rs`, + `rustfs/src/server/{event,audit}.rs`, `rustfs/src/admin/**/*.rs`, + `crates/{notify,audit,targets,iam,scanner}/**/*.rs` +- three-expert review: quality/architecture, migration preservation, and + testing/verification ## Handoff Notes -- Keep Phase 0 PRs small. Do not start Config, Storage API, Runtime, or ECStore - movement inside this `docs-only` branch. +- Keep Phase 0 PRs small. Do not move Config, Storage API, Runtime, or ECStore + code inside this `docs-only` branch. - Keep CI checks in a separate `ci-gate` PR so the PR type rule remains enforceable. - Do not add temporary compatibility code without a matching `RUSTFS_COMPAT_TODO()` marker and cleanup-register entry. +- The next config-model PR must preserve the current tuple-struct shapes and + persistence behavior before introducing narrower provider contracts. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 2264f75a5..8402cefc9 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -24,6 +24,9 @@ hot-path behavior must not drift during this migration. between StorageCore, ECStore, ClusterControlPlane, and BackgroundControllers. - [`crate-boundaries.md`](crate-boundaries.md): PR types, crate direction, compatibility rules, and migration guardrails. +- [`ecstore-config-consumer-inventory.md`](ecstore-config-consumer-inventory.md): + current `ecstore::config::{Config, KV, KVS}` definitions, consumers, + migration risks, and do-not-change contract. - [`migration-progress.md`](migration-progress.md): current task state and context handoff. - [`compat-cleanup-register.md`](compat-cleanup-register.md): temporary @@ -34,7 +37,7 @@ hot-path behavior must not drift during this migration. ```mermaid flowchart LR G["Phase 0: Baseline and guardrails"] - CFG["Phase 1a: Config model"] + CFG["Phase 1a: Config model contract"] SEC["Phase 1: Security governance"] API["Phase 2: Storage API contracts"] RT["Phase 3: Runtime and lifecycle"]