gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-07 13:53:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
a49c6b4c2e4233f4e9b64a96a6f006f773616598
rustfs/docs/architecture/migration-progress.md
T
安正超 686068d8bd refactor(storage): narrow metadata bounds (#3343)
2026-06-11 12:04:02 +08:00

338 lines
19 KiB
Markdown
Raw Blame History

# Architecture Migration Progress
Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` blocked.
## Current Context
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
- Branch: `overtrue/arch-storage-api-bound-narrowing`
- Baseline: `origin/main` at `6b86e3875c4912f3a7c27915b0bbd43399402847`
- PR type for this branch: `dependency-migration`
- Runtime behavior changes: none.
- Rust code changes: narrow metadata helper generic bounds away from full
`StorageAPI` where helpers only need object I/O, object metadata/delete,
bucket listing, object listing, or `StorageAdminApi`.
- CI/script changes: none.
- Docs changes: record API-008 completion and the current bound-narrowing
context, verification evidence, and expert review outcomes.
## Phase 0 Tasks
- [x] `G-001` Refresh `main` and record baseline.
- Acceptance: baseline commit, title, and branch are recorded.
- Verification: `git fetch upstream main --prune`; `git rev-parse upstream/main`.
- [x] `G-002` Create migration tracking checklist.
- Acceptance: this file records task state, context, verification, and handoff.
- [x] `G-003` Classify PR types.
- Acceptance: [`crate-boundaries.md`](crate-boundaries.md) lists exactly one
allowed PR type per PR.
- [x] `G-004` Define re-export and wrapper policy.
- Acceptance: temporary compatibility code must use `RUSTFS_COMPAT_TODO`.
- [x] `G-005` Add dependency direction guard.
- Acceptance: `./scripts/check_layer_dependencies.sh` passes on current
`upstream/main` while still rejecting new unaccepted layer dependencies.
- [~] `G-006` Create migration loss-prevention checks.
- Current branch: add a mechanical admin route matrix guard from
[`admin-route-action-snapshot.md`](admin-route-action-snapshot.md) and
`rustfs/src/admin/route_registration_test.rs`.
- Remaining follow-up: add checks for public re-export and storage trait
coverage before pure moves.
- [x] `G-007` Create startup timeline table.
- Acceptance: [`startup-timeline.md`](startup-timeline.md) records current
binary startup order, side effects, fatal boundaries, and readiness stages.
- [x] `G-008` Capture admin route-action snapshot.
- Acceptance: [`admin-route-action-snapshot.md`](admin-route-action-snapshot.md)
records current route families, handler ownership, authorization actions,
public exceptions, table-catalog routes, and `/minio/admin` compatibility
alias behavior.
- [x] `G-009` Enforce pre-push three-expert review.
- Acceptance: [`crate-boundaries.md`](crate-boundaries.md) requires
quality/architecture, migration-preservation, and testing/verification review
before push.
- [x] `G-010` Inventory `ecstore::config::{Config, KV, KVS}` consumers.
- Acceptance:
[`ecstore-config-consumer-inventory.md`](ecstore-config-consumer-inventory.md)
records the current model definitions, global accessors, persistence helpers,
consumer groups, migration risks, and do-not-change contract.
- [x] `TEST-PRTYPE-001` Check PR type enum consistency.
- Acceptance: `./scripts/check_architecture_migration_rules.sh` parses the
allowed PR types from [`crate-boundaries.md`](crate-boundaries.md) and fails
when `ARCHITECTURE.md` or architecture docs reference an unknown PR type.
- [x] `COMPAT-REG-001` Check temporary compatibility cleanup consistency.
- Acceptance: `./scripts/check_architecture_migration_rules.sh` fails when a
source `RUSTFS_COMPAT_TODO(<task-id>)` marker lacks a cleanup-register entry,
when a register entry lacks a source marker, or when a source marker omits a
removal condition.
## Phase 1a Config Model Tasks
- [x] `CFG-001` Inventory `ecstore::config::{Config, KV, KVS}` consumers.
- Acceptance:
[`ecstore-config-consumer-inventory.md`](ecstore-config-consumer-inventory.md)
records the current definitions, persistence helpers, global accessors,
consumer groups, migration risks, and do-not-change contract.
- [x] `CFG-002` Decide model boundary.
- Acceptance:
[`config-model-boundary-adr.md`](config-model-boundary-adr.md) records
`rustfs-config` as the target package, `server_config` as the future model
module, allowed dependencies, forbidden dependencies, preserved shape, and
extraction verification gates.
- [ ] `CFG-003` Move pure model definitions.
- Next boundary: move only `Config`, `KV`, `KVS`, and default-registration
surface into `rustfs-config`; keep persistence helpers and global
server-config state in `ecstore`.
- [ ] `CFG-004` Keep old `ecstore::config::*` compatibility path.
- Required compatibility: source must contain `RUSTFS_COMPAT_TODO(CFG-004)`
and a matching cleanup-register entry.
## Phase 1 Security Governance Tasks
- [x] `S-001` Add `crates/security-governance`.
- Acceptance: the crate is a workspace member and has no dependency on
`rustfs`, `ecstore`, admin handlers, Axum, or runtime state.
- Verification: `cargo check -p rustfs-security-governance`.
- [x] `S-002` Add admin route matrix core types.
- Acceptance: `AdminRouteSpec`, `AdminRouteAccess`, `AdminActionRef`,
`PublicRouteKind`, `RouteRiskLevel`, and validation errors model route
governance metadata without registering routes or enforcing auth.
- Verification: `cargo test -p rustfs-security-governance`.
- [x] `S-003` Add redaction contract types.
- Acceptance: `RedactionRule`, `RedactionLevel`, and validation errors model
sensitive field handling without logging, masking, or runtime integration.
- Verification: `cargo test -p rustfs-security-governance`.
- [x] `S-004` Add serde policy marker types.
- Acceptance: `SerdePolicy`, `SerdePolicyKind`, `UnknownFieldPolicy`, and
validation errors model strict ingress and compatibility serde contracts
without changing deserialization behavior.
- Verification: `cargo test -p rustfs-security-governance`.
- [x] `S-005` Add supply-chain policy contract types.
- Acceptance: `ArtifactIntegrityPolicy`, `ArtifactSourceKind`, and validation
errors model digest, signature, and provenance requirements without changing
release or CI behavior.
- Verification: `cargo test -p rustfs-security-governance`.
- [x] `S-006` Add `rustfs/src/admin/route_policy.rs` backed by these contract
types, without changing route registration or auth behavior.
- Acceptance: direct `AdminRouteSpec` entries cover routes with a single
stable admin policy action, deferred inventory records routes that need
richer contract support, and tests prove the combined inventory covers every
registered admin route.
- [x] `S-011` Add KMS action taxonomy.
- Acceptance: `KmsAction` can parse and serialize dedicated configure,
service-control, clear-cache, generate-data-key, delete, rotate, list, and
describe actions; wildcard matching still works.
- Verification: `cargo test -p rustfs-policy action --no-fail-fast`.
- [x] `S-012` Migrate KMS handlers to dedicated actions.
- Acceptance: KMS data-key, delete/cancel-delete, cache, configure,
service-control, list, and describe handlers use dedicated `kms:*` actions.
- Compatibility: legacy KMS create/status admin actions are retained only as
temporary compatibility paths and registered in
[`compat-cleanup-register.md`](compat-cleanup-register.md).
- Verification: focused handler and route policy tests, migration rules,
formatting, and `make pre-commit`.
- [x] `S-013` Apply KMS redaction.
- Acceptance: KMS Debug output and admin status response summaries contain no
Vault token, AppRole secret ID, or local master key values.
- Must preserve: internal KMS config values remain available to runtime code
and persisted config serialization still writes the original secret values.
- Verification: focused KMS redaction/status tests, full KMS tests, migration
guards, Rust quality scan, clippy, and `make pre-commit` passed.
- [x] `KMSD-001` Inventory KMS development defaults.
- Acceptance:
[`kms-development-defaults-inventory.md`](kms-development-defaults-inventory.md)
records Local and Vault defaults for missing master keys, temp key dirs,
HTTP Vault addresses, default dev-token credentials, and skip-TLS behavior.
- Must preserve: no KMS runtime behavior, config serialization,
authorization, startup order, storage path, or crate boundary changes.
- Verification: docs diff review, migration guards, metrics reference guard,
and `git diff --check`.
## Phase 2 Storage API Tasks
- [x] `API-001` Add `crates/storage-api`.
- Acceptance: `rustfs-storage-api` is a workspace member and remains a
dependency-free contract crate.
- Verification: `cargo check -p rustfs-storage-api`.
- [x] `API-002` Move public storage error/result contracts.
- Current PR: `rustfs/rustfs#3313` merged.
- Completed slice: add public `StorageErrorCode` and `StorageResult`
contracts in `rustfs-storage-api`, then make ECStore
`StorageError::to_u32/from_u32` consume the shared code table.
- Deferred: keep the full ECStore `StorageError` enum and ECStore-specific
conversions in `rustfs-ecstore` until the `DiskError`, filemeta, lock, and
`std::io::Error` downcast boundary is proven safe.
- Acceptance: storage-api contract tests pass, ECStore compatibility tests
prove numeric codes match the new contract, and
`cargo check -p rustfs-storage-api -p rustfs-ecstore` passes.
- Must preserve: storage error display, conversions, object error mapping,
quorum classification, and reserved code gaps `0x2B/0x2C`.
- Risk defense: no storage hot-path enum move in this PR; only numeric code
mapping uses the new contract.
- [x] `API-003` Move DTOs.
- Current PR: `rustfs/rustfs#3314` merged.
- Completed slice: move the pure bucket/options DTO subset:
`MakeBucketOptions`, `SRBucketDeleteOp`, `DeleteBucketOptions`,
`BucketOptions`, and `BucketInfo`.
- Acceptance: `rustfs-storage-api` exports these DTOs, ECStore re-exports
them from the old `ecstore::store_api` path, and compatibility cleanup is
registered with `RUSTFS_COMPAT_TODO(API-003)`.
- Must preserve: no `ObjectOptions`, `ObjectInfo`, reader, compression,
encryption, filemeta conversion, multipart conversion, route, storage, or
runtime behavior changes in this PR.
- [x] `API-006` Add disk inventory/admin trait.
- Current PR: `rustfs/rustfs#3330` merged.
- Completed slice: add `StorageAdminApi` and `DiskSetSelector` to
`rustfs-storage-api`.
- Acceptance: `StorageAdminApi` exposes backend info, global storage info,
local storage info, disk-set inventory, and drive-count surfaces without
depending on ECStore implementation types.
- Must preserve: no `StorageAPI::get_disks` removal, no ECStore implementation
change, no admin/readiness/capacity behavior change.
- Risk defense: use associated types for backend/storage/disk DTOs so this
contract slice does not pull `rustfs-madmin` or `rustfs-ecstore` into
`rustfs-storage-api`.
- Verification: focused storage-api tests, dependency tree, migration guards,
formatting, and diff hygiene.
- [x] `API-007` Dual-route `get_disks` consumers.
- Completed first slice: `rustfs/rustfs#3331` bound `ECStore` to
`StorageAdminApi` while keeping all consumers unchanged.
- Completed second slice: `rustfs/rustfs#3332` migrated the admin
storage-class config drive-count consumer to
`StorageAdminApi::set_drive_counts`.
- Completed third slice: `rustfs/rustfs#3333` migrated
`DefaultAdminUsecase` storage-info reads to
`StorageAdminApi::storage_info`.
- Completed fourth slice: `rustfs/rustfs#3334` migrated account-info
`backend_info`, rebalance status `storage_info`, and runtime readiness
`storage_info`.
- Completed fifth slice: `rustfs/rustfs#3335` migrated grouped observability,
RPC health, server-info, realtime metrics, and notification read-side
consumers.
- Completed sixth slice: `rustfs/rustfs#3336` migrated ECStore internal
decommission space, local-storage-info, backend-info, drive-count, and
disk-inventory admin handlers away from old `StorageAPI` method calls.
- Completed seventh slice: `rustfs/rustfs#3337` migrated maintenance and
background read-side storage inventory consumers in rebalance metadata
initialization, heal resume disk lookup, and scanner local disk scan lookup.
- Completion acceptance: admin inventory consumers no longer use old
`StorageAPI` calls for backend info, storage info, local storage info,
drive-count, or disk-set inventory when the inventory-facing
`StorageAdminApi` contract represents the same read-only operation.
- [x] `API-008` Remove duplicate old-path admin surfaces.
- Completed slice: `rustfs/rustfs#3340` removed duplicate admin-read methods
from the old `StorageAPI` trait and its ECStore/Sets/SetDisks/test
implementations after API-007 migrated their consumers.
- Acceptance: old `StorageAPI` keeps storage operation traits and
`new_ns_lock`, while admin inventory surfaces live only on
`StorageAdminApi`.
- [~] `API-009` Narrow metadata helper storage bounds.
- Current branch slice: narrow server config, tier config, rebalance
metadata, and startup metadata migration helper bounds away from full
`StorageAPI` when the helper only needs `ObjectIO`, `ObjectOperations`,
`BucketOperations`, `ListOperations`, or `StorageAdminApi`.
- Acceptance: metadata helper contracts express the actual operation group
they need, while callers and persistence behavior remain unchanged.
- Must preserve: config/tier/rebalance serialized byte shapes, read/write
object names, legacy bucket/IAM migration normalization and skip behavior,
lock behavior around rebalance metadata merge, dynamic storage-class lookup,
object APIs, scanner cache persistence, heal repair, replication
persistence, and storage hot paths.
- Risk defense: do not move traits to `rustfs-storage-api`, do not remove
`StorageAPI`, do not alter helper bodies or storage options, and keep
`save_rebalance_meta_with_merge` on full `StorageAPI` because it still needs
`new_ns_lock`.
- Verification: focused compile/tests, migration guards, Rust risk scan, and
required quality/architecture, migration-preservation, and
testing/verification review passed before push.
## Phase 8 Background Controller Tasks
- [x] `BGC-001` Inventory background services.
- Acceptance:
[`background-services-inventory.md`](background-services-inventory.md)
records scanner, heal, lifecycle, replication, config reload, metrics,
shutdown, cancellation, and side-effect surfaces before controller work.
- Must preserve: no code behavior change and no new controller contract in
this PR.
- Verification: docs-only architecture checks and diff hygiene.
- [x] `BGC-002` Define minimal controller contract.
- Acceptance:
[`background-controller-contract.md`](background-controller-contract.md)
defines desired/current/status/reconcile vocabulary, status state
semantics, service boundaries, and side-effect rules without starting
workers or changing scheduling.
- Must preserve: no Rust trait, scheduler, service registry, worker
start/stop path, storage write, readiness change, peer signal, or runtime
behavior change.
- Verification: docs-only architecture checks and diff hygiene.
## Next PRs
1. `dependency-migration`: narrow metadata/helper bounds away from full
`StorageAPI` where the helper only needs a specific operation trait.
2. `api-extraction`: move only the pure server-config model into
rustfs-config as CFG-003.
3. `api-extraction`: keep the old rustfs_ecstore::config::* path with
RUSTFS_COMPAT_TODO(CFG-004) and cleanup-register coverage.
4. `consumer-migration`: migrate external consumers one group at a time only
after the model path and compatibility shim are stable.
5. `security-change`: make Local KMS unsafe defaults explicit development
opt-ins or production failures in KMSD-002.
6. `security-change`: make Vault unsafe defaults explicit development opt-ins
or production failures in KMSD-003.
## Pre-Push Review Log
| Expert | Status | Notes |
|---|---|---|
| Quality/architecture | pass | Confirmed the Rust diff only changes imports/generic bounds, the new operation-group bounds match actual calls, `StorageAPI` and `new_ns_lock` remain where needed, and no over-abstraction, naming regression, dependency reversal, or hot-path intrusion was found. |
| Migration preservation | pass | Confirmed config/tier/rebalance byte shapes, object names, buckets, options, bucket/IAM normalization and skip behavior, dynamic storage-class lookup, and rebalance metadata merge locking remain unchanged. |
| Testing/verification | pass | Confirmed the focused compile, config/tier/rebalance/bucket migration tests, migration guards, diff hygiene, and added-line Rust risk scan are sufficient; full pre-commit can be skipped under the current larger-granularity instruction. |
## Verification Notes
Passed:
- `cargo fmt --all`.
- `cargo check -p rustfs-ecstore`.
- `cargo fmt --all --check`.
- `cargo check -p rustfs-storage-api -p rustfs-ecstore -p rustfs --lib`.
- `cargo test -p rustfs-ecstore config --lib`; 59 passed.
- `cargo test -p rustfs-ecstore tier --lib`; 27 passed.
- `cargo test -p rustfs-ecstore rebalance --lib`; 198 passed.
- `cargo test -p rustfs-ecstore bucket::migration --lib`; 2 passed.
- `./scripts/check_architecture_migration_rules.sh`.
- `./scripts/check_layer_dependencies.sh`.
- `./scripts/check_metrics_migration_refs.sh`.
- `./scripts/check_unsafe_code_allowances.sh`.
- `git diff --check`.
- Rust code-quality scan on changed `.rs` files: broad full-file matches are
pre-existing test/business-code patterns; added-line scan found no new
unwrap/expect, numeric cast, `Result<_, String>`, `Box<dyn Error>`,
println/eprintln, or `Ordering::Relaxed`.
Notes:
- Full pre-commit is intentionally skipped when the focused tests and guards
pass, per the current instruction to increase PR granularity.
- This slice changes generic bounds and imports only; helper bodies, storage
options, object names, and metadata normalization logic are unchanged.
- `save_rebalance_meta_with_merge` intentionally still requires full
`StorageAPI` because it uses `new_ns_lock`.
- No temporary compatibility shim was added.
## Handoff Notes
- Keep this API-009 slice as a `dependency-migration` PR that only narrows
metadata/helper generic bounds for config, tier, rebalance, and startup
metadata migration helpers.
- Do not remove `StorageAPI` itself, object operation traits, or
`new_ns_lock` in this PR.
- Do not move traits into `rustfs-storage-api` or introduce new compatibility
shims in this PR.
- Do not alter config/tier/rebalance serialized byte shapes, legacy bucket/IAM
migration normalization, object names, storage options, scanner cache writes,
heal object repair, object APIs, replication persistence, or storage hot-path
consumers in this PR.
- Do not add temporary compatibility code unless a matching
`RUSTFS_COMPAT_TODO(<task-id>)` marker and cleanup-register entry are added.
Reference in New Issue View Git Blame Copy Permalink