rustfs/docs/architecture/migration-progress.md

Architecture Migration Progress

Status values: [ ] not started, [~] in progress, [x] complete, [!] blocked.

Current Context

• Issue: rustfs/backlog#660
• Branch: overtrue/arch-storage-api-bucket-dtos
• Baseline: upstream/main at 5fef10548477d9d25b0d391874f8280bf259d10e
• PR type for this branch: api-extraction
• Runtime behavior changes: none.
• Rust code changes: move the pure bucket/options DTO subset from rustfs-ecstore into rustfs-storage-api, while preserving old ecstore::store_api import paths through a temporary compatibility re-export.
• CI/script changes: none
• Docs changes: record API-003 bucket DTO compatibility cleanup.

Phase 0 Tasks

• G-001 Refresh main and record baseline.
  • Acceptance: baseline commit, title, and branch are recorded.
  • Verification: git fetch upstream main --prune; git rev-parse upstream/main.
• G-002 Create migration tracking checklist.
  • Acceptance: this file records task state, context, verification, and handoff.
• G-003 Classify PR types.
  • Acceptance: crate-boundaries.md lists exactly one allowed PR type per PR.
• G-004 Define re-export and wrapper policy.
  • Acceptance: temporary compatibility code must use RUSTFS_COMPAT_TODO.
• G-005 Add dependency direction guard.
  • Acceptance: ./scripts/check_layer_dependencies.sh passes on current upstream/main while still rejecting new unaccepted layer dependencies.
• [~] G-006 Create migration loss-prevention checks.
  • Current branch: add a mechanical admin route matrix guard from admin-route-action-snapshot.md and rustfs/src/admin/route_registration_test.rs.
  • Remaining follow-up: add checks for public re-export and storage trait coverage before pure moves.
• G-007 Create startup timeline table.
  • Acceptance: startup-timeline.md records current binary startup order, side effects, fatal boundaries, and readiness stages.
• G-008 Capture admin route-action snapshot.
  • Acceptance: admin-route-action-snapshot.md records current route families, handler ownership, authorization actions, public exceptions, table-catalog routes, and /minio/admin compatibility alias behavior.
• G-009 Enforce pre-push three-expert review.
  • Acceptance: crate-boundaries.md requires quality/architecture, migration-preservation, and testing/verification review before push.
• G-010 Inventory ecstore::config::{Config, KV, KVS} consumers.
  • Acceptance: ecstore-config-consumer-inventory.md records the current model definitions, global accessors, persistence helpers, consumer groups, migration risks, and do-not-change contract.
• TEST-PRTYPE-001 Check PR type enum consistency.
  • Acceptance: ./scripts/check_architecture_migration_rules.sh parses the allowed PR types from crate-boundaries.md and fails when ARCHITECTURE.md or architecture docs reference an unknown PR type.
• COMPAT-REG-001 Check temporary compatibility cleanup consistency.
  • Acceptance: ./scripts/check_architecture_migration_rules.sh fails when a source RUSTFS_COMPAT_TODO(<task-id>) marker lacks a cleanup-register entry, when a register entry lacks a source marker, or when a source marker omits a removal condition.

Phase 1a Config Model Tasks

• CFG-001 Inventory ecstore::config::{Config, KV, KVS} consumers.
  • Acceptance: ecstore-config-consumer-inventory.md records the current definitions, persistence helpers, global accessors, consumer groups, migration risks, and do-not-change contract.
• CFG-002 Decide model boundary.
  • Acceptance: config-model-boundary-adr.md records rustfs-config as the target package, server_config as the future model module, allowed dependencies, forbidden dependencies, preserved shape, and extraction verification gates.
• CFG-003 Move pure model definitions.
  • Next boundary: move only Config, KV, KVS, and default-registration surface into rustfs-config; keep persistence helpers and global server-config state in ecstore.
• CFG-004 Keep old ecstore::config::* compatibility path.
  • Required compatibility: source must contain RUSTFS_COMPAT_TODO(CFG-004) and a matching cleanup-register entry.

Phase 1 Security Governance Tasks

• S-001 Add crates/security-governance.
  • Acceptance: the crate is a workspace member and has no dependency on rustfs, ecstore, admin handlers, Axum, or runtime state.
  • Verification: cargo check -p rustfs-security-governance.
• S-002 Add admin route matrix core types.
  • Acceptance: AdminRouteSpec, AdminRouteAccess, AdminActionRef, PublicRouteKind, RouteRiskLevel, and validation errors model route governance metadata without registering routes or enforcing auth.
  • Verification: cargo test -p rustfs-security-governance.
• S-003 Add redaction contract types.
  • Acceptance: RedactionRule, RedactionLevel, and validation errors model sensitive field handling without logging, masking, or runtime integration.
  • Verification: cargo test -p rustfs-security-governance.
• S-004 Add serde policy marker types.
  • Acceptance: SerdePolicy, SerdePolicyKind, UnknownFieldPolicy, and validation errors model strict ingress and compatibility serde contracts without changing deserialization behavior.
  • Verification: cargo test -p rustfs-security-governance.
• S-005 Add supply-chain policy contract types.
  • Acceptance: ArtifactIntegrityPolicy, ArtifactSourceKind, and validation errors model digest, signature, and provenance requirements without changing release or CI behavior.
  • Verification: cargo test -p rustfs-security-governance.
• S-006 Add rustfs/src/admin/route_policy.rs backed by these contract types, without changing route registration or auth behavior.
  • Acceptance: direct AdminRouteSpec entries cover routes with a single stable admin policy action, deferred inventory records routes that need richer contract support, and tests prove the combined inventory covers every registered admin route.
• S-011 Add KMS action taxonomy.
  • Acceptance: KmsAction can parse and serialize dedicated configure, service-control, clear-cache, generate-data-key, delete, rotate, list, and describe actions; wildcard matching still works.
  • Verification: cargo test -p rustfs-policy action --no-fail-fast.
• S-012 Migrate KMS handlers to dedicated actions.
  • Acceptance: KMS data-key, delete/cancel-delete, cache, configure, service-control, list, and describe handlers use dedicated kms:* actions.
  • Compatibility: legacy KMS create/status admin actions are retained only as temporary compatibility paths and registered in compat-cleanup-register.md.
  • Verification: focused handler and route policy tests, migration rules, formatting, and make pre-commit.
• S-013 Apply KMS redaction.
  • Acceptance: KMS Debug output and admin status response summaries contain no Vault token, AppRole secret ID, or local master key values.
  • Must preserve: internal KMS config values remain available to runtime code and persisted config serialization still writes the original secret values.
  • Verification: focused KMS redaction/status tests, full KMS tests, migration guards, Rust quality scan, clippy, and make pre-commit passed.
• KMSD-001 Inventory KMS development defaults.
  • Acceptance: kms-development-defaults-inventory.md records Local and Vault defaults for missing master keys, temp key dirs, HTTP Vault addresses, default dev-token credentials, and skip-TLS behavior.
  • Must preserve: no KMS runtime behavior, config serialization, authorization, startup order, storage path, or crate boundary changes.
  • Verification: docs diff review, migration guards, metrics reference guard, and git diff --check.

Phase 2 Storage API Tasks

• API-001 Add crates/storage-api.
  • Acceptance: rustfs-storage-api is a workspace member and remains a dependency-free contract crate.
  • Verification: cargo check -p rustfs-storage-api.
• API-002 Move public storage error/result contracts.
  • Current PR: rustfs/rustfs#3313 merged.
  • Completed slice: add public StorageErrorCode and StorageResult contracts in rustfs-storage-api, then make ECStore StorageError::to_u32/from_u32 consume the shared code table.
  • Deferred: keep the full ECStore StorageError enum and ECStore-specific conversions in rustfs-ecstore until the DiskError, filemeta, lock, and std::io::Error downcast boundary is proven safe.
  • Acceptance: storage-api contract tests pass, ECStore compatibility tests prove numeric codes match the new contract, and cargo check -p rustfs-storage-api -p rustfs-ecstore passes.
  • Must preserve: storage error display, conversions, object error mapping, quorum classification, and reserved code gaps 0x2B/0x2C.
  • Risk defense: no storage hot-path enum move in this PR; only numeric code mapping uses the new contract.
• [~] API-003 Move DTOs.
  • Current branch: move the pure bucket/options DTO subset: MakeBucketOptions, SRBucketDeleteOp, DeleteBucketOptions, BucketOptions, and BucketInfo.
  • Acceptance: rustfs-storage-api exports these DTOs, ECStore re-exports them from the old ecstore::store_api path, and compatibility cleanup is registered with RUSTFS_COMPAT_TODO(API-003).
  • Must preserve: no ObjectOptions, ObjectInfo, reader, compression, encryption, filemeta conversion, multipart conversion, route, storage, or runtime behavior changes in this PR.

Phase 8 Background Controller Tasks

• BGC-001 Inventory background services.
  • Acceptance: background-services-inventory.md records scanner, heal, lifecycle, replication, config reload, metrics, shutdown, cancellation, and side-effect surfaces before controller work.
  • Must preserve: no code behavior change and no new controller contract in this PR.
  • Verification: docs-only architecture checks and diff hygiene.
• BGC-002 Define minimal controller contract.
  • Acceptance: background-controller-contract.md defines desired/current/status/reconcile vocabulary, status state semantics, service boundaries, and side-effect rules without starting workers or changing scheduling.
  • Must preserve: no Rust trait, scheduler, service registry, worker start/stop path, storage write, readiness change, peer signal, or runtime behavior change.
  • Verification: docs-only architecture checks and diff hygiene.

Next PRs

1. api-extraction: continue API-003 with the next pure DTO subset only after the bucket/options compatibility re-export is reviewed.
2. contract: wait for API-002/#3313 before adding error-aware storage API traits.
3. test-only: add focused preservation tests before moving scanner, heal, replication, lifecycle, or disk health workers.
4. api-extraction: move only the pure server-config model into rustfs-config as CFG-003.
5. api-extraction: keep the old rustfs_ecstore::config::* path with RUSTFS_COMPAT_TODO(CFG-004) and cleanup-register coverage.
6. consumer-migration: migrate external consumers one group at a time only after the model path and compatibility shim are stable.
7. security-change: make Local KMS unsafe defaults explicit development opt-ins or production failures in KMSD-002.
8. security-change: make Vault unsafe defaults explicit development opt-ins or production failures in KMSD-003.

Pre-Push Review Log

Expert	Status	Notes
Quality/architecture	pass	Only pure bucket/options DTOs moved into rustfs-storage-api; object, reader, compression, encryption, filemeta, multipart, storage, and runtime logic stayed in ECStore.
Migration preservation	pass	Old ecstore::store_api import paths remain through RUSTFS_COMPAT_TODO(API-003) compatibility re-export, with cleanup registered.
Testing/verification	pass	Focused DTO tests, ECStore compatibility test, migration guards, formatting, rio-v2 clippy, dependency review, diff checks, and pre-commit passed.
Quality/architecture	pass	Single docs-only PR; ADR chooses existing rustfs-config, records module path and dependency boundaries, and avoids a speculative new crate.
Migration preservation	pass	No code movement; ADR explicitly keeps persistence helpers, global server-config state, startup order, and old-path compatibility requirements out of CFG-002.
Testing/verification	pass	Docs-only verification uses migration guard scripts, metrics reference guard, layer dependency guard, and whitespace checks.
Quality/architecture	pass	Single docs-only PR; the inventory is isolated under docs/architecture, uses existing KMS source files as evidence, and introduces no new abstraction or dependency edge.
Migration preservation	pass	No runtime code, config persistence, admin authorization, startup order, storage path, global state, or crate boundary changes are made.
Testing/verification	pass	Docs-only verification is bounded to diff review, architecture migration rules, metrics reference guard, layer dependency guard, and whitespace checks.

Verification Notes

Passed:

• cargo test -p rustfs-storage-api
• cargo test -p rustfs-ecstore --test storage_api_compat_test
• cargo check -p rustfs-storage-api -p rustfs-ecstore
• cargo check -p rustfs-ecstore --features rio-v2
• cargo clippy -p rustfs-ecstore --features rio-v2 --all-targets -- -D warnings
• cargo fmt --all
• cargo fmt --all --check
• cargo test -p rustfs-ecstore error -- --nocapture
• ./scripts/check_architecture_migration_rules.sh
• ./scripts/check_layer_dependencies.sh
• ./scripts/check_metrics_migration_refs.sh
• ./scripts/check_unsafe_code_allowances.sh
• git diff --check
• cargo tree -p rustfs-storage-api --edges normal
• make NUM_CORES=1 pre-commit

Notes:

• Plain make pre-commit runs fmt and unsafe-code-check concurrently via global Makefile parallelism; unsafe-code-check passes standalone, and the full pre-commit target passed when run serially with NUM_CORES=1.
• Full nextest in pre-commit: 5704 passed, 111 skipped.
• Workspace doctests passed.

Handoff Notes

• Keep this API-003 branch as a focused api-extraction PR for bucket/options DTOs only.
• Do not move ObjectOptions, ObjectInfo, CompletePart, reader types, compression/encryption helpers, filemeta conversions, S3 DTO conversions, or storage traits in this PR.
• Keep the old ecstore::store_api compatibility re-export until all consumers import bucket DTOs from rustfs_storage_api.
• Do not add temporary compatibility code without a matching RUSTFS_COMPAT_TODO(<task-id>) marker and cleanup-register entry.
• KMS production default hardening remains a separate task group; do not bundle it with this inventory PR.
• The CFG-003 extraction PR must preserve the tuple-struct shape, serde fields, hiddenIfEmpty alias, Config::new default behavior, marshal/unmarshal behavior, and old rustfs_ecstore::config::* path.
• Do not create a new config-model crate unless a later implementation attempt proves rustfs-config cannot hold the pure model boundary.