mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-06 05:17:42 +00:00
refactor: clean external DTO consumers (#3566)
* refactor: clean external DTO consumers * fix: handle empty erasure shard recovery
This commit is contained in:
安正超
@@ -5,20 +5,22 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
## Current Context
|
||||
|
||||
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
|
||||
- Branch: `overtrue/arch-storage-operation-bounds-cleanup`
|
||||
- Baseline: `main` at `a30cafa73fde3e7e88c160e70c290e74a0c2235f`
|
||||
after `rustfs/rustfs#3563` merged.
|
||||
- Branch: `overtrue/arch-storage-dto-consumer-boundaries`
|
||||
- Baseline: `main` at `99941f7e7c5e0c88532a93cc175a3ea4111d7098`
|
||||
after `rustfs/rustfs#3565` merged.
|
||||
- PR type for this branch: `consumer-migration`
|
||||
- Runtime behavior changes: no external behavior change expected.
|
||||
- Rust code changes: migrate scanner cache persistence, RustFS object
|
||||
namespace-lock helper, and table catalog storage bounds away from ECStore
|
||||
compatibility operation traits to `rustfs-storage-api` operation traits with
|
||||
ECStore concrete associated-type bindings. ECStore-owned `ObjectInfo`,
|
||||
`ObjectOptions`, readers, delete DTOs, walk filters, lock wrappers, and
|
||||
implementation behavior stay in ECStore.
|
||||
- CI/script changes: extend migration guards so outer consumers do not drift
|
||||
back to ECStore operation trait imports.
|
||||
- Docs changes: record the larger operation consumer-bound cleanup slice.
|
||||
- Runtime behavior changes: no migration behavior change expected; CI follow-up
|
||||
preserves empty-object erasure recovery by avoiding zero-byte SIMD decode.
|
||||
- Rust code changes: add crate-local semantic aliases for ECStore-owned object
|
||||
metadata, options, readers, and delete DTOs in scanner, heal, notify, Swift,
|
||||
S3 Select, and RustFS storage/app consumers so external call sites stop
|
||||
importing raw `rustfs_ecstore::store_api` DTOs outside deliberate boundary
|
||||
files. ECStore-owned DTO definitions and runtime behavior stay in ECStore.
|
||||
CI follow-up handles zero-length shards in erasure reconstruction without
|
||||
changing non-empty shard behavior.
|
||||
- CI/script changes: none.
|
||||
- Docs changes: record the larger DTO consumer-boundary cleanup slice and the
|
||||
empty-object erasure recovery CI follow-up.
|
||||
|
||||
## Phase 0 Tasks
|
||||
|
||||
@@ -775,6 +777,29 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
guards, formatting, diff hygiene, Rust risk scan, full pre-commit, and
|
||||
required three-expert review passed.
|
||||
|
||||
- [x] `API-026` Clean external DTO consumer boundaries.
|
||||
- Current branch: `overtrue/arch-storage-dto-consumer-boundaries`.
|
||||
- Completed slice: introduce crate-local semantic aliases for ECStore-owned
|
||||
object metadata/options/readers/delete DTOs in scanner, heal, notify, Swift,
|
||||
S3 Select, and RustFS storage/app consumers; update production and affected
|
||||
test call sites to use those local aliases instead of raw
|
||||
`rustfs_ecstore::store_api` DTO imports.
|
||||
- Acceptance: non-ECStore direct `rustfs_ecstore::store_api` references are
|
||||
limited to boundary alias definitions, ECStore remains the owner of
|
||||
`ObjectInfo`, `ObjectOptions`, object readers, delete DTOs, walk filters,
|
||||
lock wrappers, and implementation behavior, and external consumers express
|
||||
their local semantic dependency through crate-owned names.
|
||||
- Must preserve: object metadata shape, object option defaults, reader/writer
|
||||
behavior, delete replication DTO handling, scanner cache semantics, heal
|
||||
storage metadata semantics, Swift and S3 Select object reads, notification
|
||||
event payloads, S3 response DTO mapping, and storage/app test behavior.
|
||||
- Risk defense: this slice uses type aliases and import-boundary cleanup only;
|
||||
it does not move DTO definitions, alter serialization, change object-store
|
||||
implementations, or adjust runtime control flow.
|
||||
- Verification: focused compile/tests, migration/layer guards, formatting,
|
||||
diff hygiene, direct import scan, Rust risk scan, full pre-commit, and
|
||||
required three-expert review passed.
|
||||
|
||||
## Phase 8 Background Controller Tasks
|
||||
|
||||
- [x] `BGC-001` Inventory background services.
|
||||
@@ -1051,46 +1076,70 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
|
||||
| Expert | Status | Notes |
|
||||
|---|---|---|
|
||||
| Quality/architecture | passed | Outer scanner/RustFS operation consumers now use storage-api operation traits with explicit ECStore concrete associated-type bindings; ECStore keeps compatibility traits only for implementation and downstream compatibility. |
|
||||
| Migration preservation | passed | Scanner cache persistence, object self-copy namespace locking, and table catalog object storage keep the same ECStore DTOs, readers, lock wrappers, walk filter shape, and storage error conversion behavior. |
|
||||
| Testing/verification | passed | Focused RustFS/scanner compile/tests, migration/layer guards, formatting, diff hygiene, Rust risk scan, and full `make pre-commit` passed. |
|
||||
| Quality/architecture | passed | Scanner, heal, notify, Swift, S3 Select, and RustFS storage/app consumers now use crate-local DTO aliases; the erasure CI fix is isolated to zero-length shard reconstruction. |
|
||||
| Migration preservation | passed | ECStore remains the owner of object DTOs/readers/walk filters/implementation behavior; aliases preserve concrete types and non-empty erasure encode/decode paths still use the existing encoders. |
|
||||
| Testing/verification | passed | Focused DTO/erasure compile/tests, migration/layer guards, direct import scan, formatting, diff hygiene, Rust risk scan, and full `make pre-commit` passed. |
|
||||
|
||||
## Verification Notes
|
||||
|
||||
Passed before push:
|
||||
|
||||
- `cargo check --tests -p rustfs -p rustfs-scanner`: passed.
|
||||
- `cargo test -p rustfs-scanner`: passed; 151 unit tests passed, lifecycle
|
||||
focused test passed, 14 lifecycle integration tests ignored by default.
|
||||
- `cargo test -p rustfs --lib table_catalog`: passed; 168 passed.
|
||||
- `cargo test -p rustfs --lib app::object_usecase`: passed; 86 passed, 2
|
||||
ignored.
|
||||
- `cargo check --tests -p rustfs -p rustfs-scanner -p rustfs-heal -p
|
||||
rustfs-protocols -p rustfs-notify -p rustfs-s3select-api`: passed.
|
||||
- `cargo test -p rustfs-scanner -p rustfs-heal -p rustfs-notify -p
|
||||
rustfs-protocols -p rustfs-s3select-api`: passed.
|
||||
- `cargo test -p rustfs --lib app::object_usecase::tests`: passed; 85 passed,
|
||||
2 ignored.
|
||||
- `cargo test -p rustfs --lib app::bucket_usecase::tests`: passed; 64 passed.
|
||||
- `cargo test -p rustfs --lib app::multipart_usecase::tests`: passed; 24
|
||||
passed, 4 ignored.
|
||||
- `cargo test -p rustfs --lib storage::s3_api::bucket::tests`: passed; 22
|
||||
passed.
|
||||
- `cargo test -p rustfs --lib storage::ecfs_test::tests`: passed; 59 passed,
|
||||
14 ignored.
|
||||
- `rg -n 'rustfs_ecstore::store_api' rustfs/src crates --glob
|
||||
'!crates/ecstore/**' --glob '*.rs'`: remaining matches are deliberate
|
||||
boundary alias definitions.
|
||||
- `cargo test -p rustfs-ecstore erasure_coding::erasure::tests`: passed; 31
|
||||
passed.
|
||||
- `PROPTEST_CASES=1024 cargo test -p rustfs-ecstore
|
||||
erasure_coding::erasure::tests::decode_data_and_parity_round_trips_bounded_recoverability`:
|
||||
passed.
|
||||
- `cargo test -p rustfs-ecstore
|
||||
rpc::peer_s3_client::tests::local_get_bucket_info_survives_prior_walk_timeout`:
|
||||
passed after a full-crate `cargo test -p rustfs-ecstore` run exposed this
|
||||
unrelated ordinary-harness global-state interference.
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `./scripts/check_layer_dependencies.sh`: passed.
|
||||
- `cargo fmt --all --check`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- Rust risk scan: no new `unwrap`/`expect`, panic/todo markers, `unsafe`,
|
||||
process-spawning calls, lossy casts, println/eprintln, or relaxed ordering in
|
||||
- Rust risk scan: new hits are crate-local alias casts plus the empty-erasure
|
||||
test expectations; no new production `unwrap`/`expect`, panic/todo markers,
|
||||
`unsafe`, process-spawning calls, println/eprintln, or relaxed ordering in
|
||||
added Rust lines.
|
||||
- `make pre-commit`: passed; nextest reported 6207 tests passed and 111
|
||||
- `make pre-commit`: passed; nextest reported 6218 tests passed and 111
|
||||
skipped, and doctests passed.
|
||||
|
||||
Notes:
|
||||
|
||||
- This slice follows the shared list operation consumer cleanup and keeps the
|
||||
old aggregate facade, DTO/helper, response, and operation contract guards
|
||||
active.
|
||||
- Outer scanner/RustFS operation consumers now bind operation traits through
|
||||
`rustfs-storage-api`; ECStore keeps concrete object metadata, options,
|
||||
readers, delete DTOs, lock wrappers, filemeta-bound walk filters, and
|
||||
implementation behavior.
|
||||
- The slice does not alter scanner cache load/save behavior, object self-copy
|
||||
lock behavior, table catalog object operations, list/walk implementation,
|
||||
object reader/writer behavior, or storage error runtime behavior.
|
||||
- This slice follows the external operation consumer cleanup and keeps the old
|
||||
aggregate facade, DTO/helper, response, and operation contract guards active.
|
||||
- External DTO consumers now name ECStore-owned object DTOs through local
|
||||
semantic aliases in their owning crates or RustFS storage boundary module.
|
||||
- The slice does not alter object metadata shape, options defaults,
|
||||
reader/writer behavior, delete replication DTO handling, scanner cache
|
||||
semantics, heal storage metadata semantics, Swift/S3 Select object reads,
|
||||
notification event payloads, S3 response DTO mapping, or runtime storage
|
||||
behavior.
|
||||
- The CI follow-up preserves empty-object erasure reconstruction by filling
|
||||
missing zero-length shards before calling encoders that reject zero-byte SIMD
|
||||
shard sizes.
|
||||
|
||||
## Handoff Notes
|
||||
|
||||
- External operation consumer cleanup is based on `main` after
|
||||
`rustfs/rustfs#3563` merged.
|
||||
- Continue with larger consumer-migration batches; avoid moving ECStore-owned
|
||||
DTOs/readers/walk filters until their concrete behavior is isolated.
|
||||
- External DTO consumer cleanup is based on `main` after
|
||||
`rustfs/rustfs#3565` merged and carries the empty-object erasure recovery
|
||||
CI follow-up because the merged PR did not include that post-merge fix.
|
||||
- Continue with larger consumer-migration batches; keep ECStore-owned
|
||||
DTOs/readers/walk filters in ECStore until their concrete behavior is isolated
|
||||
enough for a pure-move slice.
|
||||
|
||||
Reference in New Issue
Block a user