mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-20 11:32:19 +00:00
refactor: flatten test harness storage compat aliases (#3596)
* refactor: flatten test harness storage compat aliases * refactor: flatten rustfs storage compat aliases (#3597) * refactor: prune runtime storage compat surface (#3598) * refactor: flatten runtime secondary storage compat (#3599) * docs: add scheduler placement profiling baselines (#3600) * feat: add observability topology capability contracts (#3601)
This commit is contained in:
安正超
@@ -5,17 +5,19 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
## Current Context
|
||||
|
||||
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
|
||||
- Branch: `overtrue/arch-storage-api-lifecycle-contracts`
|
||||
- Baseline: stacked on `rustfs/rustfs#3594` head
|
||||
(`6fc05b84e2cd22a0482adfbc042184b03f8fdfa6`).
|
||||
- PR type for this branch: `pure-move`
|
||||
- Runtime behavior changes: no migration behavior change expected.
|
||||
- Rust code changes: move lifecycle helper DTO contracts for expiration and
|
||||
transitioned object metadata into rustfs-storage-api, switch ECStore internal
|
||||
consumers to direct storage-api imports, and keep ECStore old-path re-exports.
|
||||
- CI/script changes: add migration guards rejecting reintroduced ECStore
|
||||
lifecycle helper DTO definitions and old internal consumer imports.
|
||||
- Docs changes: record the lifecycle helper pure-move slice.
|
||||
- Branch: `overtrue/arch-observability-topology-contracts`
|
||||
- Baseline: stacked on `origin/overtrue/arch-test-harness-compat-aliases`
|
||||
after `rustfs/rustfs#3600`
|
||||
(`ae6a5befcf60f2f2ebce9799ba93649032234273`).
|
||||
- PR type for this branch: `contract`
|
||||
- Runtime behavior changes: none.
|
||||
- Rust code changes: add observability and topology capability DTO/trait
|
||||
contracts to `rustfs-storage-api`.
|
||||
- CI/script changes: extend migration re-export guard coverage for the new
|
||||
contract exports.
|
||||
- Docs changes: add
|
||||
[`runtime-capability-contracts.md`](runtime-capability-contracts.md) and
|
||||
record the combined PR-08/API-013 plus PR-09/API-014 contract slice.
|
||||
|
||||
## Phase 0 Tasks
|
||||
|
||||
@@ -63,6 +65,60 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
[`ecstore-config-consumer-inventory.md`](ecstore-config-consumer-inventory.md)
|
||||
records the current model definitions, global accessors, persistence helpers,
|
||||
consumer groups, migration risks, and do-not-change contract.
|
||||
- [x] `G-011` Inventory scheduler baseline.
|
||||
- Acceptance:
|
||||
[`scheduler-baseline.md`](scheduler-baseline.md) records current owners for
|
||||
request admission, reusable scheduler/backpressure facades, workers, scanner
|
||||
budget, heal admission, and the Tokio runtime builder.
|
||||
- Must preserve: no Rust source changes, no scheduler/controller contract
|
||||
changes, and no runtime behavior changes.
|
||||
- [x] `G-012` Inventory placement and repair invariants.
|
||||
- Acceptance:
|
||||
[`placement-repair-invariants.md`](placement-repair-invariants.md) records
|
||||
object-to-set hashing, pool/set/disk assignment boundaries, set-aware
|
||||
readiness and lock quorum, scanner budget, and heal admission preservation
|
||||
gates.
|
||||
- Must preserve: no placement, repair, scanner, heal, readiness, lock, or
|
||||
storage metadata behavior changes.
|
||||
- [x] `G-013` Inventory profiling and NUMA capabilities.
|
||||
- Acceptance:
|
||||
[`profiling-numa-capability-inventory.md`](profiling-numa-capability-inventory.md)
|
||||
records current CPU/memory profiling, cgroup memory sampling, allocator
|
||||
backend, eBPF, and NUMA capability support plus no-op fallback invariants.
|
||||
- Must preserve: no startup, profiling, allocator, runtime, or platform-gate
|
||||
behavior changes.
|
||||
|
||||
## Issue #660 Capability Contract Tasks
|
||||
|
||||
- [x] `PR-08/API-013` Add observability snapshot contract.
|
||||
- Completed slice: add `CapabilityState`, `CapabilityStatus`,
|
||||
`CapabilitySnapshotError`, `ObservabilitySnapshot`,
|
||||
`UserspaceProfilingCapability`, `MemorySamplingState`,
|
||||
`PlatformSupport`, and `ObservabilitySnapshotProvider` to
|
||||
`rustfs-storage-api`.
|
||||
- Acceptance: runtime telemetry, userspace profiling, memory sampling, and
|
||||
platform support states are representable without runtime, ECStore, admin,
|
||||
profiling, exporter, sidecar, eBPF, or OTEL implementation dependencies.
|
||||
- Must preserve: no profiling, startup, admin route, exporter, sidecar, eBPF,
|
||||
OTEL, or runtime behavior changes.
|
||||
- Verification: storage-api contract tests for unknown, unsupported,
|
||||
disabled, and supported capability states; focused storage-api check;
|
||||
migration guard; formatting; diff hygiene; and three-expert review.
|
||||
|
||||
- [x] `PR-09/API-014` Add topology capability contract.
|
||||
- Completed slice: add `TopologySnapshot`, `TopologyCapabilities`,
|
||||
`TopologyPool`, `TopologySet`, `TopologyDisk`, `TopologyLabels`,
|
||||
`DiskCapabilities`, and `TopologySnapshotProvider` to
|
||||
`rustfs-storage-api`.
|
||||
- Acceptance: pool, set, and disk identity fields plus optional zone, rack,
|
||||
node, media, NUMA, and additional labels are representable without
|
||||
`rustfs-ecstore`.
|
||||
- Must preserve: no ECStore endpoint/set implementation, placement,
|
||||
membership, NUMA pinning, or runtime behavior changes.
|
||||
- Verification: storage-api contract tests for missing and additional labels
|
||||
plus supported, unsupported, unknown, and disabled capability states;
|
||||
focused storage-api check; migration guard; formatting; diff hygiene; and
|
||||
three-expert review.
|
||||
- [x] `TEST-PRTYPE-001` Check PR type enum consistency.
|
||||
- Acceptance: `./scripts/check_architecture_migration_rules.sh` parses the
|
||||
allowed PR types from [`crate-boundaries.md`](crate-boundaries.md) and fails
|
||||
@@ -1296,6 +1352,87 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
migration and layer guards, formatting check, diff hygiene, risk scan, full
|
||||
pre-commit, and required three-expert review passed before push.
|
||||
|
||||
- [x] `API-051` Flatten test harness storage compatibility aliases.
|
||||
- Current branch: `overtrue/arch-test-harness-compat-aliases`.
|
||||
- Current slice: flatten e2e, heal, scanner, and fuzz storage compatibility
|
||||
harnesses from nested `storage_compat::ecstore` modules into direct
|
||||
crate-local aliases, constants, and function imports.
|
||||
- Acceptance: no e2e, heal-test, scanner-test, or fuzz-target harness file
|
||||
may expose or consume nested `storage_compat::ecstore` paths, and migration
|
||||
rules reject reintroducing nested test/fuzz ECStore compatibility modules.
|
||||
- Must preserve: e2e bucket target/RPC/disk helper imports, heal ECStore disk
|
||||
and endpoint setup, scanner lifecycle/tier/disk/storage setup, fuzz bucket
|
||||
validation behavior, and fuzz path-containment validation behavior.
|
||||
- Risk defense: this is test-harness and fuzz-harness import cleanup only; no
|
||||
production runtime behavior, ECStore ownership, storage metadata format, or
|
||||
scanner/heal lifecycle logic is changed.
|
||||
- Verification: focused e2e/heal/scanner test compile, harness tests,
|
||||
migration and layer guards, formatting check, diff hygiene, risk scan, full
|
||||
pre-commit, and required three-expert review passed before push.
|
||||
|
||||
- [x] `API-052` Flatten RustFS runtime storage compatibility aliases.
|
||||
- Current branch: `overtrue/arch-rustfs-storage-compat-aliases`.
|
||||
- Current slice: flatten RustFS root, app, admin, and storage runtime
|
||||
compatibility facades from nested `storage_compat::ecstore` modules into
|
||||
direct crate-local aliases, constants, and function imports.
|
||||
- Acceptance: no RustFS runtime source file may expose or consume nested
|
||||
`storage_compat::ecstore` paths, and migration rules reject reintroducing
|
||||
nested RustFS runtime ECStore compatibility modules.
|
||||
- Must preserve: startup/config/bootstrap behavior, server readiness checks,
|
||||
admin replication/rebalance/tier/config handlers, app object/bucket/
|
||||
multipart usecases, storage RPC/SSE/access paths, table catalog storage
|
||||
access, and existing local compatibility ownership.
|
||||
- Risk defense: this is RustFS runtime import cleanup only; no production
|
||||
runtime behavior, ECStore ownership, storage metadata format, object I/O,
|
||||
admin authorization, or readiness semantics are changed.
|
||||
- Verification: focused RustFS compile, migration and layer guards,
|
||||
formatting check, diff hygiene, risk scan, full pre-commit, and required
|
||||
three-expert review passed before push.
|
||||
|
||||
- [x] `API-053` Flatten RustFS runtime scalar storage compatibility aliases.
|
||||
- Current branch: `overtrue/arch-runtime-compat-surface-prune`.
|
||||
- Current slice: flatten RustFS root, app, admin, and storage runtime scalar
|
||||
compatibility facades such as store, error, global, endpoints, RPC,
|
||||
metrics, notification, set-disk, and data-usage paths into direct
|
||||
crate-local aliases and functions.
|
||||
- Acceptance: RustFS runtime source no longer consumes those scalar
|
||||
compatibility surfaces through secondary modules, while higher-coupling
|
||||
bucket/config/rio compatibility modules remain unchanged; migration rules
|
||||
reject restoring the flattened scalar paths.
|
||||
- Must preserve: startup config/bootstrap behavior, server readiness checks,
|
||||
admin replication/rebalance/tier/config handlers, app object/bucket/
|
||||
multipart usecases, storage RPC/SSE/access paths, table catalog storage
|
||||
access, and existing ECStore concrete type ownership.
|
||||
- Risk defense: this is import ownership and facade-shape cleanup only; no
|
||||
production runtime behavior, ECStore ownership, storage metadata format,
|
||||
object I/O, admin authorization, or readiness semantics are changed.
|
||||
- Verification: focused RustFS compile, migration and layer guards,
|
||||
formatting check, diff hygiene, risk scan, full pre-commit, and required
|
||||
three-expert review passed before push.
|
||||
|
||||
- [x] `API-054` Flatten RustFS runtime secondary storage compatibility aliases.
|
||||
- Current branch: `overtrue/arch-runtime-secondary-compat-flatten`.
|
||||
- Current slice: flatten RustFS root, app, admin, and storage runtime
|
||||
secondary compatibility modules such as bucket, config, rio, client, tier,
|
||||
compress, disk, and rebalance into direct crate-local aliases, modules, and
|
||||
functions.
|
||||
- Acceptance: RustFS runtime source no longer consumes those compatibility
|
||||
surfaces through broad secondary modules, the runtime compatibility files no
|
||||
longer define those wrapper modules, and migration rules reject restoring
|
||||
the flattened secondary paths.
|
||||
- Must preserve: startup config/bootstrap behavior, server module-switch
|
||||
config reads, embedded startup storage initialization, admin bucket/meta/
|
||||
tier/rebalance/config handlers, app object/bucket/multipart usecases,
|
||||
storage RPC/SSE/access paths, table catalog storage access, and ECStore
|
||||
concrete type ownership.
|
||||
- Risk defense: this is import ownership and facade-shape cleanup only; no
|
||||
production runtime behavior, ECStore ownership, storage metadata format,
|
||||
object I/O, admin authorization, tier behavior, or readiness semantics are
|
||||
changed.
|
||||
- Verification: focused RustFS compile, migration and layer guards,
|
||||
formatting check, diff hygiene, risk scan, and required three-expert review
|
||||
passed before push.
|
||||
|
||||
## Phase 8 Background Controller Tasks
|
||||
|
||||
- [x] `BGC-001` Inventory background services.
|
||||
@@ -1564,7 +1701,10 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
|
||||
## Next PRs
|
||||
|
||||
1. `pure-move`/`consumer-migration`: continue larger cleanup slices with the
|
||||
1. `contract`/`consumer-migration`: wire read-only observability and topology
|
||||
snapshots to implementation owners without changing runtime, profiling,
|
||||
placement, or admin route behavior.
|
||||
2. `pure-move`/`consumer-migration`: continue larger cleanup slices with the
|
||||
loss-prevention guards active for remaining ECStore compatibility contracts
|
||||
now that broad compatibility passthroughs are fully closed.
|
||||
|
||||
@@ -1572,14 +1712,65 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
|
||||
| Expert | Status | Notes |
|
||||
|---|---|---|
|
||||
| Quality/architecture | passed | S-015 removes obsolete KMS admin policy action variants after the handler fallback cleanup; API-042/API-043/API-044/API-045/API-046/API-047/API-048/API-049/API-050 narrow notify, S3 Select, OBS, IAM, Swift, heal, scanner, RustFS runtime, test, fuzz, and lifecycle helper compatibility contracts without moving ECStore storage metadata ownership. |
|
||||
| Migration preservation | passed | KMS endpoint URLs, query aliases, request bodies, response contracts, and dedicated `kms:*` authorization behavior are preserved; event builder call sites, ECStore event bridge conversion, restore event data, version IDs, metadata filtering, config read/save semantics, S3 Select store/error/buffer semantics, OBS metrics state reads, IAM config/notification/error semantics, Swift bucket metadata access, heal disk/resume/task behavior, scanner lifecycle/replication/data-usage behavior, RustFS startup/admin/app/storage runtime access, e2e/test/fuzz import behavior, lifecycle expiration/transition helper DTO field contracts, unchanged no-op handling, and remove-event behavior are preserved. |
|
||||
| Testing/verification | passed | Focused compiles/tests, guards, formatting, diff hygiene, risk scan, and full `make pre-commit` passed for the current slice. |
|
||||
| Quality/architecture | passed | S-015 removes obsolete KMS admin policy action variants after the handler fallback cleanup; API-042/API-043/API-044/API-045/API-046/API-047/API-048/API-049/API-050/API-051/API-052/API-053/API-054 narrow notify, S3 Select, OBS, IAM, Swift, heal, scanner, RustFS runtime, test, fuzz, lifecycle helper, harness, and RustFS runtime compatibility contracts without moving ECStore storage metadata ownership; G-011/G-012/G-013 add docs-only baselines for scheduler, placement/repair, and profiling/NUMA work; Issue #660 PR-08/PR-09 add read-only observability and topology contracts in rustfs-storage-api only. |
|
||||
| Migration preservation | passed | KMS endpoint URLs, query aliases, request bodies, response contracts, and dedicated `kms:*` authorization behavior are preserved; event builder call sites, ECStore event bridge conversion, restore event data, version IDs, metadata filtering, config read/save semantics, S3 Select store/error/buffer semantics, OBS metrics state reads, IAM config/notification/error semantics, Swift bucket metadata access, heal disk/resume/task behavior, scanner lifecycle/replication/data-usage behavior, RustFS startup/admin/app/storage runtime access, e2e/test/fuzz import behavior, lifecycle expiration/transition helper DTO field contracts, flattened harness and RustFS runtime scalar/secondary alias behavior, unchanged no-op handling, remove-event behavior, scheduler/readiness/placement/profiling runtime behavior, platform gates, missing/unknown capability states, and placement/topology labels are preserved. |
|
||||
| Testing/verification | passed | Focused compiles/tests, fuzz target compile, guards, formatting, diff hygiene, risk scan, and full `make pre-commit` passed for prior code slices; current Issue #660 PR-08/PR-09 contract slice uses storage-api tests/checks, migration guard, formatting, diff hygiene, and three-expert review. |
|
||||
|
||||
## Verification Notes
|
||||
|
||||
Passed before push:
|
||||
|
||||
- Issue #660 PR-08/PR-09 current slice:
|
||||
- `cargo test -p rustfs-storage-api`: passed.
|
||||
- `cargo check -p rustfs-storage-api`: passed.
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `./scripts/check_layer_dependencies.sh`: passed.
|
||||
- `cargo fmt --all --check`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- `make pre-commit`: passed.
|
||||
- Three-expert review: passed.
|
||||
|
||||
- G-011/G-012/G-013 current slice:
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- Three-expert review: passed.
|
||||
- Full `make pre-commit`: not run because this slice is documentation-only.
|
||||
|
||||
- API-054 current slice:
|
||||
- `cargo check -p rustfs --lib`: passed.
|
||||
- `cargo check --tests -p rustfs`: passed.
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `./scripts/check_layer_dependencies.sh`: passed.
|
||||
- `cargo fmt --all --check`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- Rust risk scan: passed; only existing import and path rewrites were
|
||||
reviewed, with no new unwrap/expect, panic/todo/unsafe, risky casts,
|
||||
ad-hoc error construction, or sensitive-token handling semantics.
|
||||
|
||||
- API-053 current slice:
|
||||
- `cargo check -p rustfs --lib`: passed.
|
||||
- `cargo check --tests -p rustfs`: passed.
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `./scripts/check_layer_dependencies.sh`: passed.
|
||||
- `cargo fmt --all --check`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- Rust risk scan: passed; only existing import and path rewrites were
|
||||
reviewed, with no new unwrap/expect, panic/todo/unsafe, risky casts,
|
||||
ad-hoc error construction, or sensitive-token handling semantics.
|
||||
- `make pre-commit`: passed.
|
||||
|
||||
- API-052 current slice:
|
||||
- `cargo check -p rustfs --lib`: passed.
|
||||
- `cargo check --tests -p rustfs`: passed.
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `./scripts/check_layer_dependencies.sh`: passed.
|
||||
- `cargo fmt --all --check`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- Rust risk scan: passed; only existing-semantic path replacement hits were
|
||||
reviewed, with no new unwrap/expect, panic/todo/unsafe, risky casts,
|
||||
ad-hoc error construction, or sensitive-token handling semantics.
|
||||
- `make pre-commit`: passed, including 6250 nextest tests and doctests.
|
||||
|
||||
- API-050 current slice:
|
||||
- `cargo test -p rustfs-storage-api lifecycle_helper_defaults_preserve_existing_contracts --no-fail-fast`:
|
||||
passed.
|
||||
@@ -1598,6 +1789,24 @@ Passed before push:
|
||||
lines.
|
||||
- `make pre-commit`: passed.
|
||||
|
||||
- API-051 current slice:
|
||||
- `cargo check --tests -p e2e_test -p rustfs-heal -p rustfs-scanner`:
|
||||
passed.
|
||||
- `cargo check --manifest-path fuzz/Cargo.toml --all-targets`: passed.
|
||||
- `cargo test -p rustfs-heal --test endpoint_index_test test_endpoint_index_settings --no-fail-fast`:
|
||||
passed.
|
||||
- `cargo test -p rustfs-scanner --test lifecycle_integration_test --no-run`:
|
||||
passed.
|
||||
- `cargo test -p e2e_test --no-run`: passed.
|
||||
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||
- `./scripts/check_layer_dependencies.sh`: passed.
|
||||
- `cargo fmt --all --check`: passed.
|
||||
- `git diff --check`: passed.
|
||||
- Rust risk scan: passed; only existing test `unwrap` calls were touched by
|
||||
import path rewrites, with no new unwrap/expect, panic/todo/unsafe, risky
|
||||
casts, ad-hoc error construction, or sensitive-token handling semantics.
|
||||
- `make pre-commit`: passed.
|
||||
|
||||
- S-015 current slice:
|
||||
- `cargo test -p rustfs-policy test_legacy_kms_admin_actions_are_rejected --no-fail-fast`:
|
||||
passed.
|
||||
|
||||
Reference in New Issue
Block a user