gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-07 05:43:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor: prune consumer ecstore object aliases (#3622)

Browse Source
This commit is contained in:
安正超
2026-06-19 21:33:00 +08:00
committed by GitHub
parent 08b8f58e64
commit afe46c01e7
13 changed files with 169 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/migration-progress.md
+70 -15
View File
@@ -5,16 +5,16 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
## Current Context
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
- Branch: `overtrue/arch-iam-object-boundary-prune`
- Baseline: stacked after `rustfs/rustfs#3620`
(`c7bee5de904c9a1ea70f73ad6621895a30b3c387`).
- Branch: `overtrue/arch-final-object-boundary-prune`
- Baseline: stacked after `rustfs/rustfs#3622`
(`4300bf04f3a82ef4cc2605f417f89e7b4643b99e`).
- PR type for this branch: `consumer-migration`
- Runtime behavior changes: none.
- Rust code changes: route IAM config/store object metadata and options aliases
through `rustfs_storage_api::ObjectOperations` associated types.
- Rust code changes: route heal and RustFS storage object reader, writer,
metadata, and options aliases through `rustfs_storage_api` associated types.
- CI/script changes: shrink the remaining external `rustfs_ecstore::object_api`
compatibility alias snapshot by removing IAM object-info/options aliases.
- Docs changes: record the API-069 IAM object boundary prune slice.
compatibility alias snapshot to empty.
- Docs changes: record the API-071 final object boundary prune slice.
## Phase 0 Tasks
@@ -360,6 +360,36 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
- Verification: focused IAM compile/tests, migration and layer guards,
formatting, diff hygiene, full pre-commit, and three-expert review.
- [x] `API-070` Prune consumer direct ECStore object aliases.
- Completed slice: replace scanner, s3select, and Swift
`GetObjectReader`/`ObjectInfo`/`ObjectOptions`/`PutObjReader`
compatibility aliases with concrete store `rustfs_storage_api` associated
types.
- Acceptance: scanner, s3select, and Swift no longer name
`rustfs_ecstore::object_api::{GetObjectReader,ObjectInfo,ObjectOptions,PutObjReader}`
directly, and the remaining object-api alias allowlist shrinks by eleven
entries.
- Must preserve: scanner lifecycle/replication IO bounds and config helpers,
s3select read buffer/object error handling, Swift bucket metadata helpers,
and object reader/writer concrete types exposed through each local
compatibility boundary.
- Verification: focused consumer compile/tests, migration and layer guards,
formatting, diff hygiene, full pre-commit, and three-expert review.
- [x] `API-071` Prune final direct ECStore object aliases.
- Completed slice: replace heal and RustFS storage
`GetObjectReader`/`ObjectInfo`/`ObjectOptions`/`PutObjReader`
compatibility aliases with concrete store `rustfs_storage_api` associated
types.
- Acceptance: no external `storage_compat.rs` module names
`rustfs_ecstore::object_api::{GetObjectReader,ObjectInfo,ObjectOptions,PutObjReader}`
directly, and the external object-api alias allowlist is empty.
- Must preserve: heal object metadata and rewrite reader construction,
RustFS storage object read/write paths, S3 response metadata semantics,
SSE/encryption handling, and storage object option behavior.
- Verification: focused heal/storage compile/tests, migration and layer
guards, formatting, diff hygiene, full pre-commit, and three-expert review.
- [x] `TEST-PRTYPE-001` Check PR type enum consistency.
- Acceptance: `./scripts/check_architecture_migration_rules.sh` parses the
allowed PR types from [`crate-boundaries.md`](crate-boundaries.md) and fails
@@ -1940,27 +1970,52 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
binary/lib compile checks, formatting, migration guards, Rust risk scan,
branch freshness check, and pre-commit quality gate.
- [x] `R-020` Isolate profiling lifecycle hooks.
- Do: route BOOT-006 profiling initialization and STOP-004 profiling shutdown
through `startup_profiling` hook functions while keeping `profiling.rs` as
the CPU/memory profiling implementation and admin dump API owner.
- Acceptance: startup still initializes profiling before trusted proxies and
outbound TLS material; shutdown still stops profiling after notifier/audit
shutdown and before HTTP shutdown; unsupported targets and disabled
profiling keep their existing no-op behavior.
- Must preserve: profiling env flags, CPU/memory mode handling, target gates,
cancellation-token ownership, admin pprof routes, non-fatal startup
behavior, and shutdown ordering.
- Verification: focused startup profiling hook tests, binary/lib compile
checks, formatting, migration guards, Rust risk scan, branch freshness
check, and pre-commit quality gate.
## Next PRs
1. `consumer-migration`: connect additional scanner, repair, replication, and
metadata admission owners to the workload registry only after each owner has
dedicated preservation coverage.
2. `pure-move`/`consumer-migration`: continue larger cleanup slices with the
1. `pure-move`: continue larger lifecycle hook slices for optional runtime
sidecars while preserving startup and shutdown ordering.
2. `consumer-migration`: continue larger cleanup slices with the
loss-prevention guards active for remaining ECStore compatibility contracts
now that broad compatibility passthroughs are fully closed.
that still have dedicated preservation coverage.
## Pre-Push Review Log
| Expert | Status | Notes |
|---|---|---|
| Quality/architecture | passed | S-015 removes obsolete KMS admin policy action variants after the handler fallback cleanup; API-042/API-043/API-044/API-045/API-046/API-047/API-048/API-049/API-050/API-051/API-052/API-053/API-054 narrow notify, S3 Select, OBS, IAM, Swift, heal, scanner, RustFS runtime, test, fuzz, lifecycle helper, harness, and RustFS runtime compatibility contracts without moving ECStore storage metadata ownership; G-011/G-012/G-013 add docs-only baselines for scheduler, placement/repair, and profiling/NUMA work; Issue #660 PR-08/PR-09 add read-only observability and topology contracts in storage-api only; PR-05/PR-07 add scheduler preservation tests and workload contracts; API-055/SCH-001 adds a local storage concurrency provider; current API-056/R-016 wires runtime observability and endpoint topology providers without moving implementation ownership. |
| Migration preservation | passed | KMS endpoint URLs, query aliases, request bodies, response contracts, and dedicated `kms:*` authorization behavior are preserved; event builder call sites, ECStore event bridge conversion, restore event data, version IDs, metadata filtering, config read/save semantics, S3 Select store/error/buffer semantics, OBS metrics state reads, IAM config/notification/error semantics, Swift bucket metadata access, heal disk/resume/task behavior, scanner lifecycle/replication/data-usage behavior, RustFS startup/admin/app/storage runtime access, e2e/test/fuzz import behavior, lifecycle expiration/transition helper DTO field contracts, flattened harness and RustFS runtime scalar/secondary alias behavior, unchanged no-op handling, remove-event behavior, scheduler/readiness/placement/profiling runtime behavior, platform gates, missing/unknown capability states, placement/topology labels, scheduler thresholds, queue snapshot semantics, disk-read semaphore behavior, and admission behavior are preserved. |
| Testing/verification | passed | Focused compiles/tests, fuzz target compile, guards, formatting, diff hygiene, risk scan, and full `make pre-commit` passed for prior code slices; current Issue #660 API-056/R-016 slice uses runtime capability provider tests, focused RustFS library check, migration and layer guards, formatting, diff hygiene, and three-expert review. |
| Quality/architecture | passed | R-020 moves profiling init/shutdown call sites behind `startup_profiling` hook functions without moving CPU/memory profiling implementation ownership or admin pprof APIs. |
| Migration preservation | passed | BOOT-006 still initializes profiling before trusted proxies/provider/TLS setup; STOP-004 still stops profiling after notifier/audit shutdown and before HTTP shutdown; profiling env flags, platform gates, target unsupported behavior, and cancellation-token ownership remain in `profiling.rs`. |
| Testing/verification | passed | Focused startup profiling hook tests, RustFS library check, migration/layer guards, formatting, diff hygiene, Rust risk scan, and full `make pre-commit` validate the current R-020 slice. |
## Verification Notes
Passed before push:
- Issue #660 R-020 current slice:
- `cargo test -p rustfs --lib startup_profiling -- --nocapture`: passed.
- `cargo check -p rustfs --lib`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- Rust risk scan on changed Rust files: passed.
- `make pre-commit`: passed.
- Three-expert review: passed.
- Issue #660 API-056/R-016 current slice:
- `cargo test -p rustfs --lib runtime_capabilities -- --nocapture`: passed.
- `cargo check -p rustfs --lib`: passed.
docs/architecture/profiling-numa-capability-inventory.md
+3
View File
@@ -24,6 +24,9 @@ sidecars are designed.
The current safe baseline is:
- Profiling is opt-in through env flags and must not make startup fatal.
- Startup and shutdown call profiling through `startup_profiling` lifecycle
hooks; `profiling.rs` remains the CPU/memory profiling implementation and
admin dump API owner.
- Unsupported profiling targets return structured unsupported errors or skip
startup tasks.
- Memory observability records process/system metrics and adds cgroup split
docs/architecture/startup-timeline.md
+2 -2
View File
@@ -23,7 +23,7 @@ new startup semantics.
| `BOOT-003` | `rustfs/src/startup_entrypoint.rs` | Parse CLI command and dispatch non-server commands. | `info` and `tls` commands execute and return without server startup. | Command parse exits process with code 1; TLS command errors propagate. | None |
| `BOOT-004` | `rustfs/src/startup_preflight.rs` | Initialize config snapshot and license state. | Publishes config snapshot for later readers and initializes runtime license state. | License init is non-fallible in this path. | None |
| `BOOT-005` | `rustfs/src/startup_preflight.rs` | Initialize observability and store the global guard. | Initializes tracing/observability, stores the guard globally, and logs license/runtime telemetry status. | Fatal if observability init or guard publication fails. | None |
| `BOOT-006` | `rustfs/src/startup_runtime.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None |
| `BOOT-006` | `rustfs/src/startup_runtime.rs` and `rustfs/src/startup_profiling.rs` | Log startup logo, initialize profiling, trusted proxies, rustls provider, and outbound TLS material. | Starts optional profiling tasks, trusted proxy config, default rustls provider, outbound TLS global state, TLS generation metric, and TLS metrics when enabled. | Profiling/proxy/provider setup is non-fatal; configured TLS material load is fatal on error. | None |
| `RUN-001` | `rustfs/src/startup_server.rs` | Enter startup run orchestration and create `GlobalReadiness`. | Allocates the readiness tracker shared with HTTP readiness gates. | Non-fatal. | Initial readiness state is not ready |
| `RUN-002` | `rustfs/src/startup_server.rs` | Parse and publish the configured region. | Updates ECStore global region when configured. | Fatal if the configured region is invalid. | None |
| `RUN-003` | `rustfs/src/startup_server.rs` | Resolve server address and warn on default credentials. | Computes server port/address and emits production credential warning when defaults are used. | Address parse is fatal; default credentials warning is non-fatal. | None |
@@ -72,7 +72,7 @@ new startup semantics.
| `STOP-001` | `rustfs/src/startup_services.rs` | Cancel runtime token and move service state to `Stopping`. | Notifies cancellation-aware background tasks. | Non-fatal. | Service state moves to `Stopping`; readiness stages are not cleared here |
| `STOP-002` | `rustfs/src/startup_services.rs` | Stop scanner/background services and AHM services according to enable flags. | Calls ECStore background shutdown and heal/scanner shutdown helpers. | Non-fatal in this path. | No readiness-stage change |
| `STOP-003` | `rustfs/src/startup_services.rs` | Signal optional FTP/FTPS/WebDAV/SFTP protocol servers. | Collects protocol shutdown futures. | Non-fatal in this path. | No readiness-stage change |
| `STOP-004` | `rustfs/src/startup_services.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change |
| `STOP-004` | `rustfs/src/startup_services.rs` and `rustfs/src/startup_profiling.rs` | Stop event notifier, audit system, and profiling tasks. | Stops notifier and profiling tasks; audit stop failures are logged. | Non-fatal in this path. | No readiness-stage change |
| `STOP-005` | `rustfs/src/startup_services.rs` | Stop S3 and console HTTP servers, wait for protocol shutdowns, then mark service state `Stopped`. | HTTP shutdown happens after notifier/audit/profiling shutdown in current order. | Join failures are logged by shutdown handles; this path does not return errors. | Service state moves to `Stopped`; readiness stages are not cleared here |
## Migration Rules