mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-29 16:37:07 +00:00
refactor: prune root bucket compat modules (#3686)
This commit is contained in:
@@ -142,6 +142,10 @@ well as remaining bitrot, erasure coding, object DTO/reader, event, list, and
|
|||||||
batch processor root modules once their facade groups exist.
|
batch processor root modules once their facade groups exist.
|
||||||
ECStore root `global` re-exports must also stay removed once consumers use
|
ECStore root `global` re-exports must also stay removed once consumers use
|
||||||
`rustfs_ecstore::api::global` or crate-internal `crate::global` paths.
|
`rustfs_ecstore::api::global` or crate-internal `crate::global` paths.
|
||||||
|
RustFS root `storage_compat.rs` must expose bucket metadata and quota contracts
|
||||||
|
as explicit aliases only. Broad `metadata`, `metadata_sys`, and `quota` module
|
||||||
|
passthroughs are reserved to narrower app/admin/storage compatibility
|
||||||
|
boundaries that still need module-local owner cleanup.
|
||||||
|
|
||||||
ECStore ClusterControlPlane read models must stay owned by the crate-private
|
ECStore ClusterControlPlane read models must stay owned by the crate-private
|
||||||
`cluster` module. Public access goes through `rustfs_ecstore::api::cluster` so
|
`cluster` module. Public access goes through `rustfs_ecstore::api::cluster` so
|
||||||
|
|||||||
@@ -5,17 +5,16 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
|||||||
## Current Context
|
## Current Context
|
||||||
|
|
||||||
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
|
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
|
||||||
- Branch: `overtrue/arch-workload-admission-provider-compose`
|
- Branch: `overtrue/arch-root-runtime-compat-boundaries`
|
||||||
- Baseline: completed `C-011/C-012/API-055/API-059`.
|
- Baseline: completed `C-011/C-012/C-013/API-055/API-059`.
|
||||||
- Stacked on: storage concurrency policy consumers.
|
- Stacked on: main after workload admission provider composition.
|
||||||
- PR type for this branch: `api-extraction`
|
- PR type for this branch: `pure-move`
|
||||||
- Runtime behavior changes: none.
|
- Runtime behavior changes: none.
|
||||||
- Rust code changes: route storage object backpressure and request hang/deadlock
|
- Rust code changes: prune broad root bucket metadata/quota compatibility module
|
||||||
runtime config consumption through shared concurrency facade policies, then
|
passthroughs into explicit aliases for the current RustFS root consumers.
|
||||||
compose RustFS workload admission through provider registries.
|
- CI/script changes: guard against restoring broad root bucket compatibility
|
||||||
- CI/script changes: none.
|
module passthroughs.
|
||||||
- Docs changes: record the C-013 admission provider composition precondition
|
- Docs changes: record the API-079 root runtime bucket compatibility boundary.
|
||||||
for later controller status work.
|
|
||||||
|
|
||||||
## Phase 0 Tasks
|
## Phase 0 Tasks
|
||||||
|
|
||||||
@@ -111,6 +110,20 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
|||||||
- Verification: workload contract tests, RustFS workload admission tests,
|
- Verification: workload contract tests, RustFS workload admission tests,
|
||||||
compile coverage, formatting, diff hygiene, risk scan, architecture guard,
|
compile coverage, formatting, diff hygiene, risk scan, architecture guard,
|
||||||
pre-commit quality gate, and three-expert review.
|
pre-commit quality gate, and three-expert review.
|
||||||
|
- [x] `API-079` Prune root runtime bucket compatibility modules.
|
||||||
|
- Completed slice: collapse RustFS root `storage_compat.rs` bucket
|
||||||
|
metadata/quota module passthroughs into explicit notification-config,
|
||||||
|
table-catalog metadata, and quota-error aliases, and guard the boundary
|
||||||
|
against broad module restores.
|
||||||
|
- Acceptance: root runtime consumers use direct compatibility aliases for
|
||||||
|
bucket notification loading, table-catalog metadata checks, and quota error
|
||||||
|
mapping, while app/admin/storage owner-local compatibility modules keep
|
||||||
|
their narrower module paths until their own cleanup slices.
|
||||||
|
- Must preserve: bucket notification loading, notifier event registration,
|
||||||
|
table-bucket mutation guards, quota error to S3 error mapping, ECStore
|
||||||
|
bucket metadata ownership, and all app/admin/storage compatibility paths.
|
||||||
|
- Verification: RustFS compile coverage, formatting, diff hygiene, risk
|
||||||
|
scan, architecture guard, pre-commit quality gate, and three-expert review.
|
||||||
- [x] `G-012` Inventory placement and repair invariants.
|
- [x] `G-012` Inventory placement and repair invariants.
|
||||||
- Acceptance:
|
- Acceptance:
|
||||||
[`placement-repair-invariants.md`](placement-repair-invariants.md) records
|
[`placement-repair-invariants.md`](placement-repair-invariants.md) records
|
||||||
@@ -3144,9 +3157,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
|||||||
|
|
||||||
| Expert | Status | Notes |
|
| Expert | Status | Notes |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Quality/architecture | passed | C-013 composes workload admission providers through the shared registry contract without adding ownership cycles or ECStore dependencies. |
|
| Quality/architecture | passed | API-079 narrows the root RustFS bucket compatibility boundary to explicit aliases without adding new ECStore ownership cycles. |
|
||||||
| Migration preservation | passed | Provider composition preserves storage foreground-read ownership, scanner activity, heal counters, replication stats, metadata runtime checks, and object/queue behavior. |
|
| Migration preservation | passed | Bucket notification loading, table-catalog metadata checks, quota error mapping, and app/admin/storage compatibility paths remain preserved. |
|
||||||
| Testing/verification | passed | Focused workload admission tests, RustFS/concurrency compile coverage, migration guard, formatting, diff hygiene, added-line risk scan, full pre-commit, and three-expert review passed. |
|
| Testing/verification | passed | RustFS compile coverage, migration guard, formatting, diff hygiene, added-line risk scan, full pre-commit, and three-expert review passed. |
|
||||||
|
|
||||||
## Verification Notes
|
## Verification Notes
|
||||||
|
|
||||||
@@ -3165,6 +3178,15 @@ Passed before push:
|
|||||||
- Rust added-line risk scan on changed Rust files: passed.
|
- Rust added-line risk scan on changed Rust files: passed.
|
||||||
- `make pre-commit`: passed.
|
- `make pre-commit`: passed.
|
||||||
|
|
||||||
|
- Issue #660 API-079 current slice:
|
||||||
|
- `cargo check -p rustfs --lib`: passed.
|
||||||
|
- `cargo fmt --all --check`: passed.
|
||||||
|
- `git diff --check`: passed.
|
||||||
|
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
|
||||||
|
- `./scripts/check_architecture_migration_rules.sh`: passed.
|
||||||
|
- Rust added-line risk scan on changed Rust files and guard script: passed.
|
||||||
|
- `make pre-commit`: passed.
|
||||||
|
|
||||||
- Issue #660 C-012 current slice:
|
- Issue #660 C-012 current slice:
|
||||||
- `cargo test -p rustfs --lib storage::backpressure::tests:: -- --nocapture`: passed.
|
- `cargo test -p rustfs --lib storage::backpressure::tests:: -- --nocapture`: passed.
|
||||||
- `cargo test -p rustfs --lib storage::deadlock_detector::tests:: -- --nocapture`: passed.
|
- `cargo test -p rustfs --lib storage::deadlock_detector::tests:: -- --nocapture`: passed.
|
||||||
|
|||||||
+1
-2
@@ -12,8 +12,7 @@
|
|||||||
// See the License for the specific language governing permissions and
|
// See the License for the specific language governing permissions and
|
||||||
// limitations under the License.
|
// limitations under the License.
|
||||||
|
|
||||||
use crate::storage_compat::StorageError;
|
use crate::storage_compat::{QuotaError, StorageError};
|
||||||
use crate::storage_compat::quota::QuotaError;
|
|
||||||
use rustfs_storage_api::HTTPRangeError;
|
use rustfs_storage_api::HTTPRangeError;
|
||||||
use s3s::{S3Error, S3ErrorCode};
|
use s3s::{S3Error, S3ErrorCode};
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -14,7 +14,7 @@
|
|||||||
|
|
||||||
use crate::server::ShutdownHandle;
|
use crate::server::ShutdownHandle;
|
||||||
use crate::storage::{process_lambda_configurations, process_queue_configurations, process_topic_configurations};
|
use crate::storage::{process_lambda_configurations, process_queue_configurations, process_topic_configurations};
|
||||||
use crate::storage_compat::metadata_sys;
|
use crate::storage_compat::get_notification_config;
|
||||||
use crate::{admin, config, version};
|
use crate::{admin, config, version};
|
||||||
use rustfs_config::{
|
use rustfs_config::{
|
||||||
DEFAULT_BUFFER_MAX_SIZE, DEFAULT_BUFFER_MIN_SIZE, DEFAULT_BUFFER_PROFILE, DEFAULT_BUFFER_UNKNOWN_SIZE, DEFAULT_UPDATE_CHECK,
|
DEFAULT_BUFFER_MAX_SIZE, DEFAULT_BUFFER_MIN_SIZE, DEFAULT_BUFFER_PROFILE, DEFAULT_BUFFER_UNKNOWN_SIZE, DEFAULT_UPDATE_CHECK,
|
||||||
@@ -174,7 +174,7 @@ pub async fn add_bucket_notification_configuration(buckets: Vec<String>) {
|
|||||||
RUSTFS_REGION
|
RUSTFS_REGION
|
||||||
});
|
});
|
||||||
for bucket in buckets.iter() {
|
for bucket in buckets.iter() {
|
||||||
let has_notification_config = metadata_sys::get_notification_config(bucket).await.unwrap_or_else(|err| {
|
let has_notification_config = get_notification_config(bucket).await.unwrap_or_else(|err| {
|
||||||
warn!(
|
warn!(
|
||||||
target: "rustfs::init",
|
target: "rustfs::init",
|
||||||
event = "notification_config_load_failed",
|
event = "notification_config_load_failed",
|
||||||
|
|||||||
@@ -12,12 +12,18 @@
|
|||||||
// See the License for the specific language governing permissions and
|
// See the License for the specific language governing permissions and
|
||||||
// limitations under the License.
|
// limitations under the License.
|
||||||
|
|
||||||
pub(crate) use rustfs_ecstore::api::bucket::metadata_sys::{get_global_bucket_metadata_sys, init_bucket_metadata_sys};
|
pub(crate) use rustfs_ecstore::api::bucket::metadata::{
|
||||||
|
BUCKET_TABLE_CATALOG_META_PREFIX, BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX, BUCKET_TABLE_CONFIG,
|
||||||
|
BUCKET_TABLE_RESERVED_PREFIX, table_catalog_path_hash,
|
||||||
|
};
|
||||||
|
pub(crate) use rustfs_ecstore::api::bucket::metadata_sys::{
|
||||||
|
get as get_bucket_metadata, get_global_bucket_metadata_sys, get_notification_config, init_bucket_metadata_sys,
|
||||||
|
};
|
||||||
pub(crate) use rustfs_ecstore::api::bucket::migration::{try_migrate_bucket_metadata, try_migrate_iam_config};
|
pub(crate) use rustfs_ecstore::api::bucket::migration::{try_migrate_bucket_metadata, try_migrate_iam_config};
|
||||||
|
pub(crate) use rustfs_ecstore::api::bucket::quota::QuotaError;
|
||||||
pub(crate) use rustfs_ecstore::api::bucket::replication::{
|
pub(crate) use rustfs_ecstore::api::bucket::replication::{
|
||||||
GLOBAL_REPLICATION_STATS, get_global_replication_pool, init_background_replication,
|
GLOBAL_REPLICATION_STATS, get_global_replication_pool, init_background_replication,
|
||||||
};
|
};
|
||||||
pub(crate) use rustfs_ecstore::api::bucket::{metadata, metadata_sys, quota};
|
|
||||||
pub(crate) use rustfs_ecstore::api::cluster::topology_snapshot_from_endpoint_pools_with_capabilities;
|
pub(crate) use rustfs_ecstore::api::cluster::topology_snapshot_from_endpoint_pools_with_capabilities;
|
||||||
pub(crate) use rustfs_ecstore::api::config::{com, init, init_global_config_sys, try_migrate_server_config};
|
pub(crate) use rustfs_ecstore::api::config::{com, init, init_global_config_sys, try_migrate_server_config};
|
||||||
pub(crate) use rustfs_ecstore::api::disk::{DiskAPI, RUSTFS_META_BUCKET, endpoint::Endpoint};
|
pub(crate) use rustfs_ecstore::api::disk::{DiskAPI, RUSTFS_META_BUCKET, endpoint::Endpoint};
|
||||||
|
|||||||
@@ -29,13 +29,9 @@ use std::{
|
|||||||
|
|
||||||
use crate::storage_compat::RUSTFS_META_BUCKET;
|
use crate::storage_compat::RUSTFS_META_BUCKET;
|
||||||
use crate::storage_compat::get_lock_acquire_timeout;
|
use crate::storage_compat::get_lock_acquire_timeout;
|
||||||
use crate::storage_compat::{EcstoreError, StorageError};
|
|
||||||
use crate::storage_compat::{
|
use crate::storage_compat::{
|
||||||
metadata::{
|
BUCKET_TABLE_CATALOG_META_PREFIX, BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX, BUCKET_TABLE_CONFIG,
|
||||||
BUCKET_TABLE_CATALOG_META_PREFIX, BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX, BUCKET_TABLE_CONFIG,
|
BUCKET_TABLE_RESERVED_PREFIX, EcstoreError, StorageError, get_bucket_metadata, table_catalog_path_hash,
|
||||||
BUCKET_TABLE_RESERVED_PREFIX, table_catalog_path_hash,
|
|
||||||
},
|
|
||||||
metadata_sys,
|
|
||||||
};
|
};
|
||||||
use bytes::Bytes;
|
use bytes::Bytes;
|
||||||
use datafusion::{
|
use datafusion::{
|
||||||
@@ -7007,7 +7003,7 @@ pub fn validate_object_mutation(table_bucket_enabled: bool, object_key: &str) ->
|
|||||||
}
|
}
|
||||||
|
|
||||||
pub(crate) async fn validate_bucket_object_mutation(bucket: &str, object_key: &str) -> Result<(), TableObjectMutationError> {
|
pub(crate) async fn validate_bucket_object_mutation(bucket: &str, object_key: &str) -> Result<(), TableObjectMutationError> {
|
||||||
let table_bucket_enabled = metadata_sys::get(bucket)
|
let table_bucket_enabled = get_bucket_metadata(bucket)
|
||||||
.await
|
.await
|
||||||
.is_ok_and(|metadata| metadata.table_bucket_enabled());
|
.is_ok_and(|metadata| metadata.table_bucket_enabled());
|
||||||
|
|
||||||
|
|||||||
@@ -78,6 +78,7 @@ TEST_HARNESS_NESTED_STORAGE_COMPAT_HITS_FILE="${TMP_DIR}/test_harness_nested_sto
|
|||||||
RUSTFS_NESTED_STORAGE_COMPAT_HITS_FILE="${TMP_DIR}/rustfs_nested_storage_compat_hits.txt"
|
RUSTFS_NESTED_STORAGE_COMPAT_HITS_FILE="${TMP_DIR}/rustfs_nested_storage_compat_hits.txt"
|
||||||
RUSTFS_RUNTIME_SCALAR_STORAGE_COMPAT_HITS_FILE="${TMP_DIR}/rustfs_runtime_scalar_storage_compat_hits.txt"
|
RUSTFS_RUNTIME_SCALAR_STORAGE_COMPAT_HITS_FILE="${TMP_DIR}/rustfs_runtime_scalar_storage_compat_hits.txt"
|
||||||
RUSTFS_RUNTIME_SECONDARY_STORAGE_COMPAT_HITS_FILE="${TMP_DIR}/rustfs_runtime_secondary_storage_compat_hits.txt"
|
RUSTFS_RUNTIME_SECONDARY_STORAGE_COMPAT_HITS_FILE="${TMP_DIR}/rustfs_runtime_secondary_storage_compat_hits.txt"
|
||||||
|
RUSTFS_ROOT_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_root_bucket_storage_compat_module_hits.txt"
|
||||||
PRODUCTION_UNUSED_COMPAT_ALLOW_HITS_FILE="${TMP_DIR}/production_unused_compat_allow_hits.txt"
|
PRODUCTION_UNUSED_COMPAT_ALLOW_HITS_FILE="${TMP_DIR}/production_unused_compat_allow_hits.txt"
|
||||||
BROAD_STORE_API_COMPAT_REEXPORT_HITS_FILE="${TMP_DIR}/broad_store_api_compat_reexport_hits.txt"
|
BROAD_STORE_API_COMPAT_REEXPORT_HITS_FILE="${TMP_DIR}/broad_store_api_compat_reexport_hits.txt"
|
||||||
NESTED_STORE_API_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/nested_store_api_compat_module_hits.txt"
|
NESTED_STORE_API_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/nested_store_api_compat_module_hits.txt"
|
||||||
@@ -765,6 +766,16 @@ if [[ -s "$RUSTFS_RUNTIME_SECONDARY_STORAGE_COMPAT_HITS_FILE" ]]; then
|
|||||||
report_failure "RustFS runtime secondary storage compatibility paths must use direct aliases instead of bucket/config/rio/client/tier/compress/disk/rebalance modules: $(paste -sd '; ' "$RUSTFS_RUNTIME_SECONDARY_STORAGE_COMPAT_HITS_FILE")"
|
report_failure "RustFS runtime secondary storage compatibility paths must use direct aliases instead of bucket/config/rio/client/tier/compress/disk/rebalance modules: $(paste -sd '; ' "$RUSTFS_RUNTIME_SECONDARY_STORAGE_COMPAT_HITS_FILE")"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
(
|
||||||
|
cd "$ROOT_DIR"
|
||||||
|
rg -n --no-heading 'pub\(crate\)\s+use rustfs_ecstore::api::bucket::\{[^}]*\b(?:metadata|metadata_sys|quota)\b[^}]*\}\s*;|pub\(crate\)\s+use rustfs_ecstore::api::bucket::(?:metadata|metadata_sys|quota)\s*;' \
|
||||||
|
rustfs/src/storage_compat.rs || true
|
||||||
|
) >"$RUSTFS_ROOT_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE"
|
||||||
|
|
||||||
|
if [[ -s "$RUSTFS_ROOT_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE" ]]; then
|
||||||
|
report_failure "RustFS root storage compatibility must expose bucket metadata/quota contracts as explicit aliases: $(paste -sd '; ' "$RUSTFS_ROOT_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE")"
|
||||||
|
fi
|
||||||
|
|
||||||
(
|
(
|
||||||
cd "$ROOT_DIR"
|
cd "$ROOT_DIR"
|
||||||
rg -n --no-heading '#!\[allow\(unused_imports\)\]' \
|
rg -n --no-heading '#!\[allow\(unused_imports\)\]' \
|
||||||
|
|||||||
Reference in New Issue
Block a user