refactor: retire root storage compat facade (#3708)

This commit is contained in:
安正超
2026-06-22 07:06:08 +08:00
committed by GitHub
parent c13f42e9a0
commit 1b718498f4
35 changed files with 523 additions and 297 deletions
+92 -10
View File
@@ -5,17 +5,17 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
## Current Context
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
- Branch: `overtrue/arch-disk-rpc-method-wrappers`
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096`.
- Stacked on: `overtrue/arch-compat-trait-method-wrappers` pending API-096 merge.
- Branch: `overtrue/arch-runtime-compat-consumer-batch`
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100`.
- Stacked on: `overtrue/arch-disk-rpc-method-wrappers` pending API-097 merge.
- PR type for this branch: `pure-move`
- Runtime behavior changes: none.
- Rust code changes: move remaining disk RPC, peer S3 RPC, heal/scanner disk,
and warm-backend test method access behind local compatibility traits or
aliases.
- CI/script changes: remove the final direct ECStore method-resolution import
exceptions outside compatibility boundaries.
- Docs changes: record the API-097 disk/RPC/warm-backend wrapper cleanup.
- Rust code changes: move capacity, server, startup, table catalog,
runtime-capability, workload-admission, error, and config-test ECStore
compatibility consumers from the root runtime facade into local boundaries.
- CI/script changes: guard against restoring consumer wrappers in the root
runtime compatibility facade and tolerate retiring the empty root facade.
- Docs changes: record the API-098/API-099/API-100 runtime consumer boundary cleanup.
## Phase 0 Tasks
@@ -377,7 +377,7 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
import residual scan, migration guard, formatting, diff hygiene, Rust risk
scan, pre-commit quality gate, and three-expert review.
- [x] `API-097` Prune disk/RPC/warm-backend method imports.
- Current slice: move disk RPC, peer S3 RPC, heal/scanner disk, and
- Completed slice: move disk RPC, peer S3 RPC, heal/scanner disk, and
warm-backend test method access behind local compatibility traits or
aliases in the owning boundaries.
- Acceptance: non-compat RustFS, scanner, heal, and test sources no longer
@@ -390,6 +390,51 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
disk/RPC/warm-backend trait import residual scan, migration guard,
formatting, diff hygiene, Rust risk scan, pre-commit quality gate, and
three-expert review.
- [x] `API-098` Prune root runtime capacity/server compat consumers.
- Completed slice: move capacity disk access, HTTP RPC signature verification,
event dispatch bridging, module-switch config persistence, and readiness
storage/lock quorum lookups into local `capacity` and `server`
compatibility boundaries.
- Acceptance: root runtime `storage_compat.rs` no longer owns
capacity/server-only ECStore wrapper functions, trait shims, or constants;
migration rules reject restoring those wrappers to the root facade.
- Must preserve: capacity background refresh disk discovery, internode RPC
signature verification, live event dispatch, module-switch persistence,
storage readiness, and distributed lock quorum behavior.
- Verification: RustFS test-target compile coverage, capacity/server residual
scan, migration and layer guards, formatting, diff hygiene, Rust risk scan,
pre-commit quality gate, and three-expert review.
- [x] `API-099` Prune root runtime startup compat consumers.
- Completed slice: move startup storage bootstrap, bucket metadata migration,
notification initialization, global region/port setup, background shutdown,
and startup service ECStore aliases into a dedicated startup compatibility
boundary.
- Acceptance: startup and init modules no longer consume root
`storage_compat.rs`; root runtime `storage_compat.rs` no longer owns
startup-only ECStore wrapper functions or aliases; migration rules reject
restoring those wrappers to the root facade.
- Must preserve: endpoint parsing, unsupported filesystem policy,
local-disk and lock-client initialization, global config migration,
bucket metadata migration, IAM migration, notification registration,
default-region fallback, background replication, and shutdown behavior.
- Verification: RustFS test-target compile coverage, startup residual scan,
migration and layer guards, formatting, diff hygiene, Rust risk scan,
pre-commit quality gate, and three-expert review.
- [x] `API-100` Retire root runtime storage compatibility consumers.
- Completed slice: move table catalog metadata constants and bucket metadata
reads, runtime topology capability mapping, workload admission runtime
state probes, S3 error mapping aliases, and config test disk-layout aliases
into local compatibility boundaries, then remove the root
`storage_compat.rs` module.
- Acceptance: no RustFS source consumes `crate::storage_compat`; root
runtime compatibility file is removed; migration rules still reject direct
ECStore imports outside `*storage_compat.rs` boundaries.
- Must preserve: table catalog internal metadata paths, lock timeout lookup,
runtime topology snapshots, workload admission status reporting, quota and
storage error mapping, and config disk-layout parsing tests.
- Verification: RustFS test-target compile coverage, direct root compatibility
consumer residual scan, migration and layer guards, formatting, diff
hygiene, Rust risk scan, pre-commit quality gate, and three-expert review.
- [x] `G-012` Inventory placement and repair invariants.
- Acceptance:
[`placement-repair-invariants.md`](placement-repair-invariants.md) records
@@ -3431,6 +3476,43 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
Passed before push:
- Issue #660 API-098 current slice:
- `cargo check -p rustfs --tests`: passed.
- `cargo fmt --all`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- Direct root capacity/server compatibility consumer residual scan: passed.
- Rust risk scan on changed Rust files and guard script: passed.
- `make pre-commit`: passed.
- Issue #660 API-099 current slice:
- `cargo check -p rustfs --tests`: passed.
- `cargo fmt --all`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- Startup/init root compatibility consumer residual scan: passed.
- Root startup consumer wrapper residual scan: passed.
- Rust risk scan on changed Rust files and guard script: passed.
- `make pre-commit`: passed.
- Issue #660 API-100 current slice:
- `cargo check -p rustfs --tests`: passed.
- `cargo fmt --all`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `bash -n scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- Direct root compatibility consumer residual scan: passed.
- Rust risk scan on changed Rust files and guard script: passed.
- `make pre-commit`: passed.
- Issue #660 API-097 current slice:
- `cargo check -p rustfs -p rustfs-scanner -p rustfs-heal -p e2e_test --tests`: passed.
- `cargo fmt --all --check`: passed.
+1
View File
@@ -52,6 +52,7 @@
pub mod capacity_integration;
pub mod service;
mod storage_compat;
pub use service::{
capacity_disk_ref, get_cached_capacity_with_metrics, init_capacity_management_for_local_disks, record_capacity_write,
+2 -2
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage_compat::DiskAPI;
use super::storage_compat::CapacityDiskExt as _;
use rustfs_io_metrics::capacity_metrics::{
record_capacity_cache_hit, record_capacity_cache_miss, record_capacity_cache_served, record_capacity_refresh_request,
record_capacity_scan_mode,
@@ -263,7 +263,7 @@ pub async fn init_capacity_management_for_local_disks() {
"Capacity manager state changed"
);
let disks = crate::storage_compat::all_local_disk().await;
let disks = super::storage_compat::all_local_disk().await;
if disks.is_empty() {
warn!(
component = LOG_COMPONENT_CAPACITY,
+39
View File
@@ -0,0 +1,39 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_ecstore::api::{disk as ecstore_disk, storage as ecstore_storage};
pub(crate) type Endpoint = ecstore_disk::endpoint::Endpoint;
pub(crate) trait CapacityDiskExt {
fn endpoint(&self) -> Endpoint;
fn to_string(&self) -> String;
}
impl<T> CapacityDiskExt for T
where
T: ecstore_disk::DiskAPI,
{
fn endpoint(&self) -> Endpoint {
ecstore_disk::DiskAPI::endpoint(self)
}
fn to_string(&self) -> String {
ecstore_disk::DiskAPI::to_string(self)
}
}
pub(crate) async fn all_local_disk() -> Vec<ecstore_disk::DiskStore> {
ecstore_storage::all_local_disk().await
}
+2 -2
View File
@@ -16,7 +16,7 @@
#[allow(unsafe_op_in_unsafe_fn)]
mod tests {
use crate::config::{CommandResult, Config, Opt, TlsCommands};
use crate::storage_compat::DisksLayout;
use crate::config_storage_compat::DisksLayout;
use rustfs_config::{DEFAULT_CONSOLE_ADDRESS, DEFAULT_CONSOLE_ENABLE, DEFAULT_OBS_ENDPOINT, RUSTFS_REGION};
use rustfs_credentials::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY};
use serial_test::serial;
@@ -262,7 +262,7 @@ mod tests {
#[test]
#[serial]
fn test_volumes_and_disk_layout_parsing() {
use crate::storage_compat::DisksLayout;
use crate::config_storage_compat::DisksLayout;
// Test case 1: Single volume path
let args = vec!["rustfs", "/data/vol1"];
+19
View File
@@ -0,0 +1,19 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(test)]
use rustfs_ecstore::api::layout as ecstore_layout;
#[cfg(test)]
pub(crate) type DisksLayout = ecstore_layout::DisksLayout;
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage_compat::{QuotaError, StorageError};
use crate::error_storage_compat::{QuotaError, StorageError};
use rustfs_storage_api::HTTPRangeError;
use s3s::{S3Error, S3ErrorCode};
+18
View File
@@ -0,0 +1,18 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_ecstore::api::{bucket as ecstore_bucket, error as ecstore_error};
pub(crate) type QuotaError = ecstore_bucket::quota::QuotaError;
pub(crate) type StorageError = ecstore_error::StorageError;
+2 -2
View File
@@ -13,8 +13,8 @@
// limitations under the License.
use crate::server::ShutdownHandle;
use crate::startup_storage_compat::{get_global_region, get_notification_config};
use crate::storage::{process_lambda_configurations, process_queue_configurations, process_topic_configurations};
use crate::storage_compat::get_notification_config;
use crate::{admin, config, version};
use rustfs_config::{
DEFAULT_BUFFER_MAX_SIZE, DEFAULT_BUFFER_MIN_SIZE, DEFAULT_BUFFER_PROFILE, DEFAULT_BUFFER_UNKNOWN_SIZE, DEFAULT_UPDATE_CHECK,
@@ -157,7 +157,7 @@ fn arn_to_target_id(arn_str: &str) -> Result<rustfs_targets::arn::TargetID, Targ
/// * `buckets` - A vector of bucket names to process
#[instrument(skip_all)]
pub async fn add_bucket_notification_configuration(buckets: Vec<String>) {
let global_region = crate::storage_compat::get_global_region();
let global_region = get_global_region();
let region = global_region
.as_ref()
.filter(|r| !r.as_str().is_empty())
+6 -1
View File
@@ -57,9 +57,11 @@ pub mod auth;
pub mod auth_keystone;
pub mod capacity;
pub mod config;
pub(crate) mod config_storage_compat;
pub mod delete_tail_activity;
pub mod embedded;
pub mod error;
pub(crate) mod error_storage_compat;
pub mod init;
pub mod license;
pub mod memory_observability;
@@ -67,6 +69,7 @@ pub mod profiling;
#[cfg(any(feature = "ftps", feature = "webdav", feature = "sftp"))]
pub mod protocols;
pub mod runtime_capabilities;
pub(crate) mod runtime_capabilities_storage_compat;
pub mod server;
pub(crate) mod startup_audit;
pub(crate) mod startup_auth;
@@ -90,14 +93,16 @@ pub(crate) mod startup_server;
pub(crate) mod startup_services;
pub(crate) mod startup_shutdown;
pub(crate) mod startup_storage;
pub(crate) mod startup_storage_compat;
pub(crate) mod startup_tls_material;
pub mod storage;
pub(crate) mod storage_compat;
pub(crate) mod table_catalog;
pub(crate) mod table_catalog_storage_compat;
pub mod tls;
pub mod update;
pub mod version;
pub mod workload_admission;
pub(crate) mod workload_admission_storage_compat;
// Re-export from rustfs_utils so that config sub-modules can use
// `crate::apply_external_env_compat` without breaking.
+3 -3
View File
@@ -18,7 +18,7 @@ use rustfs_storage_api::{
UserspaceProfilingCapability,
};
use crate::storage_compat::EndpointServerPools;
use crate::runtime_capabilities_storage_compat::{EndpointServerPools, topology_snapshot_from_endpoint_pools_with_capabilities};
const NOT_WIRED_INTO_RUNTIME: &str = "not wired into runtime";
const STORAGE_MEDIA_NOT_REPORTED: &str = "storage media not reported by endpoints";
@@ -80,7 +80,7 @@ impl TopologySnapshotProvider for EndpointTopologySnapshotProvider {
}
pub fn topology_snapshot_from_endpoint_pools(endpoint_pools: &EndpointServerPools) -> TopologySnapshot {
crate::storage_compat::topology_snapshot_from_endpoint_pools_with_capabilities(
topology_snapshot_from_endpoint_pools_with_capabilities(
endpoint_pools,
TopologyCapabilities {
profiling: cpu_profiling_status(),
@@ -132,7 +132,7 @@ fn cgroup_memory_status() -> CapabilityStatus {
#[cfg(test)]
mod tests {
use super::*;
use crate::storage_compat::{Endpoint, Endpoints, PoolEndpoints};
use crate::runtime_capabilities_storage_compat::{Endpoint, Endpoints, PoolEndpoints};
use rustfs_storage_api::{CapabilityState, ObservabilitySnapshotProvider, TopologySnapshotProvider};
#[tokio::test]
@@ -0,0 +1,33 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(test)]
use rustfs_ecstore::api::disk as ecstore_disk;
use rustfs_ecstore::api::{cluster as ecstore_cluster, layout as ecstore_layout};
#[cfg(test)]
pub(crate) type Endpoint = ecstore_disk::endpoint::Endpoint;
pub(crate) type EndpointServerPools = ecstore_layout::EndpointServerPools;
#[cfg(test)]
pub(crate) type Endpoints = ecstore_layout::Endpoints;
#[cfg(test)]
pub(crate) type PoolEndpoints = ecstore_layout::PoolEndpoints;
pub(crate) fn topology_snapshot_from_endpoint_pools_with_capabilities(
endpoint_pools: &EndpointServerPools,
capabilities: rustfs_storage_api::TopologyCapabilities,
disk_capabilities: rustfs_storage_api::DiskCapabilities,
) -> rustfs_storage_api::TopologySnapshot {
ecstore_cluster::topology_snapshot_from_endpoint_pools_with_capabilities(endpoint_pools, capabilities, disk_capabilities)
}
+1 -1
View File
@@ -12,10 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::storage_compat::{EcstoreEventArgs, register_event_dispatch_hook};
use super::{module_switch::resolve_notify_module_state, refresh_persisted_module_switches_from_store};
use crate::app::context::resolve_server_config;
use crate::storage::StorageObjectInfo;
use crate::storage_compat::{EcstoreEventArgs, register_event_dispatch_hook};
use chrono::{DateTime, Utc};
use rustfs_notify::{EventArgs as NotifyEventArgs, NotifyObjectInfo};
use rustfs_s3_types::EventName;
+1 -1
View File
@@ -13,6 +13,7 @@
// limitations under the License.
// Import HTTP server components and compression configuration
use super::storage_compat::{TONIC_RPC_PREFIX, verify_rpc_signature};
use crate::admin;
use crate::auth::IAMAuth;
use crate::auth_keystone;
@@ -33,7 +34,6 @@ use crate::server::{
use crate::storage;
use crate::storage::rpc::InternodeRpcService;
use crate::storage::tonic_service::make_server;
use crate::storage_compat::{TONIC_RPC_PREFIX, verify_rpc_signature};
use bytes::Bytes;
use http::{HeaderMap, Method, Request as HttpRequest, Response};
use hyper_util::{
+1
View File
@@ -24,6 +24,7 @@ mod prefix;
mod readiness;
mod runtime;
mod service_state;
mod storage_compat;
pub mod tls_material;
use tracing::warn;
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage_compat::{
use super::storage_compat::{
EcstoreError as StorageError, read_ecstore_config, resolve_object_store_handle, save_ecstore_config,
};
use serde::{Deserialize, Serialize};
+9 -13
View File
@@ -12,11 +12,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::storage_compat::{
Endpoint, EndpointServerPools, get_global_endpoints_opt, get_global_lock_clients, is_dist_erasure,
resolve_object_store_handle,
};
use crate::server::{ServiceState, ServiceStateManager};
use crate::server::{has_path_prefix, is_table_catalog_path};
use crate::storage_compat::is_dist_erasure;
use crate::storage_compat::resolve_object_store_handle;
use crate::storage_compat::{get_global_endpoints_opt, get_global_lock_clients};
use bytes::Bytes;
use http::{Request as HttpRequest, Response, StatusCode};
use http_body::Body;
@@ -496,10 +497,10 @@ async fn collect_storage_readiness_uncached() -> bool {
}
}
fn set_lock_quorum_status(online_hosts: &HashSet<String>, set_endpoints: &[crate::storage_compat::Endpoint]) -> LockQuorumStatus {
fn set_lock_quorum_status(online_hosts: &HashSet<String>, set_endpoints: &[Endpoint]) -> LockQuorumStatus {
let total_clients = set_endpoints
.iter()
.map(crate::storage_compat::Endpoint::host_port)
.map(Endpoint::host_port)
.filter(|host| !host.is_empty())
.collect::<HashSet<_>>();
let total_clients_len = total_clients.len();
@@ -522,10 +523,7 @@ fn set_lock_quorum_status(online_hosts: &HashSet<String>, set_endpoints: &[crate
}
}
fn aggregate_lock_quorum_status(
pool_endpoints: &crate::storage_compat::EndpointServerPools,
online_hosts: &HashSet<String>,
) -> LockQuorumStatus {
fn aggregate_lock_quorum_status(pool_endpoints: &EndpointServerPools, online_hosts: &HashSet<String>) -> LockQuorumStatus {
let mut connected_clients = 0usize;
let mut total_clients = 0usize;
let mut required_quorum = 0usize;
@@ -839,8 +837,7 @@ mod tests {
#[test]
fn aggregate_lock_quorum_status_requires_each_set_to_meet_quorum() {
use crate::storage_compat::Endpoint;
use crate::storage_compat::{EndpointServerPools, Endpoints, PoolEndpoints};
use crate::server::storage_compat::{Endpoint, EndpointServerPools, Endpoints, PoolEndpoints};
let endpoints = vec![
Endpoint {
@@ -897,8 +894,7 @@ mod tests {
#[test]
fn aggregate_lock_quorum_status_fails_when_any_set_loses_quorum() {
use crate::storage_compat::Endpoint;
use crate::storage_compat::{EndpointServerPools, Endpoints, PoolEndpoints};
use crate::server::storage_compat::{Endpoint, EndpointServerPools, Endpoints, PoolEndpoints};
let endpoints = vec![
Endpoint {
+69
View File
@@ -0,0 +1,69 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::{collections::HashMap, sync::Arc};
use rustfs_ecstore::api::{
config as ecstore_config, disk as ecstore_disk, error as ecstore_error, event as ecstore_event, global as ecstore_global,
layout as ecstore_layout, rpc as ecstore_rpc, storage as ecstore_storage,
};
pub(crate) const TONIC_RPC_PREFIX: &str = ecstore_rpc::TONIC_RPC_PREFIX;
pub(crate) type ECStore = ecstore_storage::ECStore;
pub(crate) type EcstoreError = ecstore_error::Error;
pub(crate) type EcstoreEventArgs = ecstore_event::EventArgs;
pub(crate) type EcstoreResult<T> = ecstore_error::Result<T>;
pub(crate) type Endpoint = ecstore_disk::endpoint::Endpoint;
pub(crate) type EndpointServerPools = ecstore_layout::EndpointServerPools;
#[cfg(test)]
pub(crate) type Endpoints = ecstore_layout::Endpoints;
#[cfg(test)]
pub(crate) type PoolEndpoints = ecstore_layout::PoolEndpoints;
pub(crate) async fn read_ecstore_config(api: Arc<ECStore>, file: &str) -> EcstoreResult<Vec<u8>> {
ecstore_config::com::read_config(api, file).await
}
pub(crate) async fn save_ecstore_config(api: Arc<ECStore>, file: &str, data: Vec<u8>) -> EcstoreResult<()> {
ecstore_config::com::save_config(api, file, data).await
}
pub(crate) fn register_event_dispatch_hook<F>(hook: F) -> bool
where
F: Fn(EcstoreEventArgs) + Send + Sync + 'static,
{
ecstore_event::register_event_dispatch_hook(hook)
}
pub(crate) fn get_global_endpoints_opt() -> Option<EndpointServerPools> {
ecstore_global::get_global_endpoints_opt()
}
pub(crate) fn get_global_lock_clients() -> Option<&'static HashMap<String, Arc<dyn rustfs_lock::client::LockClient>>> {
ecstore_global::get_global_lock_clients()
}
pub(crate) async fn is_dist_erasure() -> bool {
ecstore_global::is_dist_erasure().await
}
pub(crate) fn resolve_object_store_handle() -> Option<Arc<ECStore>> {
ecstore_global::resolve_object_store_handle()
}
pub(crate) fn verify_rpc_signature(url: &str, method: &http::Method, headers: &http::HeaderMap) -> std::io::Result<()> {
ecstore_rpc::verify_rpc_signature(url, method, headers)
}
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage_compat::ECStore;
use crate::startup_storage_compat::ECStore;
use rustfs_heal::{create_ahm_services_cancel_token, heal::storage::ECStoreHealStorage, init_heal_manager};
use rustfs_utils::get_env_bool_with_aliases;
use std::{io::Result, sync::Arc};
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage_compat::{
use crate::startup_storage_compat::{
ECStore, get_global_replication_pool, init_bucket_metadata_sys, try_migrate_bucket_metadata, try_migrate_iam_config,
};
use rustfs_storage_api::{BucketOperations, BucketOptions};
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage_compat::EndpointServerPools;
use crate::startup_storage_compat::EndpointServerPools;
use rustfs_config::{
DEFAULT_RUSTFS_UNSUPPORTED_FS_POLICY, ENV_RUSTFS_UNSUPPORTED_FS_POLICY, RUSTFS_UNSUPPORTED_FS_POLICY_FAIL,
RUSTFS_UNSUPPORTED_FS_POLICY_WARN,
+1 -1
View File
@@ -14,7 +14,7 @@
use crate::app::context::AppContext;
use crate::server::{ServiceStateManager, publish_ready_when_runtime_ready};
use crate::storage_compat::ECStore;
use crate::startup_storage_compat::ECStore;
use rustfs_common::{GlobalReadiness, SystemStage};
use rustfs_iam::init_iam_sys;
use rustfs_kms::KmsServiceManager;
+1 -1
View File
@@ -17,7 +17,7 @@ use crate::{
startup_iam::{IamBootstrapDisposition, publish_ready_for_iam_bootstrap},
startup_services::StartupServiceRuntime,
startup_shutdown::run_startup_shutdown_sequence,
storage_compat::ECStore,
startup_storage_compat::ECStore,
};
use rustfs_common::GlobalReadiness;
use rustfs_scanner::init_data_scanner;
+6 -5
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use crate::init::add_bucket_notification_configuration;
use crate::storage_compat::{EndpointServerPools, new_global_notification_sys};
use crate::startup_storage_compat::{EcstoreResult, EndpointServerPools, new_global_notification_sys};
use std::{
future::Future,
io::{Error, Result},
@@ -50,14 +50,14 @@ pub(crate) async fn init_notification_runtime(endpoint_pools: EndpointServerPool
})
}
pub(crate) async fn init_notification_system(endpoint_pools: EndpointServerPools) -> crate::storage_compat::EcstoreResult<()> {
pub(crate) async fn init_notification_system(endpoint_pools: EndpointServerPools) -> EcstoreResult<()> {
init_notification_system_with(|| new_global_notification_sys(endpoint_pools)).await
}
async fn init_notification_system_with<InitFn, InitFuture>(init_notification: InitFn) -> crate::storage_compat::EcstoreResult<()>
async fn init_notification_system_with<InitFn, InitFuture>(init_notification: InitFn) -> EcstoreResult<()>
where
InitFn: FnOnce() -> InitFuture,
InitFuture: Future<Output = crate::storage_compat::EcstoreResult<()>>,
InitFuture: Future<Output = EcstoreResult<()>>,
{
init_notification().await
}
@@ -79,7 +79,8 @@ mod tests {
#[tokio::test]
async fn notification_system_returns_source_error() {
let result = init_notification_system_with(|| async { Err(crate::storage_compat::EcstoreError::FaultyDisk) }).await;
let result =
init_notification_system_with(|| async { Err(crate::startup_storage_compat::EcstoreError::FaultyDisk) }).await;
assert!(result.is_err());
}
+3 -3
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage_compat::set_global_rustfs_port;
use crate::startup_storage_compat::{set_global_region, set_global_rustfs_port};
use crate::{
capacity::capacity_integration::init_capacity_management,
config::Config,
@@ -82,7 +82,7 @@ pub(crate) async fn init_startup_listen_context(config: &Config) -> Result<Start
if let Some(region_str) = &config.region {
region_str
.parse::<s3s::region::Region>()
.map(crate::storage_compat::set_global_region)
.map(set_global_region)
.map_err(|err| Error::other(format!("invalid region '{}': {}", region_str, err)))?;
}
@@ -190,7 +190,7 @@ pub(crate) async fn init_embedded_startup_listen_context(config: &Config) -> Res
if let Some(region_str) = &config.region {
region_str
.parse::<s3s::region::Region>()
.map(crate::storage_compat::set_global_region)
.map(set_global_region)
.map_err(|err| Error::other(format!("invalid region '{region_str}': {err}")))?;
}
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage_compat::{ECStore, EndpointServerPools};
use crate::startup_storage_compat::{ECStore, EndpointServerPools};
use crate::{
config::Config,
init::{init_buffer_profile_system, init_kms_system},
+1 -1
View File
@@ -17,7 +17,7 @@ use crate::{
startup_optional_runtime_sidecars::{
OptionalRuntimeServices, prepare_optional_runtime_shutdowns, shutdown_optional_runtime_services,
},
storage_compat::shutdown_background_services,
startup_storage_compat::shutdown_background_services,
};
use rustfs_heal::shutdown_ahm_services;
use rustfs_utils::get_env_bool_with_aliases;
+1 -1
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use crate::startup_fs_guard::enforce_unsupported_fs_policy;
use crate::storage_compat::{
use crate::startup_storage_compat::{
ECStore, EndpointServerPools, init_background_replication, init_ecstore_config, init_global_config_sys, init_local_disks,
init_lock_clients, prewarm_local_disk_id_map, set_global_endpoints, try_migrate_server_config, update_erasure_type,
};
+102
View File
@@ -0,0 +1,102 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::Arc;
use rustfs_ecstore::api::{
bucket as ecstore_bucket, config as ecstore_config, error as ecstore_error, global as ecstore_global,
layout as ecstore_layout, notification as ecstore_notification, storage as ecstore_storage,
};
pub(crate) type ECStore = ecstore_storage::ECStore;
#[cfg(test)]
pub(crate) type EcstoreError = ecstore_error::Error;
pub(crate) type EcstoreResult<T> = ecstore_error::Result<T>;
pub(crate) type EndpointServerPools = ecstore_layout::EndpointServerPools;
pub(crate) async fn get_notification_config(bucket: &str) -> ecstore_error::Result<Option<s3s::dto::NotificationConfiguration>> {
ecstore_bucket::metadata_sys::get_notification_config(bucket).await
}
pub(crate) async fn init_bucket_metadata_sys(api: Arc<ECStore>, buckets: Vec<String>) {
ecstore_bucket::metadata_sys::init_bucket_metadata_sys(api, buckets).await;
}
pub(crate) async fn try_migrate_bucket_metadata(store: Arc<ECStore>) {
ecstore_bucket::migration::try_migrate_bucket_metadata(store).await;
}
pub(crate) async fn try_migrate_iam_config(store: Arc<ECStore>) {
ecstore_bucket::migration::try_migrate_iam_config(store).await;
}
pub(crate) fn get_global_replication_pool() -> Option<Arc<ecstore_bucket::replication::DynReplicationPool>> {
ecstore_bucket::replication::get_global_replication_pool()
}
pub(crate) async fn init_background_replication(storage: Arc<ECStore>) {
ecstore_bucket::replication::init_background_replication(storage).await;
}
pub(crate) fn init_ecstore_config() {
ecstore_config::init();
}
pub(crate) async fn init_global_config_sys(api: Arc<ECStore>) -> EcstoreResult<()> {
ecstore_config::init_global_config_sys(api).await
}
pub(crate) async fn try_migrate_server_config(api: Arc<ECStore>) {
ecstore_config::try_migrate_server_config(api).await;
}
pub(crate) fn get_global_region() -> Option<s3s::region::Region> {
ecstore_global::get_global_region()
}
pub(crate) fn set_global_endpoints(eps: Vec<ecstore_layout::PoolEndpoints>) {
ecstore_global::set_global_endpoints(eps);
}
pub(crate) fn set_global_region(region: s3s::region::Region) {
ecstore_global::set_global_region(region);
}
pub(crate) fn set_global_rustfs_port(value: u16) {
ecstore_global::set_global_rustfs_port(value);
}
pub(crate) fn shutdown_background_services() {
ecstore_global::shutdown_background_services();
}
pub(crate) async fn update_erasure_type(setup_type: ecstore_layout::SetupType) {
ecstore_global::update_erasure_type(setup_type).await
}
pub(crate) async fn new_global_notification_sys(eps: EndpointServerPools) -> EcstoreResult<()> {
ecstore_notification::new_global_notification_sys(eps).await
}
pub(crate) async fn init_local_disks(endpoint_pools: EndpointServerPools) -> EcstoreResult<()> {
ecstore_storage::init_local_disks(endpoint_pools).await
}
pub(crate) fn init_lock_clients(endpoint_pools: EndpointServerPools) {
ecstore_storage::init_lock_clients(endpoint_pools);
}
pub(crate) async fn prewarm_local_disk_id_map() {
ecstore_storage::prewarm_local_disk_id_map().await;
}
-213
View File
@@ -1,213 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::{collections::HashMap, sync::Arc, time::Duration};
use rustfs_ecstore::api::{
bucket as ecstore_bucket, cluster as ecstore_cluster, config as ecstore_config, disk as ecstore_disk, error as ecstore_error,
event as ecstore_event, global as ecstore_global, layout as ecstore_layout, notification as ecstore_notification,
rpc as ecstore_rpc, set_disk as ecstore_set_disk, storage as ecstore_storage,
};
pub(crate) const BUCKET_TABLE_CATALOG_META_PREFIX: &str = ecstore_bucket::metadata::BUCKET_TABLE_CATALOG_META_PREFIX;
pub(crate) const BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX: &str =
ecstore_bucket::metadata::BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX;
pub(crate) const BUCKET_TABLE_CONFIG: &str = ecstore_bucket::metadata::BUCKET_TABLE_CONFIG;
pub(crate) const BUCKET_TABLE_RESERVED_PREFIX: &str = ecstore_bucket::metadata::BUCKET_TABLE_RESERVED_PREFIX;
pub(crate) const RUSTFS_META_BUCKET: &str = ecstore_disk::RUSTFS_META_BUCKET;
pub(crate) const TONIC_RPC_PREFIX: &str = ecstore_rpc::TONIC_RPC_PREFIX;
pub(crate) type ECStore = ecstore_storage::ECStore;
pub(crate) type EcstoreError = ecstore_error::Error;
pub(crate) type EcstoreEventArgs = ecstore_event::EventArgs;
pub(crate) type EcstoreResult<T> = ecstore_error::Result<T>;
pub(crate) type Endpoint = ecstore_disk::endpoint::Endpoint;
pub(crate) type EndpointServerPools = ecstore_layout::EndpointServerPools;
pub(crate) type QuotaError = ecstore_bucket::quota::QuotaError;
pub(crate) type StorageError = ecstore_error::StorageError;
#[cfg(test)]
pub(crate) type DisksLayout = ecstore_layout::DisksLayout;
#[cfg(test)]
pub(crate) type Endpoints = ecstore_layout::Endpoints;
#[cfg(test)]
pub(crate) type PoolEndpoints = ecstore_layout::PoolEndpoints;
pub(crate) trait DiskAPI {
fn endpoint(&self) -> Endpoint;
fn to_string(&self) -> String;
}
impl<T> DiskAPI for T
where
T: ecstore_disk::DiskAPI,
{
fn endpoint(&self) -> Endpoint {
ecstore_disk::DiskAPI::endpoint(self)
}
fn to_string(&self) -> String {
ecstore_disk::DiskAPI::to_string(self)
}
}
pub(crate) fn table_catalog_path_hash(value: &str) -> String {
ecstore_bucket::metadata::table_catalog_path_hash(value)
}
pub(crate) async fn get_bucket_metadata(bucket: &str) -> ecstore_error::Result<Arc<ecstore_bucket::metadata::BucketMetadata>> {
ecstore_bucket::metadata_sys::get(bucket).await
}
pub(crate) fn get_global_bucket_metadata_sys() -> Option<Arc<tokio::sync::RwLock<ecstore_bucket::metadata_sys::BucketMetadataSys>>>
{
ecstore_bucket::metadata_sys::get_global_bucket_metadata_sys()
}
pub(crate) async fn get_notification_config(bucket: &str) -> ecstore_error::Result<Option<s3s::dto::NotificationConfiguration>> {
ecstore_bucket::metadata_sys::get_notification_config(bucket).await
}
pub(crate) async fn init_bucket_metadata_sys(api: Arc<ECStore>, buckets: Vec<String>) {
ecstore_bucket::metadata_sys::init_bucket_metadata_sys(api, buckets).await;
}
pub(crate) async fn try_migrate_bucket_metadata(store: Arc<ECStore>) {
ecstore_bucket::migration::try_migrate_bucket_metadata(store).await;
}
pub(crate) async fn try_migrate_iam_config(store: Arc<ECStore>) {
ecstore_bucket::migration::try_migrate_iam_config(store).await;
}
pub(crate) fn get_global_replication_pool() -> Option<Arc<ecstore_bucket::replication::DynReplicationPool>> {
ecstore_bucket::replication::get_global_replication_pool()
}
pub(crate) fn replication_queue_current_count() -> Option<i64> {
ecstore_bucket::replication::GLOBAL_REPLICATION_STATS.get().and_then(|stats| {
stats
.q_cache
.try_lock()
.ok()
.map(|cache| cache.sr_queue_stats.curr.get_current_count())
})
}
pub(crate) async fn init_background_replication(storage: Arc<ECStore>) {
ecstore_bucket::replication::init_background_replication(storage).await;
}
pub(crate) fn topology_snapshot_from_endpoint_pools_with_capabilities(
endpoint_pools: &EndpointServerPools,
capabilities: rustfs_storage_api::TopologyCapabilities,
disk_capabilities: rustfs_storage_api::DiskCapabilities,
) -> rustfs_storage_api::TopologySnapshot {
ecstore_cluster::topology_snapshot_from_endpoint_pools_with_capabilities(endpoint_pools, capabilities, disk_capabilities)
}
pub(crate) async fn read_ecstore_config(api: Arc<ECStore>, file: &str) -> EcstoreResult<Vec<u8>> {
ecstore_config::com::read_config(api, file).await
}
pub(crate) async fn save_ecstore_config(api: Arc<ECStore>, file: &str, data: Vec<u8>) -> EcstoreResult<()> {
ecstore_config::com::save_config(api, file, data).await
}
pub(crate) fn init_ecstore_config() {
ecstore_config::init();
}
pub(crate) async fn init_global_config_sys(api: Arc<ECStore>) -> EcstoreResult<()> {
ecstore_config::init_global_config_sys(api).await
}
pub(crate) async fn try_migrate_server_config(api: Arc<ECStore>) {
ecstore_config::try_migrate_server_config(api).await;
}
pub(crate) fn register_event_dispatch_hook<F>(hook: F) -> bool
where
F: Fn(EcstoreEventArgs) + Send + Sync + 'static,
{
ecstore_event::register_event_dispatch_hook(hook)
}
pub(crate) fn get_global_endpoints_opt() -> Option<EndpointServerPools> {
ecstore_global::get_global_endpoints_opt()
}
pub(crate) fn get_global_lock_clients() -> Option<&'static HashMap<String, Arc<dyn rustfs_lock::client::LockClient>>> {
ecstore_global::get_global_lock_clients()
}
pub(crate) fn get_global_region() -> Option<s3s::region::Region> {
ecstore_global::get_global_region()
}
pub(crate) async fn is_dist_erasure() -> bool {
ecstore_global::is_dist_erasure().await
}
pub(crate) fn resolve_object_store_handle() -> Option<Arc<ECStore>> {
ecstore_global::resolve_object_store_handle()
}
pub(crate) fn set_global_endpoints(eps: Vec<ecstore_layout::PoolEndpoints>) {
ecstore_global::set_global_endpoints(eps);
}
pub(crate) fn set_global_region(region: s3s::region::Region) {
ecstore_global::set_global_region(region);
}
pub(crate) fn set_global_rustfs_port(value: u16) {
ecstore_global::set_global_rustfs_port(value);
}
pub(crate) fn shutdown_background_services() {
ecstore_global::shutdown_background_services();
}
pub(crate) async fn update_erasure_type(setup_type: ecstore_layout::SetupType) {
ecstore_global::update_erasure_type(setup_type).await;
}
pub(crate) async fn new_global_notification_sys(eps: EndpointServerPools) -> EcstoreResult<()> {
ecstore_notification::new_global_notification_sys(eps).await
}
pub(crate) fn verify_rpc_signature(url: &str, method: &http::Method, headers: &http::HeaderMap) -> std::io::Result<()> {
ecstore_rpc::verify_rpc_signature(url, method, headers)
}
pub(crate) fn get_lock_acquire_timeout() -> Duration {
ecstore_set_disk::get_lock_acquire_timeout()
}
pub(crate) async fn all_local_disk() -> Vec<ecstore_disk::DiskStore> {
ecstore_storage::all_local_disk().await
}
pub(crate) async fn init_local_disks(endpoint_pools: EndpointServerPools) -> EcstoreResult<()> {
ecstore_storage::init_local_disks(endpoint_pools).await
}
pub(crate) fn init_lock_clients(endpoint_pools: EndpointServerPools) {
ecstore_storage::init_lock_clients(endpoint_pools);
}
pub(crate) async fn prewarm_local_disk_id_map() {
ecstore_storage::prewarm_local_disk_id_map().await;
}
+11 -26
View File
@@ -27,11 +27,10 @@ use std::{
time::{Duration as StdDuration, Instant},
};
use crate::storage_compat::RUSTFS_META_BUCKET;
use crate::storage_compat::get_lock_acquire_timeout;
use crate::storage_compat::{
use crate::table_catalog_storage_compat::{
BUCKET_TABLE_CATALOG_META_PREFIX, BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX, BUCKET_TABLE_CONFIG,
BUCKET_TABLE_RESERVED_PREFIX, EcstoreError, StorageError, get_bucket_metadata, table_catalog_path_hash,
BUCKET_TABLE_RESERVED_PREFIX, EcstoreError, RUSTFS_META_BUCKET, StorageError, get_bucket_metadata, get_lock_acquire_timeout,
table_catalog_path_hash,
};
use bytes::Bytes;
use datafusion::{
@@ -7886,7 +7885,7 @@ mod tests {
.map(|(bucket, _)| bucket.as_str())
.collect::<BTreeSet<_>>();
assert_eq!(object_buckets, BTreeSet::from([crate::storage_compat::RUSTFS_META_BUCKET]));
assert_eq!(object_buckets, BTreeSet::from([RUSTFS_META_BUCKET]));
}
#[tokio::test]
@@ -10931,9 +10930,7 @@ mod tests {
.await
.unwrap();
backend.seed_object(bucket, &new_metadata, b"{}".to_vec()).await;
backend
.fail_put_attempt(crate::storage_compat::RUSTFS_META_BUCKET, &idempotency_path, 1)
.await;
backend.fail_put_attempt(RUSTFS_META_BUCKET, &idempotency_path, 1).await;
let err = store
.commit_table(TableCommitRequest {
@@ -10981,9 +10978,7 @@ mod tests {
.await
.unwrap();
backend.seed_object(bucket, &new_metadata, b"{}".to_vec()).await;
backend
.fail_put_attempt(crate::storage_compat::RUSTFS_META_BUCKET, &commit_path, 2)
.await;
backend.fail_put_attempt(RUSTFS_META_BUCKET, &commit_path, 2).await;
let request = TableCommitRequest {
table_bucket: bucket.to_string(),
@@ -11036,9 +11031,7 @@ mod tests {
.await
.unwrap();
backend.seed_object(bucket, &new_metadata, b"{}".to_vec()).await;
backend
.fail_put_attempt(crate::storage_compat::RUSTFS_META_BUCKET, &commit_path, 2)
.await;
backend.fail_put_attempt(RUSTFS_META_BUCKET, &commit_path, 2).await;
let result = store
.commit_table(TableCommitRequest {
@@ -11090,9 +11083,7 @@ mod tests {
.await
.unwrap();
backend.seed_object(bucket, &new_metadata, b"{}".to_vec()).await;
backend
.fail_put_attempt(crate::storage_compat::RUSTFS_META_BUCKET, &commit_path, 2)
.await;
backend.fail_put_attempt(RUSTFS_META_BUCKET, &commit_path, 2).await;
store
.commit_table(TableCommitRequest {
@@ -11144,9 +11135,7 @@ mod tests {
.await
.unwrap();
backend.seed_object(bucket, &new_metadata, b"{}".to_vec()).await;
backend
.fail_put_attempt(crate::storage_compat::RUSTFS_META_BUCKET, &commit_path, 2)
.await;
backend.fail_put_attempt(RUSTFS_META_BUCKET, &commit_path, 2).await;
store
.commit_table(TableCommitRequest {
@@ -11195,9 +11184,7 @@ mod tests {
.unwrap();
backend.seed_object(bucket, &current_metadata, b"{}".to_vec()).await;
backend.seed_object(bucket, &new_metadata, b"{}".to_vec()).await;
backend
.fail_put_attempt(crate::storage_compat::RUSTFS_META_BUCKET, &table_path, 2)
.await;
backend.fail_put_attempt(RUSTFS_META_BUCKET, &table_path, 2).await;
let err = store
.commit_table(TableCommitRequest {
@@ -11252,9 +11239,7 @@ mod tests {
.await
.unwrap();
backend.seed_object(bucket, &new_metadata, b"{}".to_vec()).await;
backend
.fail_put_attempt(crate::storage_compat::RUSTFS_META_BUCKET, &idempotency_path, 2)
.await;
backend.fail_put_attempt(RUSTFS_META_BUCKET, &idempotency_path, 2).await;
let result = store
.commit_table(TableCommitRequest {
@@ -0,0 +1,38 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::{sync::Arc, time::Duration};
use rustfs_ecstore::api::{bucket as ecstore_bucket, disk as ecstore_disk, error as ecstore_error, set_disk as ecstore_set_disk};
pub(crate) const BUCKET_TABLE_CATALOG_META_PREFIX: &str = ecstore_bucket::metadata::BUCKET_TABLE_CATALOG_META_PREFIX;
pub(crate) const BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX: &str =
ecstore_bucket::metadata::BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX;
pub(crate) const BUCKET_TABLE_CONFIG: &str = ecstore_bucket::metadata::BUCKET_TABLE_CONFIG;
pub(crate) const BUCKET_TABLE_RESERVED_PREFIX: &str = ecstore_bucket::metadata::BUCKET_TABLE_RESERVED_PREFIX;
pub(crate) const RUSTFS_META_BUCKET: &str = ecstore_disk::RUSTFS_META_BUCKET;
pub(crate) type EcstoreError = ecstore_error::Error;
pub(crate) type StorageError = ecstore_error::StorageError;
pub(crate) fn table_catalog_path_hash(value: &str) -> String {
ecstore_bucket::metadata::table_catalog_path_hash(value)
}
pub(crate) async fn get_bucket_metadata(bucket: &str) -> ecstore_error::Result<Arc<ecstore_bucket::metadata::BucketMetadata>> {
ecstore_bucket::metadata_sys::get(bucket).await
}
pub(crate) fn get_lock_acquire_timeout() -> Duration {
ecstore_set_disk::get_lock_acquire_timeout()
}
+3 -1
View File
@@ -18,7 +18,9 @@ use rustfs_concurrency::{
};
use crate::storage::concurrency::get_concurrency_manager;
use crate::storage_compat::{get_global_bucket_metadata_sys, get_global_replication_pool, replication_queue_current_count};
use crate::workload_admission_storage_compat::{
get_global_bucket_metadata_sys, get_global_replication_pool, replication_queue_current_count,
};
const BUCKET_METADATA_RUNTIME_NOT_INITIALIZED: &str = "bucket metadata runtime not initialized";
const HEAL_MANAGER_NOT_INITIALIZED: &str = "heal manager not initialized";
@@ -0,0 +1,36 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::Arc;
use rustfs_ecstore::api::bucket as ecstore_bucket;
pub(crate) fn get_global_bucket_metadata_sys() -> Option<Arc<tokio::sync::RwLock<ecstore_bucket::metadata_sys::BucketMetadataSys>>>
{
ecstore_bucket::metadata_sys::get_global_bucket_metadata_sys()
}
pub(crate) fn get_global_replication_pool() -> Option<Arc<ecstore_bucket::replication::DynReplicationPool>> {
ecstore_bucket::replication::get_global_replication_pool()
}
pub(crate) fn replication_queue_current_count() -> Option<i64> {
ecstore_bucket::replication::GLOBAL_REPLICATION_STATS.get().and_then(|stats| {
stats
.q_cache
.try_lock()
.ok()
.map(|cache| cache.sr_queue_stats.curr.get_current_count())
})
}
+16 -4
View File
@@ -81,6 +81,7 @@ RUSTFS_RUNTIME_SECONDARY_STORAGE_COMPAT_HITS_FILE="${TMP_DIR}/rustfs_runtime_sec
RUSTFS_ROOT_STORAGE_COMPAT_REEXPORT_HITS_FILE="${TMP_DIR}/rustfs_root_storage_compat_reexport_hits.txt"
RUSTFS_ROOT_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_root_bucket_storage_compat_module_hits.txt"
RUSTFS_ROOT_RUNTIME_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_root_runtime_storage_compat_module_hits.txt"
RUSTFS_ROOT_CONSUMER_COMPAT_HITS_FILE="${TMP_DIR}/rustfs_root_consumer_compat_hits.txt"
RUSTFS_ADMIN_CONFIG_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_admin_config_storage_compat_module_hits.txt"
RUSTFS_STORAGE_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_storage_bucket_storage_compat_module_hits.txt"
RUSTFS_STORAGE_OWNER_COMPAT_REEXPORT_HITS_FILE="${TMP_DIR}/rustfs_storage_owner_compat_reexport_hits.txt"
@@ -689,6 +690,7 @@ fi
rustfs/src crates fuzz/fuzz_targets \
--glob '!crates/ecstore/**' \
--glob '!**/storage_compat.rs' \
--glob '!**/*storage_compat.rs' \
--glob '!target/**' || true
) |
cat >"$DIRECT_ECSTORE_IMPORT_HITS_FILE"
@@ -788,7 +790,7 @@ fi
(
cd "$ROOT_DIR"
rg -n --no-heading 'pub\(crate\)\s+use\s+rustfs_ecstore::api::' \
rustfs/src/storage_compat.rs || true
rustfs/src/storage_compat.rs 2>/dev/null || true
) >"$RUSTFS_ROOT_STORAGE_COMPAT_REEXPORT_HITS_FILE"
if [[ -s "$RUSTFS_ROOT_STORAGE_COMPAT_REEXPORT_HITS_FILE" ]]; then
@@ -798,7 +800,7 @@ fi
(
cd "$ROOT_DIR"
rg -n --no-heading 'pub\(crate\)\s+use rustfs_ecstore::api::bucket::\{[^}]*\b(?:metadata|metadata_sys|quota)\b[^}]*\}\s*;|pub\(crate\)\s+use rustfs_ecstore::api::bucket::(?:metadata|metadata_sys|quota)\s*;' \
rustfs/src/storage_compat.rs || true
rustfs/src/storage_compat.rs 2>/dev/null || true
) >"$RUSTFS_ROOT_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE"
if [[ -s "$RUSTFS_ROOT_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE" ]]; then
@@ -808,13 +810,23 @@ fi
(
cd "$ROOT_DIR"
rg -n --no-heading 'pub\(crate\)\s+use rustfs_ecstore::api::config::\{[^}]*\bcom\b[^}]*\}\s*;|pub\(crate\)\s+use rustfs_ecstore::api::config::\{[^}]*\binit\s*(?:,|})[^}]*\}\s*;|pub\(crate\)\s+use rustfs_ecstore::api::disk::\{[^}]*\bendpoint::Endpoint\b[^}]*\}\s*;' \
rustfs/src/storage_compat.rs || true
rustfs/src/storage_compat.rs 2>/dev/null || true
) >"$RUSTFS_ROOT_RUNTIME_STORAGE_COMPAT_MODULE_HITS_FILE"
if [[ -s "$RUSTFS_ROOT_RUNTIME_STORAGE_COMPAT_MODULE_HITS_FILE" ]]; then
report_failure "RustFS root storage compatibility must expose config init and disk endpoint contracts as explicit aliases: $(paste -sd '; ' "$RUSTFS_ROOT_RUNTIME_STORAGE_COMPAT_MODULE_HITS_FILE")"
fi
(
cd "$ROOT_DIR"
rg -n --no-heading 'pub\(crate\)\s+(?:async\s+)?fn\s+(?:all_local_disk|read_ecstore_config|save_ecstore_config|get_global_endpoints_opt|get_global_lock_clients|is_dist_erasure|resolve_object_store_handle|register_event_dispatch_hook|verify_rpc_signature|get_notification_config|init_bucket_metadata_sys|try_migrate_bucket_metadata|try_migrate_iam_config|init_background_replication|init_ecstore_config|init_global_config_sys|try_migrate_server_config|get_global_region|set_global_endpoints|set_global_region|set_global_rustfs_port|shutdown_background_services|update_erasure_type|new_global_notification_sys|init_local_disks|init_lock_clients|prewarm_local_disk_id_map)\b|pub\(crate\)\s+(?:const|type)\s+(?:TONIC_RPC_PREFIX|EcstoreEventArgs|ECStore|EcstoreResult)\b|pub\(crate\)\s+trait\s+DiskAPI\b' \
rustfs/src/storage_compat.rs 2>/dev/null || true
) >"$RUSTFS_ROOT_CONSUMER_COMPAT_HITS_FILE"
if [[ -s "$RUSTFS_ROOT_CONSUMER_COMPAT_HITS_FILE" ]]; then
report_failure "RustFS root storage compatibility must not own capacity/server/startup consumer wrappers: $(paste -sd '; ' "$RUSTFS_ROOT_CONSUMER_COMPAT_HITS_FILE")"
fi
(
cd "$ROOT_DIR"
rg -n --no-heading 'pub\(crate\)\s+use rustfs_ecstore::api::config::\{[^}]*\bcom\b[^}]*\}\s*;|pub\(crate\)\s+use rustfs_ecstore::api::config::\{[^}]*\binit\s*(?:,|})[^}]*\}\s*;' \
@@ -942,7 +954,7 @@ fi
(
cd "$ROOT_DIR"
rg -n --no-heading 'rustfs_ecstore::api::' rustfs/src/storage_compat.rs crates/e2e_test/src/storage_compat.rs \
rg -n --with-filename --no-heading 'rustfs_ecstore::api::' crates/e2e_test/src/storage_compat.rs \
| rg -v '^[^:]+:[0-9]+:use rustfs_ecstore::api::\{' || true
) >"$RUSTFS_ROOT_E2E_COMPAT_RAW_FACADE_PATH_HITS_FILE"