gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-22 04:16:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor: route app facade boundaries (#3865)

Browse Source 
* refactor: route app storage error helpers

* refactor: route app bucket owner facades

* refactor: route app runtime facades
This commit is contained in:
Zhengchao An
2026-06-25 21:57:42 +08:00
committed by GitHub
parent 60c00f7622
commit 1c973bc1d9
22 changed files with 866 additions and 775 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/migration-progress.md
+68 -7
View File
@@ -5,9 +5,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
## Current Context ## Current Context
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
- Branch: `overtrue/arch-app-storage-io-helper-boundary` - Branch: `overtrue/arch-app-runtime-global-facade-cleanup`
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213`. - Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213/API-214/API-215`.
- Based on: stacked on API-212 after PR #3854 merged. - Based on: stacked on API-215 while PR #3856 is pending.
- PR type for this branch: `consumer-migration` - PR type for this branch: `consumer-migration`
- Runtime behavior changes: none. - Runtime behavior changes: none.
- Rust code changes: route replication pool, outbound TLS generation, runtime - Rust code changes: route replication pool, outbound TLS generation, runtime
@@ -45,7 +45,7 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
readiness-time, observability, metrics, buffer, and TLS runtime source readiness-time, observability, metrics, buffer, and TLS runtime source
helpers, plus server readiness/audit/event/module-switch runtime source helpers, plus server readiness/audit/event/module-switch runtime source
helpers, storage request/RPC/SSE runtime source helpers, and admin helpers, storage request/RPC/SSE runtime source helpers, and admin
handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, app bucket/object/multipart usecase explicit storage imports, app select/bucket/object/multipart explicit S3 DTO plus ECFS owner imports, app-local S3 API response helper imports for bucket/object/multipart usecases, app-local storage helper boundary imports for select/bucket/object/multipart usecases, admin/app test storage helper boundary imports for request context, authorization, object options, and ECFS test harness consumers, and app storage IO/compression/set-disk helper boundary imports for object and multipart usecases, handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, app bucket/object/multipart usecase explicit storage imports, app select/bucket/object/multipart explicit S3 DTO plus ECFS owner imports, app-local S3 API response helper imports for bucket/object/multipart usecases, app-local storage helper boundary imports for select/bucket/object/multipart usecases, admin/app test storage helper boundary imports for request context, authorization, object options, and ECFS test harness consumers, app storage IO/compression/set-disk helper boundary imports for object and multipart usecases, app storage error, ETag, and storage-class helper boundary imports for bucket/object/multipart usecases and lifecycle transition tests, app bucket owner facade imports for lifecycle, metadata, object-lock, quota, replication, tagging, target, versioning, and transition test helpers, and app/admin runtime, capacity, data-usage, endpoint, and global facade imports,
through AppContext-first or owner-crate resolver boundaries. through AppContext-first or owner-crate resolver boundaries.
- CI/script changes: lock completed owner and test/fuzz boundaries against - CI/script changes: lock completed owner and test/fuzz boundaries against
bare/glob imports, scattered raw ECStore facade subpaths, and startup bare/glob imports, scattered raw ECStore facade subpaths, and startup
@@ -56,8 +56,8 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
event-bridge thin module regressions, plus IAM runtime-source bypasses; event-bridge thin module regressions, plus IAM runtime-source bypasses;
accept the reviewed AppContext resolver reverse dependencies in the layer accept the reviewed AppContext resolver reverse dependencies in the layer
baseline, and block direct admin AppContext resolver consumers outside the baseline, and block direct admin AppContext resolver consumers outside the
admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, narrow the object-usecase ECFS layer baseline entry to `FS`, reject direct storage S3 API helper imports from app usecase files, reject direct storage helper imports from app select/usecase files, reject completed app/admin storage helper bypasses, and reject app usecase bypasses for migrated storage IO/compression/set-disk helpers. admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, narrow the object-usecase ECFS layer baseline entry to `FS`, reject direct storage S3 API helper imports from app usecase files, reject direct storage helper imports from app select/usecase files, reject completed app/admin storage helper bypasses, reject app usecase bypasses for migrated storage IO/compression/set-disk helpers, reject app usecase/test bypasses for migrated storage error, ETag, and storage-class helpers, reject app root bucket owner facade bypasses from migrated app consumers, and reject app/admin runtime/data-usage root facade regressions.
- Docs changes: record the API-136 through API-213 owner facade and lifecycle - Docs changes: record the API-136 through API-216 owner facade and lifecycle
runtime-source cleanup. runtime-source cleanup.
## Phase 0 Tasks ## Phase 0 Tasks
@@ -5069,14 +5069,75 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
layer guards, diff hygiene, residual migrated helper scan, Rust risk scan, layer guards, diff hygiene, residual migrated helper scan, Rust risk scan,
fast PR gate, and full PR gate before PR. fast PR gate, and full PR gate before PR.
- [x] `API-214` Route app storage error helpers through app boundary.
- Do: extend the app-local storage helper boundary for storage errors, error
classification helpers, ETag conversion, and storage-class constants, then
route bucket, object, multipart, and lifecycle transition test consumers
through it instead of app root compatibility aliases.
- Acceptance: completed app consumers no longer import the migrated storage
error, ETag, or storage-class helpers from the app root, the app root no
longer exposes those helper aliases, and migration rules reject
reintroducing the bypasses.
- Must preserve: bucket config-not-found handling, multipart upload-not-found
and version-not-found classification, object delete/copy error mapping,
list response ETag conversion, storage-class response projection, and
lifecycle transition test ETag formatting.
- Verification: focused RustFS app compile/tests, formatting, migration and
layer guards, diff hygiene, residual migrated helper scan, Rust risk scan,
fast PR gate, and full PR gate before PR.
- [x] `API-215` Route app bucket owner facades through storage_api boundary.
- Do: move app-facing lifecycle, metadata, object-lock, quota, replication,
tagging, target, versioning, and transition test helper facades behind
`storage_api::bucket`, then route bucket, object, multipart, context, and
app integration-test consumers through that boundary.
- Acceptance: app root no longer exposes the migrated bucket owner facade
modules or object-info/object-options compatibility aliases, and migration
rules reject consumers that reintroduce root-level bypasses.
- Must preserve: bucket metadata CRUD, lifecycle transition enqueueing and
restore validation, object-lock deletion checks, quota checks, replication
scheduling/decision helpers, XML serialization, target validation, and
bucket metadata runtime-source lookup.
- Verification: focused RustFS app compile/tests, formatting, migration and
layer guards, diff hygiene, residual migrated helper scan, Rust risk scan,
fast PR gate, and full PR gate before PR.
- [x] `API-216` Route app runtime/global facades through storage_api boundary.
- Do: move app/admin-facing admin info, capacity, data-usage, endpoint,
runtime global, storage-class, scanner metric, tier, replication, and
notification facades behind local `storage_api` modules, then route app
usecases, context modules, tests, admin quota, admin config service, and
storage node-service consumers through those boundaries.
- Acceptance: app root no longer exposes the migrated runtime/global facade
aliases or wrappers, admin root no longer exposes the migrated data-usage
wrapper, and migration rules reject root-level regressions for the
completed app/admin runtime/data-usage paths.
- Must preserve: admin server info and capacity math, data-usage memory
overlay and bucket accounting, object-store resolver publication,
AppContext fallback runtime sources, lifecycle test disk initialization,
tier warm-backend test contracts, scanner metrics report collection, and
admin quota current-usage reads.
- Verification: focused RustFS app compile/tests, formatting, migration and
layer guards, diff hygiene, residual migrated facade scan, Rust risk scan,
fast PR gate, and full PR gate before PR.
## Next PRs ## Next PRs
1. `consumer-migration`: batch remaining app/admin/storage helper boundary and owner facade cleanup. 1. `consumer-migration`: batch remaining app/admin facade cleanup after API-216.
## Pre-Push Review Log ## Pre-Push Review Log
| Expert | Status | Notes | | Expert | Status | Notes |
|---|---|---| |---|---|---|
| Quality/architecture | pass | API-216 removes app root runtime/global facades and routes app/admin data-usage and runtime consumers through local storage_api boundaries. |
| Migration preservation | pass | Admin info/capacity, data-usage accounting, AppContext fallback runtime sources, lifecycle test disk initialization, tier warm-backend tests, and scanner metrics keep the same storage implementations. |
| Testing/verification | pass | Focused RustFS app compile/tests, formatting, migration/layer guards, residual migrated facade scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. |
| Quality/architecture | pass | API-215 consolidates bucket owner facades under app storage_api and removes the app root compatibility modules. |
| Migration preservation | pass | Lifecycle, metadata, object-lock, quota, replication, tagging, target, versioning, and transition test helpers keep the same storage implementations. |
| Testing/verification | pass | Focused app compile/tests, formatting, migration/layer guards, residual migrated facade scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. |
| Quality/architecture | pass | API-214 moves app storage error, ETag, and storage-class helpers behind the app storage_api boundary and removes the matching app root aliases. |
| Migration preservation | pass | Bucket config handling, multipart/object not-found classification, object delete/copy mapping, list ETag conversion, storage-class response projection, and lifecycle transition test formatting keep the same storage implementations. |
| Testing/verification | pass | Focused app compile/tests, formatting, migration/layer guards, residual migrated helper scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. |
| Quality/architecture | pass | API-213 removes app root IO/compression/set-disk compatibility aliases and routes object/multipart usecases through the app storage_api boundary. | | Quality/architecture | pass | API-213 removes app root IO/compression/set-disk compatibility aliases and routes object/multipart usecases through the app storage_api boundary. |
| Migration preservation | pass | Object compression metadata, hash-reader checksum handling, multipart write-encryption planning, storage-class checks, lock timeouts, and test reader wrappers keep the same storage implementations. | | Migration preservation | pass | Object compression metadata, hash-reader checksum handling, multipart write-encryption planning, storage-class checks, lock timeouts, and test reader wrappers keep the same storage implementations. |
| Testing/verification | pass | Focused app compile/tests, formatting, migration/layer guards, residual migrated helper scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. | | Testing/verification | pass | Focused app compile/tests, formatting, migration/layer guards, residual migrated helper scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. |
rustfs/src/admin/handlers/quota.rs
+1 -1
View File
@@ -175,7 +175,7 @@ async fn current_usage_from_context(bucket: &str) -> u64 {
return 0; return 0;
}; };
match super::super::load_data_usage_from_backend(store).await { match super::super::storage_api::data_usage::load_data_usage_from_backend(store).await {
Ok(data_usage_info) => data_usage_info Ok(data_usage_info) => data_usage_info
.buckets_usage .buckets_usage
.get(bucket) .get(bucket)
rustfs/src/admin/mod.rs
-10
View File
@@ -133,10 +133,6 @@ mod ecstore_config {
pub(crate) use crate::storage::ecstore_config::{com, init, storageclass}; pub(crate) use crate::storage::ecstore_config::{com, init, storageclass};
} }
mod ecstore_data_usage {
pub(crate) use crate::storage::ecstore_data_usage::load_data_usage_from_backend;
}
#[allow(unused_imports)] #[allow(unused_imports)]
mod ecstore_disk { mod ecstore_disk {
pub(crate) use crate::storage::ecstore_disk::{RUSTFS_META_BUCKET, endpoint}; pub(crate) use crate::storage::ecstore_disk::{RUSTFS_META_BUCKET, endpoint};
@@ -473,12 +469,6 @@ pub(crate) fn init_admin_config_defaults() {
ecstore_config::init(); ecstore_config::init();
} }
pub(crate) async fn load_data_usage_from_backend(
store: Arc<ECStore>,
) -> std::result::Result<rustfs_data_usage::DataUsageInfo, Error> {
ecstore_data_usage::load_data_usage_from_backend(store).await
}
pub(crate) async fn collect_local_metrics( pub(crate) async fn collect_local_metrics(
types: MetricType, types: MetricType,
opts: &CollectMetricsOpts, opts: &CollectMetricsOpts,
rustfs/src/admin/service/config.rs
+3 -1
View File
@@ -67,7 +67,9 @@ fn invalid_request(message: impl Into<String>) -> S3Error {
S3Error::with_message(S3ErrorCode::InvalidRequest, message.into()) S3Error::with_message(S3ErrorCode::InvalidRequest, message.into())
} }
fn resolve_runtime_config_store_for_context(context: Option<&AppContext>) -> S3Result<std::sync::Arc<crate::app::ECStore>> { fn resolve_runtime_config_store_for_context(
context: Option<&AppContext>,
) -> S3Result<std::sync::Arc<crate::app::storage_api::ECStore>> {
resolve_object_store_handle_for_context(context).ok_or_else(|| internal_error("storage layer not initialized")) resolve_object_store_handle_for_context(context).ok_or_else(|| internal_error("storage layer not initialized"))
} }
rustfs/src/admin/storage_api.rs
+10
View File
@@ -15,3 +15,13 @@
pub(crate) use crate::storage::StorageObjectOptions; pub(crate) use crate::storage::StorageObjectOptions;
pub(crate) use crate::storage::access::{ReqInfo, authorize_request}; pub(crate) use crate::storage::access::{ReqInfo, authorize_request};
pub(crate) use crate::storage::request_context::{RequestContext, spawn_traced}; pub(crate) use crate::storage::request_context::{RequestContext, spawn_traced};
pub(crate) mod data_usage {
use std::sync::Arc;
pub(crate) async fn load_data_usage_from_backend(
store: Arc<crate::storage::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::StorageError> {
crate::storage::ecstore_data_usage::load_data_usage_from_backend(store).await
}
}
rustfs/src/app/admin_usecase.rs
+7 -6
View File
@@ -14,11 +14,12 @@
//! Admin application use-case contracts. //! Admin application use-case contracts.
use super::ECStore; use super::storage_api::admin::get_server_info;
use super::EndpointServerPools; use super::storage_api::capacity::{
use super::get_server_info; PoolDecommissionInfo, PoolStatus, RebalStatus, get_total_usable_capacity, get_total_usable_capacity_free,
use super::{PoolDecommissionInfo, PoolStatus, RebalStatus, get_total_usable_capacity, get_total_usable_capacity_free}; };
use super::{apply_bucket_usage_memory_overlay, load_data_usage_from_backend}; use super::storage_api::data_usage::{apply_bucket_usage_memory_overlay, load_data_usage_from_backend};
use super::storage_api::{ECStore, EndpointServerPools};
use crate::app::runtime_sources::{ use crate::app::runtime_sources::{
AppContext, get_global_app_context, resolve_endpoints_handle, resolve_object_store_handle_for_context, AppContext, get_global_app_context, resolve_endpoints_handle, resolve_object_store_handle_for_context,
}; };
@@ -596,7 +597,7 @@ impl DefaultAdminUsecase {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::super::{PoolDecommissionInfo, PoolStatus}; use super::super::storage_api::capacity::{PoolDecommissionInfo, PoolStatus};
use super::*; use super::*;
use time::OffsetDateTime; use time::OffsetDateTime;
rustfs/src/app/bucket_usecase.rs
+15 -12
View File
@@ -14,23 +14,18 @@
//! Bucket application use-case contracts. //! Bucket application use-case contracts.
use super::ECStore;
use super::StorageError;
use super::object_api_utils::to_s3s_etag;
use super::s3_api::bucket::{ use super::s3_api::bucket::{
ListObjectVersionsParams, ListObjectsV2Params, build_list_buckets_output, build_list_object_versions_output, ListObjectVersionsParams, ListObjectsV2Params, build_list_buckets_output, build_list_object_versions_output,
build_list_objects_output, build_list_objects_v2_output, parse_list_object_versions_params, parse_list_objects_v2_params, build_list_objects_output, build_list_objects_v2_output, parse_list_object_versions_params, parse_list_objects_v2_params,
rustfs_owner, rustfs_owner,
}; };
use super::storage_api::ECStore;
use super::storage_api::StorageObjectInfo as ObjectInfo; use super::storage_api::StorageObjectInfo as ObjectInfo;
use super::storage_api::access::{ReqInfo, authorize_request, req_info_ref}; use super::storage_api::access::{ReqInfo, authorize_request, req_info_ref};
use super::storage_api::helper::{OperationHelper, spawn_background_with_context}; #[cfg(test)]
use super::storage_api::{ use super::storage_api::bucket::target::BucketTarget;
get_validated_store, process_lambda_configurations, process_queue_configurations, process_topic_configurations, use super::storage_api::bucket::{
request_context, validate_list_object_unordered_with_delimiter, ObjectLockConfigExt as _, VersioningConfigExt as _,
};
use super::{AppObjectLockConfigExt as _, AppVersioningConfigExt as _};
use super::{
bucket_target_sys::BucketTargetSys, bucket_target_sys::BucketTargetSys,
lifecycle::bucket_lifecycle_ops::{ lifecycle::bucket_lifecycle_ops::{
enqueue_expiry_for_existing_objects, enqueue_transition_for_existing_objects, validate_lifecycle_config, enqueue_expiry_for_existing_objects, enqueue_transition_for_existing_objects, validate_lifecycle_config,
@@ -47,6 +42,14 @@ use super::{
utils::serialize, utils::serialize,
versioning_sys::BucketVersioningSys, versioning_sys::BucketVersioningSys,
}; };
use super::storage_api::data_usage::remove_bucket_usage_from_backend;
use super::storage_api::error::StorageError;
use super::storage_api::helper::{OperationHelper, spawn_background_with_context};
use super::storage_api::object_utils::to_s3s_etag;
use super::storage_api::{
get_validated_store, process_lambda_configurations, process_queue_configurations, process_topic_configurations,
request_context, validate_list_object_unordered_with_delimiter,
};
use crate::admin::handlers::site_replication::{ use crate::admin::handlers::site_replication::{
site_replication_bucket_meta_hook, site_replication_delete_bucket_hook, site_replication_make_bucket_hook, site_replication_bucket_meta_hook, site_replication_delete_bucket_hook, site_replication_make_bucket_hook,
}; };
@@ -865,7 +868,7 @@ impl DefaultBucketUsecase {
.await .await
.map_err(ApiError::from)?; .map_err(ApiError::from)?;
rustfs_scanner::clear_dirty_usage_bucket(&input.bucket); rustfs_scanner::clear_dirty_usage_bucket(&input.bucket);
if let Err(err) = super::remove_bucket_usage_from_backend(store.clone(), &input.bucket).await { if let Err(err) = remove_bucket_usage_from_backend(store.clone(), &input.bucket).await {
warn!(bucket = %input.bucket, error = ?err, "failed to remove deleted bucket from data usage"); warn!(bucket = %input.bucket, error = ?err, "failed to remove deleted bucket from data usage");
} }
@@ -2315,7 +2318,7 @@ mod tests {
BucketTargets { BucketTargets {
targets: arns targets: arns
.iter() .iter()
.map(|arn| super::super::target::BucketTarget { .map(|arn| BucketTarget {
arn: (*arn).to_string(), arn: (*arn).to_string(),
target_type: BucketTargetType::ReplicationService, target_type: BucketTargetType::ReplicationService,
..Default::default() ..Default::default()
rustfs/src/app/capacity_dirty_scope_test.rs
+5 -2
View File
@@ -12,7 +12,8 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
use super::{ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints, metadata_sys}; use super::storage_api::bucket::metadata_sys;
use super::storage_api::{ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints};
use rustfs_common::heal_channel::{HealOpts, HealScanMode}; use rustfs_common::heal_channel::{HealOpts, HealScanMode};
use rustfs_object_capacity::capacity_manager::{HybridStrategyConfig, create_isolated_manager}; use rustfs_object_capacity::capacity_manager::{HybridStrategyConfig, create_isolated_manager};
use rustfs_storage_api::{BucketOperations, BucketOptions, HealOperations as _, MakeBucketOptions, ObjectIO as _}; use rustfs_storage_api::{BucketOperations, BucketOptions, HealOperations as _, MakeBucketOptions, ObjectIO as _};
@@ -77,7 +78,9 @@ async fn setup_capacity_dirty_scope_env() -> (Vec<PathBuf>, Arc<ECStore>) {
}; };
let endpoint_pools = EndpointServerPools(vec![pool_endpoints]); let endpoint_pools = EndpointServerPools(vec![pool_endpoints]);
super::init_local_disks(endpoint_pools.clone()).await.unwrap(); super::storage_api::runtime::init_local_disks(endpoint_pools.clone())
.await
.unwrap();
let server_addr: std::net::SocketAddr = "127.0.0.1:0".parse().unwrap(); let server_addr: std::net::SocketAddr = "127.0.0.1:0".parse().unwrap();
let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new()) let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
rustfs/src/app/context.rs
+8 -11
View File
@@ -26,14 +26,12 @@ pub use global::*;
pub use handles::*; pub use handles::*;
pub use interfaces::*; pub use interfaces::*;
use super::DailyAllTierStats; use super::storage_api::bucket::metadata_sys::BucketMetadataSys;
use super::ECStore; use super::storage_api::runtime::{
use super::EndpointServerPools; BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats,
use super::ScannerMetricsReport; ScannerMetricsReport, StorageClassConfig, TierConfigMgr,
use super::StorageClassConfig; };
use super::TierConfigMgr; use super::storage_api::{ECStore, EndpointServerPools};
use super::metadata_sys::BucketMetadataSys;
use super::{BucketBandwidthMonitor, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats};
use crate::config::RustFSBufferConfig; use crate::config::RustFSBufferConfig;
use rustfs_config::server_config::Config; use rustfs_config::server_config::Config;
use rustfs_credentials::Credentials; use rustfs_credentials::Credentials;
@@ -562,9 +560,8 @@ fn resolve_buffer_config_with(
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::super::Endpoint; use super::super::storage_api::runtime::init_local_disks;
use super::super::init_local_disks; use super::super::storage_api::{Endpoint, Endpoints, PoolEndpoints};
use super::super::{Endpoints, PoolEndpoints};
use super::*; use super::*;
use crate::app::context::global::AppContextTestInterfaces; use crate::app::context::global::AppContextTestInterfaces;
use crate::app::context::handles::{ use crate::app::context::handles::{
rustfs/src/app/context/global.rs
+2 -1
View File
@@ -12,7 +12,8 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
use super::super::{ECStore, set_object_store_resolver}; use super::super::storage_api::ECStore;
use super::super::storage_api::runtime::set_object_store_resolver;
use super::handles::{ use super::handles::{
IamHandle, KmsHandle, default_action_credential_interface, default_boot_time_interface, default_bucket_metadata_interface, IamHandle, KmsHandle, default_action_credential_interface, default_boot_time_interface, default_bucket_metadata_interface,
default_bucket_monitor_interface, default_buffer_config_interface, default_deployment_id_interface, default_bucket_monitor_interface, default_buffer_config_interface, default_deployment_id_interface,
rustfs/src/app/context/handles.rs
+13 -11
View File
@@ -12,10 +12,12 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
use super::super::EndpointServerPools; use super::super::storage_api::EndpointServerPools;
use super::super::StorageClassConfig; use super::super::storage_api::bucket::metadata_sys::BucketMetadataSys;
use super::super::TierConfigMgr; use super::super::storage_api::runtime::{
use super::super::metadata_sys::BucketMetadataSys; BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats,
ScannerMetricsReport, StorageClassConfig, TierConfigMgr,
};
use super::interfaces::{ use super::interfaces::{
ActionCredentialInterface, BootTimeInterface, BucketMetadataInterface, BucketMonitorInterface, BufferConfigInterface, ActionCredentialInterface, BootTimeInterface, BucketMetadataInterface, BucketMonitorInterface, BufferConfigInterface,
DeploymentIdInterface, EndpointsInterface, ExpiryStateInterface, IamInterface, InternodeMetricsInterface, KmsInterface, DeploymentIdInterface, EndpointsInterface, ExpiryStateInterface, IamInterface, InternodeMetricsInterface, KmsInterface,
@@ -153,7 +155,7 @@ impl NotifyInterface for NotifyHandle {
pub struct NotificationSystemHandle; pub struct NotificationSystemHandle;
impl NotificationSystemInterface for NotificationSystemHandle { impl NotificationSystemInterface for NotificationSystemHandle {
fn handle(&self) -> Option<&'static super::super::NotificationSys> { fn handle(&self) -> Option<&'static NotificationSys> {
runtime_sources::notification_system() runtime_sources::notification_system()
} }
} }
@@ -173,7 +175,7 @@ impl BucketMetadataInterface for BucketMetadataHandle {
pub struct BucketMonitorHandle; pub struct BucketMonitorHandle;
impl BucketMonitorInterface for BucketMonitorHandle { impl BucketMonitorInterface for BucketMonitorHandle {
fn handle(&self) -> Option<Arc<super::super::BucketBandwidthMonitor>> { fn handle(&self) -> Option<Arc<BucketBandwidthMonitor>> {
runtime_sources::bucket_monitor() runtime_sources::bucket_monitor()
} }
} }
@@ -183,7 +185,7 @@ impl BucketMonitorInterface for BucketMonitorHandle {
pub struct ReplicationPoolHandle; pub struct ReplicationPoolHandle;
impl ReplicationPoolInterface for ReplicationPoolHandle { impl ReplicationPoolInterface for ReplicationPoolHandle {
fn handle(&self) -> Option<Arc<super::super::DynReplicationPool>> { fn handle(&self) -> Option<Arc<DynReplicationPool>> {
runtime_sources::replication_pool() runtime_sources::replication_pool()
} }
} }
@@ -193,7 +195,7 @@ impl ReplicationPoolInterface for ReplicationPoolHandle {
pub struct ReplicationStatsHandle; pub struct ReplicationStatsHandle;
impl ReplicationStatsInterface for ReplicationStatsHandle { impl ReplicationStatsInterface for ReplicationStatsHandle {
fn handle(&self) -> Option<Arc<super::super::ReplicationStats>> { fn handle(&self) -> Option<Arc<ReplicationStats>> {
runtime_sources::replication_stats() runtime_sources::replication_stats()
} }
} }
@@ -213,7 +215,7 @@ impl BootTimeInterface for BootTimeHandle {
pub struct TierStatsHandle; pub struct TierStatsHandle;
impl TierStatsInterface for TierStatsHandle { impl TierStatsInterface for TierStatsHandle {
fn daily_all(&self) -> super::super::DailyAllTierStats { fn daily_all(&self) -> DailyAllTierStats {
runtime_sources::daily_tier_stats() runtime_sources::daily_tier_stats()
} }
} }
@@ -224,7 +226,7 @@ pub struct ScannerMetricsHandle;
#[async_trait] #[async_trait]
impl ScannerMetricsInterface for ScannerMetricsHandle { impl ScannerMetricsInterface for ScannerMetricsHandle {
async fn report(&self) -> super::super::ScannerMetricsReport { async fn report(&self) -> ScannerMetricsReport {
runtime_sources::scanner_metrics_report().await runtime_sources::scanner_metrics_report().await
} }
} }
@@ -360,7 +362,7 @@ impl TierConfigInterface for TierConfigHandle {
pub struct ExpiryStateHandle; pub struct ExpiryStateHandle;
impl ExpiryStateInterface for ExpiryStateHandle { impl ExpiryStateInterface for ExpiryStateHandle {
fn handle(&self) -> Arc<RwLock<super::super::ExpiryState>> { fn handle(&self) -> Arc<RwLock<ExpiryState>> {
runtime_sources::expiry_state() runtime_sources::expiry_state()
} }
} }
rustfs/src/app/context/interfaces.rs
+6 -7
View File
@@ -12,13 +12,12 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
use super::super::DailyAllTierStats; use super::super::storage_api::EndpointServerPools;
use super::super::EndpointServerPools; use super::super::storage_api::bucket::metadata_sys::BucketMetadataSys;
use super::super::ScannerMetricsReport; use super::super::storage_api::runtime::{
use super::super::StorageClassConfig; BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats,
use super::super::TierConfigMgr; ScannerMetricsReport, StorageClassConfig, TierConfigMgr,
use super::super::metadata_sys::BucketMetadataSys; };
use super::super::{BucketBandwidthMonitor, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats};
use crate::config::RustFSBufferConfig; use crate::config::RustFSBufferConfig;
use async_trait::async_trait; use async_trait::async_trait;
use rustfs_config::server_config::Config; use rustfs_config::server_config::Config;
rustfs/src/app/context/runtime_sources.rs
+9 -9
View File
@@ -12,15 +12,15 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
use super::super::metadata_sys::BucketMetadataSys; use super::super::storage_api::bucket::metadata_sys::BucketMetadataSys;
use super::super::{ use super::super::storage_api::runtime::{
BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ECStore, EndpointServerPools, ExpiryState, NotificationSys, BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats,
ReplicationStats, ScannerMetricsReport, StorageClassConfig, TierConfigMgr, collect_scanner_metrics_report, ScannerMetricsReport, StorageClassConfig, TierConfigMgr, collect_scanner_metrics_report, get_daily_all_tier_stats,
get_daily_all_tier_stats, get_global_boot_time, get_global_bucket_monitor, get_global_deployment_id, get_global_boot_time, get_global_bucket_monitor, get_global_deployment_id, get_global_endpoints_opt, get_global_expiry_state,
get_global_endpoints_opt, get_global_expiry_state, get_global_lock_client, get_global_lock_clients, get_global_lock_client, get_global_lock_clients, get_global_notification_sys, get_global_region, get_global_replication_pool,
get_global_notification_sys, get_global_region, get_global_replication_pool, get_global_replication_stats, get_global_replication_stats, get_global_tier_config_mgr, global_rustfs_port, new_object_layer_fn, set_global_storage_class,
get_global_tier_config_mgr, global_rustfs_port, new_object_layer_fn, set_global_storage_class,
}; };
use super::super::storage_api::{ECStore, EndpointServerPools};
use crate::config::{RustFSBufferConfig, get_global_buffer_config}; use crate::config::{RustFSBufferConfig, get_global_buffer_config};
use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config}; use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
use rustfs_credentials::{Credentials, get_global_action_cred}; use rustfs_credentials::{Credentials, get_global_action_cred};
@@ -108,7 +108,7 @@ pub fn notification_system() -> Option<&'static NotificationSys> {
} }
pub fn bucket_metadata() -> Option<Arc<RwLock<BucketMetadataSys>>> { pub fn bucket_metadata() -> Option<Arc<RwLock<BucketMetadataSys>>> {
super::super::metadata_sys::get_global_bucket_metadata_sys() super::super::storage_api::bucket::metadata_sys::get_global_bucket_metadata_sys()
} }
pub fn bucket_monitor() -> Option<Arc<BucketBandwidthMonitor>> { pub fn bucket_monitor() -> Option<Arc<BucketBandwidthMonitor>> {
rustfs/src/app/context/startup.rs
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
use super::super::ECStore; use super::super::storage_api::ECStore;
use super::global::{AppContext, get_global_app_context, init_global_app_context}; use super::global::{AppContext, get_global_app_context, init_global_app_context};
use super::runtime_sources; use super::runtime_sources;
use rustfs_kms::KmsServiceManager; use rustfs_kms::KmsServiceManager;
rustfs/src/app/lifecycle_transition_api_test.rs
+15 -11
View File
@@ -12,17 +12,19 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
use super::storage_api::ecfs::FS; use super::storage_api::bucket::{
use super::storage_api::{ lifecycle,
StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
};
use super::{
AppWarmBackend, ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints, TierConfig, TierType, WarmBackendGetOpts,
metadata::{BUCKET_LIFECYCLE_CONFIG, OBJECT_LOCK_CONFIG}, metadata::{BUCKET_LIFECYCLE_CONFIG, OBJECT_LOCK_CONFIG},
metadata_sys, metadata_sys,
object_api_utils::to_s3s_etag,
transition_api::{ReadCloser, ReaderImpl}, transition_api::{ReadCloser, ReaderImpl},
}; };
use super::storage_api::ecfs::FS;
use super::storage_api::object_utils::to_s3s_etag;
use super::storage_api::runtime::{AppWarmBackend, TierConfig, TierType, WarmBackendGetOpts};
use super::storage_api::{
ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints, StorageObjectInfo as ObjectInfo,
StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
};
use super::{multipart_usecase::DefaultMultipartUsecase, object_usecase::DefaultObjectUsecase}; use super::{multipart_usecase::DefaultMultipartUsecase, object_usecase::DefaultObjectUsecase};
use crate::app::bucket_usecase::DefaultBucketUsecase; use crate::app::bucket_usecase::DefaultBucketUsecase;
use crate::app::runtime_sources::resolve_tier_config_handle; use crate::app::runtime_sources::resolve_tier_config_handle;
@@ -107,7 +109,9 @@ async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>) {
let endpoint_pools = EndpointServerPools(vec![pool_endpoints]); let endpoint_pools = EndpointServerPools(vec![pool_endpoints]);
super::init_local_disks(endpoint_pools.clone()).await.unwrap(); super::storage_api::runtime::init_local_disks(endpoint_pools.clone())
.await
.unwrap();
let server_addr: std::net::SocketAddr = "127.0.0.1:9003".parse().unwrap(); let server_addr: std::net::SocketAddr = "127.0.0.1:9003".parse().unwrap();
let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new()) let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
@@ -124,7 +128,7 @@ async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>) {
let buckets = buckets_list.into_iter().map(|v| v.name).collect(); let buckets = buckets_list.into_iter().map(|v| v.name).collect();
metadata_sys::init_bucket_metadata_sys(ecstore.clone(), buckets).await; metadata_sys::init_bucket_metadata_sys(ecstore.clone(), buckets).await;
super::lifecycle::bucket_lifecycle_ops::init_background_expiry(ecstore.clone()).await; lifecycle::bucket_lifecycle_ops::init_background_expiry(ecstore.clone()).await;
let _ = GLOBAL_ENV.set((disk_paths.clone(), ecstore.clone())); let _ = GLOBAL_ENV.set((disk_paths.clone(), ecstore.clone()));
@@ -925,7 +929,7 @@ async fn delete_transitioned_object_removes_remote_tier_copy_via_usecase() {
.expect("Failed to set lifecycle configuration"); .expect("Failed to set lifecycle configuration");
let _ = upload_test_object(&ecstore, bucket.as_str(), object, payload).await; let _ = upload_test_object(&ecstore, bucket.as_str(), object, payload).await;
super::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(ecstore.clone(), bucket.as_str()) lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(ecstore.clone(), bucket.as_str())
.await .await
.expect("Failed to enqueue transitioned object"); .expect("Failed to enqueue transitioned object");
@@ -982,7 +986,7 @@ async fn lifecycle_transition_marks_dirty_disks_for_capacity_manager() {
.expect("Failed to set lifecycle configuration"); .expect("Failed to set lifecycle configuration");
let _ = upload_test_object(&ecstore, bucket.as_str(), object, payload).await; let _ = upload_test_object(&ecstore, bucket.as_str(), object, payload).await;
super::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(ecstore.clone(), bucket.as_str()) lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(ecstore.clone(), bucket.as_str())
.await .await
.expect("Failed to enqueue transitioned object"); .expect("Failed to enqueue transitioned object");
rustfs/src/app/mod.rs
-635
View File
@@ -29,638 +29,3 @@ pub(crate) mod storage_api;
mod capacity_dirty_scope_test; mod capacity_dirty_scope_test;
#[cfg(test)] #[cfg(test)]
mod lifecycle_transition_api_test; mod lifecycle_transition_api_test;
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::{collections::HashMap, sync::Arc};
mod ecstore_admin {
pub(crate) use crate::storage::ecstore_admin::get_server_info;
}
mod ecstore_bucket {
pub(crate) use crate::storage::ecstore_bucket::{
bucket_target_sys, lifecycle, metadata, metadata_sys, object_lock, policy_sys, quota, replication, tagging, target,
utils, versioning, versioning_sys,
};
}
mod ecstore_capacity {
pub(crate) use crate::storage::ecstore_capacity::{
PoolDecommissionInfo, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free,
};
}
#[allow(unused_imports)]
mod ecstore_client {
pub(crate) use crate::storage::ecstore_client::{object_api_utils, transition_api};
}
mod ecstore_config {
pub(crate) use crate::storage::ecstore_config::storageclass;
}
mod ecstore_data_usage {
pub(crate) use crate::storage::ecstore_data_usage::{
apply_bucket_usage_memory_overlay, load_data_usage_from_backend, record_bucket_object_delete_memory,
record_bucket_object_write_memory, remove_bucket_usage_from_backend,
};
}
#[allow(unused_imports)]
mod ecstore_tier {
pub(crate) use crate::storage::ecstore_tier::{tier, tier_config, warm_backend};
}
pub(crate) type DiskError = crate::storage::DiskError;
pub(crate) type DynReplicationPool = crate::storage::DynReplicationPool;
pub(crate) type ECStore = crate::storage::ECStore;
pub(crate) type EndpointServerPools = crate::storage::EndpointServerPools;
pub(crate) type ExpiryState = crate::storage::ExpiryState;
pub(crate) type NotificationSys = crate::storage::NotificationSys;
pub(crate) type BucketBandwidthMonitor = crate::storage::BucketBandwidthMonitor;
pub(crate) type ObjectStoreResolver = crate::storage::ObjectStoreResolver;
pub(crate) type ObjectInfo = <ECStore as rustfs_storage_api::ObjectOperations>::ObjectInfo;
pub(crate) type ObjectOptions = <ECStore as rustfs_storage_api::ObjectOperations>::ObjectOptions;
pub(crate) type PoolDecommissionInfo = ecstore_capacity::PoolDecommissionInfo;
pub(crate) type PoolStatus = ecstore_capacity::PoolStatus;
pub(crate) type RebalStatus = crate::storage::ecstore_rebalance::RebalStatus;
pub(crate) type StorageError = crate::storage::StorageError;
pub(crate) type Error = StorageError;
pub(crate) type TierConfigMgr = crate::storage::TierConfigMgr;
#[cfg(test)]
pub(crate) type Endpoint = crate::storage::Endpoint;
#[cfg(test)]
pub(crate) type Endpoints = crate::storage::Endpoints;
#[cfg(test)]
pub(crate) type PoolEndpoints = crate::storage::PoolEndpoints;
#[cfg(test)]
pub(crate) type TierConfig = ecstore_tier::tier_config::TierConfig;
#[cfg(test)]
pub(crate) type TierType = ecstore_tier::tier_config::TierType;
#[cfg(test)]
pub(crate) use ecstore_tier::warm_backend::WarmBackend as AppWarmBackend;
#[cfg(test)]
pub(crate) type WarmBackendGetOpts = ecstore_tier::warm_backend::WarmBackendGetOpts;
#[cfg(test)]
#[allow(non_snake_case)]
pub(crate) fn EndpointServerPools(pools: Vec<PoolEndpoints>) -> EndpointServerPools {
crate::storage::EndpointServerPools::from(pools)
}
pub(crate) trait AppObjectLockConfigExt {
fn enabled(&self) -> bool;
}
impl AppObjectLockConfigExt for s3s::dto::ObjectLockConfiguration {
fn enabled(&self) -> bool {
<s3s::dto::ObjectLockConfiguration as ecstore_bucket::object_lock::ObjectLockApi>::enabled(self)
}
}
pub(crate) trait AppReplicationConfigExt {
fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec<String>;
fn replicate(&self, opts: &replication::ObjectOpts) -> bool;
}
impl AppReplicationConfigExt for s3s::dto::ReplicationConfiguration {
fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec<String> {
<s3s::dto::ReplicationConfiguration as ecstore_bucket::replication::ReplicationConfigurationExt>::filter_target_arns(
self, obj,
)
}
fn replicate(&self, opts: &replication::ObjectOpts) -> bool {
<s3s::dto::ReplicationConfiguration as ecstore_bucket::replication::ReplicationConfigurationExt>::replicate(self, opts)
}
}
pub(crate) trait AppVersioningConfigExt {
fn prefix_enabled(&self, prefix: &str) -> bool;
fn suspended(&self) -> bool;
}
impl AppVersioningConfigExt for s3s::dto::VersioningConfiguration {
fn prefix_enabled(&self, prefix: &str) -> bool {
<s3s::dto::VersioningConfiguration as ecstore_bucket::versioning::VersioningApi>::prefix_enabled(self, prefix)
}
fn suspended(&self) -> bool {
<s3s::dto::VersioningConfiguration as ecstore_bucket::versioning::VersioningApi>::suspended(self)
}
}
pub(crate) async fn predict_lifecycle_expiration(
lifecycle: &s3s::dto::BucketLifecycleConfiguration,
obj: &lifecycle::lifecycle::ObjectOpts,
) -> lifecycle::lifecycle::Event {
ecstore_bucket::lifecycle::lifecycle::Lifecycle::predict_expiration(lifecycle, obj).await
}
pub(crate) fn validate_restore_request(request: &s3s::dto::RestoreRequest, api: Arc<ECStore>) -> std::io::Result<()> {
<s3s::dto::RestoreRequest as ecstore_bucket::lifecycle::bucket_lifecycle_ops::RestoreRequestOps>::validate(request, api)
}
pub(crate) async fn get_server_info(get_pools: bool) -> rustfs_madmin::InfoMessage {
ecstore_admin::get_server_info(get_pools).await
}
pub(crate) mod bucket_target_sys {
pub(crate) type BucketTargetSys = super::ecstore_bucket::bucket_target_sys::BucketTargetSys;
}
pub(crate) mod lifecycle {
use super::ECStore;
pub(crate) mod bucket_lifecycle_audit {
pub(crate) type LcEventSrc = super::super::ecstore_bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc;
}
pub(crate) mod bucket_lifecycle_ops {
use std::sync::Arc;
use super::ECStore;
use super::bucket_lifecycle_audit::LcEventSrc;
#[cfg(test)]
pub(crate) async fn init_background_expiry(api: Arc<ECStore>) {
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::init_background_expiry(api).await;
}
pub(crate) async fn enqueue_expiry_for_existing_objects(
api: Arc<ECStore>,
bucket: &str,
) -> Result<(), super::super::Error> {
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_expiry_for_existing_objects(api, bucket).await
}
pub(crate) async fn enqueue_transition_for_existing_objects(
api: Arc<ECStore>,
bucket: &str,
) -> Result<(), super::super::Error> {
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(api, bucket)
.await
}
pub(crate) async fn enqueue_transition_immediate(oi: &super::super::ObjectInfo, src: LcEventSrc) {
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_immediate(oi, src).await;
}
pub(crate) async fn post_restore_opts(
version_id: &str,
bucket: &str,
object: &str,
) -> Result<super::super::ObjectOptions, std::io::Error> {
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::post_restore_opts(version_id, bucket, object).await
}
pub(crate) async fn validate_transition_tier(lc: &s3s::dto::BucketLifecycleConfiguration) -> Result<(), std::io::Error> {
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::validate_transition_tier(lc).await
}
pub(crate) async fn validate_lifecycle_config(
lc: &s3s::dto::BucketLifecycleConfiguration,
lock_config: &s3s::dto::ObjectLockConfiguration,
) -> Result<(), std::io::Error> {
use super::super::ecstore_bucket::lifecycle::lifecycle::Lifecycle as _;
lc.validate(lock_config).await
}
}
pub(crate) mod lifecycle_contract {
#[cfg(test)]
pub(crate) type IlmAction = super::super::ecstore_bucket::lifecycle::lifecycle::IlmAction;
pub(crate) type Event = super::super::ecstore_bucket::lifecycle::lifecycle::Event;
pub(crate) type ObjectOpts = super::super::ecstore_bucket::lifecycle::lifecycle::ObjectOpts;
pub(crate) type TransitionOptions = super::super::ecstore_bucket::lifecycle::lifecycle::TransitionOptions;
pub(crate) const TRANSITION_COMPLETE: &str = super::super::ecstore_bucket::lifecycle::lifecycle::TRANSITION_COMPLETE;
pub(crate) fn expected_expiry_time(mod_time: time::OffsetDateTime, days: i32) -> time::OffsetDateTime {
super::super::ecstore_bucket::lifecycle::lifecycle::expected_expiry_time(mod_time, days)
}
}
pub(crate) use lifecycle_contract as lifecycle;
pub(crate) mod tier_delete_journal {
use std::sync::Arc;
use super::ECStore;
pub(crate) async fn persist_tier_delete_journal_entry(
api: Arc<ECStore>,
je: &super::tier_sweeper::Jentry,
) -> std::io::Result<()> {
super::super::ecstore_bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry(api, je).await
}
}
pub(crate) mod tier_sweeper {
pub(crate) type Jentry = super::super::ecstore_bucket::lifecycle::tier_sweeper::Jentry;
pub(crate) fn transitioned_delete_journal_entry(
version_id: Option<uuid::Uuid>,
versioned: bool,
suspended: bool,
transitioned: &rustfs_storage_api::TransitionedObject,
) -> Option<Jentry> {
super::super::ecstore_bucket::lifecycle::tier_sweeper::transitioned_delete_journal_entry(
version_id,
versioned,
suspended,
transitioned,
)
}
pub(crate) fn transitioned_force_delete_journal_entry(
transitioned: &rustfs_storage_api::TransitionedObject,
) -> Option<Jentry> {
super::super::ecstore_bucket::lifecycle::tier_sweeper::transitioned_force_delete_journal_entry(transitioned)
}
}
}
pub(crate) mod metadata {
pub(crate) const BUCKET_CORS_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_CORS_CONFIG;
pub(crate) const BUCKET_LIFECYCLE_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
pub(crate) const BUCKET_NOTIFICATION_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_NOTIFICATION_CONFIG;
pub(crate) const BUCKET_POLICY_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_POLICY_CONFIG;
pub(crate) const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG;
pub(crate) const BUCKET_REPLICATION_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_REPLICATION_CONFIG;
pub(crate) const BUCKET_SSECONFIG: &str = super::ecstore_bucket::metadata::BUCKET_SSECONFIG;
pub(crate) const BUCKET_TAGGING_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_TAGGING_CONFIG;
pub(crate) const BUCKET_TARGETS_FILE: &str = super::ecstore_bucket::metadata::BUCKET_TARGETS_FILE;
pub(crate) const BUCKET_VERSIONING_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_VERSIONING_CONFIG;
#[cfg(test)]
pub(crate) const OBJECT_LOCK_CONFIG: &str = super::ecstore_bucket::metadata::OBJECT_LOCK_CONFIG;
}
pub(crate) mod metadata_sys {
use std::sync::Arc;
use rustfs_policy::policy::BucketPolicy;
use s3s::dto::{
BucketLifecycleConfiguration, CORSConfiguration, NotificationConfiguration, ObjectLockConfiguration,
PublicAccessBlockConfiguration, ServerSideEncryptionConfiguration, Tagging,
};
use time::OffsetDateTime;
use tokio::sync::RwLock;
#[cfg(test)]
use super::ECStore;
use super::Error;
use super::target::BucketTargets;
pub(crate) type BucketMetadataSys = super::ecstore_bucket::metadata_sys::BucketMetadataSys;
#[cfg(test)]
pub(crate) async fn init_bucket_metadata_sys(api: Arc<ECStore>, buckets: Vec<String>) {
super::ecstore_bucket::metadata_sys::init_bucket_metadata_sys(api, buckets).await;
}
pub(crate) async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime, Error> {
super::ecstore_bucket::metadata_sys::delete(bucket, config_file).await
}
pub(crate) async fn get_bucket_policy(bucket: &str) -> Result<(BucketPolicy, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_bucket_policy(bucket).await
}
pub(crate) async fn get_bucket_policy_raw(bucket: &str) -> Result<(String, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_bucket_policy_raw(bucket).await
}
pub(crate) async fn get_bucket_targets_config(bucket: &str) -> Result<BucketTargets, Error> {
super::ecstore_bucket::metadata_sys::get_bucket_targets_config(bucket).await
}
pub(crate) async fn get_cors_config(bucket: &str) -> Result<(CORSConfiguration, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_cors_config(bucket).await
}
pub(crate) fn get_global_bucket_metadata_sys() -> Option<Arc<RwLock<BucketMetadataSys>>> {
super::ecstore_bucket::metadata_sys::get_global_bucket_metadata_sys()
}
pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_lifecycle_config(bucket).await
}
pub(crate) async fn get_notification_config(bucket: &str) -> Result<Option<NotificationConfiguration>, Error> {
super::ecstore_bucket::metadata_sys::get_notification_config(bucket).await
}
pub(crate) async fn get_object_lock_config(bucket: &str) -> Result<(ObjectLockConfiguration, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_object_lock_config(bucket).await
}
pub(crate) async fn get_public_access_block_config(
bucket: &str,
) -> Result<(PublicAccessBlockConfiguration, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_public_access_block_config(bucket).await
}
pub(crate) async fn get_replication_config(
bucket: &str,
) -> Result<(s3s::dto::ReplicationConfiguration, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_replication_config(bucket).await
}
pub(crate) async fn get_sse_config(bucket: &str) -> Result<(ServerSideEncryptionConfiguration, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_sse_config(bucket).await
}
pub(crate) async fn get_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime), Error> {
super::ecstore_bucket::metadata_sys::get_tagging_config(bucket).await
}
pub(crate) async fn update(bucket: &str, config_file: &str, data: Vec<u8>) -> Result<OffsetDateTime, Error> {
super::ecstore_bucket::metadata_sys::update(bucket, config_file, data).await
}
}
pub(crate) mod object_api_utils {
pub(crate) fn to_s3s_etag(etag: &str) -> s3s::dto::ETag {
crate::storage::to_s3s_etag(etag)
}
}
pub(crate) mod object_lock {
pub(crate) mod objectlock {
pub(crate) fn get_object_legalhold_meta(
meta: &std::collections::HashMap<String, String>,
) -> s3s::dto::ObjectLockLegalHold {
super::super::ecstore_bucket::object_lock::objectlock::get_object_legalhold_meta(meta)
}
pub(crate) fn get_object_retention_meta(
meta: &std::collections::HashMap<String, String>,
) -> s3s::dto::ObjectLockRetention {
super::super::ecstore_bucket::object_lock::objectlock::get_object_retention_meta(meta)
}
}
pub(crate) mod objectlock_sys {
pub(crate) type BucketObjectLockSys = super::super::ecstore_bucket::object_lock::objectlock_sys::BucketObjectLockSys;
pub(crate) type ObjectLockBlockReason = super::super::ecstore_bucket::object_lock::objectlock_sys::ObjectLockBlockReason;
pub(crate) async fn check_object_lock_for_deletion(
bucket: &str,
obj_info: &super::super::ObjectInfo,
bypass_governance: bool,
) -> Option<ObjectLockBlockReason> {
super::super::ecstore_bucket::object_lock::objectlock_sys::check_object_lock_for_deletion(
bucket,
obj_info,
bypass_governance,
)
.await
}
pub(crate) fn is_retention_active(mode: &str, retain_until_date: Option<&s3s::dto::Date>) -> bool {
super::super::ecstore_bucket::object_lock::objectlock_sys::is_retention_active(mode, retain_until_date)
}
}
}
pub(crate) mod policy_sys {
pub(crate) type PolicySys = super::ecstore_bucket::policy_sys::PolicySys;
}
pub(crate) mod quota {
pub(crate) mod checker {
pub(crate) type QuotaChecker = super::super::ecstore_bucket::quota::checker::QuotaChecker;
}
pub(crate) type QuotaOperation = super::ecstore_bucket::quota::QuotaOperation;
}
pub(crate) mod replication {
use std::collections::HashMap;
use std::sync::Arc;
pub(crate) type DeletedObjectReplicationInfo = super::ecstore_bucket::replication::DeletedObjectReplicationInfo;
pub(crate) type MustReplicateOptions = super::ecstore_bucket::replication::MustReplicateOptions;
pub(crate) type ObjectOpts = super::ecstore_bucket::replication::ObjectOpts;
pub(crate) type ReplicateDecision = rustfs_filemeta::ReplicateDecision;
pub(crate) async fn check_replicate_delete(
bucket: &str,
dobj: &rustfs_storage_api::ObjectToDelete,
oi: &super::ObjectInfo,
del_opts: &super::ObjectOptions,
gerr: Option<String>,
) -> ReplicateDecision {
super::ecstore_bucket::replication::check_replicate_delete(bucket, dobj, oi, del_opts, gerr).await
}
pub(crate) fn get_must_replicate_options(
user_defined: &HashMap<String, String>,
user_tags: String,
status: rustfs_filemeta::ReplicationStatusType,
op_type: rustfs_filemeta::ReplicationType,
opts: super::ObjectOptions,
) -> MustReplicateOptions {
super::ecstore_bucket::replication::get_must_replicate_options(user_defined, user_tags, status, op_type, opts)
}
pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplicateOptions) -> ReplicateDecision {
super::ecstore_bucket::replication::must_replicate(bucket, object, mopts).await
}
pub(crate) async fn schedule_replication(
oi: super::ObjectInfo,
store: Arc<super::ECStore>,
dsc: ReplicateDecision,
op_type: rustfs_filemeta::ReplicationType,
) {
super::ecstore_bucket::replication::schedule_replication(oi, store, dsc, op_type).await;
}
pub(crate) async fn schedule_replication_delete(dv: DeletedObjectReplicationInfo) {
super::ecstore_bucket::replication::schedule_replication_delete(dv).await;
}
}
pub(crate) mod tagging {
pub(crate) fn decode_tags(tags: &str) -> Vec<s3s::dto::Tag> {
super::ecstore_bucket::tagging::decode_tags(tags)
}
}
pub(crate) mod target {
#[cfg(test)]
pub(crate) type BucketTarget = super::ecstore_bucket::target::BucketTarget;
pub(crate) type BucketTargetType = super::ecstore_bucket::target::BucketTargetType;
pub(crate) type BucketTargets = super::ecstore_bucket::target::BucketTargets;
}
pub(crate) mod utils {
pub(crate) fn serialize<T: s3s::xml::Serialize>(val: &T) -> s3s::xml::SerResult<Vec<u8>> {
super::ecstore_bucket::utils::serialize(val)
}
}
pub(crate) mod versioning {}
pub(crate) mod versioning_sys {
pub(crate) type BucketVersioningSys = super::ecstore_bucket::versioning_sys::BucketVersioningSys;
}
#[cfg(test)]
pub(crate) mod transition_api {
pub(crate) type ReadCloser = super::ecstore_client::transition_api::ReadCloser;
pub(crate) type ReaderImpl = super::ecstore_client::transition_api::ReaderImpl;
}
pub(crate) mod storageclass {
pub(crate) const STANDARD: &str = super::ecstore_config::storageclass::STANDARD;
#[cfg(test)]
pub(crate) const STANDARD_IA: &str = super::ecstore_config::storageclass::STANDARD_IA;
}
pub(crate) type StorageClassConfig = crate::storage::ecstore_config::storageclass::Config;
pub(crate) fn set_global_storage_class(cfg: StorageClassConfig) {
crate::storage::ecstore_config::set_global_storage_class(cfg);
}
pub(crate) fn get_total_usable_capacity(disks: &[rustfs_madmin::Disk], info: &rustfs_madmin::StorageInfo) -> usize {
ecstore_capacity::get_total_usable_capacity(disks, info)
}
pub(crate) fn get_total_usable_capacity_free(disks: &[rustfs_madmin::Disk], info: &rustfs_madmin::StorageInfo) -> usize {
ecstore_capacity::get_total_usable_capacity_free(disks, info)
}
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
}
pub(crate) async fn load_data_usage_from_backend(
store: Arc<ECStore>,
) -> std::result::Result<rustfs_data_usage::DataUsageInfo, Error> {
ecstore_data_usage::load_data_usage_from_backend(store).await
}
pub(crate) async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) {
ecstore_data_usage::record_bucket_object_delete_memory(bucket, deleted_size, removed_current_object).await;
}
pub(crate) async fn record_bucket_object_write_memory(bucket: &str, previous_current_size: Option<u64>, new_size: u64) {
ecstore_data_usage::record_bucket_object_write_memory(bucket, previous_current_size, new_size).await;
}
pub(crate) async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str) -> std::result::Result<(), Error> {
ecstore_data_usage::remove_bucket_usage_from_backend(store, bucket).await
}
pub(crate) fn is_all_buckets_not_found(errs: &[Option<DiskError>]) -> bool {
crate::storage::is_all_buckets_not_found(errs)
}
pub(crate) fn is_err_bucket_not_found(err: &Error) -> bool {
crate::storage::is_err_bucket_not_found(err)
}
pub(crate) fn is_err_object_not_found(err: &Error) -> bool {
crate::storage::is_err_object_not_found(err)
}
pub(crate) fn is_err_version_not_found(err: &Error) -> bool {
crate::storage::is_err_version_not_found(err)
}
pub(crate) fn get_global_endpoints_opt() -> Option<EndpointServerPools> {
crate::storage::get_global_endpoints_opt()
}
pub(crate) fn get_global_deployment_id() -> Option<String> {
crate::storage::get_global_deployment_id()
}
pub(crate) fn get_global_lock_client() -> Option<Arc<dyn rustfs_lock::client::LockClient>> {
crate::storage::get_global_lock_client()
}
pub(crate) fn get_global_lock_clients() -> Option<&'static HashMap<String, Arc<dyn rustfs_lock::client::LockClient>>> {
crate::storage::get_global_lock_clients()
}
pub(crate) fn get_global_region() -> Option<s3s::region::Region> {
crate::storage::get_global_region()
}
pub(crate) fn global_rustfs_port() -> u16 {
crate::storage::global_rustfs_port()
}
pub(crate) fn get_global_tier_config_mgr() -> Arc<tokio::sync::RwLock<TierConfigMgr>> {
crate::storage::get_global_tier_config_mgr()
}
pub(crate) fn get_global_expiry_state() -> Arc<tokio::sync::RwLock<ExpiryState>> {
crate::storage::get_global_expiry_state()
}
pub(crate) fn new_object_layer_fn() -> Option<Arc<ECStore>> {
crate::storage::new_object_layer_fn()
}
pub(crate) fn set_object_store_resolver(resolver: Arc<ObjectStoreResolver>) -> bool {
crate::storage::set_object_store_resolver(resolver)
}
pub(crate) fn get_global_notification_sys() -> Option<&'static NotificationSys> {
crate::storage::get_global_notification_sys()
}
pub(crate) fn get_global_bucket_monitor() -> Option<Arc<BucketBandwidthMonitor>> {
crate::storage::get_global_bucket_monitor()
}
pub(crate) fn get_global_replication_pool() -> Option<Arc<DynReplicationPool>> {
crate::storage::get_global_replication_pool()
}
pub(crate) type ReplicationStats = crate::storage::ReplicationStats;
pub(crate) fn get_global_replication_stats() -> Option<Arc<ReplicationStats>> {
crate::storage::get_global_replication_stats()
}
pub(crate) type DailyAllTierStats = crate::storage::DailyAllTierStats;
pub(crate) fn get_global_boot_time() -> Option<std::time::SystemTime> {
crate::storage::get_global_boot_time()
}
pub(crate) fn get_daily_all_tier_stats() -> DailyAllTierStats {
crate::storage::get_daily_all_tier_stats()
}
pub(crate) type ScannerMetricsReport = rustfs_common::metrics::ScannerMetricsReport;
pub(crate) async fn collect_scanner_metrics_report() -> ScannerMetricsReport {
rustfs_common::metrics::global_metrics().report().await
}
#[cfg(test)]
pub(crate) async fn init_local_disks(endpoint_pools: EndpointServerPools) -> Result<(), Error> {
crate::storage::init_local_disks(endpoint_pools).await
}
rustfs/src/app/multipart_usecase.rs
+13 -12
View File
@@ -14,19 +14,28 @@
//! Multipart application use-case contracts. //! Multipart application use-case contracts.
use super::ECStore;
use super::object_api_utils::to_s3s_etag;
use super::quota::checker::QuotaChecker;
use super::s3_api::multipart::{ use super::s3_api::multipart::{
ListMultipartUploadsParams, build_list_multipart_uploads_output, build_list_parts_output, ListMultipartUploadsParams, build_list_multipart_uploads_output, build_list_parts_output,
parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number, parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number,
}; };
use super::storage_api::ECStore;
use super::storage_api::access::has_bypass_governance_header; use super::storage_api::access::has_bypass_governance_header;
use super::storage_api::bucket::quota::checker::QuotaChecker;
use super::storage_api::bucket::{
lifecycle::{bucket_lifecycle_audit::LcEventSrc, bucket_lifecycle_ops::enqueue_transition_immediate},
metadata_sys,
quota::QuotaOperation,
replication::{get_must_replicate_options, must_replicate, schedule_replication},
versioning_sys::BucketVersioningSys,
};
use super::storage_api::compression::is_disk_compressible; use super::storage_api::compression::is_disk_compressible;
use super::storage_api::data_usage::record_bucket_object_write_memory;
use super::storage_api::error::{StorageError, is_err_object_not_found, is_err_version_not_found};
use super::storage_api::helper::OperationHelper; use super::storage_api::helper::OperationHelper;
#[cfg(test)] #[cfg(test)]
use super::storage_api::io::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader, wrap_reader}; use super::storage_api::io::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader, wrap_reader};
use super::storage_api::io::{HashReader, WriteEncryption, WritePlan, compression_metadata_value}; use super::storage_api::io::{HashReader, WriteEncryption, WritePlan, compression_metadata_value};
use super::storage_api::object_utils::to_s3s_etag;
use super::storage_api::options::{ use super::storage_api::options::{
copy_src_opts, extract_metadata_from_mime, get_complete_multipart_upload_opts, get_content_sha256_with_query, get_opts, copy_src_opts, extract_metadata_from_mime, get_complete_multipart_upload_opts, get_content_sha256_with_query, get_opts,
parse_copy_source_range, put_opts, validate_archive_content_encoding, parse_copy_source_range, put_opts, validate_archive_content_encoding,
@@ -39,14 +48,6 @@ use super::storage_api::sse::{
mark_encrypted_multipart_metadata, sse_decryption, sse_prepare_encryption, mark_encrypted_multipart_metadata, sse_decryption, sse_prepare_encryption,
}; };
use super::storage_api::{StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader}; use super::storage_api::{StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader};
use super::{StorageError, is_err_object_not_found, is_err_version_not_found};
use super::{
lifecycle::{bucket_lifecycle_audit::LcEventSrc, bucket_lifecycle_ops::enqueue_transition_immediate},
metadata_sys,
quota::QuotaOperation,
replication::{get_must_replicate_options, must_replicate, schedule_replication},
versioning_sys::BucketVersioningSys,
};
use crate::app::object_usecase::{build_put_like_object_lock_metadata, validate_existing_object_lock_for_write}; use crate::app::object_usecase::{build_put_like_object_lock_metadata, validate_existing_object_lock_for_write};
use crate::app::runtime_sources::{AppContext, get_global_app_context, resolve_object_store_handle_for_context}; use crate::app::runtime_sources::{AppContext, get_global_app_context, resolve_object_store_handle_for_context};
use crate::capacity::record_capacity_write; use crate::capacity::record_capacity_write;
@@ -488,7 +489,7 @@ impl DefaultMultipartUsecase {
)); ));
} }
// Update quota tracking after successful multipart upload // Update quota tracking after successful multipart upload
super::record_bucket_object_write_memory(&bucket, previous_current_size, obj_info.size.max(0) as u64).await; record_bucket_object_write_memory(&bucket, previous_current_size, obj_info.size.max(0) as u64).await;
} }
Err(e) => { Err(e) => {
warn!("Quota check failed for bucket {}: {}, allowing operation", bucket, e); warn!("Quota check failed for bucket {}: {}, allowing operation", bucket, e);
rustfs/src/app/object_usecase.rs
+39 -33
View File
@@ -14,23 +14,50 @@
//! Object application use-case contracts. //! Object application use-case contracts.
use super::object_api_utils::to_s3s_etag;
use super::quota::checker::QuotaChecker;
use super::storageclass;
// Performance metrics recording (with zero-copy-metrics integration) // Performance metrics recording (with zero-copy-metrics integration)
use super::ECStore;
use super::s3_api::multipart::parse_list_parts_params; use super::s3_api::multipart::parse_list_parts_params;
use super::storage_api::ECStore;
use super::storage_api::access::{PostObjectRequestMarker, authorize_request, has_bypass_governance_header, req_info_mut}; use super::storage_api::access::{PostObjectRequestMarker, authorize_request, has_bypass_governance_header, req_info_mut};
use super::storage_api::bucket::quota::checker::QuotaChecker;
use super::storage_api::bucket::{
ReplicationConfigExt as _, VersioningConfigExt as _,
lifecycle::{
bucket_lifecycle_audit::LcEventSrc,
bucket_lifecycle_ops::{enqueue_transition_immediate, post_restore_opts},
lifecycle::{self, TransitionOptions},
tier_delete_journal, tier_sweeper,
},
metadata_sys,
object_lock::{
objectlock::{get_object_legalhold_meta, get_object_retention_meta},
objectlock_sys::{BucketObjectLockSys, check_object_lock_for_deletion, is_retention_active},
},
predict_lifecycle_expiration,
quota::QuotaOperation,
replication::{
DeletedObjectReplicationInfo, ObjectOpts as ReplicationObjectOpts, check_replicate_delete, get_must_replicate_options,
must_replicate, schedule_replication, schedule_replication_delete,
},
tagging::decode_tags,
validate_restore_request,
versioning_sys::BucketVersioningSys,
};
use super::storage_api::compression::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible}; use super::storage_api::compression::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible};
use super::storage_api::concurrency::{ use super::storage_api::concurrency::{
self, ConcurrencyManager, GetObjectGuard, PutObjectGuard, get_concurrency_aware_buffer_size, get_concurrency_manager, self, ConcurrencyManager, GetObjectGuard, PutObjectGuard, get_concurrency_aware_buffer_size, get_concurrency_manager,
get_put_concurrency_aware_buffer_size, get_put_concurrency_aware_buffer_size,
}; };
use super::storage_api::data_usage::{record_bucket_object_delete_memory, record_bucket_object_write_memory};
use super::storage_api::deadlock_detector; use super::storage_api::deadlock_detector;
use super::storage_api::ecfs::FS; use super::storage_api::ecfs::FS;
use super::storage_api::error::{
DiskError, Error as EcstoreError, StorageError, is_all_buckets_not_found, is_err_bucket_not_found, is_err_object_not_found,
is_err_version_not_found,
};
use super::storage_api::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children}; use super::storage_api::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children};
use super::storage_api::helper::{OperationHelper, spawn_background_with_context}; use super::storage_api::helper::{OperationHelper, spawn_background_with_context};
use super::storage_api::io::{DynReader, HashReader, WritePlan, compression_metadata_value, wrap_reader}; use super::storage_api::io::{DynReader, HashReader, WritePlan, compression_metadata_value, wrap_reader};
use super::storage_api::object_utils::to_s3s_etag;
use super::storage_api::options::{ use super::storage_api::options::{
copy_dst_opts, copy_src_opts, del_opts, extract_metadata, extract_metadata_from_mime_with_object_name, copy_dst_opts, copy_src_opts, del_opts, extract_metadata, extract_metadata_from_mime_with_object_name,
filter_object_metadata, get_content_sha256_with_query, get_opts, normalize_content_encoding_for_storage, put_opts, filter_object_metadata, get_content_sha256_with_query, get_opts, normalize_content_encoding_for_storage, put_opts,
@@ -42,6 +69,7 @@ use super::storage_api::sse::{
encryption_material_to_metadata, extract_server_side_encryption_from_headers, extract_ssec_params_from_headers, encryption_material_to_metadata, extract_server_side_encryption_from_headers, extract_ssec_params_from_headers,
extract_ssekms_context_from_headers, get_buffer_size_opt_in, map_get_object_reader_error, sse_decryption, sse_encryption, extract_ssekms_context_from_headers, get_buffer_size_opt_in, map_get_object_reader_error, sse_decryption, sse_encryption,
}; };
use super::storage_api::storage_class as storageclass;
use super::storage_api::timeout_wrapper::{GetObjectTimeoutPolicy, RequestTimeoutWrapper}; use super::storage_api::timeout_wrapper::{GetObjectTimeoutPolicy, RequestTimeoutWrapper};
use super::storage_api::{ use super::storage_api::{
RFC1123, check_preconditions, get_validated_store, has_replication_rules, parse_object_lock_legal_hold, RFC1123, check_preconditions, get_validated_store, has_replication_rules, parse_object_lock_legal_hold,
@@ -49,28 +77,6 @@ use super::storage_api::{
strip_managed_encryption_metadata, validate_bucket_object_lock_enabled, validate_object_key, validate_sse_headers_for_read, strip_managed_encryption_metadata, validate_bucket_object_lock_enabled, validate_object_key, validate_sse_headers_for_read,
validate_sse_headers_for_write, validate_ssec_for_read, wrap_response_with_cors, validate_sse_headers_for_write, validate_ssec_for_read, wrap_response_with_cors,
}; };
use super::{AppReplicationConfigExt as _, AppVersioningConfigExt as _, predict_lifecycle_expiration, validate_restore_request};
use super::{DiskError, is_all_buckets_not_found};
use super::{Error as EcstoreError, StorageError, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found};
use super::{
lifecycle::{
bucket_lifecycle_audit::LcEventSrc,
bucket_lifecycle_ops::{enqueue_transition_immediate, post_restore_opts},
lifecycle::{self, TransitionOptions},
},
metadata_sys,
object_lock::{
objectlock::{get_object_legalhold_meta, get_object_retention_meta},
objectlock_sys::{BucketObjectLockSys, check_object_lock_for_deletion, is_retention_active},
},
quota::QuotaOperation,
replication::{
DeletedObjectReplicationInfo, ObjectOpts as ReplicationObjectOpts, check_replicate_delete, get_must_replicate_options,
must_replicate, schedule_replication, schedule_replication_delete,
},
tagging::decode_tags,
versioning_sys::BucketVersioningSys,
};
use crate::app::runtime_sources::{ use crate::app::runtime_sources::{
AppContext, get_global_app_context, resolve_expiry_state_handle, resolve_notify_interface_for_context, AppContext, get_global_app_context, resolve_expiry_state_handle, resolve_notify_interface_for_context,
resolve_object_store_handle_for_context, resolve_object_store_handle_for_context,
@@ -316,10 +322,10 @@ async fn enqueue_transitioned_delete_cleanup(
let _activity_guard = DeleteTailActivityGuard::new(DeleteTailStage::Cleanup); let _activity_guard = DeleteTailActivityGuard::new(DeleteTailStage::Cleanup);
let je = if opts.delete_prefix { let je = if opts.delete_prefix {
super::lifecycle::tier_sweeper::transitioned_force_delete_journal_entry(&existing.transitioned_object) tier_sweeper::transitioned_force_delete_journal_entry(&existing.transitioned_object)
} else { } else {
let version_id = opts.version_id.as_ref().and_then(|v| Uuid::parse_str(v).ok()); let version_id = opts.version_id.as_ref().and_then(|v| Uuid::parse_str(v).ok());
super::lifecycle::tier_sweeper::transitioned_delete_journal_entry( tier_sweeper::transitioned_delete_journal_entry(
version_id, version_id,
opts.versioned, opts.versioned,
opts.version_suspended, opts.version_suspended,
@@ -330,7 +336,7 @@ async fn enqueue_transitioned_delete_cleanup(
return Ok(()); return Ok(());
}; };
super::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry(store, &je).await?; tier_delete_journal::persist_tier_delete_journal_entry(store, &je).await?;
let expiry_state = resolve_expiry_state_handle(); let expiry_state = resolve_expiry_state_handle();
let mut expiry_state = expiry_state.write().await; let mut expiry_state = expiry_state.write().await;
@@ -2591,7 +2597,7 @@ impl DefaultObjectUsecase {
maybe_enqueue_transition_immediate(&obj_info, LcEventSrc::S3PutObject).await; maybe_enqueue_transition_immediate(&obj_info, LcEventSrc::S3PutObject).await;
// Fast in-memory update for immediate quota and admin usage consistency // Fast in-memory update for immediate quota and admin usage consistency
super::record_bucket_object_write_memory(&bucket, previous_current_size, obj_info.size.max(0) as u64).await; record_bucket_object_write_memory(&bucket, previous_current_size, obj_info.size.max(0) as u64).await;
let raw_version = obj_info.version_id.map(|v| v.to_string()); let raw_version = obj_info.version_id.map(|v| v.to_string());
@@ -3558,7 +3564,7 @@ impl DefaultObjectUsecase {
// Update quota tracking after successful copy // Update quota tracking after successful copy
if has_bucket_metadata { if has_bucket_metadata {
super::record_bucket_object_write_memory(&bucket, previous_current_size, oi.size.max(0) as u64).await; record_bucket_object_write_memory(&bucket, previous_current_size, oi.size.max(0) as u64).await;
} }
let raw_dest_version = oi.version_id.map(|v| v.to_string()); let raw_dest_version = oi.version_id.map(|v| v.to_string());
@@ -3835,7 +3841,7 @@ impl DefaultObjectUsecase {
); );
} }
let size = object_sizes[i].max(0) as u64; let size = object_sizes[i].max(0) as u64;
super::record_bucket_object_delete_memory( record_bucket_object_delete_memory(
&bucket, &bucket,
size, size,
existing_object_infos[i].is_some() && object_to_delete[i].version_id.is_none(), existing_object_infos[i].is_some() && object_to_delete[i].version_id.is_none(),
@@ -4072,7 +4078,7 @@ impl DefaultObjectUsecase {
} }
// Fast in-memory update for immediate quota and admin usage consistency // Fast in-memory update for immediate quota and admin usage consistency
super::record_bucket_object_delete_memory(&bucket, obj_info.size.max(0) as u64, opts.version_id.is_none()).await; record_bucket_object_delete_memory(&bucket, obj_info.size.max(0) as u64, opts.version_id.is_none()).await;
if obj_info.name.is_empty() { if obj_info.name.is_empty() {
if replicate_force_delete { if replicate_force_delete {
rustfs/src/app/storage_api.rs
+614
View File
@@ -14,12 +14,607 @@
//! App-local boundary for storage-layer helper APIs used by S3 use cases. //! App-local boundary for storage-layer helper APIs used by S3 use cases.
pub(crate) use crate::storage::ECStore;
#[cfg(test)]
pub(crate) use crate::storage::{Endpoint, Endpoints, PoolEndpoints};
pub(crate) type EndpointServerPools = crate::storage::EndpointServerPools;
#[cfg(test)]
#[allow(non_snake_case)]
pub(crate) fn EndpointServerPools(pools: Vec<PoolEndpoints>) -> EndpointServerPools {
crate::storage::EndpointServerPools::from(pools)
}
pub(crate) mod admin {
pub(crate) async fn get_server_info(get_pools: bool) -> rustfs_madmin::InfoMessage {
crate::storage::ecstore_admin::get_server_info(get_pools).await
}
}
pub(crate) mod capacity {
pub(crate) type PoolDecommissionInfo = crate::storage::ecstore_capacity::PoolDecommissionInfo;
pub(crate) type PoolStatus = crate::storage::ecstore_capacity::PoolStatus;
pub(crate) type RebalStatus = crate::storage::ecstore_rebalance::RebalStatus;
pub(crate) fn get_total_usable_capacity(disks: &[rustfs_madmin::Disk], info: &rustfs_madmin::StorageInfo) -> usize {
crate::storage::ecstore_capacity::get_total_usable_capacity(disks, info)
}
pub(crate) fn get_total_usable_capacity_free(disks: &[rustfs_madmin::Disk], info: &rustfs_madmin::StorageInfo) -> usize {
crate::storage::ecstore_capacity::get_total_usable_capacity_free(disks, info)
}
}
pub(crate) mod data_usage {
use std::sync::Arc;
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
crate::storage::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
}
pub(crate) async fn load_data_usage_from_backend(
store: Arc<super::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::StorageError> {
crate::storage::ecstore_data_usage::load_data_usage_from_backend(store).await
}
pub(crate) async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) {
crate::storage::ecstore_data_usage::record_bucket_object_delete_memory(bucket, deleted_size, removed_current_object)
.await;
}
pub(crate) async fn record_bucket_object_write_memory(bucket: &str, previous_current_size: Option<u64>, new_size: u64) {
crate::storage::ecstore_data_usage::record_bucket_object_write_memory(bucket, previous_current_size, new_size).await;
}
pub(crate) async fn remove_bucket_usage_from_backend(
store: Arc<super::ECStore>,
bucket: &str,
) -> Result<(), crate::storage::StorageError> {
crate::storage::ecstore_data_usage::remove_bucket_usage_from_backend(store, bucket).await
}
}
pub(crate) mod runtime {
use std::{collections::HashMap, sync::Arc};
pub(crate) type BucketBandwidthMonitor = crate::storage::BucketBandwidthMonitor;
pub(crate) type DailyAllTierStats = crate::storage::DailyAllTierStats;
pub(crate) type DynReplicationPool = crate::storage::DynReplicationPool;
pub(crate) type ExpiryState = crate::storage::ExpiryState;
pub(crate) type NotificationSys = crate::storage::NotificationSys;
pub(crate) type ObjectStoreResolver = crate::storage::ObjectStoreResolver;
pub(crate) type ReplicationStats = crate::storage::ReplicationStats;
pub(crate) type ScannerMetricsReport = rustfs_common::metrics::ScannerMetricsReport;
pub(crate) type StorageClassConfig = crate::storage::ecstore_config::storageclass::Config;
pub(crate) type TierConfigMgr = crate::storage::TierConfigMgr;
#[cfg(test)]
pub(crate) type TierConfig = crate::storage::ecstore_tier::tier_config::TierConfig;
#[cfg(test)]
pub(crate) type TierType = crate::storage::ecstore_tier::tier_config::TierType;
#[cfg(test)]
pub(crate) use crate::storage::ecstore_tier::warm_backend::WarmBackend as AppWarmBackend;
#[cfg(test)]
pub(crate) type WarmBackendGetOpts = crate::storage::ecstore_tier::warm_backend::WarmBackendGetOpts;
pub(crate) fn set_global_storage_class(cfg: StorageClassConfig) {
crate::storage::ecstore_config::set_global_storage_class(cfg);
}
pub(crate) fn get_global_endpoints_opt() -> Option<super::EndpointServerPools> {
crate::storage::get_global_endpoints_opt()
}
pub(crate) fn get_global_deployment_id() -> Option<String> {
crate::storage::get_global_deployment_id()
}
pub(crate) fn get_global_lock_client() -> Option<Arc<dyn rustfs_lock::client::LockClient>> {
crate::storage::get_global_lock_client()
}
pub(crate) fn get_global_lock_clients() -> Option<&'static HashMap<String, Arc<dyn rustfs_lock::client::LockClient>>> {
crate::storage::get_global_lock_clients()
}
pub(crate) fn get_global_region() -> Option<s3s::region::Region> {
crate::storage::get_global_region()
}
pub(crate) fn global_rustfs_port() -> u16 {
crate::storage::global_rustfs_port()
}
pub(crate) fn get_global_tier_config_mgr() -> Arc<tokio::sync::RwLock<TierConfigMgr>> {
crate::storage::get_global_tier_config_mgr()
}
pub(crate) fn get_global_expiry_state() -> Arc<tokio::sync::RwLock<ExpiryState>> {
crate::storage::get_global_expiry_state()
}
pub(crate) fn new_object_layer_fn() -> Option<Arc<super::ECStore>> {
crate::storage::new_object_layer_fn()
}
pub(crate) fn set_object_store_resolver(resolver: Arc<ObjectStoreResolver>) -> bool {
crate::storage::set_object_store_resolver(resolver)
}
pub(crate) fn get_global_notification_sys() -> Option<&'static NotificationSys> {
crate::storage::get_global_notification_sys()
}
pub(crate) fn get_global_bucket_monitor() -> Option<Arc<BucketBandwidthMonitor>> {
crate::storage::get_global_bucket_monitor()
}
pub(crate) fn get_global_replication_pool() -> Option<Arc<DynReplicationPool>> {
crate::storage::get_global_replication_pool()
}
pub(crate) fn get_global_replication_stats() -> Option<Arc<ReplicationStats>> {
crate::storage::get_global_replication_stats()
}
pub(crate) fn get_global_boot_time() -> Option<std::time::SystemTime> {
crate::storage::get_global_boot_time()
}
pub(crate) fn get_daily_all_tier_stats() -> DailyAllTierStats {
crate::storage::get_daily_all_tier_stats()
}
pub(crate) async fn collect_scanner_metrics_report() -> ScannerMetricsReport {
rustfs_common::metrics::global_metrics().report().await
}
#[cfg(test)]
pub(crate) async fn init_local_disks(endpoint_pools: super::EndpointServerPools) -> Result<(), crate::storage::StorageError> {
crate::storage::init_local_disks(endpoint_pools).await
}
}
pub(crate) mod access { pub(crate) mod access {
pub(crate) use crate::storage::access::{ pub(crate) use crate::storage::access::{
PostObjectRequestMarker, ReqInfo, authorize_request, has_bypass_governance_header, req_info_mut, req_info_ref, PostObjectRequestMarker, ReqInfo, authorize_request, has_bypass_governance_header, req_info_mut, req_info_ref,
}; };
} }
pub(crate) mod bucket {
use std::sync::Arc;
pub(crate) trait ObjectLockConfigExt {
fn enabled(&self) -> bool;
}
impl ObjectLockConfigExt for s3s::dto::ObjectLockConfiguration {
fn enabled(&self) -> bool {
<s3s::dto::ObjectLockConfiguration as crate::storage::ecstore_bucket::object_lock::ObjectLockApi>::enabled(self)
}
}
pub(crate) trait ReplicationConfigExt {
fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec<String>;
fn replicate(&self, opts: &replication::ObjectOpts) -> bool;
}
impl ReplicationConfigExt for s3s::dto::ReplicationConfiguration {
fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec<String> {
<s3s::dto::ReplicationConfiguration as crate::storage::ecstore_bucket::replication::ReplicationConfigurationExt>::filter_target_arns(
self, obj,
)
}
fn replicate(&self, opts: &replication::ObjectOpts) -> bool {
<s3s::dto::ReplicationConfiguration as crate::storage::ecstore_bucket::replication::ReplicationConfigurationExt>::replicate(
self, opts,
)
}
}
pub(crate) trait VersioningConfigExt {
fn prefix_enabled(&self, prefix: &str) -> bool;
fn suspended(&self) -> bool;
}
impl VersioningConfigExt for s3s::dto::VersioningConfiguration {
fn prefix_enabled(&self, prefix: &str) -> bool {
<s3s::dto::VersioningConfiguration as crate::storage::ecstore_bucket::versioning::VersioningApi>::prefix_enabled(
self, prefix,
)
}
fn suspended(&self) -> bool {
<s3s::dto::VersioningConfiguration as crate::storage::ecstore_bucket::versioning::VersioningApi>::suspended(self)
}
}
pub(crate) async fn predict_lifecycle_expiration(
lifecycle: &s3s::dto::BucketLifecycleConfiguration,
obj: &lifecycle::lifecycle::ObjectOpts,
) -> lifecycle::lifecycle::Event {
crate::storage::ecstore_bucket::lifecycle::lifecycle::Lifecycle::predict_expiration(lifecycle, obj).await
}
pub(crate) fn validate_restore_request(
request: &s3s::dto::RestoreRequest,
api: Arc<crate::storage::ECStore>,
) -> std::io::Result<()> {
<s3s::dto::RestoreRequest as crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::RestoreRequestOps>::validate(
request, api,
)
}
pub(crate) mod bucket_target_sys {
pub(crate) type BucketTargetSys = crate::storage::ecstore_bucket::bucket_target_sys::BucketTargetSys;
}
pub(crate) mod lifecycle {
pub(crate) mod bucket_lifecycle_audit {
pub(crate) type LcEventSrc = crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc;
}
pub(crate) mod bucket_lifecycle_ops {
use std::sync::Arc;
use super::bucket_lifecycle_audit::LcEventSrc;
#[cfg(test)]
pub(crate) async fn init_background_expiry(api: Arc<crate::storage::ECStore>) {
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::init_background_expiry(api).await;
}
pub(crate) async fn enqueue_expiry_for_existing_objects(
api: Arc<crate::storage::ECStore>,
bucket: &str,
) -> Result<(), crate::storage::StorageError> {
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_expiry_for_existing_objects(api, bucket)
.await
}
pub(crate) async fn enqueue_transition_for_existing_objects(
api: Arc<crate::storage::ECStore>,
bucket: &str,
) -> Result<(), crate::storage::StorageError> {
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(
api, bucket,
)
.await
}
pub(crate) async fn enqueue_transition_immediate(oi: &crate::storage::StorageObjectInfo, src: LcEventSrc) {
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_immediate(oi, src).await;
}
pub(crate) async fn post_restore_opts(
version_id: &str,
bucket: &str,
object: &str,
) -> Result<crate::storage::StorageObjectOptions, std::io::Error> {
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::post_restore_opts(version_id, bucket, object)
.await
}
pub(crate) async fn validate_transition_tier(
lc: &s3s::dto::BucketLifecycleConfiguration,
) -> Result<(), std::io::Error> {
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::validate_transition_tier(lc).await
}
pub(crate) async fn validate_lifecycle_config(
lc: &s3s::dto::BucketLifecycleConfiguration,
lock_config: &s3s::dto::ObjectLockConfiguration,
) -> Result<(), std::io::Error> {
use crate::storage::ecstore_bucket::lifecycle::lifecycle::Lifecycle as _;
lc.validate(lock_config).await
}
}
pub(crate) mod lifecycle_contract {
#[cfg(test)]
pub(crate) type IlmAction = crate::storage::ecstore_bucket::lifecycle::lifecycle::IlmAction;
pub(crate) type Event = crate::storage::ecstore_bucket::lifecycle::lifecycle::Event;
pub(crate) type ObjectOpts = crate::storage::ecstore_bucket::lifecycle::lifecycle::ObjectOpts;
pub(crate) type TransitionOptions = crate::storage::ecstore_bucket::lifecycle::lifecycle::TransitionOptions;
pub(crate) const TRANSITION_COMPLETE: &str =
crate::storage::ecstore_bucket::lifecycle::lifecycle::TRANSITION_COMPLETE;
pub(crate) fn expected_expiry_time(mod_time: time::OffsetDateTime, days: i32) -> time::OffsetDateTime {
crate::storage::ecstore_bucket::lifecycle::lifecycle::expected_expiry_time(mod_time, days)
}
}
pub(crate) use lifecycle_contract as lifecycle;
pub(crate) mod tier_delete_journal {
use std::sync::Arc;
pub(crate) async fn persist_tier_delete_journal_entry(
api: Arc<crate::storage::ECStore>,
je: &super::tier_sweeper::Jentry,
) -> std::io::Result<()> {
crate::storage::ecstore_bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry(api, je).await
}
}
pub(crate) mod tier_sweeper {
pub(crate) type Jentry = crate::storage::ecstore_bucket::lifecycle::tier_sweeper::Jentry;
pub(crate) fn transitioned_delete_journal_entry(
version_id: Option<uuid::Uuid>,
versioned: bool,
suspended: bool,
transitioned: &rustfs_storage_api::TransitionedObject,
) -> Option<Jentry> {
crate::storage::ecstore_bucket::lifecycle::tier_sweeper::transitioned_delete_journal_entry(
version_id,
versioned,
suspended,
transitioned,
)
}
pub(crate) fn transitioned_force_delete_journal_entry(
transitioned: &rustfs_storage_api::TransitionedObject,
) -> Option<Jentry> {
crate::storage::ecstore_bucket::lifecycle::tier_sweeper::transitioned_force_delete_journal_entry(transitioned)
}
}
}
pub(crate) mod metadata {
pub(crate) const BUCKET_CORS_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_CORS_CONFIG;
pub(crate) const BUCKET_LIFECYCLE_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
pub(crate) const BUCKET_NOTIFICATION_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_NOTIFICATION_CONFIG;
pub(crate) const BUCKET_POLICY_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_POLICY_CONFIG;
pub(crate) const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str =
crate::storage::ecstore_bucket::metadata::BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG;
pub(crate) const BUCKET_REPLICATION_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_REPLICATION_CONFIG;
pub(crate) const BUCKET_SSECONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_SSECONFIG;
pub(crate) const BUCKET_TAGGING_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_TAGGING_CONFIG;
pub(crate) const BUCKET_TARGETS_FILE: &str = crate::storage::ecstore_bucket::metadata::BUCKET_TARGETS_FILE;
pub(crate) const BUCKET_VERSIONING_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_VERSIONING_CONFIG;
#[cfg(test)]
pub(crate) const OBJECT_LOCK_CONFIG: &str = crate::storage::ecstore_bucket::metadata::OBJECT_LOCK_CONFIG;
}
pub(crate) mod metadata_sys {
use std::sync::Arc;
use rustfs_policy::policy::BucketPolicy;
use s3s::dto::{
BucketLifecycleConfiguration, CORSConfiguration, NotificationConfiguration, ObjectLockConfiguration,
PublicAccessBlockConfiguration, ServerSideEncryptionConfiguration, Tagging,
};
use time::OffsetDateTime;
use tokio::sync::RwLock;
use super::target::BucketTargets;
pub(crate) type BucketMetadataSys = crate::storage::ecstore_bucket::metadata_sys::BucketMetadataSys;
#[cfg(test)]
pub(crate) async fn init_bucket_metadata_sys(api: Arc<crate::storage::ECStore>, buckets: Vec<String>) {
crate::storage::ecstore_bucket::metadata_sys::init_bucket_metadata_sys(api, buckets).await;
}
pub(crate) async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime, crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::delete(bucket, config_file).await
}
pub(crate) async fn get_bucket_policy(
bucket: &str,
) -> Result<(BucketPolicy, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_bucket_policy(bucket).await
}
pub(crate) async fn get_bucket_policy_raw(
bucket: &str,
) -> Result<(String, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_bucket_policy_raw(bucket).await
}
pub(crate) async fn get_bucket_targets_config(bucket: &str) -> Result<BucketTargets, crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_bucket_targets_config(bucket).await
}
pub(crate) async fn get_cors_config(
bucket: &str,
) -> Result<(CORSConfiguration, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_cors_config(bucket).await
}
pub(crate) fn get_global_bucket_metadata_sys() -> Option<Arc<RwLock<BucketMetadataSys>>> {
crate::storage::ecstore_bucket::metadata_sys::get_global_bucket_metadata_sys()
}
pub(crate) async fn get_lifecycle_config(
bucket: &str,
) -> Result<(BucketLifecycleConfiguration, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_lifecycle_config(bucket).await
}
pub(crate) async fn get_notification_config(
bucket: &str,
) -> Result<Option<NotificationConfiguration>, crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_notification_config(bucket).await
}
pub(crate) async fn get_object_lock_config(
bucket: &str,
) -> Result<(ObjectLockConfiguration, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_object_lock_config(bucket).await
}
pub(crate) async fn get_public_access_block_config(
bucket: &str,
) -> Result<(PublicAccessBlockConfiguration, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_public_access_block_config(bucket).await
}
pub(crate) async fn get_replication_config(
bucket: &str,
) -> Result<(s3s::dto::ReplicationConfiguration, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_replication_config(bucket).await
}
pub(crate) async fn get_sse_config(
bucket: &str,
) -> Result<(ServerSideEncryptionConfiguration, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_sse_config(bucket).await
}
pub(crate) async fn get_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime), crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::get_tagging_config(bucket).await
}
pub(crate) async fn update(
bucket: &str,
config_file: &str,
data: Vec<u8>,
) -> Result<OffsetDateTime, crate::storage::StorageError> {
crate::storage::ecstore_bucket::metadata_sys::update(bucket, config_file, data).await
}
}
pub(crate) mod object_lock {
pub(crate) mod objectlock {
pub(crate) fn get_object_legalhold_meta(
meta: &std::collections::HashMap<String, String>,
) -> s3s::dto::ObjectLockLegalHold {
crate::storage::ecstore_bucket::object_lock::objectlock::get_object_legalhold_meta(meta)
}
pub(crate) fn get_object_retention_meta(
meta: &std::collections::HashMap<String, String>,
) -> s3s::dto::ObjectLockRetention {
crate::storage::ecstore_bucket::object_lock::objectlock::get_object_retention_meta(meta)
}
}
pub(crate) mod objectlock_sys {
pub(crate) type BucketObjectLockSys =
crate::storage::ecstore_bucket::object_lock::objectlock_sys::BucketObjectLockSys;
pub(crate) type ObjectLockBlockReason =
crate::storage::ecstore_bucket::object_lock::objectlock_sys::ObjectLockBlockReason;
pub(crate) async fn check_object_lock_for_deletion(
bucket: &str,
obj_info: &crate::storage::StorageObjectInfo,
bypass_governance: bool,
) -> Option<ObjectLockBlockReason> {
crate::storage::ecstore_bucket::object_lock::objectlock_sys::check_object_lock_for_deletion(
bucket,
obj_info,
bypass_governance,
)
.await
}
pub(crate) fn is_retention_active(mode: &str, retain_until_date: Option<&s3s::dto::Date>) -> bool {
crate::storage::ecstore_bucket::object_lock::objectlock_sys::is_retention_active(mode, retain_until_date)
}
}
}
pub(crate) mod policy_sys {
pub(crate) type PolicySys = crate::storage::ecstore_bucket::policy_sys::PolicySys;
}
pub(crate) mod quota {
pub(crate) mod checker {
pub(crate) type QuotaChecker = crate::storage::ecstore_bucket::quota::checker::QuotaChecker;
}
pub(crate) type QuotaOperation = crate::storage::ecstore_bucket::quota::QuotaOperation;
}
pub(crate) mod replication {
use std::collections::HashMap;
use std::sync::Arc;
pub(crate) type DeletedObjectReplicationInfo = crate::storage::ecstore_bucket::replication::DeletedObjectReplicationInfo;
pub(crate) type MustReplicateOptions = crate::storage::ecstore_bucket::replication::MustReplicateOptions;
pub(crate) type ObjectOpts = crate::storage::ecstore_bucket::replication::ObjectOpts;
pub(crate) type ReplicateDecision = rustfs_filemeta::ReplicateDecision;
pub(crate) async fn check_replicate_delete(
bucket: &str,
dobj: &rustfs_storage_api::ObjectToDelete,
oi: &crate::storage::StorageObjectInfo,
del_opts: &crate::storage::StorageObjectOptions,
gerr: Option<String>,
) -> ReplicateDecision {
crate::storage::ecstore_bucket::replication::check_replicate_delete(bucket, dobj, oi, del_opts, gerr).await
}
pub(crate) fn get_must_replicate_options(
user_defined: &HashMap<String, String>,
user_tags: String,
status: rustfs_filemeta::ReplicationStatusType,
op_type: rustfs_filemeta::ReplicationType,
opts: crate::storage::StorageObjectOptions,
) -> MustReplicateOptions {
crate::storage::ecstore_bucket::replication::get_must_replicate_options(
user_defined,
user_tags,
status,
op_type,
opts,
)
}
pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplicateOptions) -> ReplicateDecision {
crate::storage::ecstore_bucket::replication::must_replicate(bucket, object, mopts).await
}
pub(crate) async fn schedule_replication(
oi: crate::storage::StorageObjectInfo,
store: Arc<crate::storage::ECStore>,
dsc: ReplicateDecision,
op_type: rustfs_filemeta::ReplicationType,
) {
crate::storage::ecstore_bucket::replication::schedule_replication(oi, store, dsc, op_type).await;
}
pub(crate) async fn schedule_replication_delete(dv: DeletedObjectReplicationInfo) {
crate::storage::ecstore_bucket::replication::schedule_replication_delete(dv).await;
}
}
pub(crate) mod tagging {
pub(crate) fn decode_tags(tags: &str) -> Vec<s3s::dto::Tag> {
crate::storage::ecstore_bucket::tagging::decode_tags(tags)
}
}
pub(crate) mod target {
#[cfg(test)]
pub(crate) type BucketTarget = crate::storage::ecstore_bucket::target::BucketTarget;
pub(crate) type BucketTargetType = crate::storage::ecstore_bucket::target::BucketTargetType;
pub(crate) type BucketTargets = crate::storage::ecstore_bucket::target::BucketTargets;
}
pub(crate) mod utils {
pub(crate) fn serialize<T: s3s::xml::Serialize>(val: &T) -> s3s::xml::SerResult<Vec<u8>> {
crate::storage::ecstore_bucket::utils::serialize(val)
}
}
#[cfg(test)]
pub(crate) mod transition_api {
pub(crate) type ReadCloser = crate::storage::ecstore_client::transition_api::ReadCloser;
pub(crate) type ReaderImpl = crate::storage::ecstore_client::transition_api::ReaderImpl;
}
pub(crate) mod versioning_sys {
pub(crate) type BucketVersioningSys = crate::storage::ecstore_bucket::versioning_sys::BucketVersioningSys;
}
}
pub(crate) mod concurrency { pub(crate) mod concurrency {
pub(crate) use crate::storage::concurrency::{ pub(crate) use crate::storage::concurrency::{
ConcurrencyManager, GetObjectGuard, IoQueueStatus, IoStrategy, PutObjectGuard, get_concurrency_aware_buffer_size, ConcurrencyManager, GetObjectGuard, IoQueueStatus, IoStrategy, PutObjectGuard, get_concurrency_aware_buffer_size,
@@ -41,6 +636,15 @@ pub(crate) mod ecfs {
pub(crate) use crate::storage::ecfs::FS; pub(crate) use crate::storage::ecfs::FS;
} }
pub(crate) mod error {
pub(crate) use crate::storage::{
DiskError, StorageError, is_all_buckets_not_found, is_err_bucket_not_found, is_err_object_not_found,
is_err_version_not_found,
};
pub(crate) type Error = StorageError;
}
pub(crate) mod head_prefix { pub(crate) mod head_prefix {
pub(crate) use crate::storage::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children}; pub(crate) use crate::storage::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children};
} }
@@ -49,6 +653,10 @@ pub(crate) mod helper {
pub(crate) use crate::storage::helper::{OperationHelper, spawn_background_with_context}; pub(crate) use crate::storage::helper::{OperationHelper, spawn_background_with_context};
} }
pub(crate) mod object_utils {
pub(crate) use crate::storage::to_s3s_etag;
}
pub(crate) mod io { pub(crate) mod io {
#[cfg(test)] #[cfg(test)]
pub(crate) use crate::storage::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader}; pub(crate) use crate::storage::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader};
@@ -84,6 +692,12 @@ pub(crate) mod set_disk {
pub(crate) use crate::storage::{get_lock_acquire_timeout, is_valid_storage_class}; pub(crate) use crate::storage::{get_lock_acquire_timeout, is_valid_storage_class};
} }
pub(crate) mod storage_class {
pub(crate) use crate::storage::ecstore_config::storageclass::STANDARD;
#[cfg(test)]
pub(crate) use crate::storage::ecstore_config::storageclass::STANDARD_IA;
}
pub(crate) mod timeout_wrapper { pub(crate) mod timeout_wrapper {
pub(crate) use crate::storage::timeout_wrapper::{GetObjectTimeoutPolicy, RequestTimeoutWrapper}; pub(crate) use crate::storage::timeout_wrapper::{GetObjectTimeoutPolicy, RequestTimeoutWrapper};
} }
rustfs/src/storage/mod.rs
+6 -2
View File
@@ -77,7 +77,9 @@ pub(crate) mod ecstore_capacity {
} }
pub(crate) mod ecstore_client { pub(crate) mod ecstore_client {
pub(crate) use rustfs_ecstore::api::client::{admin_handler_utils, object_api_utils, transition_api}; #[cfg(test)]
pub(crate) use rustfs_ecstore::api::client::transition_api;
pub(crate) use rustfs_ecstore::api::client::{admin_handler_utils, object_api_utils};
} }
pub(crate) mod ecstore_compression { pub(crate) mod ecstore_compression {
@@ -188,7 +190,9 @@ pub(crate) mod ecstore_storage {
pub(crate) mod ecstore_tier { pub(crate) mod ecstore_tier {
pub(crate) use rustfs_ecstore::api::tier::tier::TierConfigMgr; pub(crate) use rustfs_ecstore::api::tier::tier::TierConfigMgr;
pub(crate) use rustfs_ecstore::api::tier::{tier, tier_admin, tier_config, tier_handlers, warm_backend}; #[cfg(test)]
pub(crate) use rustfs_ecstore::api::tier::warm_backend;
pub(crate) use rustfs_ecstore::api::tier::{tier, tier_admin, tier_config, tier_handlers};
} }
pub(crate) const BUCKET_ACCELERATE_CONFIG: &str = ecstore_bucket::metadata::BUCKET_ACCELERATE_CONFIG; pub(crate) const BUCKET_ACCELERATE_CONFIG: &str = ecstore_bucket::metadata::BUCKET_ACCELERATE_CONFIG;
rustfs/src/storage/rpc/node_service.rs
+1 -1
View File
@@ -196,7 +196,7 @@ pub fn make_server_for_context(context: Option<Arc<runtime_sources::AppContext>>
} }
impl NodeService { impl NodeService {
fn resolve_object_store(&self) -> Option<Arc<crate::app::ECStore>> { fn resolve_object_store(&self) -> Option<Arc<crate::app::storage_api::ECStore>> {
let context = self.context.clone().or_else(runtime_sources::get_global_app_context); let context = self.context.clone().or_else(runtime_sources::get_global_app_context);
runtime_sources::object_store_handle_for_context(context.as_deref()) runtime_sources::object_store_handle_for_context(context.as_deref())
} }
scripts/check_architecture_migration_rules.sh
+30 -2
View File
@@ -153,6 +153,7 @@ ECSTORE_COMPAT_PASSTHROUGH_DIFF_FILE="${TMP_DIR}/ecstore_compat_passthrough_diff
RUSTFS_WORKLOAD_DIRECT_FOREGROUND_MAPPING_HITS_FILE="${TMP_DIR}/rustfs_workload_direct_foreground_mapping_hits.txt" RUSTFS_WORKLOAD_DIRECT_FOREGROUND_MAPPING_HITS_FILE="${TMP_DIR}/rustfs_workload_direct_foreground_mapping_hits.txt"
IAM_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/iam_runtime_source_bypass_hits.txt" IAM_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/iam_runtime_source_bypass_hits.txt"
RUSTFS_APP_CONTEXT_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_context_runtime_source_bypass_hits.txt" RUSTFS_APP_CONTEXT_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_context_runtime_source_bypass_hits.txt"
RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_runtime_storage_api_bypass_hits.txt"
RUSTFS_STARTUP_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_startup_runtime_source_bypass_hits.txt" RUSTFS_STARTUP_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_startup_runtime_source_bypass_hits.txt"
RUSTFS_SERVER_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_server_runtime_source_bypass_hits.txt" RUSTFS_SERVER_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_server_runtime_source_bypass_hits.txt"
RUSTFS_STORAGE_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_storage_runtime_source_bypass_hits.txt" RUSTFS_STORAGE_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_storage_runtime_source_bypass_hits.txt"
@@ -1447,8 +1448,11 @@ fi
'(use crate::storage::(access|helper|options|request_context|sse|timeout_wrapper|head_prefix|concurrency|ecfs)|crate::storage::sse::EncryptionKeyKind|use crate::storage::\{|use crate::storage::[A-Z])' \ '(use crate::storage::(access|helper|options|request_context|sse|timeout_wrapper|head_prefix|concurrency|ecfs)|crate::storage::sse::EncryptionKeyKind|use crate::storage::\{|use crate::storage::[A-Z])' \
rustfs/src/app/select_object.rs rustfs/src/app/*_usecase.rs || true rustfs/src/app/select_object.rs rustfs/src/app/*_usecase.rs || true
rg -n --with-filename \ rg -n --with-filename \
'use super::(?:\{[^}]*\b(?:DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class)\b|(?:DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class)\b)' \ 'use super::(?:\{[^}]*\b(?:DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class|StorageError|DiskError|is_all_buckets_not_found|is_err_bucket_not_found|is_err_object_not_found|is_err_version_not_found)\b|(?:object_api_utils::to_s3s_etag|storageclass|StorageError|DiskError|DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class|is_all_buckets_not_found|is_err_bucket_not_found|is_err_object_not_found|is_err_version_not_found)\b)' \
rustfs/src/app/object_usecase.rs rustfs/src/app/multipart_usecase.rs || true rustfs/src/app/bucket_usecase.rs rustfs/src/app/object_usecase.rs rustfs/src/app/multipart_usecase.rs rustfs/src/app/lifecycle_transition_api_test.rs || true
rg -n --with-filename \
'use super::(?:\{[^}]*\b(?:AppObjectLockConfigExt|AppReplicationConfigExt|AppVersioningConfigExt|predict_lifecycle_expiration|validate_restore_request|bucket_target_sys|lifecycle|metadata|metadata_sys|object_lock|policy_sys|quota|replication|tagging|target|utils|versioning_sys|transition_api|ObjectInfo|ObjectOptions)\b|(?:AppObjectLockConfigExt|AppReplicationConfigExt|AppVersioningConfigExt|predict_lifecycle_expiration|validate_restore_request|bucket_target_sys|lifecycle|metadata|metadata_sys|object_lock|policy_sys|quota|replication|tagging|target|utils|versioning_sys|transition_api|ObjectInfo|ObjectOptions)\b)|super::(?:lifecycle|metadata_sys|object_lock|quota|replication|tagging|target|utils|versioning_sys|transition_api)::|super::super::(?:metadata_sys|lifecycle|target)::' \
rustfs/src/app/bucket_usecase.rs rustfs/src/app/object_usecase.rs rustfs/src/app/multipart_usecase.rs rustfs/src/app/lifecycle_transition_api_test.rs rustfs/src/app/capacity_dirty_scope_test.rs rustfs/src/app/context.rs rustfs/src/app/context/handles.rs rustfs/src/app/context/interfaces.rs rustfs/src/app/context/runtime_sources.rs || true
} }
) >"$RUSTFS_APP_USECASE_STORAGE_API_BYPASS_HITS_FILE" ) >"$RUSTFS_APP_USECASE_STORAGE_API_BYPASS_HITS_FILE"
@@ -1472,6 +1476,30 @@ if [[ -s "$RUSTFS_APP_ADMIN_STORAGE_API_BYPASS_HITS_FILE" ]]; then
report_failure "RustFS app/admin completed storage helper consumers must use local storage_api boundaries: $(paste -sd '; ' "$RUSTFS_APP_ADMIN_STORAGE_API_BYPASS_HITS_FILE")" report_failure "RustFS app/admin completed storage helper consumers must use local storage_api boundaries: $(paste -sd '; ' "$RUSTFS_APP_ADMIN_STORAGE_API_BYPASS_HITS_FILE")"
fi fi
(
cd "$ROOT_DIR"
{
rg -n --with-filename \
'use super::(?:\{[^}]*\b(?:ECStore|EndpointServerPools|Endpoint|Endpoints|PoolEndpoints|StorageClassConfig|TierConfigMgr|DailyAllTierStats|ScannerMetricsReport|ReplicationStats|DynReplicationPool|NotificationSys|BucketBandwidthMonitor|ExpiryState|AppWarmBackend|WarmBackendGetOpts|TierConfig|TierType|PoolStatus|PoolDecommissionInfo|RebalStatus)\b|(?:ECStore|EndpointServerPools|Endpoint|Endpoints|PoolEndpoints|StorageClassConfig|TierConfigMgr|DailyAllTierStats|ScannerMetricsReport|ReplicationStats|DynReplicationPool|NotificationSys|BucketBandwidthMonitor|ExpiryState|AppWarmBackend|WarmBackendGetOpts|TierConfig|TierType|PoolStatus|PoolDecommissionInfo|RebalStatus)\b)' \
rustfs/src/app/*.rs rustfs/src/app/context/*.rs \
-g '!storage_api.rs' || true
rg -n --with-filename \
'super::(?:get_server_info|get_total_usable_capacity|get_total_usable_capacity_free|apply_bucket_usage_memory_overlay|load_data_usage_from_backend|record_bucket_object_delete_memory|record_bucket_object_write_memory|remove_bucket_usage_from_backend|get_global_|global_rustfs_port|new_object_layer_fn|set_object_store_resolver|set_global_storage_class|collect_scanner_metrics_report|init_local_disks)|super::super::(?:get_server_info|get_total_usable_capacity|get_total_usable_capacity_free|apply_bucket_usage_memory_overlay|load_data_usage_from_backend|record_bucket_object_delete_memory|record_bucket_object_write_memory|remove_bucket_usage_from_backend|get_global_|global_rustfs_port|new_object_layer_fn|set_object_store_resolver|set_global_storage_class|collect_scanner_metrics_report|init_local_disks)' \
rustfs/src/app/*.rs rustfs/src/app/context/*.rs \
-g '!storage_api.rs' || true
rg -n --with-filename \
'crate::app::(?:ECStore|EndpointServerPools|Endpoint|Endpoints|PoolEndpoints|StorageClassConfig|TierConfigMgr|DailyAllTierStats|ScannerMetricsReport|ReplicationStats|DynReplicationPool|NotificationSys|BucketBandwidthMonitor|ExpiryState|get_global_|set_object_store_resolver)' \
rustfs/src --glob '*.rs' || true
rg -n --with-filename \
'ecstore_data_usage|pub\(crate\) async fn load_data_usage_from_backend|super::super::load_data_usage_from_backend|crate::admin::load_data_usage_from_backend' \
rustfs/src/admin/mod.rs rustfs/src/admin/handlers/*.rs || true
}
) >"$RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE"
if [[ -s "$RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE" ]]; then
report_failure "RustFS app/admin runtime and data-usage facades must stay behind local storage_api boundaries: $(paste -sd '; ' "$RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE")"
fi
( (
cd "$ROOT_DIR" cd "$ROOT_DIR"
rg -n --with-filename 'crate::app::context::|use crate::app::context|app::context::' rustfs/src/storage --glob '*.rs' | rg -n --with-filename 'crate::app::context::|use crate::app::context|app::context::' rustfs/src/storage --glob '*.rs' |