mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
refactor: route app facade boundaries (#3865)
* refactor: route app storage error helpers * refactor: route app bucket owner facades * refactor: route app runtime facades
This commit is contained in:
Zhengchao An
@@ -5,9 +5,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
## Current Context
|
||||
|
||||
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
|
||||
- Branch: `overtrue/arch-app-storage-io-helper-boundary`
|
||||
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213`.
|
||||
- Based on: stacked on API-212 after PR #3854 merged.
|
||||
- Branch: `overtrue/arch-app-runtime-global-facade-cleanup`
|
||||
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213/API-214/API-215`.
|
||||
- Based on: stacked on API-215 while PR #3856 is pending.
|
||||
- PR type for this branch: `consumer-migration`
|
||||
- Runtime behavior changes: none.
|
||||
- Rust code changes: route replication pool, outbound TLS generation, runtime
|
||||
@@ -45,7 +45,7 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
readiness-time, observability, metrics, buffer, and TLS runtime source
|
||||
helpers, plus server readiness/audit/event/module-switch runtime source
|
||||
helpers, storage request/RPC/SSE runtime source helpers, and admin
|
||||
handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, app bucket/object/multipart usecase explicit storage imports, app select/bucket/object/multipart explicit S3 DTO plus ECFS owner imports, app-local S3 API response helper imports for bucket/object/multipart usecases, app-local storage helper boundary imports for select/bucket/object/multipart usecases, admin/app test storage helper boundary imports for request context, authorization, object options, and ECFS test harness consumers, and app storage IO/compression/set-disk helper boundary imports for object and multipart usecases,
|
||||
handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, app bucket/object/multipart usecase explicit storage imports, app select/bucket/object/multipart explicit S3 DTO plus ECFS owner imports, app-local S3 API response helper imports for bucket/object/multipart usecases, app-local storage helper boundary imports for select/bucket/object/multipart usecases, admin/app test storage helper boundary imports for request context, authorization, object options, and ECFS test harness consumers, app storage IO/compression/set-disk helper boundary imports for object and multipart usecases, app storage error, ETag, and storage-class helper boundary imports for bucket/object/multipart usecases and lifecycle transition tests, app bucket owner facade imports for lifecycle, metadata, object-lock, quota, replication, tagging, target, versioning, and transition test helpers, and app/admin runtime, capacity, data-usage, endpoint, and global facade imports,
|
||||
through AppContext-first or owner-crate resolver boundaries.
|
||||
- CI/script changes: lock completed owner and test/fuzz boundaries against
|
||||
bare/glob imports, scattered raw ECStore facade subpaths, and startup
|
||||
@@ -56,8 +56,8 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
event-bridge thin module regressions, plus IAM runtime-source bypasses;
|
||||
accept the reviewed AppContext resolver reverse dependencies in the layer
|
||||
baseline, and block direct admin AppContext resolver consumers outside the
|
||||
admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, narrow the object-usecase ECFS layer baseline entry to `FS`, reject direct storage S3 API helper imports from app usecase files, reject direct storage helper imports from app select/usecase files, reject completed app/admin storage helper bypasses, and reject app usecase bypasses for migrated storage IO/compression/set-disk helpers.
|
||||
- Docs changes: record the API-136 through API-213 owner facade and lifecycle
|
||||
admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, narrow the object-usecase ECFS layer baseline entry to `FS`, reject direct storage S3 API helper imports from app usecase files, reject direct storage helper imports from app select/usecase files, reject completed app/admin storage helper bypasses, reject app usecase bypasses for migrated storage IO/compression/set-disk helpers, reject app usecase/test bypasses for migrated storage error, ETag, and storage-class helpers, reject app root bucket owner facade bypasses from migrated app consumers, and reject app/admin runtime/data-usage root facade regressions.
|
||||
- Docs changes: record the API-136 through API-216 owner facade and lifecycle
|
||||
runtime-source cleanup.
|
||||
|
||||
## Phase 0 Tasks
|
||||
@@ -5069,14 +5069,75 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
|
||||
layer guards, diff hygiene, residual migrated helper scan, Rust risk scan,
|
||||
fast PR gate, and full PR gate before PR.
|
||||
|
||||
- [x] `API-214` Route app storage error helpers through app boundary.
|
||||
- Do: extend the app-local storage helper boundary for storage errors, error
|
||||
classification helpers, ETag conversion, and storage-class constants, then
|
||||
route bucket, object, multipart, and lifecycle transition test consumers
|
||||
through it instead of app root compatibility aliases.
|
||||
- Acceptance: completed app consumers no longer import the migrated storage
|
||||
error, ETag, or storage-class helpers from the app root, the app root no
|
||||
longer exposes those helper aliases, and migration rules reject
|
||||
reintroducing the bypasses.
|
||||
- Must preserve: bucket config-not-found handling, multipart upload-not-found
|
||||
and version-not-found classification, object delete/copy error mapping,
|
||||
list response ETag conversion, storage-class response projection, and
|
||||
lifecycle transition test ETag formatting.
|
||||
- Verification: focused RustFS app compile/tests, formatting, migration and
|
||||
layer guards, diff hygiene, residual migrated helper scan, Rust risk scan,
|
||||
fast PR gate, and full PR gate before PR.
|
||||
|
||||
- [x] `API-215` Route app bucket owner facades through storage_api boundary.
|
||||
- Do: move app-facing lifecycle, metadata, object-lock, quota, replication,
|
||||
tagging, target, versioning, and transition test helper facades behind
|
||||
`storage_api::bucket`, then route bucket, object, multipart, context, and
|
||||
app integration-test consumers through that boundary.
|
||||
- Acceptance: app root no longer exposes the migrated bucket owner facade
|
||||
modules or object-info/object-options compatibility aliases, and migration
|
||||
rules reject consumers that reintroduce root-level bypasses.
|
||||
- Must preserve: bucket metadata CRUD, lifecycle transition enqueueing and
|
||||
restore validation, object-lock deletion checks, quota checks, replication
|
||||
scheduling/decision helpers, XML serialization, target validation, and
|
||||
bucket metadata runtime-source lookup.
|
||||
- Verification: focused RustFS app compile/tests, formatting, migration and
|
||||
layer guards, diff hygiene, residual migrated helper scan, Rust risk scan,
|
||||
fast PR gate, and full PR gate before PR.
|
||||
|
||||
- [x] `API-216` Route app runtime/global facades through storage_api boundary.
|
||||
- Do: move app/admin-facing admin info, capacity, data-usage, endpoint,
|
||||
runtime global, storage-class, scanner metric, tier, replication, and
|
||||
notification facades behind local `storage_api` modules, then route app
|
||||
usecases, context modules, tests, admin quota, admin config service, and
|
||||
storage node-service consumers through those boundaries.
|
||||
- Acceptance: app root no longer exposes the migrated runtime/global facade
|
||||
aliases or wrappers, admin root no longer exposes the migrated data-usage
|
||||
wrapper, and migration rules reject root-level regressions for the
|
||||
completed app/admin runtime/data-usage paths.
|
||||
- Must preserve: admin server info and capacity math, data-usage memory
|
||||
overlay and bucket accounting, object-store resolver publication,
|
||||
AppContext fallback runtime sources, lifecycle test disk initialization,
|
||||
tier warm-backend test contracts, scanner metrics report collection, and
|
||||
admin quota current-usage reads.
|
||||
- Verification: focused RustFS app compile/tests, formatting, migration and
|
||||
layer guards, diff hygiene, residual migrated facade scan, Rust risk scan,
|
||||
fast PR gate, and full PR gate before PR.
|
||||
|
||||
## Next PRs
|
||||
|
||||
1. `consumer-migration`: batch remaining app/admin/storage helper boundary and owner facade cleanup.
|
||||
1. `consumer-migration`: batch remaining app/admin facade cleanup after API-216.
|
||||
|
||||
## Pre-Push Review Log
|
||||
|
||||
| Expert | Status | Notes |
|
||||
|---|---|---|
|
||||
| Quality/architecture | pass | API-216 removes app root runtime/global facades and routes app/admin data-usage and runtime consumers through local storage_api boundaries. |
|
||||
| Migration preservation | pass | Admin info/capacity, data-usage accounting, AppContext fallback runtime sources, lifecycle test disk initialization, tier warm-backend tests, and scanner metrics keep the same storage implementations. |
|
||||
| Testing/verification | pass | Focused RustFS app compile/tests, formatting, migration/layer guards, residual migrated facade scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. |
|
||||
| Quality/architecture | pass | API-215 consolidates bucket owner facades under app storage_api and removes the app root compatibility modules. |
|
||||
| Migration preservation | pass | Lifecycle, metadata, object-lock, quota, replication, tagging, target, versioning, and transition test helpers keep the same storage implementations. |
|
||||
| Testing/verification | pass | Focused app compile/tests, formatting, migration/layer guards, residual migrated facade scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. |
|
||||
| Quality/architecture | pass | API-214 moves app storage error, ETag, and storage-class helpers behind the app storage_api boundary and removes the matching app root aliases. |
|
||||
| Migration preservation | pass | Bucket config handling, multipart/object not-found classification, object delete/copy mapping, list ETag conversion, storage-class response projection, and lifecycle transition test formatting keep the same storage implementations. |
|
||||
| Testing/verification | pass | Focused app compile/tests, formatting, migration/layer guards, residual migrated helper scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. |
|
||||
| Quality/architecture | pass | API-213 removes app root IO/compression/set-disk compatibility aliases and routes object/multipart usecases through the app storage_api boundary. |
|
||||
| Migration preservation | pass | Object compression metadata, hash-reader checksum handling, multipart write-encryption planning, storage-class checks, lock timeouts, and test reader wrappers keep the same storage implementations. |
|
||||
| Testing/verification | pass | Focused app compile/tests, formatting, migration/layer guards, residual migrated helper scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. |
|
||||
|
||||
@@ -175,7 +175,7 @@ async fn current_usage_from_context(bucket: &str) -> u64 {
|
||||
return 0;
|
||||
};
|
||||
|
||||
match super::super::load_data_usage_from_backend(store).await {
|
||||
match super::super::storage_api::data_usage::load_data_usage_from_backend(store).await {
|
||||
Ok(data_usage_info) => data_usage_info
|
||||
.buckets_usage
|
||||
.get(bucket)
|
||||
|
||||
@@ -133,10 +133,6 @@ mod ecstore_config {
|
||||
pub(crate) use crate::storage::ecstore_config::{com, init, storageclass};
|
||||
}
|
||||
|
||||
mod ecstore_data_usage {
|
||||
pub(crate) use crate::storage::ecstore_data_usage::load_data_usage_from_backend;
|
||||
}
|
||||
|
||||
#[allow(unused_imports)]
|
||||
mod ecstore_disk {
|
||||
pub(crate) use crate::storage::ecstore_disk::{RUSTFS_META_BUCKET, endpoint};
|
||||
@@ -473,12 +469,6 @@ pub(crate) fn init_admin_config_defaults() {
|
||||
ecstore_config::init();
|
||||
}
|
||||
|
||||
pub(crate) async fn load_data_usage_from_backend(
|
||||
store: Arc<ECStore>,
|
||||
) -> std::result::Result<rustfs_data_usage::DataUsageInfo, Error> {
|
||||
ecstore_data_usage::load_data_usage_from_backend(store).await
|
||||
}
|
||||
|
||||
pub(crate) async fn collect_local_metrics(
|
||||
types: MetricType,
|
||||
opts: &CollectMetricsOpts,
|
||||
|
||||
@@ -67,7 +67,9 @@ fn invalid_request(message: impl Into<String>) -> S3Error {
|
||||
S3Error::with_message(S3ErrorCode::InvalidRequest, message.into())
|
||||
}
|
||||
|
||||
fn resolve_runtime_config_store_for_context(context: Option<&AppContext>) -> S3Result<std::sync::Arc<crate::app::ECStore>> {
|
||||
fn resolve_runtime_config_store_for_context(
|
||||
context: Option<&AppContext>,
|
||||
) -> S3Result<std::sync::Arc<crate::app::storage_api::ECStore>> {
|
||||
resolve_object_store_handle_for_context(context).ok_or_else(|| internal_error("storage layer not initialized"))
|
||||
}
|
||||
|
||||
|
||||
@@ -15,3 +15,13 @@
|
||||
pub(crate) use crate::storage::StorageObjectOptions;
|
||||
pub(crate) use crate::storage::access::{ReqInfo, authorize_request};
|
||||
pub(crate) use crate::storage::request_context::{RequestContext, spawn_traced};
|
||||
|
||||
pub(crate) mod data_usage {
|
||||
use std::sync::Arc;
|
||||
|
||||
pub(crate) async fn load_data_usage_from_backend(
|
||||
store: Arc<crate::storage::ECStore>,
|
||||
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::StorageError> {
|
||||
crate::storage::ecstore_data_usage::load_data_usage_from_backend(store).await
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,11 +14,12 @@
|
||||
|
||||
//! Admin application use-case contracts.
|
||||
|
||||
use super::ECStore;
|
||||
use super::EndpointServerPools;
|
||||
use super::get_server_info;
|
||||
use super::{PoolDecommissionInfo, PoolStatus, RebalStatus, get_total_usable_capacity, get_total_usable_capacity_free};
|
||||
use super::{apply_bucket_usage_memory_overlay, load_data_usage_from_backend};
|
||||
use super::storage_api::admin::get_server_info;
|
||||
use super::storage_api::capacity::{
|
||||
PoolDecommissionInfo, PoolStatus, RebalStatus, get_total_usable_capacity, get_total_usable_capacity_free,
|
||||
};
|
||||
use super::storage_api::data_usage::{apply_bucket_usage_memory_overlay, load_data_usage_from_backend};
|
||||
use super::storage_api::{ECStore, EndpointServerPools};
|
||||
use crate::app::runtime_sources::{
|
||||
AppContext, get_global_app_context, resolve_endpoints_handle, resolve_object_store_handle_for_context,
|
||||
};
|
||||
@@ -596,7 +597,7 @@ impl DefaultAdminUsecase {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::{PoolDecommissionInfo, PoolStatus};
|
||||
use super::super::storage_api::capacity::{PoolDecommissionInfo, PoolStatus};
|
||||
use super::*;
|
||||
use time::OffsetDateTime;
|
||||
|
||||
|
||||
@@ -14,23 +14,18 @@
|
||||
|
||||
//! Bucket application use-case contracts.
|
||||
|
||||
use super::ECStore;
|
||||
use super::StorageError;
|
||||
use super::object_api_utils::to_s3s_etag;
|
||||
use super::s3_api::bucket::{
|
||||
ListObjectVersionsParams, ListObjectsV2Params, build_list_buckets_output, build_list_object_versions_output,
|
||||
build_list_objects_output, build_list_objects_v2_output, parse_list_object_versions_params, parse_list_objects_v2_params,
|
||||
rustfs_owner,
|
||||
};
|
||||
use super::storage_api::ECStore;
|
||||
use super::storage_api::StorageObjectInfo as ObjectInfo;
|
||||
use super::storage_api::access::{ReqInfo, authorize_request, req_info_ref};
|
||||
use super::storage_api::helper::{OperationHelper, spawn_background_with_context};
|
||||
use super::storage_api::{
|
||||
get_validated_store, process_lambda_configurations, process_queue_configurations, process_topic_configurations,
|
||||
request_context, validate_list_object_unordered_with_delimiter,
|
||||
};
|
||||
use super::{AppObjectLockConfigExt as _, AppVersioningConfigExt as _};
|
||||
use super::{
|
||||
#[cfg(test)]
|
||||
use super::storage_api::bucket::target::BucketTarget;
|
||||
use super::storage_api::bucket::{
|
||||
ObjectLockConfigExt as _, VersioningConfigExt as _,
|
||||
bucket_target_sys::BucketTargetSys,
|
||||
lifecycle::bucket_lifecycle_ops::{
|
||||
enqueue_expiry_for_existing_objects, enqueue_transition_for_existing_objects, validate_lifecycle_config,
|
||||
@@ -47,6 +42,14 @@ use super::{
|
||||
utils::serialize,
|
||||
versioning_sys::BucketVersioningSys,
|
||||
};
|
||||
use super::storage_api::data_usage::remove_bucket_usage_from_backend;
|
||||
use super::storage_api::error::StorageError;
|
||||
use super::storage_api::helper::{OperationHelper, spawn_background_with_context};
|
||||
use super::storage_api::object_utils::to_s3s_etag;
|
||||
use super::storage_api::{
|
||||
get_validated_store, process_lambda_configurations, process_queue_configurations, process_topic_configurations,
|
||||
request_context, validate_list_object_unordered_with_delimiter,
|
||||
};
|
||||
use crate::admin::handlers::site_replication::{
|
||||
site_replication_bucket_meta_hook, site_replication_delete_bucket_hook, site_replication_make_bucket_hook,
|
||||
};
|
||||
@@ -865,7 +868,7 @@ impl DefaultBucketUsecase {
|
||||
.await
|
||||
.map_err(ApiError::from)?;
|
||||
rustfs_scanner::clear_dirty_usage_bucket(&input.bucket);
|
||||
if let Err(err) = super::remove_bucket_usage_from_backend(store.clone(), &input.bucket).await {
|
||||
if let Err(err) = remove_bucket_usage_from_backend(store.clone(), &input.bucket).await {
|
||||
warn!(bucket = %input.bucket, error = ?err, "failed to remove deleted bucket from data usage");
|
||||
}
|
||||
|
||||
@@ -2315,7 +2318,7 @@ mod tests {
|
||||
BucketTargets {
|
||||
targets: arns
|
||||
.iter()
|
||||
.map(|arn| super::super::target::BucketTarget {
|
||||
.map(|arn| BucketTarget {
|
||||
arn: (*arn).to_string(),
|
||||
target_type: BucketTargetType::ReplicationService,
|
||||
..Default::default()
|
||||
|
||||
@@ -12,7 +12,8 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::{ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints, metadata_sys};
|
||||
use super::storage_api::bucket::metadata_sys;
|
||||
use super::storage_api::{ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints};
|
||||
use rustfs_common::heal_channel::{HealOpts, HealScanMode};
|
||||
use rustfs_object_capacity::capacity_manager::{HybridStrategyConfig, create_isolated_manager};
|
||||
use rustfs_storage_api::{BucketOperations, BucketOptions, HealOperations as _, MakeBucketOptions, ObjectIO as _};
|
||||
@@ -77,7 +78,9 @@ async fn setup_capacity_dirty_scope_env() -> (Vec<PathBuf>, Arc<ECStore>) {
|
||||
};
|
||||
|
||||
let endpoint_pools = EndpointServerPools(vec![pool_endpoints]);
|
||||
super::init_local_disks(endpoint_pools.clone()).await.unwrap();
|
||||
super::storage_api::runtime::init_local_disks(endpoint_pools.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let server_addr: std::net::SocketAddr = "127.0.0.1:0".parse().unwrap();
|
||||
let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
|
||||
|
||||
@@ -26,14 +26,12 @@ pub use global::*;
|
||||
pub use handles::*;
|
||||
pub use interfaces::*;
|
||||
|
||||
use super::DailyAllTierStats;
|
||||
use super::ECStore;
|
||||
use super::EndpointServerPools;
|
||||
use super::ScannerMetricsReport;
|
||||
use super::StorageClassConfig;
|
||||
use super::TierConfigMgr;
|
||||
use super::metadata_sys::BucketMetadataSys;
|
||||
use super::{BucketBandwidthMonitor, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats};
|
||||
use super::storage_api::bucket::metadata_sys::BucketMetadataSys;
|
||||
use super::storage_api::runtime::{
|
||||
BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats,
|
||||
ScannerMetricsReport, StorageClassConfig, TierConfigMgr,
|
||||
};
|
||||
use super::storage_api::{ECStore, EndpointServerPools};
|
||||
use crate::config::RustFSBufferConfig;
|
||||
use rustfs_config::server_config::Config;
|
||||
use rustfs_credentials::Credentials;
|
||||
@@ -562,9 +560,8 @@ fn resolve_buffer_config_with(
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::Endpoint;
|
||||
use super::super::init_local_disks;
|
||||
use super::super::{Endpoints, PoolEndpoints};
|
||||
use super::super::storage_api::runtime::init_local_disks;
|
||||
use super::super::storage_api::{Endpoint, Endpoints, PoolEndpoints};
|
||||
use super::*;
|
||||
use crate::app::context::global::AppContextTestInterfaces;
|
||||
use crate::app::context::handles::{
|
||||
|
||||
@@ -12,7 +12,8 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::super::{ECStore, set_object_store_resolver};
|
||||
use super::super::storage_api::ECStore;
|
||||
use super::super::storage_api::runtime::set_object_store_resolver;
|
||||
use super::handles::{
|
||||
IamHandle, KmsHandle, default_action_credential_interface, default_boot_time_interface, default_bucket_metadata_interface,
|
||||
default_bucket_monitor_interface, default_buffer_config_interface, default_deployment_id_interface,
|
||||
|
||||
@@ -12,10 +12,12 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::super::EndpointServerPools;
|
||||
use super::super::StorageClassConfig;
|
||||
use super::super::TierConfigMgr;
|
||||
use super::super::metadata_sys::BucketMetadataSys;
|
||||
use super::super::storage_api::EndpointServerPools;
|
||||
use super::super::storage_api::bucket::metadata_sys::BucketMetadataSys;
|
||||
use super::super::storage_api::runtime::{
|
||||
BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats,
|
||||
ScannerMetricsReport, StorageClassConfig, TierConfigMgr,
|
||||
};
|
||||
use super::interfaces::{
|
||||
ActionCredentialInterface, BootTimeInterface, BucketMetadataInterface, BucketMonitorInterface, BufferConfigInterface,
|
||||
DeploymentIdInterface, EndpointsInterface, ExpiryStateInterface, IamInterface, InternodeMetricsInterface, KmsInterface,
|
||||
@@ -153,7 +155,7 @@ impl NotifyInterface for NotifyHandle {
|
||||
pub struct NotificationSystemHandle;
|
||||
|
||||
impl NotificationSystemInterface for NotificationSystemHandle {
|
||||
fn handle(&self) -> Option<&'static super::super::NotificationSys> {
|
||||
fn handle(&self) -> Option<&'static NotificationSys> {
|
||||
runtime_sources::notification_system()
|
||||
}
|
||||
}
|
||||
@@ -173,7 +175,7 @@ impl BucketMetadataInterface for BucketMetadataHandle {
|
||||
pub struct BucketMonitorHandle;
|
||||
|
||||
impl BucketMonitorInterface for BucketMonitorHandle {
|
||||
fn handle(&self) -> Option<Arc<super::super::BucketBandwidthMonitor>> {
|
||||
fn handle(&self) -> Option<Arc<BucketBandwidthMonitor>> {
|
||||
runtime_sources::bucket_monitor()
|
||||
}
|
||||
}
|
||||
@@ -183,7 +185,7 @@ impl BucketMonitorInterface for BucketMonitorHandle {
|
||||
pub struct ReplicationPoolHandle;
|
||||
|
||||
impl ReplicationPoolInterface for ReplicationPoolHandle {
|
||||
fn handle(&self) -> Option<Arc<super::super::DynReplicationPool>> {
|
||||
fn handle(&self) -> Option<Arc<DynReplicationPool>> {
|
||||
runtime_sources::replication_pool()
|
||||
}
|
||||
}
|
||||
@@ -193,7 +195,7 @@ impl ReplicationPoolInterface for ReplicationPoolHandle {
|
||||
pub struct ReplicationStatsHandle;
|
||||
|
||||
impl ReplicationStatsInterface for ReplicationStatsHandle {
|
||||
fn handle(&self) -> Option<Arc<super::super::ReplicationStats>> {
|
||||
fn handle(&self) -> Option<Arc<ReplicationStats>> {
|
||||
runtime_sources::replication_stats()
|
||||
}
|
||||
}
|
||||
@@ -213,7 +215,7 @@ impl BootTimeInterface for BootTimeHandle {
|
||||
pub struct TierStatsHandle;
|
||||
|
||||
impl TierStatsInterface for TierStatsHandle {
|
||||
fn daily_all(&self) -> super::super::DailyAllTierStats {
|
||||
fn daily_all(&self) -> DailyAllTierStats {
|
||||
runtime_sources::daily_tier_stats()
|
||||
}
|
||||
}
|
||||
@@ -224,7 +226,7 @@ pub struct ScannerMetricsHandle;
|
||||
|
||||
#[async_trait]
|
||||
impl ScannerMetricsInterface for ScannerMetricsHandle {
|
||||
async fn report(&self) -> super::super::ScannerMetricsReport {
|
||||
async fn report(&self) -> ScannerMetricsReport {
|
||||
runtime_sources::scanner_metrics_report().await
|
||||
}
|
||||
}
|
||||
@@ -360,7 +362,7 @@ impl TierConfigInterface for TierConfigHandle {
|
||||
pub struct ExpiryStateHandle;
|
||||
|
||||
impl ExpiryStateInterface for ExpiryStateHandle {
|
||||
fn handle(&self) -> Arc<RwLock<super::super::ExpiryState>> {
|
||||
fn handle(&self) -> Arc<RwLock<ExpiryState>> {
|
||||
runtime_sources::expiry_state()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,13 +12,12 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::super::DailyAllTierStats;
|
||||
use super::super::EndpointServerPools;
|
||||
use super::super::ScannerMetricsReport;
|
||||
use super::super::StorageClassConfig;
|
||||
use super::super::TierConfigMgr;
|
||||
use super::super::metadata_sys::BucketMetadataSys;
|
||||
use super::super::{BucketBandwidthMonitor, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats};
|
||||
use super::super::storage_api::EndpointServerPools;
|
||||
use super::super::storage_api::bucket::metadata_sys::BucketMetadataSys;
|
||||
use super::super::storage_api::runtime::{
|
||||
BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats,
|
||||
ScannerMetricsReport, StorageClassConfig, TierConfigMgr,
|
||||
};
|
||||
use crate::config::RustFSBufferConfig;
|
||||
use async_trait::async_trait;
|
||||
use rustfs_config::server_config::Config;
|
||||
|
||||
@@ -12,15 +12,15 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::super::metadata_sys::BucketMetadataSys;
|
||||
use super::super::{
|
||||
BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ECStore, EndpointServerPools, ExpiryState, NotificationSys,
|
||||
ReplicationStats, ScannerMetricsReport, StorageClassConfig, TierConfigMgr, collect_scanner_metrics_report,
|
||||
get_daily_all_tier_stats, get_global_boot_time, get_global_bucket_monitor, get_global_deployment_id,
|
||||
get_global_endpoints_opt, get_global_expiry_state, get_global_lock_client, get_global_lock_clients,
|
||||
get_global_notification_sys, get_global_region, get_global_replication_pool, get_global_replication_stats,
|
||||
get_global_tier_config_mgr, global_rustfs_port, new_object_layer_fn, set_global_storage_class,
|
||||
use super::super::storage_api::bucket::metadata_sys::BucketMetadataSys;
|
||||
use super::super::storage_api::runtime::{
|
||||
BucketBandwidthMonitor, DailyAllTierStats, DynReplicationPool, ExpiryState, NotificationSys, ReplicationStats,
|
||||
ScannerMetricsReport, StorageClassConfig, TierConfigMgr, collect_scanner_metrics_report, get_daily_all_tier_stats,
|
||||
get_global_boot_time, get_global_bucket_monitor, get_global_deployment_id, get_global_endpoints_opt, get_global_expiry_state,
|
||||
get_global_lock_client, get_global_lock_clients, get_global_notification_sys, get_global_region, get_global_replication_pool,
|
||||
get_global_replication_stats, get_global_tier_config_mgr, global_rustfs_port, new_object_layer_fn, set_global_storage_class,
|
||||
};
|
||||
use super::super::storage_api::{ECStore, EndpointServerPools};
|
||||
use crate::config::{RustFSBufferConfig, get_global_buffer_config};
|
||||
use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
|
||||
use rustfs_credentials::{Credentials, get_global_action_cred};
|
||||
@@ -108,7 +108,7 @@ pub fn notification_system() -> Option<&'static NotificationSys> {
|
||||
}
|
||||
|
||||
pub fn bucket_metadata() -> Option<Arc<RwLock<BucketMetadataSys>>> {
|
||||
super::super::metadata_sys::get_global_bucket_metadata_sys()
|
||||
super::super::storage_api::bucket::metadata_sys::get_global_bucket_metadata_sys()
|
||||
}
|
||||
|
||||
pub fn bucket_monitor() -> Option<Arc<BucketBandwidthMonitor>> {
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::super::ECStore;
|
||||
use super::super::storage_api::ECStore;
|
||||
use super::global::{AppContext, get_global_app_context, init_global_app_context};
|
||||
use super::runtime_sources;
|
||||
use rustfs_kms::KmsServiceManager;
|
||||
|
||||
@@ -12,17 +12,19 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::storage_api::ecfs::FS;
|
||||
use super::storage_api::{
|
||||
StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
|
||||
};
|
||||
use super::{
|
||||
AppWarmBackend, ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints, TierConfig, TierType, WarmBackendGetOpts,
|
||||
use super::storage_api::bucket::{
|
||||
lifecycle,
|
||||
metadata::{BUCKET_LIFECYCLE_CONFIG, OBJECT_LOCK_CONFIG},
|
||||
metadata_sys,
|
||||
object_api_utils::to_s3s_etag,
|
||||
transition_api::{ReadCloser, ReaderImpl},
|
||||
};
|
||||
use super::storage_api::ecfs::FS;
|
||||
use super::storage_api::object_utils::to_s3s_etag;
|
||||
use super::storage_api::runtime::{AppWarmBackend, TierConfig, TierType, WarmBackendGetOpts};
|
||||
use super::storage_api::{
|
||||
ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints, StorageObjectInfo as ObjectInfo,
|
||||
StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
|
||||
};
|
||||
use super::{multipart_usecase::DefaultMultipartUsecase, object_usecase::DefaultObjectUsecase};
|
||||
use crate::app::bucket_usecase::DefaultBucketUsecase;
|
||||
use crate::app::runtime_sources::resolve_tier_config_handle;
|
||||
@@ -107,7 +109,9 @@ async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>) {
|
||||
|
||||
let endpoint_pools = EndpointServerPools(vec![pool_endpoints]);
|
||||
|
||||
super::init_local_disks(endpoint_pools.clone()).await.unwrap();
|
||||
super::storage_api::runtime::init_local_disks(endpoint_pools.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let server_addr: std::net::SocketAddr = "127.0.0.1:9003".parse().unwrap();
|
||||
let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
|
||||
@@ -124,7 +128,7 @@ async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>) {
|
||||
let buckets = buckets_list.into_iter().map(|v| v.name).collect();
|
||||
metadata_sys::init_bucket_metadata_sys(ecstore.clone(), buckets).await;
|
||||
|
||||
super::lifecycle::bucket_lifecycle_ops::init_background_expiry(ecstore.clone()).await;
|
||||
lifecycle::bucket_lifecycle_ops::init_background_expiry(ecstore.clone()).await;
|
||||
|
||||
let _ = GLOBAL_ENV.set((disk_paths.clone(), ecstore.clone()));
|
||||
|
||||
@@ -925,7 +929,7 @@ async fn delete_transitioned_object_removes_remote_tier_copy_via_usecase() {
|
||||
.expect("Failed to set lifecycle configuration");
|
||||
let _ = upload_test_object(&ecstore, bucket.as_str(), object, payload).await;
|
||||
|
||||
super::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(ecstore.clone(), bucket.as_str())
|
||||
lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(ecstore.clone(), bucket.as_str())
|
||||
.await
|
||||
.expect("Failed to enqueue transitioned object");
|
||||
|
||||
@@ -982,7 +986,7 @@ async fn lifecycle_transition_marks_dirty_disks_for_capacity_manager() {
|
||||
.expect("Failed to set lifecycle configuration");
|
||||
let _ = upload_test_object(&ecstore, bucket.as_str(), object, payload).await;
|
||||
|
||||
super::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(ecstore.clone(), bucket.as_str())
|
||||
lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(ecstore.clone(), bucket.as_str())
|
||||
.await
|
||||
.expect("Failed to enqueue transitioned object");
|
||||
|
||||
|
||||
@@ -29,638 +29,3 @@ pub(crate) mod storage_api;
|
||||
mod capacity_dirty_scope_test;
|
||||
#[cfg(test)]
|
||||
mod lifecycle_transition_api_test;
|
||||
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::{collections::HashMap, sync::Arc};
|
||||
|
||||
mod ecstore_admin {
|
||||
pub(crate) use crate::storage::ecstore_admin::get_server_info;
|
||||
}
|
||||
|
||||
mod ecstore_bucket {
|
||||
pub(crate) use crate::storage::ecstore_bucket::{
|
||||
bucket_target_sys, lifecycle, metadata, metadata_sys, object_lock, policy_sys, quota, replication, tagging, target,
|
||||
utils, versioning, versioning_sys,
|
||||
};
|
||||
}
|
||||
|
||||
mod ecstore_capacity {
|
||||
pub(crate) use crate::storage::ecstore_capacity::{
|
||||
PoolDecommissionInfo, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free,
|
||||
};
|
||||
}
|
||||
|
||||
#[allow(unused_imports)]
|
||||
mod ecstore_client {
|
||||
pub(crate) use crate::storage::ecstore_client::{object_api_utils, transition_api};
|
||||
}
|
||||
|
||||
mod ecstore_config {
|
||||
pub(crate) use crate::storage::ecstore_config::storageclass;
|
||||
}
|
||||
|
||||
mod ecstore_data_usage {
|
||||
pub(crate) use crate::storage::ecstore_data_usage::{
|
||||
apply_bucket_usage_memory_overlay, load_data_usage_from_backend, record_bucket_object_delete_memory,
|
||||
record_bucket_object_write_memory, remove_bucket_usage_from_backend,
|
||||
};
|
||||
}
|
||||
|
||||
#[allow(unused_imports)]
|
||||
mod ecstore_tier {
|
||||
pub(crate) use crate::storage::ecstore_tier::{tier, tier_config, warm_backend};
|
||||
}
|
||||
|
||||
pub(crate) type DiskError = crate::storage::DiskError;
|
||||
pub(crate) type DynReplicationPool = crate::storage::DynReplicationPool;
|
||||
pub(crate) type ECStore = crate::storage::ECStore;
|
||||
pub(crate) type EndpointServerPools = crate::storage::EndpointServerPools;
|
||||
pub(crate) type ExpiryState = crate::storage::ExpiryState;
|
||||
pub(crate) type NotificationSys = crate::storage::NotificationSys;
|
||||
pub(crate) type BucketBandwidthMonitor = crate::storage::BucketBandwidthMonitor;
|
||||
pub(crate) type ObjectStoreResolver = crate::storage::ObjectStoreResolver;
|
||||
pub(crate) type ObjectInfo = <ECStore as rustfs_storage_api::ObjectOperations>::ObjectInfo;
|
||||
pub(crate) type ObjectOptions = <ECStore as rustfs_storage_api::ObjectOperations>::ObjectOptions;
|
||||
pub(crate) type PoolDecommissionInfo = ecstore_capacity::PoolDecommissionInfo;
|
||||
pub(crate) type PoolStatus = ecstore_capacity::PoolStatus;
|
||||
pub(crate) type RebalStatus = crate::storage::ecstore_rebalance::RebalStatus;
|
||||
pub(crate) type StorageError = crate::storage::StorageError;
|
||||
pub(crate) type Error = StorageError;
|
||||
pub(crate) type TierConfigMgr = crate::storage::TierConfigMgr;
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) type Endpoint = crate::storage::Endpoint;
|
||||
#[cfg(test)]
|
||||
pub(crate) type Endpoints = crate::storage::Endpoints;
|
||||
#[cfg(test)]
|
||||
pub(crate) type PoolEndpoints = crate::storage::PoolEndpoints;
|
||||
#[cfg(test)]
|
||||
pub(crate) type TierConfig = ecstore_tier::tier_config::TierConfig;
|
||||
#[cfg(test)]
|
||||
pub(crate) type TierType = ecstore_tier::tier_config::TierType;
|
||||
#[cfg(test)]
|
||||
pub(crate) use ecstore_tier::warm_backend::WarmBackend as AppWarmBackend;
|
||||
#[cfg(test)]
|
||||
pub(crate) type WarmBackendGetOpts = ecstore_tier::warm_backend::WarmBackendGetOpts;
|
||||
|
||||
#[cfg(test)]
|
||||
#[allow(non_snake_case)]
|
||||
pub(crate) fn EndpointServerPools(pools: Vec<PoolEndpoints>) -> EndpointServerPools {
|
||||
crate::storage::EndpointServerPools::from(pools)
|
||||
}
|
||||
|
||||
pub(crate) trait AppObjectLockConfigExt {
|
||||
fn enabled(&self) -> bool;
|
||||
}
|
||||
|
||||
impl AppObjectLockConfigExt for s3s::dto::ObjectLockConfiguration {
|
||||
fn enabled(&self) -> bool {
|
||||
<s3s::dto::ObjectLockConfiguration as ecstore_bucket::object_lock::ObjectLockApi>::enabled(self)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) trait AppReplicationConfigExt {
|
||||
fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec<String>;
|
||||
fn replicate(&self, opts: &replication::ObjectOpts) -> bool;
|
||||
}
|
||||
|
||||
impl AppReplicationConfigExt for s3s::dto::ReplicationConfiguration {
|
||||
fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec<String> {
|
||||
<s3s::dto::ReplicationConfiguration as ecstore_bucket::replication::ReplicationConfigurationExt>::filter_target_arns(
|
||||
self, obj,
|
||||
)
|
||||
}
|
||||
|
||||
fn replicate(&self, opts: &replication::ObjectOpts) -> bool {
|
||||
<s3s::dto::ReplicationConfiguration as ecstore_bucket::replication::ReplicationConfigurationExt>::replicate(self, opts)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) trait AppVersioningConfigExt {
|
||||
fn prefix_enabled(&self, prefix: &str) -> bool;
|
||||
fn suspended(&self) -> bool;
|
||||
}
|
||||
|
||||
impl AppVersioningConfigExt for s3s::dto::VersioningConfiguration {
|
||||
fn prefix_enabled(&self, prefix: &str) -> bool {
|
||||
<s3s::dto::VersioningConfiguration as ecstore_bucket::versioning::VersioningApi>::prefix_enabled(self, prefix)
|
||||
}
|
||||
|
||||
fn suspended(&self) -> bool {
|
||||
<s3s::dto::VersioningConfiguration as ecstore_bucket::versioning::VersioningApi>::suspended(self)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn predict_lifecycle_expiration(
|
||||
lifecycle: &s3s::dto::BucketLifecycleConfiguration,
|
||||
obj: &lifecycle::lifecycle::ObjectOpts,
|
||||
) -> lifecycle::lifecycle::Event {
|
||||
ecstore_bucket::lifecycle::lifecycle::Lifecycle::predict_expiration(lifecycle, obj).await
|
||||
}
|
||||
|
||||
pub(crate) fn validate_restore_request(request: &s3s::dto::RestoreRequest, api: Arc<ECStore>) -> std::io::Result<()> {
|
||||
<s3s::dto::RestoreRequest as ecstore_bucket::lifecycle::bucket_lifecycle_ops::RestoreRequestOps>::validate(request, api)
|
||||
}
|
||||
|
||||
pub(crate) async fn get_server_info(get_pools: bool) -> rustfs_madmin::InfoMessage {
|
||||
ecstore_admin::get_server_info(get_pools).await
|
||||
}
|
||||
|
||||
pub(crate) mod bucket_target_sys {
|
||||
pub(crate) type BucketTargetSys = super::ecstore_bucket::bucket_target_sys::BucketTargetSys;
|
||||
}
|
||||
|
||||
pub(crate) mod lifecycle {
|
||||
use super::ECStore;
|
||||
|
||||
pub(crate) mod bucket_lifecycle_audit {
|
||||
pub(crate) type LcEventSrc = super::super::ecstore_bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc;
|
||||
}
|
||||
|
||||
pub(crate) mod bucket_lifecycle_ops {
|
||||
use std::sync::Arc;
|
||||
|
||||
use super::ECStore;
|
||||
use super::bucket_lifecycle_audit::LcEventSrc;
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn init_background_expiry(api: Arc<ECStore>) {
|
||||
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::init_background_expiry(api).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_expiry_for_existing_objects(
|
||||
api: Arc<ECStore>,
|
||||
bucket: &str,
|
||||
) -> Result<(), super::super::Error> {
|
||||
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_expiry_for_existing_objects(api, bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_transition_for_existing_objects(
|
||||
api: Arc<ECStore>,
|
||||
bucket: &str,
|
||||
) -> Result<(), super::super::Error> {
|
||||
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(api, bucket)
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_transition_immediate(oi: &super::super::ObjectInfo, src: LcEventSrc) {
|
||||
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_immediate(oi, src).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn post_restore_opts(
|
||||
version_id: &str,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
) -> Result<super::super::ObjectOptions, std::io::Error> {
|
||||
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::post_restore_opts(version_id, bucket, object).await
|
||||
}
|
||||
|
||||
pub(crate) async fn validate_transition_tier(lc: &s3s::dto::BucketLifecycleConfiguration) -> Result<(), std::io::Error> {
|
||||
super::super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::validate_transition_tier(lc).await
|
||||
}
|
||||
|
||||
pub(crate) async fn validate_lifecycle_config(
|
||||
lc: &s3s::dto::BucketLifecycleConfiguration,
|
||||
lock_config: &s3s::dto::ObjectLockConfiguration,
|
||||
) -> Result<(), std::io::Error> {
|
||||
use super::super::ecstore_bucket::lifecycle::lifecycle::Lifecycle as _;
|
||||
|
||||
lc.validate(lock_config).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod lifecycle_contract {
|
||||
#[cfg(test)]
|
||||
pub(crate) type IlmAction = super::super::ecstore_bucket::lifecycle::lifecycle::IlmAction;
|
||||
pub(crate) type Event = super::super::ecstore_bucket::lifecycle::lifecycle::Event;
|
||||
pub(crate) type ObjectOpts = super::super::ecstore_bucket::lifecycle::lifecycle::ObjectOpts;
|
||||
pub(crate) type TransitionOptions = super::super::ecstore_bucket::lifecycle::lifecycle::TransitionOptions;
|
||||
|
||||
pub(crate) const TRANSITION_COMPLETE: &str = super::super::ecstore_bucket::lifecycle::lifecycle::TRANSITION_COMPLETE;
|
||||
|
||||
pub(crate) fn expected_expiry_time(mod_time: time::OffsetDateTime, days: i32) -> time::OffsetDateTime {
|
||||
super::super::ecstore_bucket::lifecycle::lifecycle::expected_expiry_time(mod_time, days)
|
||||
}
|
||||
}
|
||||
pub(crate) use lifecycle_contract as lifecycle;
|
||||
|
||||
pub(crate) mod tier_delete_journal {
|
||||
use std::sync::Arc;
|
||||
|
||||
use super::ECStore;
|
||||
|
||||
pub(crate) async fn persist_tier_delete_journal_entry(
|
||||
api: Arc<ECStore>,
|
||||
je: &super::tier_sweeper::Jentry,
|
||||
) -> std::io::Result<()> {
|
||||
super::super::ecstore_bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry(api, je).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod tier_sweeper {
|
||||
pub(crate) type Jentry = super::super::ecstore_bucket::lifecycle::tier_sweeper::Jentry;
|
||||
|
||||
pub(crate) fn transitioned_delete_journal_entry(
|
||||
version_id: Option<uuid::Uuid>,
|
||||
versioned: bool,
|
||||
suspended: bool,
|
||||
transitioned: &rustfs_storage_api::TransitionedObject,
|
||||
) -> Option<Jentry> {
|
||||
super::super::ecstore_bucket::lifecycle::tier_sweeper::transitioned_delete_journal_entry(
|
||||
version_id,
|
||||
versioned,
|
||||
suspended,
|
||||
transitioned,
|
||||
)
|
||||
}
|
||||
|
||||
pub(crate) fn transitioned_force_delete_journal_entry(
|
||||
transitioned: &rustfs_storage_api::TransitionedObject,
|
||||
) -> Option<Jentry> {
|
||||
super::super::ecstore_bucket::lifecycle::tier_sweeper::transitioned_force_delete_journal_entry(transitioned)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod metadata {
|
||||
pub(crate) const BUCKET_CORS_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_CORS_CONFIG;
|
||||
pub(crate) const BUCKET_LIFECYCLE_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
|
||||
pub(crate) const BUCKET_NOTIFICATION_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_NOTIFICATION_CONFIG;
|
||||
pub(crate) const BUCKET_POLICY_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_POLICY_CONFIG;
|
||||
pub(crate) const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG;
|
||||
pub(crate) const BUCKET_REPLICATION_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_REPLICATION_CONFIG;
|
||||
pub(crate) const BUCKET_SSECONFIG: &str = super::ecstore_bucket::metadata::BUCKET_SSECONFIG;
|
||||
pub(crate) const BUCKET_TAGGING_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_TAGGING_CONFIG;
|
||||
pub(crate) const BUCKET_TARGETS_FILE: &str = super::ecstore_bucket::metadata::BUCKET_TARGETS_FILE;
|
||||
pub(crate) const BUCKET_VERSIONING_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_VERSIONING_CONFIG;
|
||||
#[cfg(test)]
|
||||
pub(crate) const OBJECT_LOCK_CONFIG: &str = super::ecstore_bucket::metadata::OBJECT_LOCK_CONFIG;
|
||||
}
|
||||
|
||||
pub(crate) mod metadata_sys {
|
||||
use std::sync::Arc;
|
||||
|
||||
use rustfs_policy::policy::BucketPolicy;
|
||||
use s3s::dto::{
|
||||
BucketLifecycleConfiguration, CORSConfiguration, NotificationConfiguration, ObjectLockConfiguration,
|
||||
PublicAccessBlockConfiguration, ServerSideEncryptionConfiguration, Tagging,
|
||||
};
|
||||
use time::OffsetDateTime;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
#[cfg(test)]
|
||||
use super::ECStore;
|
||||
use super::Error;
|
||||
use super::target::BucketTargets;
|
||||
|
||||
pub(crate) type BucketMetadataSys = super::ecstore_bucket::metadata_sys::BucketMetadataSys;
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn init_bucket_metadata_sys(api: Arc<ECStore>, buckets: Vec<String>) {
|
||||
super::ecstore_bucket::metadata_sys::init_bucket_metadata_sys(api, buckets).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime, Error> {
|
||||
super::ecstore_bucket::metadata_sys::delete(bucket, config_file).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_bucket_policy(bucket: &str) -> Result<(BucketPolicy, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_bucket_policy(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_bucket_policy_raw(bucket: &str) -> Result<(String, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_bucket_policy_raw(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_bucket_targets_config(bucket: &str) -> Result<BucketTargets, Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_bucket_targets_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_cors_config(bucket: &str) -> Result<(CORSConfiguration, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_cors_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_bucket_metadata_sys() -> Option<Arc<RwLock<BucketMetadataSys>>> {
|
||||
super::ecstore_bucket::metadata_sys::get_global_bucket_metadata_sys()
|
||||
}
|
||||
|
||||
pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_lifecycle_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_notification_config(bucket: &str) -> Result<Option<NotificationConfiguration>, Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_notification_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_object_lock_config(bucket: &str) -> Result<(ObjectLockConfiguration, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_object_lock_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_public_access_block_config(
|
||||
bucket: &str,
|
||||
) -> Result<(PublicAccessBlockConfiguration, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_public_access_block_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_replication_config(
|
||||
bucket: &str,
|
||||
) -> Result<(s3s::dto::ReplicationConfiguration, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_replication_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_sse_config(bucket: &str) -> Result<(ServerSideEncryptionConfiguration, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_sse_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime), Error> {
|
||||
super::ecstore_bucket::metadata_sys::get_tagging_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn update(bucket: &str, config_file: &str, data: Vec<u8>) -> Result<OffsetDateTime, Error> {
|
||||
super::ecstore_bucket::metadata_sys::update(bucket, config_file, data).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod object_api_utils {
|
||||
pub(crate) fn to_s3s_etag(etag: &str) -> s3s::dto::ETag {
|
||||
crate::storage::to_s3s_etag(etag)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod object_lock {
|
||||
pub(crate) mod objectlock {
|
||||
pub(crate) fn get_object_legalhold_meta(
|
||||
meta: &std::collections::HashMap<String, String>,
|
||||
) -> s3s::dto::ObjectLockLegalHold {
|
||||
super::super::ecstore_bucket::object_lock::objectlock::get_object_legalhold_meta(meta)
|
||||
}
|
||||
|
||||
pub(crate) fn get_object_retention_meta(
|
||||
meta: &std::collections::HashMap<String, String>,
|
||||
) -> s3s::dto::ObjectLockRetention {
|
||||
super::super::ecstore_bucket::object_lock::objectlock::get_object_retention_meta(meta)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod objectlock_sys {
|
||||
pub(crate) type BucketObjectLockSys = super::super::ecstore_bucket::object_lock::objectlock_sys::BucketObjectLockSys;
|
||||
pub(crate) type ObjectLockBlockReason = super::super::ecstore_bucket::object_lock::objectlock_sys::ObjectLockBlockReason;
|
||||
|
||||
pub(crate) async fn check_object_lock_for_deletion(
|
||||
bucket: &str,
|
||||
obj_info: &super::super::ObjectInfo,
|
||||
bypass_governance: bool,
|
||||
) -> Option<ObjectLockBlockReason> {
|
||||
super::super::ecstore_bucket::object_lock::objectlock_sys::check_object_lock_for_deletion(
|
||||
bucket,
|
||||
obj_info,
|
||||
bypass_governance,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) fn is_retention_active(mode: &str, retain_until_date: Option<&s3s::dto::Date>) -> bool {
|
||||
super::super::ecstore_bucket::object_lock::objectlock_sys::is_retention_active(mode, retain_until_date)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod policy_sys {
|
||||
pub(crate) type PolicySys = super::ecstore_bucket::policy_sys::PolicySys;
|
||||
}
|
||||
|
||||
pub(crate) mod quota {
|
||||
pub(crate) mod checker {
|
||||
pub(crate) type QuotaChecker = super::super::ecstore_bucket::quota::checker::QuotaChecker;
|
||||
}
|
||||
|
||||
pub(crate) type QuotaOperation = super::ecstore_bucket::quota::QuotaOperation;
|
||||
}
|
||||
|
||||
pub(crate) mod replication {
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
pub(crate) type DeletedObjectReplicationInfo = super::ecstore_bucket::replication::DeletedObjectReplicationInfo;
|
||||
pub(crate) type MustReplicateOptions = super::ecstore_bucket::replication::MustReplicateOptions;
|
||||
pub(crate) type ObjectOpts = super::ecstore_bucket::replication::ObjectOpts;
|
||||
pub(crate) type ReplicateDecision = rustfs_filemeta::ReplicateDecision;
|
||||
|
||||
pub(crate) async fn check_replicate_delete(
|
||||
bucket: &str,
|
||||
dobj: &rustfs_storage_api::ObjectToDelete,
|
||||
oi: &super::ObjectInfo,
|
||||
del_opts: &super::ObjectOptions,
|
||||
gerr: Option<String>,
|
||||
) -> ReplicateDecision {
|
||||
super::ecstore_bucket::replication::check_replicate_delete(bucket, dobj, oi, del_opts, gerr).await
|
||||
}
|
||||
|
||||
pub(crate) fn get_must_replicate_options(
|
||||
user_defined: &HashMap<String, String>,
|
||||
user_tags: String,
|
||||
status: rustfs_filemeta::ReplicationStatusType,
|
||||
op_type: rustfs_filemeta::ReplicationType,
|
||||
opts: super::ObjectOptions,
|
||||
) -> MustReplicateOptions {
|
||||
super::ecstore_bucket::replication::get_must_replicate_options(user_defined, user_tags, status, op_type, opts)
|
||||
}
|
||||
|
||||
pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplicateOptions) -> ReplicateDecision {
|
||||
super::ecstore_bucket::replication::must_replicate(bucket, object, mopts).await
|
||||
}
|
||||
|
||||
pub(crate) async fn schedule_replication(
|
||||
oi: super::ObjectInfo,
|
||||
store: Arc<super::ECStore>,
|
||||
dsc: ReplicateDecision,
|
||||
op_type: rustfs_filemeta::ReplicationType,
|
||||
) {
|
||||
super::ecstore_bucket::replication::schedule_replication(oi, store, dsc, op_type).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn schedule_replication_delete(dv: DeletedObjectReplicationInfo) {
|
||||
super::ecstore_bucket::replication::schedule_replication_delete(dv).await;
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod tagging {
|
||||
pub(crate) fn decode_tags(tags: &str) -> Vec<s3s::dto::Tag> {
|
||||
super::ecstore_bucket::tagging::decode_tags(tags)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod target {
|
||||
#[cfg(test)]
|
||||
pub(crate) type BucketTarget = super::ecstore_bucket::target::BucketTarget;
|
||||
pub(crate) type BucketTargetType = super::ecstore_bucket::target::BucketTargetType;
|
||||
pub(crate) type BucketTargets = super::ecstore_bucket::target::BucketTargets;
|
||||
}
|
||||
|
||||
pub(crate) mod utils {
|
||||
pub(crate) fn serialize<T: s3s::xml::Serialize>(val: &T) -> s3s::xml::SerResult<Vec<u8>> {
|
||||
super::ecstore_bucket::utils::serialize(val)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod versioning {}
|
||||
|
||||
pub(crate) mod versioning_sys {
|
||||
pub(crate) type BucketVersioningSys = super::ecstore_bucket::versioning_sys::BucketVersioningSys;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) mod transition_api {
|
||||
pub(crate) type ReadCloser = super::ecstore_client::transition_api::ReadCloser;
|
||||
pub(crate) type ReaderImpl = super::ecstore_client::transition_api::ReaderImpl;
|
||||
}
|
||||
|
||||
pub(crate) mod storageclass {
|
||||
pub(crate) const STANDARD: &str = super::ecstore_config::storageclass::STANDARD;
|
||||
#[cfg(test)]
|
||||
pub(crate) const STANDARD_IA: &str = super::ecstore_config::storageclass::STANDARD_IA;
|
||||
}
|
||||
|
||||
pub(crate) type StorageClassConfig = crate::storage::ecstore_config::storageclass::Config;
|
||||
|
||||
pub(crate) fn set_global_storage_class(cfg: StorageClassConfig) {
|
||||
crate::storage::ecstore_config::set_global_storage_class(cfg);
|
||||
}
|
||||
|
||||
pub(crate) fn get_total_usable_capacity(disks: &[rustfs_madmin::Disk], info: &rustfs_madmin::StorageInfo) -> usize {
|
||||
ecstore_capacity::get_total_usable_capacity(disks, info)
|
||||
}
|
||||
|
||||
pub(crate) fn get_total_usable_capacity_free(disks: &[rustfs_madmin::Disk], info: &rustfs_madmin::StorageInfo) -> usize {
|
||||
ecstore_capacity::get_total_usable_capacity_free(disks, info)
|
||||
}
|
||||
|
||||
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
|
||||
ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn load_data_usage_from_backend(
|
||||
store: Arc<ECStore>,
|
||||
) -> std::result::Result<rustfs_data_usage::DataUsageInfo, Error> {
|
||||
ecstore_data_usage::load_data_usage_from_backend(store).await
|
||||
}
|
||||
|
||||
pub(crate) async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) {
|
||||
ecstore_data_usage::record_bucket_object_delete_memory(bucket, deleted_size, removed_current_object).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn record_bucket_object_write_memory(bucket: &str, previous_current_size: Option<u64>, new_size: u64) {
|
||||
ecstore_data_usage::record_bucket_object_write_memory(bucket, previous_current_size, new_size).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str) -> std::result::Result<(), Error> {
|
||||
ecstore_data_usage::remove_bucket_usage_from_backend(store, bucket).await
|
||||
}
|
||||
|
||||
pub(crate) fn is_all_buckets_not_found(errs: &[Option<DiskError>]) -> bool {
|
||||
crate::storage::is_all_buckets_not_found(errs)
|
||||
}
|
||||
|
||||
pub(crate) fn is_err_bucket_not_found(err: &Error) -> bool {
|
||||
crate::storage::is_err_bucket_not_found(err)
|
||||
}
|
||||
|
||||
pub(crate) fn is_err_object_not_found(err: &Error) -> bool {
|
||||
crate::storage::is_err_object_not_found(err)
|
||||
}
|
||||
|
||||
pub(crate) fn is_err_version_not_found(err: &Error) -> bool {
|
||||
crate::storage::is_err_version_not_found(err)
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_endpoints_opt() -> Option<EndpointServerPools> {
|
||||
crate::storage::get_global_endpoints_opt()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_deployment_id() -> Option<String> {
|
||||
crate::storage::get_global_deployment_id()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_lock_client() -> Option<Arc<dyn rustfs_lock::client::LockClient>> {
|
||||
crate::storage::get_global_lock_client()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_lock_clients() -> Option<&'static HashMap<String, Arc<dyn rustfs_lock::client::LockClient>>> {
|
||||
crate::storage::get_global_lock_clients()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_region() -> Option<s3s::region::Region> {
|
||||
crate::storage::get_global_region()
|
||||
}
|
||||
|
||||
pub(crate) fn global_rustfs_port() -> u16 {
|
||||
crate::storage::global_rustfs_port()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_tier_config_mgr() -> Arc<tokio::sync::RwLock<TierConfigMgr>> {
|
||||
crate::storage::get_global_tier_config_mgr()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_expiry_state() -> Arc<tokio::sync::RwLock<ExpiryState>> {
|
||||
crate::storage::get_global_expiry_state()
|
||||
}
|
||||
|
||||
pub(crate) fn new_object_layer_fn() -> Option<Arc<ECStore>> {
|
||||
crate::storage::new_object_layer_fn()
|
||||
}
|
||||
|
||||
pub(crate) fn set_object_store_resolver(resolver: Arc<ObjectStoreResolver>) -> bool {
|
||||
crate::storage::set_object_store_resolver(resolver)
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_notification_sys() -> Option<&'static NotificationSys> {
|
||||
crate::storage::get_global_notification_sys()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_bucket_monitor() -> Option<Arc<BucketBandwidthMonitor>> {
|
||||
crate::storage::get_global_bucket_monitor()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_replication_pool() -> Option<Arc<DynReplicationPool>> {
|
||||
crate::storage::get_global_replication_pool()
|
||||
}
|
||||
|
||||
pub(crate) type ReplicationStats = crate::storage::ReplicationStats;
|
||||
|
||||
pub(crate) fn get_global_replication_stats() -> Option<Arc<ReplicationStats>> {
|
||||
crate::storage::get_global_replication_stats()
|
||||
}
|
||||
|
||||
pub(crate) type DailyAllTierStats = crate::storage::DailyAllTierStats;
|
||||
|
||||
pub(crate) fn get_global_boot_time() -> Option<std::time::SystemTime> {
|
||||
crate::storage::get_global_boot_time()
|
||||
}
|
||||
|
||||
pub(crate) fn get_daily_all_tier_stats() -> DailyAllTierStats {
|
||||
crate::storage::get_daily_all_tier_stats()
|
||||
}
|
||||
|
||||
pub(crate) type ScannerMetricsReport = rustfs_common::metrics::ScannerMetricsReport;
|
||||
|
||||
pub(crate) async fn collect_scanner_metrics_report() -> ScannerMetricsReport {
|
||||
rustfs_common::metrics::global_metrics().report().await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn init_local_disks(endpoint_pools: EndpointServerPools) -> Result<(), Error> {
|
||||
crate::storage::init_local_disks(endpoint_pools).await
|
||||
}
|
||||
|
||||
@@ -14,19 +14,28 @@
|
||||
|
||||
//! Multipart application use-case contracts.
|
||||
|
||||
use super::ECStore;
|
||||
use super::object_api_utils::to_s3s_etag;
|
||||
use super::quota::checker::QuotaChecker;
|
||||
use super::s3_api::multipart::{
|
||||
ListMultipartUploadsParams, build_list_multipart_uploads_output, build_list_parts_output,
|
||||
parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number,
|
||||
};
|
||||
use super::storage_api::ECStore;
|
||||
use super::storage_api::access::has_bypass_governance_header;
|
||||
use super::storage_api::bucket::quota::checker::QuotaChecker;
|
||||
use super::storage_api::bucket::{
|
||||
lifecycle::{bucket_lifecycle_audit::LcEventSrc, bucket_lifecycle_ops::enqueue_transition_immediate},
|
||||
metadata_sys,
|
||||
quota::QuotaOperation,
|
||||
replication::{get_must_replicate_options, must_replicate, schedule_replication},
|
||||
versioning_sys::BucketVersioningSys,
|
||||
};
|
||||
use super::storage_api::compression::is_disk_compressible;
|
||||
use super::storage_api::data_usage::record_bucket_object_write_memory;
|
||||
use super::storage_api::error::{StorageError, is_err_object_not_found, is_err_version_not_found};
|
||||
use super::storage_api::helper::OperationHelper;
|
||||
#[cfg(test)]
|
||||
use super::storage_api::io::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader, wrap_reader};
|
||||
use super::storage_api::io::{HashReader, WriteEncryption, WritePlan, compression_metadata_value};
|
||||
use super::storage_api::object_utils::to_s3s_etag;
|
||||
use super::storage_api::options::{
|
||||
copy_src_opts, extract_metadata_from_mime, get_complete_multipart_upload_opts, get_content_sha256_with_query, get_opts,
|
||||
parse_copy_source_range, put_opts, validate_archive_content_encoding,
|
||||
@@ -39,14 +48,6 @@ use super::storage_api::sse::{
|
||||
mark_encrypted_multipart_metadata, sse_decryption, sse_prepare_encryption,
|
||||
};
|
||||
use super::storage_api::{StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader};
|
||||
use super::{StorageError, is_err_object_not_found, is_err_version_not_found};
|
||||
use super::{
|
||||
lifecycle::{bucket_lifecycle_audit::LcEventSrc, bucket_lifecycle_ops::enqueue_transition_immediate},
|
||||
metadata_sys,
|
||||
quota::QuotaOperation,
|
||||
replication::{get_must_replicate_options, must_replicate, schedule_replication},
|
||||
versioning_sys::BucketVersioningSys,
|
||||
};
|
||||
use crate::app::object_usecase::{build_put_like_object_lock_metadata, validate_existing_object_lock_for_write};
|
||||
use crate::app::runtime_sources::{AppContext, get_global_app_context, resolve_object_store_handle_for_context};
|
||||
use crate::capacity::record_capacity_write;
|
||||
@@ -488,7 +489,7 @@ impl DefaultMultipartUsecase {
|
||||
));
|
||||
}
|
||||
// Update quota tracking after successful multipart upload
|
||||
super::record_bucket_object_write_memory(&bucket, previous_current_size, obj_info.size.max(0) as u64).await;
|
||||
record_bucket_object_write_memory(&bucket, previous_current_size, obj_info.size.max(0) as u64).await;
|
||||
}
|
||||
Err(e) => {
|
||||
warn!("Quota check failed for bucket {}: {}, allowing operation", bucket, e);
|
||||
|
||||
@@ -14,23 +14,50 @@
|
||||
|
||||
//! Object application use-case contracts.
|
||||
|
||||
use super::object_api_utils::to_s3s_etag;
|
||||
use super::quota::checker::QuotaChecker;
|
||||
use super::storageclass;
|
||||
// Performance metrics recording (with zero-copy-metrics integration)
|
||||
use super::ECStore;
|
||||
use super::s3_api::multipart::parse_list_parts_params;
|
||||
use super::storage_api::ECStore;
|
||||
use super::storage_api::access::{PostObjectRequestMarker, authorize_request, has_bypass_governance_header, req_info_mut};
|
||||
use super::storage_api::bucket::quota::checker::QuotaChecker;
|
||||
use super::storage_api::bucket::{
|
||||
ReplicationConfigExt as _, VersioningConfigExt as _,
|
||||
lifecycle::{
|
||||
bucket_lifecycle_audit::LcEventSrc,
|
||||
bucket_lifecycle_ops::{enqueue_transition_immediate, post_restore_opts},
|
||||
lifecycle::{self, TransitionOptions},
|
||||
tier_delete_journal, tier_sweeper,
|
||||
},
|
||||
metadata_sys,
|
||||
object_lock::{
|
||||
objectlock::{get_object_legalhold_meta, get_object_retention_meta},
|
||||
objectlock_sys::{BucketObjectLockSys, check_object_lock_for_deletion, is_retention_active},
|
||||
},
|
||||
predict_lifecycle_expiration,
|
||||
quota::QuotaOperation,
|
||||
replication::{
|
||||
DeletedObjectReplicationInfo, ObjectOpts as ReplicationObjectOpts, check_replicate_delete, get_must_replicate_options,
|
||||
must_replicate, schedule_replication, schedule_replication_delete,
|
||||
},
|
||||
tagging::decode_tags,
|
||||
validate_restore_request,
|
||||
versioning_sys::BucketVersioningSys,
|
||||
};
|
||||
use super::storage_api::compression::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible};
|
||||
use super::storage_api::concurrency::{
|
||||
self, ConcurrencyManager, GetObjectGuard, PutObjectGuard, get_concurrency_aware_buffer_size, get_concurrency_manager,
|
||||
get_put_concurrency_aware_buffer_size,
|
||||
};
|
||||
use super::storage_api::data_usage::{record_bucket_object_delete_memory, record_bucket_object_write_memory};
|
||||
use super::storage_api::deadlock_detector;
|
||||
use super::storage_api::ecfs::FS;
|
||||
use super::storage_api::error::{
|
||||
DiskError, Error as EcstoreError, StorageError, is_all_buckets_not_found, is_err_bucket_not_found, is_err_object_not_found,
|
||||
is_err_version_not_found,
|
||||
};
|
||||
use super::storage_api::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children};
|
||||
use super::storage_api::helper::{OperationHelper, spawn_background_with_context};
|
||||
use super::storage_api::io::{DynReader, HashReader, WritePlan, compression_metadata_value, wrap_reader};
|
||||
use super::storage_api::object_utils::to_s3s_etag;
|
||||
use super::storage_api::options::{
|
||||
copy_dst_opts, copy_src_opts, del_opts, extract_metadata, extract_metadata_from_mime_with_object_name,
|
||||
filter_object_metadata, get_content_sha256_with_query, get_opts, normalize_content_encoding_for_storage, put_opts,
|
||||
@@ -42,6 +69,7 @@ use super::storage_api::sse::{
|
||||
encryption_material_to_metadata, extract_server_side_encryption_from_headers, extract_ssec_params_from_headers,
|
||||
extract_ssekms_context_from_headers, get_buffer_size_opt_in, map_get_object_reader_error, sse_decryption, sse_encryption,
|
||||
};
|
||||
use super::storage_api::storage_class as storageclass;
|
||||
use super::storage_api::timeout_wrapper::{GetObjectTimeoutPolicy, RequestTimeoutWrapper};
|
||||
use super::storage_api::{
|
||||
RFC1123, check_preconditions, get_validated_store, has_replication_rules, parse_object_lock_legal_hold,
|
||||
@@ -49,28 +77,6 @@ use super::storage_api::{
|
||||
strip_managed_encryption_metadata, validate_bucket_object_lock_enabled, validate_object_key, validate_sse_headers_for_read,
|
||||
validate_sse_headers_for_write, validate_ssec_for_read, wrap_response_with_cors,
|
||||
};
|
||||
use super::{AppReplicationConfigExt as _, AppVersioningConfigExt as _, predict_lifecycle_expiration, validate_restore_request};
|
||||
use super::{DiskError, is_all_buckets_not_found};
|
||||
use super::{Error as EcstoreError, StorageError, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found};
|
||||
use super::{
|
||||
lifecycle::{
|
||||
bucket_lifecycle_audit::LcEventSrc,
|
||||
bucket_lifecycle_ops::{enqueue_transition_immediate, post_restore_opts},
|
||||
lifecycle::{self, TransitionOptions},
|
||||
},
|
||||
metadata_sys,
|
||||
object_lock::{
|
||||
objectlock::{get_object_legalhold_meta, get_object_retention_meta},
|
||||
objectlock_sys::{BucketObjectLockSys, check_object_lock_for_deletion, is_retention_active},
|
||||
},
|
||||
quota::QuotaOperation,
|
||||
replication::{
|
||||
DeletedObjectReplicationInfo, ObjectOpts as ReplicationObjectOpts, check_replicate_delete, get_must_replicate_options,
|
||||
must_replicate, schedule_replication, schedule_replication_delete,
|
||||
},
|
||||
tagging::decode_tags,
|
||||
versioning_sys::BucketVersioningSys,
|
||||
};
|
||||
use crate::app::runtime_sources::{
|
||||
AppContext, get_global_app_context, resolve_expiry_state_handle, resolve_notify_interface_for_context,
|
||||
resolve_object_store_handle_for_context,
|
||||
@@ -316,10 +322,10 @@ async fn enqueue_transitioned_delete_cleanup(
|
||||
let _activity_guard = DeleteTailActivityGuard::new(DeleteTailStage::Cleanup);
|
||||
|
||||
let je = if opts.delete_prefix {
|
||||
super::lifecycle::tier_sweeper::transitioned_force_delete_journal_entry(&existing.transitioned_object)
|
||||
tier_sweeper::transitioned_force_delete_journal_entry(&existing.transitioned_object)
|
||||
} else {
|
||||
let version_id = opts.version_id.as_ref().and_then(|v| Uuid::parse_str(v).ok());
|
||||
super::lifecycle::tier_sweeper::transitioned_delete_journal_entry(
|
||||
tier_sweeper::transitioned_delete_journal_entry(
|
||||
version_id,
|
||||
opts.versioned,
|
||||
opts.version_suspended,
|
||||
@@ -330,7 +336,7 @@ async fn enqueue_transitioned_delete_cleanup(
|
||||
return Ok(());
|
||||
};
|
||||
|
||||
super::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry(store, &je).await?;
|
||||
tier_delete_journal::persist_tier_delete_journal_entry(store, &je).await?;
|
||||
|
||||
let expiry_state = resolve_expiry_state_handle();
|
||||
let mut expiry_state = expiry_state.write().await;
|
||||
@@ -2591,7 +2597,7 @@ impl DefaultObjectUsecase {
|
||||
maybe_enqueue_transition_immediate(&obj_info, LcEventSrc::S3PutObject).await;
|
||||
|
||||
// Fast in-memory update for immediate quota and admin usage consistency
|
||||
super::record_bucket_object_write_memory(&bucket, previous_current_size, obj_info.size.max(0) as u64).await;
|
||||
record_bucket_object_write_memory(&bucket, previous_current_size, obj_info.size.max(0) as u64).await;
|
||||
|
||||
let raw_version = obj_info.version_id.map(|v| v.to_string());
|
||||
|
||||
@@ -3558,7 +3564,7 @@ impl DefaultObjectUsecase {
|
||||
|
||||
// Update quota tracking after successful copy
|
||||
if has_bucket_metadata {
|
||||
super::record_bucket_object_write_memory(&bucket, previous_current_size, oi.size.max(0) as u64).await;
|
||||
record_bucket_object_write_memory(&bucket, previous_current_size, oi.size.max(0) as u64).await;
|
||||
}
|
||||
|
||||
let raw_dest_version = oi.version_id.map(|v| v.to_string());
|
||||
@@ -3835,7 +3841,7 @@ impl DefaultObjectUsecase {
|
||||
);
|
||||
}
|
||||
let size = object_sizes[i].max(0) as u64;
|
||||
super::record_bucket_object_delete_memory(
|
||||
record_bucket_object_delete_memory(
|
||||
&bucket,
|
||||
size,
|
||||
existing_object_infos[i].is_some() && object_to_delete[i].version_id.is_none(),
|
||||
@@ -4072,7 +4078,7 @@ impl DefaultObjectUsecase {
|
||||
}
|
||||
|
||||
// Fast in-memory update for immediate quota and admin usage consistency
|
||||
super::record_bucket_object_delete_memory(&bucket, obj_info.size.max(0) as u64, opts.version_id.is_none()).await;
|
||||
record_bucket_object_delete_memory(&bucket, obj_info.size.max(0) as u64, opts.version_id.is_none()).await;
|
||||
|
||||
if obj_info.name.is_empty() {
|
||||
if replicate_force_delete {
|
||||
|
||||
@@ -14,12 +14,607 @@
|
||||
|
||||
//! App-local boundary for storage-layer helper APIs used by S3 use cases.
|
||||
|
||||
pub(crate) use crate::storage::ECStore;
|
||||
#[cfg(test)]
|
||||
pub(crate) use crate::storage::{Endpoint, Endpoints, PoolEndpoints};
|
||||
pub(crate) type EndpointServerPools = crate::storage::EndpointServerPools;
|
||||
|
||||
#[cfg(test)]
|
||||
#[allow(non_snake_case)]
|
||||
pub(crate) fn EndpointServerPools(pools: Vec<PoolEndpoints>) -> EndpointServerPools {
|
||||
crate::storage::EndpointServerPools::from(pools)
|
||||
}
|
||||
|
||||
pub(crate) mod admin {
|
||||
pub(crate) async fn get_server_info(get_pools: bool) -> rustfs_madmin::InfoMessage {
|
||||
crate::storage::ecstore_admin::get_server_info(get_pools).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod capacity {
|
||||
pub(crate) type PoolDecommissionInfo = crate::storage::ecstore_capacity::PoolDecommissionInfo;
|
||||
pub(crate) type PoolStatus = crate::storage::ecstore_capacity::PoolStatus;
|
||||
pub(crate) type RebalStatus = crate::storage::ecstore_rebalance::RebalStatus;
|
||||
|
||||
pub(crate) fn get_total_usable_capacity(disks: &[rustfs_madmin::Disk], info: &rustfs_madmin::StorageInfo) -> usize {
|
||||
crate::storage::ecstore_capacity::get_total_usable_capacity(disks, info)
|
||||
}
|
||||
|
||||
pub(crate) fn get_total_usable_capacity_free(disks: &[rustfs_madmin::Disk], info: &rustfs_madmin::StorageInfo) -> usize {
|
||||
crate::storage::ecstore_capacity::get_total_usable_capacity_free(disks, info)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod data_usage {
|
||||
use std::sync::Arc;
|
||||
|
||||
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
|
||||
crate::storage::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn load_data_usage_from_backend(
|
||||
store: Arc<super::ECStore>,
|
||||
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::StorageError> {
|
||||
crate::storage::ecstore_data_usage::load_data_usage_from_backend(store).await
|
||||
}
|
||||
|
||||
pub(crate) async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) {
|
||||
crate::storage::ecstore_data_usage::record_bucket_object_delete_memory(bucket, deleted_size, removed_current_object)
|
||||
.await;
|
||||
}
|
||||
|
||||
pub(crate) async fn record_bucket_object_write_memory(bucket: &str, previous_current_size: Option<u64>, new_size: u64) {
|
||||
crate::storage::ecstore_data_usage::record_bucket_object_write_memory(bucket, previous_current_size, new_size).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn remove_bucket_usage_from_backend(
|
||||
store: Arc<super::ECStore>,
|
||||
bucket: &str,
|
||||
) -> Result<(), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_data_usage::remove_bucket_usage_from_backend(store, bucket).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod runtime {
|
||||
use std::{collections::HashMap, sync::Arc};
|
||||
|
||||
pub(crate) type BucketBandwidthMonitor = crate::storage::BucketBandwidthMonitor;
|
||||
pub(crate) type DailyAllTierStats = crate::storage::DailyAllTierStats;
|
||||
pub(crate) type DynReplicationPool = crate::storage::DynReplicationPool;
|
||||
pub(crate) type ExpiryState = crate::storage::ExpiryState;
|
||||
pub(crate) type NotificationSys = crate::storage::NotificationSys;
|
||||
pub(crate) type ObjectStoreResolver = crate::storage::ObjectStoreResolver;
|
||||
pub(crate) type ReplicationStats = crate::storage::ReplicationStats;
|
||||
pub(crate) type ScannerMetricsReport = rustfs_common::metrics::ScannerMetricsReport;
|
||||
pub(crate) type StorageClassConfig = crate::storage::ecstore_config::storageclass::Config;
|
||||
pub(crate) type TierConfigMgr = crate::storage::TierConfigMgr;
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) type TierConfig = crate::storage::ecstore_tier::tier_config::TierConfig;
|
||||
#[cfg(test)]
|
||||
pub(crate) type TierType = crate::storage::ecstore_tier::tier_config::TierType;
|
||||
#[cfg(test)]
|
||||
pub(crate) use crate::storage::ecstore_tier::warm_backend::WarmBackend as AppWarmBackend;
|
||||
#[cfg(test)]
|
||||
pub(crate) type WarmBackendGetOpts = crate::storage::ecstore_tier::warm_backend::WarmBackendGetOpts;
|
||||
|
||||
pub(crate) fn set_global_storage_class(cfg: StorageClassConfig) {
|
||||
crate::storage::ecstore_config::set_global_storage_class(cfg);
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_endpoints_opt() -> Option<super::EndpointServerPools> {
|
||||
crate::storage::get_global_endpoints_opt()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_deployment_id() -> Option<String> {
|
||||
crate::storage::get_global_deployment_id()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_lock_client() -> Option<Arc<dyn rustfs_lock::client::LockClient>> {
|
||||
crate::storage::get_global_lock_client()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_lock_clients() -> Option<&'static HashMap<String, Arc<dyn rustfs_lock::client::LockClient>>> {
|
||||
crate::storage::get_global_lock_clients()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_region() -> Option<s3s::region::Region> {
|
||||
crate::storage::get_global_region()
|
||||
}
|
||||
|
||||
pub(crate) fn global_rustfs_port() -> u16 {
|
||||
crate::storage::global_rustfs_port()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_tier_config_mgr() -> Arc<tokio::sync::RwLock<TierConfigMgr>> {
|
||||
crate::storage::get_global_tier_config_mgr()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_expiry_state() -> Arc<tokio::sync::RwLock<ExpiryState>> {
|
||||
crate::storage::get_global_expiry_state()
|
||||
}
|
||||
|
||||
pub(crate) fn new_object_layer_fn() -> Option<Arc<super::ECStore>> {
|
||||
crate::storage::new_object_layer_fn()
|
||||
}
|
||||
|
||||
pub(crate) fn set_object_store_resolver(resolver: Arc<ObjectStoreResolver>) -> bool {
|
||||
crate::storage::set_object_store_resolver(resolver)
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_notification_sys() -> Option<&'static NotificationSys> {
|
||||
crate::storage::get_global_notification_sys()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_bucket_monitor() -> Option<Arc<BucketBandwidthMonitor>> {
|
||||
crate::storage::get_global_bucket_monitor()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_replication_pool() -> Option<Arc<DynReplicationPool>> {
|
||||
crate::storage::get_global_replication_pool()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_replication_stats() -> Option<Arc<ReplicationStats>> {
|
||||
crate::storage::get_global_replication_stats()
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_boot_time() -> Option<std::time::SystemTime> {
|
||||
crate::storage::get_global_boot_time()
|
||||
}
|
||||
|
||||
pub(crate) fn get_daily_all_tier_stats() -> DailyAllTierStats {
|
||||
crate::storage::get_daily_all_tier_stats()
|
||||
}
|
||||
|
||||
pub(crate) async fn collect_scanner_metrics_report() -> ScannerMetricsReport {
|
||||
rustfs_common::metrics::global_metrics().report().await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn init_local_disks(endpoint_pools: super::EndpointServerPools) -> Result<(), crate::storage::StorageError> {
|
||||
crate::storage::init_local_disks(endpoint_pools).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod access {
|
||||
pub(crate) use crate::storage::access::{
|
||||
PostObjectRequestMarker, ReqInfo, authorize_request, has_bypass_governance_header, req_info_mut, req_info_ref,
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) mod bucket {
|
||||
use std::sync::Arc;
|
||||
|
||||
pub(crate) trait ObjectLockConfigExt {
|
||||
fn enabled(&self) -> bool;
|
||||
}
|
||||
|
||||
impl ObjectLockConfigExt for s3s::dto::ObjectLockConfiguration {
|
||||
fn enabled(&self) -> bool {
|
||||
<s3s::dto::ObjectLockConfiguration as crate::storage::ecstore_bucket::object_lock::ObjectLockApi>::enabled(self)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) trait ReplicationConfigExt {
|
||||
fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec<String>;
|
||||
fn replicate(&self, opts: &replication::ObjectOpts) -> bool;
|
||||
}
|
||||
|
||||
impl ReplicationConfigExt for s3s::dto::ReplicationConfiguration {
|
||||
fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec<String> {
|
||||
<s3s::dto::ReplicationConfiguration as crate::storage::ecstore_bucket::replication::ReplicationConfigurationExt>::filter_target_arns(
|
||||
self, obj,
|
||||
)
|
||||
}
|
||||
|
||||
fn replicate(&self, opts: &replication::ObjectOpts) -> bool {
|
||||
<s3s::dto::ReplicationConfiguration as crate::storage::ecstore_bucket::replication::ReplicationConfigurationExt>::replicate(
|
||||
self, opts,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) trait VersioningConfigExt {
|
||||
fn prefix_enabled(&self, prefix: &str) -> bool;
|
||||
fn suspended(&self) -> bool;
|
||||
}
|
||||
|
||||
impl VersioningConfigExt for s3s::dto::VersioningConfiguration {
|
||||
fn prefix_enabled(&self, prefix: &str) -> bool {
|
||||
<s3s::dto::VersioningConfiguration as crate::storage::ecstore_bucket::versioning::VersioningApi>::prefix_enabled(
|
||||
self, prefix,
|
||||
)
|
||||
}
|
||||
|
||||
fn suspended(&self) -> bool {
|
||||
<s3s::dto::VersioningConfiguration as crate::storage::ecstore_bucket::versioning::VersioningApi>::suspended(self)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn predict_lifecycle_expiration(
|
||||
lifecycle: &s3s::dto::BucketLifecycleConfiguration,
|
||||
obj: &lifecycle::lifecycle::ObjectOpts,
|
||||
) -> lifecycle::lifecycle::Event {
|
||||
crate::storage::ecstore_bucket::lifecycle::lifecycle::Lifecycle::predict_expiration(lifecycle, obj).await
|
||||
}
|
||||
|
||||
pub(crate) fn validate_restore_request(
|
||||
request: &s3s::dto::RestoreRequest,
|
||||
api: Arc<crate::storage::ECStore>,
|
||||
) -> std::io::Result<()> {
|
||||
<s3s::dto::RestoreRequest as crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::RestoreRequestOps>::validate(
|
||||
request, api,
|
||||
)
|
||||
}
|
||||
|
||||
pub(crate) mod bucket_target_sys {
|
||||
pub(crate) type BucketTargetSys = crate::storage::ecstore_bucket::bucket_target_sys::BucketTargetSys;
|
||||
}
|
||||
|
||||
pub(crate) mod lifecycle {
|
||||
pub(crate) mod bucket_lifecycle_audit {
|
||||
pub(crate) type LcEventSrc = crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc;
|
||||
}
|
||||
|
||||
pub(crate) mod bucket_lifecycle_ops {
|
||||
use std::sync::Arc;
|
||||
|
||||
use super::bucket_lifecycle_audit::LcEventSrc;
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn init_background_expiry(api: Arc<crate::storage::ECStore>) {
|
||||
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::init_background_expiry(api).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_expiry_for_existing_objects(
|
||||
api: Arc<crate::storage::ECStore>,
|
||||
bucket: &str,
|
||||
) -> Result<(), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_expiry_for_existing_objects(api, bucket)
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_transition_for_existing_objects(
|
||||
api: Arc<crate::storage::ECStore>,
|
||||
bucket: &str,
|
||||
) -> Result<(), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(
|
||||
api, bucket,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_transition_immediate(oi: &crate::storage::StorageObjectInfo, src: LcEventSrc) {
|
||||
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_immediate(oi, src).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn post_restore_opts(
|
||||
version_id: &str,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
) -> Result<crate::storage::StorageObjectOptions, std::io::Error> {
|
||||
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::post_restore_opts(version_id, bucket, object)
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) async fn validate_transition_tier(
|
||||
lc: &s3s::dto::BucketLifecycleConfiguration,
|
||||
) -> Result<(), std::io::Error> {
|
||||
crate::storage::ecstore_bucket::lifecycle::bucket_lifecycle_ops::validate_transition_tier(lc).await
|
||||
}
|
||||
|
||||
pub(crate) async fn validate_lifecycle_config(
|
||||
lc: &s3s::dto::BucketLifecycleConfiguration,
|
||||
lock_config: &s3s::dto::ObjectLockConfiguration,
|
||||
) -> Result<(), std::io::Error> {
|
||||
use crate::storage::ecstore_bucket::lifecycle::lifecycle::Lifecycle as _;
|
||||
|
||||
lc.validate(lock_config).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod lifecycle_contract {
|
||||
#[cfg(test)]
|
||||
pub(crate) type IlmAction = crate::storage::ecstore_bucket::lifecycle::lifecycle::IlmAction;
|
||||
pub(crate) type Event = crate::storage::ecstore_bucket::lifecycle::lifecycle::Event;
|
||||
pub(crate) type ObjectOpts = crate::storage::ecstore_bucket::lifecycle::lifecycle::ObjectOpts;
|
||||
pub(crate) type TransitionOptions = crate::storage::ecstore_bucket::lifecycle::lifecycle::TransitionOptions;
|
||||
|
||||
pub(crate) const TRANSITION_COMPLETE: &str =
|
||||
crate::storage::ecstore_bucket::lifecycle::lifecycle::TRANSITION_COMPLETE;
|
||||
|
||||
pub(crate) fn expected_expiry_time(mod_time: time::OffsetDateTime, days: i32) -> time::OffsetDateTime {
|
||||
crate::storage::ecstore_bucket::lifecycle::lifecycle::expected_expiry_time(mod_time, days)
|
||||
}
|
||||
}
|
||||
pub(crate) use lifecycle_contract as lifecycle;
|
||||
|
||||
pub(crate) mod tier_delete_journal {
|
||||
use std::sync::Arc;
|
||||
|
||||
pub(crate) async fn persist_tier_delete_journal_entry(
|
||||
api: Arc<crate::storage::ECStore>,
|
||||
je: &super::tier_sweeper::Jentry,
|
||||
) -> std::io::Result<()> {
|
||||
crate::storage::ecstore_bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry(api, je).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod tier_sweeper {
|
||||
pub(crate) type Jentry = crate::storage::ecstore_bucket::lifecycle::tier_sweeper::Jentry;
|
||||
|
||||
pub(crate) fn transitioned_delete_journal_entry(
|
||||
version_id: Option<uuid::Uuid>,
|
||||
versioned: bool,
|
||||
suspended: bool,
|
||||
transitioned: &rustfs_storage_api::TransitionedObject,
|
||||
) -> Option<Jentry> {
|
||||
crate::storage::ecstore_bucket::lifecycle::tier_sweeper::transitioned_delete_journal_entry(
|
||||
version_id,
|
||||
versioned,
|
||||
suspended,
|
||||
transitioned,
|
||||
)
|
||||
}
|
||||
|
||||
pub(crate) fn transitioned_force_delete_journal_entry(
|
||||
transitioned: &rustfs_storage_api::TransitionedObject,
|
||||
) -> Option<Jentry> {
|
||||
crate::storage::ecstore_bucket::lifecycle::tier_sweeper::transitioned_force_delete_journal_entry(transitioned)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod metadata {
|
||||
pub(crate) const BUCKET_CORS_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_CORS_CONFIG;
|
||||
pub(crate) const BUCKET_LIFECYCLE_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
|
||||
pub(crate) const BUCKET_NOTIFICATION_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_NOTIFICATION_CONFIG;
|
||||
pub(crate) const BUCKET_POLICY_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_POLICY_CONFIG;
|
||||
pub(crate) const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str =
|
||||
crate::storage::ecstore_bucket::metadata::BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG;
|
||||
pub(crate) const BUCKET_REPLICATION_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_REPLICATION_CONFIG;
|
||||
pub(crate) const BUCKET_SSECONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_SSECONFIG;
|
||||
pub(crate) const BUCKET_TAGGING_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_TAGGING_CONFIG;
|
||||
pub(crate) const BUCKET_TARGETS_FILE: &str = crate::storage::ecstore_bucket::metadata::BUCKET_TARGETS_FILE;
|
||||
pub(crate) const BUCKET_VERSIONING_CONFIG: &str = crate::storage::ecstore_bucket::metadata::BUCKET_VERSIONING_CONFIG;
|
||||
#[cfg(test)]
|
||||
pub(crate) const OBJECT_LOCK_CONFIG: &str = crate::storage::ecstore_bucket::metadata::OBJECT_LOCK_CONFIG;
|
||||
}
|
||||
|
||||
pub(crate) mod metadata_sys {
|
||||
use std::sync::Arc;
|
||||
|
||||
use rustfs_policy::policy::BucketPolicy;
|
||||
use s3s::dto::{
|
||||
BucketLifecycleConfiguration, CORSConfiguration, NotificationConfiguration, ObjectLockConfiguration,
|
||||
PublicAccessBlockConfiguration, ServerSideEncryptionConfiguration, Tagging,
|
||||
};
|
||||
use time::OffsetDateTime;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use super::target::BucketTargets;
|
||||
|
||||
pub(crate) type BucketMetadataSys = crate::storage::ecstore_bucket::metadata_sys::BucketMetadataSys;
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn init_bucket_metadata_sys(api: Arc<crate::storage::ECStore>, buckets: Vec<String>) {
|
||||
crate::storage::ecstore_bucket::metadata_sys::init_bucket_metadata_sys(api, buckets).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime, crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::delete(bucket, config_file).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_bucket_policy(
|
||||
bucket: &str,
|
||||
) -> Result<(BucketPolicy, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_bucket_policy(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_bucket_policy_raw(
|
||||
bucket: &str,
|
||||
) -> Result<(String, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_bucket_policy_raw(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_bucket_targets_config(bucket: &str) -> Result<BucketTargets, crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_bucket_targets_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_cors_config(
|
||||
bucket: &str,
|
||||
) -> Result<(CORSConfiguration, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_cors_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) fn get_global_bucket_metadata_sys() -> Option<Arc<RwLock<BucketMetadataSys>>> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_global_bucket_metadata_sys()
|
||||
}
|
||||
|
||||
pub(crate) async fn get_lifecycle_config(
|
||||
bucket: &str,
|
||||
) -> Result<(BucketLifecycleConfiguration, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_lifecycle_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_notification_config(
|
||||
bucket: &str,
|
||||
) -> Result<Option<NotificationConfiguration>, crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_notification_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_object_lock_config(
|
||||
bucket: &str,
|
||||
) -> Result<(ObjectLockConfiguration, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_object_lock_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_public_access_block_config(
|
||||
bucket: &str,
|
||||
) -> Result<(PublicAccessBlockConfiguration, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_public_access_block_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_replication_config(
|
||||
bucket: &str,
|
||||
) -> Result<(s3s::dto::ReplicationConfiguration, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_replication_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_sse_config(
|
||||
bucket: &str,
|
||||
) -> Result<(ServerSideEncryptionConfiguration, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_sse_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn get_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime), crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::get_tagging_config(bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn update(
|
||||
bucket: &str,
|
||||
config_file: &str,
|
||||
data: Vec<u8>,
|
||||
) -> Result<OffsetDateTime, crate::storage::StorageError> {
|
||||
crate::storage::ecstore_bucket::metadata_sys::update(bucket, config_file, data).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod object_lock {
|
||||
pub(crate) mod objectlock {
|
||||
pub(crate) fn get_object_legalhold_meta(
|
||||
meta: &std::collections::HashMap<String, String>,
|
||||
) -> s3s::dto::ObjectLockLegalHold {
|
||||
crate::storage::ecstore_bucket::object_lock::objectlock::get_object_legalhold_meta(meta)
|
||||
}
|
||||
|
||||
pub(crate) fn get_object_retention_meta(
|
||||
meta: &std::collections::HashMap<String, String>,
|
||||
) -> s3s::dto::ObjectLockRetention {
|
||||
crate::storage::ecstore_bucket::object_lock::objectlock::get_object_retention_meta(meta)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod objectlock_sys {
|
||||
pub(crate) type BucketObjectLockSys =
|
||||
crate::storage::ecstore_bucket::object_lock::objectlock_sys::BucketObjectLockSys;
|
||||
pub(crate) type ObjectLockBlockReason =
|
||||
crate::storage::ecstore_bucket::object_lock::objectlock_sys::ObjectLockBlockReason;
|
||||
|
||||
pub(crate) async fn check_object_lock_for_deletion(
|
||||
bucket: &str,
|
||||
obj_info: &crate::storage::StorageObjectInfo,
|
||||
bypass_governance: bool,
|
||||
) -> Option<ObjectLockBlockReason> {
|
||||
crate::storage::ecstore_bucket::object_lock::objectlock_sys::check_object_lock_for_deletion(
|
||||
bucket,
|
||||
obj_info,
|
||||
bypass_governance,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) fn is_retention_active(mode: &str, retain_until_date: Option<&s3s::dto::Date>) -> bool {
|
||||
crate::storage::ecstore_bucket::object_lock::objectlock_sys::is_retention_active(mode, retain_until_date)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod policy_sys {
|
||||
pub(crate) type PolicySys = crate::storage::ecstore_bucket::policy_sys::PolicySys;
|
||||
}
|
||||
|
||||
pub(crate) mod quota {
|
||||
pub(crate) mod checker {
|
||||
pub(crate) type QuotaChecker = crate::storage::ecstore_bucket::quota::checker::QuotaChecker;
|
||||
}
|
||||
|
||||
pub(crate) type QuotaOperation = crate::storage::ecstore_bucket::quota::QuotaOperation;
|
||||
}
|
||||
|
||||
pub(crate) mod replication {
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
pub(crate) type DeletedObjectReplicationInfo = crate::storage::ecstore_bucket::replication::DeletedObjectReplicationInfo;
|
||||
pub(crate) type MustReplicateOptions = crate::storage::ecstore_bucket::replication::MustReplicateOptions;
|
||||
pub(crate) type ObjectOpts = crate::storage::ecstore_bucket::replication::ObjectOpts;
|
||||
pub(crate) type ReplicateDecision = rustfs_filemeta::ReplicateDecision;
|
||||
|
||||
pub(crate) async fn check_replicate_delete(
|
||||
bucket: &str,
|
||||
dobj: &rustfs_storage_api::ObjectToDelete,
|
||||
oi: &crate::storage::StorageObjectInfo,
|
||||
del_opts: &crate::storage::StorageObjectOptions,
|
||||
gerr: Option<String>,
|
||||
) -> ReplicateDecision {
|
||||
crate::storage::ecstore_bucket::replication::check_replicate_delete(bucket, dobj, oi, del_opts, gerr).await
|
||||
}
|
||||
|
||||
pub(crate) fn get_must_replicate_options(
|
||||
user_defined: &HashMap<String, String>,
|
||||
user_tags: String,
|
||||
status: rustfs_filemeta::ReplicationStatusType,
|
||||
op_type: rustfs_filemeta::ReplicationType,
|
||||
opts: crate::storage::StorageObjectOptions,
|
||||
) -> MustReplicateOptions {
|
||||
crate::storage::ecstore_bucket::replication::get_must_replicate_options(
|
||||
user_defined,
|
||||
user_tags,
|
||||
status,
|
||||
op_type,
|
||||
opts,
|
||||
)
|
||||
}
|
||||
|
||||
pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplicateOptions) -> ReplicateDecision {
|
||||
crate::storage::ecstore_bucket::replication::must_replicate(bucket, object, mopts).await
|
||||
}
|
||||
|
||||
pub(crate) async fn schedule_replication(
|
||||
oi: crate::storage::StorageObjectInfo,
|
||||
store: Arc<crate::storage::ECStore>,
|
||||
dsc: ReplicateDecision,
|
||||
op_type: rustfs_filemeta::ReplicationType,
|
||||
) {
|
||||
crate::storage::ecstore_bucket::replication::schedule_replication(oi, store, dsc, op_type).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn schedule_replication_delete(dv: DeletedObjectReplicationInfo) {
|
||||
crate::storage::ecstore_bucket::replication::schedule_replication_delete(dv).await;
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod tagging {
|
||||
pub(crate) fn decode_tags(tags: &str) -> Vec<s3s::dto::Tag> {
|
||||
crate::storage::ecstore_bucket::tagging::decode_tags(tags)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod target {
|
||||
#[cfg(test)]
|
||||
pub(crate) type BucketTarget = crate::storage::ecstore_bucket::target::BucketTarget;
|
||||
pub(crate) type BucketTargetType = crate::storage::ecstore_bucket::target::BucketTargetType;
|
||||
pub(crate) type BucketTargets = crate::storage::ecstore_bucket::target::BucketTargets;
|
||||
}
|
||||
|
||||
pub(crate) mod utils {
|
||||
pub(crate) fn serialize<T: s3s::xml::Serialize>(val: &T) -> s3s::xml::SerResult<Vec<u8>> {
|
||||
crate::storage::ecstore_bucket::utils::serialize(val)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) mod transition_api {
|
||||
pub(crate) type ReadCloser = crate::storage::ecstore_client::transition_api::ReadCloser;
|
||||
pub(crate) type ReaderImpl = crate::storage::ecstore_client::transition_api::ReaderImpl;
|
||||
}
|
||||
|
||||
pub(crate) mod versioning_sys {
|
||||
pub(crate) type BucketVersioningSys = crate::storage::ecstore_bucket::versioning_sys::BucketVersioningSys;
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod concurrency {
|
||||
pub(crate) use crate::storage::concurrency::{
|
||||
ConcurrencyManager, GetObjectGuard, IoQueueStatus, IoStrategy, PutObjectGuard, get_concurrency_aware_buffer_size,
|
||||
@@ -41,6 +636,15 @@ pub(crate) mod ecfs {
|
||||
pub(crate) use crate::storage::ecfs::FS;
|
||||
}
|
||||
|
||||
pub(crate) mod error {
|
||||
pub(crate) use crate::storage::{
|
||||
DiskError, StorageError, is_all_buckets_not_found, is_err_bucket_not_found, is_err_object_not_found,
|
||||
is_err_version_not_found,
|
||||
};
|
||||
|
||||
pub(crate) type Error = StorageError;
|
||||
}
|
||||
|
||||
pub(crate) mod head_prefix {
|
||||
pub(crate) use crate::storage::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children};
|
||||
}
|
||||
@@ -49,6 +653,10 @@ pub(crate) mod helper {
|
||||
pub(crate) use crate::storage::helper::{OperationHelper, spawn_background_with_context};
|
||||
}
|
||||
|
||||
pub(crate) mod object_utils {
|
||||
pub(crate) use crate::storage::to_s3s_etag;
|
||||
}
|
||||
|
||||
pub(crate) mod io {
|
||||
#[cfg(test)]
|
||||
pub(crate) use crate::storage::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader};
|
||||
@@ -84,6 +692,12 @@ pub(crate) mod set_disk {
|
||||
pub(crate) use crate::storage::{get_lock_acquire_timeout, is_valid_storage_class};
|
||||
}
|
||||
|
||||
pub(crate) mod storage_class {
|
||||
pub(crate) use crate::storage::ecstore_config::storageclass::STANDARD;
|
||||
#[cfg(test)]
|
||||
pub(crate) use crate::storage::ecstore_config::storageclass::STANDARD_IA;
|
||||
}
|
||||
|
||||
pub(crate) mod timeout_wrapper {
|
||||
pub(crate) use crate::storage::timeout_wrapper::{GetObjectTimeoutPolicy, RequestTimeoutWrapper};
|
||||
}
|
||||
|
||||
@@ -77,7 +77,9 @@ pub(crate) mod ecstore_capacity {
|
||||
}
|
||||
|
||||
pub(crate) mod ecstore_client {
|
||||
pub(crate) use rustfs_ecstore::api::client::{admin_handler_utils, object_api_utils, transition_api};
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::client::transition_api;
|
||||
pub(crate) use rustfs_ecstore::api::client::{admin_handler_utils, object_api_utils};
|
||||
}
|
||||
|
||||
pub(crate) mod ecstore_compression {
|
||||
@@ -188,7 +190,9 @@ pub(crate) mod ecstore_storage {
|
||||
|
||||
pub(crate) mod ecstore_tier {
|
||||
pub(crate) use rustfs_ecstore::api::tier::tier::TierConfigMgr;
|
||||
pub(crate) use rustfs_ecstore::api::tier::{tier, tier_admin, tier_config, tier_handlers, warm_backend};
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::tier::warm_backend;
|
||||
pub(crate) use rustfs_ecstore::api::tier::{tier, tier_admin, tier_config, tier_handlers};
|
||||
}
|
||||
|
||||
pub(crate) const BUCKET_ACCELERATE_CONFIG: &str = ecstore_bucket::metadata::BUCKET_ACCELERATE_CONFIG;
|
||||
|
||||
@@ -196,7 +196,7 @@ pub fn make_server_for_context(context: Option<Arc<runtime_sources::AppContext>>
|
||||
}
|
||||
|
||||
impl NodeService {
|
||||
fn resolve_object_store(&self) -> Option<Arc<crate::app::ECStore>> {
|
||||
fn resolve_object_store(&self) -> Option<Arc<crate::app::storage_api::ECStore>> {
|
||||
let context = self.context.clone().or_else(runtime_sources::get_global_app_context);
|
||||
runtime_sources::object_store_handle_for_context(context.as_deref())
|
||||
}
|
||||
|
||||
@@ -153,6 +153,7 @@ ECSTORE_COMPAT_PASSTHROUGH_DIFF_FILE="${TMP_DIR}/ecstore_compat_passthrough_diff
|
||||
RUSTFS_WORKLOAD_DIRECT_FOREGROUND_MAPPING_HITS_FILE="${TMP_DIR}/rustfs_workload_direct_foreground_mapping_hits.txt"
|
||||
IAM_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/iam_runtime_source_bypass_hits.txt"
|
||||
RUSTFS_APP_CONTEXT_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_context_runtime_source_bypass_hits.txt"
|
||||
RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_runtime_storage_api_bypass_hits.txt"
|
||||
RUSTFS_STARTUP_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_startup_runtime_source_bypass_hits.txt"
|
||||
RUSTFS_SERVER_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_server_runtime_source_bypass_hits.txt"
|
||||
RUSTFS_STORAGE_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_storage_runtime_source_bypass_hits.txt"
|
||||
@@ -1447,8 +1448,11 @@ fi
|
||||
'(use crate::storage::(access|helper|options|request_context|sse|timeout_wrapper|head_prefix|concurrency|ecfs)|crate::storage::sse::EncryptionKeyKind|use crate::storage::\{|use crate::storage::[A-Z])' \
|
||||
rustfs/src/app/select_object.rs rustfs/src/app/*_usecase.rs || true
|
||||
rg -n --with-filename \
|
||||
'use super::(?:\{[^}]*\b(?:DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class)\b|(?:DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class)\b)' \
|
||||
rustfs/src/app/object_usecase.rs rustfs/src/app/multipart_usecase.rs || true
|
||||
'use super::(?:\{[^}]*\b(?:DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class|StorageError|DiskError|is_all_buckets_not_found|is_err_bucket_not_found|is_err_object_not_found|is_err_version_not_found)\b|(?:object_api_utils::to_s3s_etag|storageclass|StorageError|DiskError|DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class|is_all_buckets_not_found|is_err_bucket_not_found|is_err_object_not_found|is_err_version_not_found)\b)' \
|
||||
rustfs/src/app/bucket_usecase.rs rustfs/src/app/object_usecase.rs rustfs/src/app/multipart_usecase.rs rustfs/src/app/lifecycle_transition_api_test.rs || true
|
||||
rg -n --with-filename \
|
||||
'use super::(?:\{[^}]*\b(?:AppObjectLockConfigExt|AppReplicationConfigExt|AppVersioningConfigExt|predict_lifecycle_expiration|validate_restore_request|bucket_target_sys|lifecycle|metadata|metadata_sys|object_lock|policy_sys|quota|replication|tagging|target|utils|versioning_sys|transition_api|ObjectInfo|ObjectOptions)\b|(?:AppObjectLockConfigExt|AppReplicationConfigExt|AppVersioningConfigExt|predict_lifecycle_expiration|validate_restore_request|bucket_target_sys|lifecycle|metadata|metadata_sys|object_lock|policy_sys|quota|replication|tagging|target|utils|versioning_sys|transition_api|ObjectInfo|ObjectOptions)\b)|super::(?:lifecycle|metadata_sys|object_lock|quota|replication|tagging|target|utils|versioning_sys|transition_api)::|super::super::(?:metadata_sys|lifecycle|target)::' \
|
||||
rustfs/src/app/bucket_usecase.rs rustfs/src/app/object_usecase.rs rustfs/src/app/multipart_usecase.rs rustfs/src/app/lifecycle_transition_api_test.rs rustfs/src/app/capacity_dirty_scope_test.rs rustfs/src/app/context.rs rustfs/src/app/context/handles.rs rustfs/src/app/context/interfaces.rs rustfs/src/app/context/runtime_sources.rs || true
|
||||
}
|
||||
) >"$RUSTFS_APP_USECASE_STORAGE_API_BYPASS_HITS_FILE"
|
||||
|
||||
@@ -1472,6 +1476,30 @@ if [[ -s "$RUSTFS_APP_ADMIN_STORAGE_API_BYPASS_HITS_FILE" ]]; then
|
||||
report_failure "RustFS app/admin completed storage helper consumers must use local storage_api boundaries: $(paste -sd '; ' "$RUSTFS_APP_ADMIN_STORAGE_API_BYPASS_HITS_FILE")"
|
||||
fi
|
||||
|
||||
(
|
||||
cd "$ROOT_DIR"
|
||||
{
|
||||
rg -n --with-filename \
|
||||
'use super::(?:\{[^}]*\b(?:ECStore|EndpointServerPools|Endpoint|Endpoints|PoolEndpoints|StorageClassConfig|TierConfigMgr|DailyAllTierStats|ScannerMetricsReport|ReplicationStats|DynReplicationPool|NotificationSys|BucketBandwidthMonitor|ExpiryState|AppWarmBackend|WarmBackendGetOpts|TierConfig|TierType|PoolStatus|PoolDecommissionInfo|RebalStatus)\b|(?:ECStore|EndpointServerPools|Endpoint|Endpoints|PoolEndpoints|StorageClassConfig|TierConfigMgr|DailyAllTierStats|ScannerMetricsReport|ReplicationStats|DynReplicationPool|NotificationSys|BucketBandwidthMonitor|ExpiryState|AppWarmBackend|WarmBackendGetOpts|TierConfig|TierType|PoolStatus|PoolDecommissionInfo|RebalStatus)\b)' \
|
||||
rustfs/src/app/*.rs rustfs/src/app/context/*.rs \
|
||||
-g '!storage_api.rs' || true
|
||||
rg -n --with-filename \
|
||||
'super::(?:get_server_info|get_total_usable_capacity|get_total_usable_capacity_free|apply_bucket_usage_memory_overlay|load_data_usage_from_backend|record_bucket_object_delete_memory|record_bucket_object_write_memory|remove_bucket_usage_from_backend|get_global_|global_rustfs_port|new_object_layer_fn|set_object_store_resolver|set_global_storage_class|collect_scanner_metrics_report|init_local_disks)|super::super::(?:get_server_info|get_total_usable_capacity|get_total_usable_capacity_free|apply_bucket_usage_memory_overlay|load_data_usage_from_backend|record_bucket_object_delete_memory|record_bucket_object_write_memory|remove_bucket_usage_from_backend|get_global_|global_rustfs_port|new_object_layer_fn|set_object_store_resolver|set_global_storage_class|collect_scanner_metrics_report|init_local_disks)' \
|
||||
rustfs/src/app/*.rs rustfs/src/app/context/*.rs \
|
||||
-g '!storage_api.rs' || true
|
||||
rg -n --with-filename \
|
||||
'crate::app::(?:ECStore|EndpointServerPools|Endpoint|Endpoints|PoolEndpoints|StorageClassConfig|TierConfigMgr|DailyAllTierStats|ScannerMetricsReport|ReplicationStats|DynReplicationPool|NotificationSys|BucketBandwidthMonitor|ExpiryState|get_global_|set_object_store_resolver)' \
|
||||
rustfs/src --glob '*.rs' || true
|
||||
rg -n --with-filename \
|
||||
'ecstore_data_usage|pub\(crate\) async fn load_data_usage_from_backend|super::super::load_data_usage_from_backend|crate::admin::load_data_usage_from_backend' \
|
||||
rustfs/src/admin/mod.rs rustfs/src/admin/handlers/*.rs || true
|
||||
}
|
||||
) >"$RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE"
|
||||
|
||||
if [[ -s "$RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE" ]]; then
|
||||
report_failure "RustFS app/admin runtime and data-usage facades must stay behind local storage_api boundaries: $(paste -sd '; ' "$RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE")"
|
||||
fi
|
||||
|
||||
(
|
||||
cd "$ROOT_DIR"
|
||||
rg -n --with-filename 'crate::app::context::|use crate::app::context|app::context::' rustfs/src/storage --glob '*.rs' |
|
||||
|
||||
Reference in New Issue
Block a user