diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index a935dbcde..da52d11dd 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,9 +5,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-app-runtime-global-facade-cleanup` -- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213/API-214/API-215`. -- Based on: stacked on API-215 while PR #3856 is pending. +- Branch: `overtrue/arch-admin-facade-storage-api-cleanup` +- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213/API-214/API-215/API-216`. +- Based on: API-216 merged; branch replays the admin storage_api cleanup on latest `main`. - PR type for this branch: `consumer-migration` - Runtime behavior changes: none. - Rust code changes: route replication pool, outbound TLS generation, runtime @@ -46,7 +46,10 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block helpers, plus server readiness/audit/event/module-switch runtime source helpers, storage request/RPC/SSE runtime source helpers, and admin handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, app bucket/object/multipart usecase explicit storage imports, app select/bucket/object/multipart explicit S3 DTO plus ECFS owner imports, app-local S3 API response helper imports for bucket/object/multipart usecases, app-local storage helper boundary imports for select/bucket/object/multipart usecases, admin/app test storage helper boundary imports for request context, authorization, object options, and ECFS test harness consumers, app storage IO/compression/set-disk helper boundary imports for object and multipart usecases, app storage error, ETag, and storage-class helper boundary imports for bucket/object/multipart usecases and lifecycle transition tests, app bucket owner facade imports for lifecycle, metadata, object-lock, quota, replication, tagging, target, versioning, and transition test helpers, and app/admin runtime, capacity, data-usage, endpoint, and global facade imports, - through AppContext-first or owner-crate resolver boundaries. + admin root storage facade consumers for config, bucket metadata, replication, + rebalancing, tier, quota, metrics, object zip, site replication, and admin + service config paths, through AppContext-first or owner-crate resolver + boundaries. - CI/script changes: lock completed owner and test/fuzz boundaries against bare/glob imports, scattered raw ECStore facade subpaths, and startup runtime/root-server/table/S3/app shared/app bucket/app ECStore/admin facade @@ -56,8 +59,8 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block event-bridge thin module regressions, plus IAM runtime-source bypasses; accept the reviewed AppContext resolver reverse dependencies in the layer baseline, and block direct admin AppContext resolver consumers outside the - admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, narrow the object-usecase ECFS layer baseline entry to `FS`, reject direct storage S3 API helper imports from app usecase files, reject direct storage helper imports from app select/usecase files, reject completed app/admin storage helper bypasses, reject app usecase bypasses for migrated storage IO/compression/set-disk helpers, reject app usecase/test bypasses for migrated storage error, ETag, and storage-class helpers, reject app root bucket owner facade bypasses from migrated app consumers, and reject app/admin runtime/data-usage root facade regressions. -- Docs changes: record the API-136 through API-216 owner facade and lifecycle + admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, narrow the object-usecase ECFS layer baseline entry to `FS`, reject direct storage S3 API helper imports from app usecase files, reject direct storage helper imports from app select/usecase files, reject completed app/admin storage helper bypasses, reject app usecase bypasses for migrated storage IO/compression/set-disk helpers, reject app usecase/test bypasses for migrated storage error, ETag, and storage-class helpers, reject app root bucket owner facade bypasses from migrated app consumers, reject app/admin runtime/data-usage root facade regressions, and reject admin root storage facade regressions from migrated admin consumers. +- Docs changes: record the API-136 through API-217 owner facade and lifecycle runtime-source cleanup. ## Phase 0 Tasks @@ -5121,14 +5124,33 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block layer guards, diff hygiene, residual migrated facade scan, Rust risk scan, fast PR gate, and full PR gate before PR. +- [x] `API-217` Route admin storage facades through storage_api boundary. + - Do: move admin-facing ECStore bucket/config/rebalance/tier/metrics/error + facades from the admin root module into `admin::storage_api`, then route + admin handlers, router, and config/site-replication services through that + boundary. + - Acceptance: admin root no longer exposes the migrated storage facade + modules, aliases, constants, or wrappers, and migration rules reject + root-level regressions for completed admin storage facade consumers. + - Must preserve: admin config IO/default initialization, bucket metadata and + target management, replication/resync DTO handling, rebalance status + projection, quota usage reads, tier admin contracts, metrics collection, + object-zip error mapping, and site-replication metadata serialization. + - Verification: focused RustFS admin compile/tests, formatting, migration and + layer guards, diff hygiene, residual migrated facade scan, Rust risk scan, + fast PR gate, and full PR gate before PR. + ## Next PRs -1. `consumer-migration`: batch remaining app/admin facade cleanup after API-216. +1. `consumer-migration`: continue larger app/admin/storage facade batches after API-217. ## Pre-Push Review Log | Expert | Status | Notes | |---|---|---| +| Quality/architecture | pass | API-217 moves admin storage facade ownership behind admin storage_api and removes the admin root compatibility surface. | +| Migration preservation | pass | Config IO/defaults, bucket metadata/targets, replication/resync, rebalance, quota, tier, metrics, object zip, and site replication keep the same storage implementations. | +| Testing/verification | pass | Focused RustFS admin compile/tests, formatting, migration/layer guards, residual migrated facade scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. | | Quality/architecture | pass | API-216 removes app root runtime/global facades and routes app/admin data-usage and runtime consumers through local storage_api boundaries. | | Migration preservation | pass | Admin info/capacity, data-usage accounting, AppContext fallback runtime sources, lifecycle test disk initialization, tier warm-backend tests, and scanner metrics keep the same storage implementations. | | Testing/verification | pass | Focused RustFS app compile/tests, formatting, migration/layer guards, residual migrated facade scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. | diff --git a/rustfs/src/admin/handlers/account_info.rs b/rustfs/src/admin/handlers/account_info.rs index 7b50aab3a..be4538fd8 100644 --- a/rustfs/src/admin/handlers/account_info.rs +++ b/rustfs/src/admin/handlers/account_info.rs @@ -12,10 +12,10 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::versioning_sys::BucketVersioningSys; use crate::admin::auth::authenticate_request; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::{resolve_action_credentials, resolve_object_store_handle}; +use crate::admin::storage_api::versioning_sys::BucketVersioningSys; use crate::auth::get_condition_values; use crate::server::{ADMIN_PREFIX, RemoteAddr}; use http::{HeaderMap, HeaderValue}; diff --git a/rustfs/src/admin/handlers/audit_runtime_config.rs b/rustfs/src/admin/handlers/audit_runtime_config.rs index 9fec069cb..0920882fe 100644 --- a/rustfs/src/admin/handlers/audit_runtime_config.rs +++ b/rustfs/src/admin/handlers/audit_runtime_config.rs @@ -12,9 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::{read_admin_config_without_migrate, save_admin_server_config}; use crate::admin::handlers::target_descriptor::AdminTargetSpec; use crate::admin::runtime_sources::resolve_object_store_handle; +use crate::admin::storage_api::{read_admin_config_without_migrate, save_admin_server_config}; use rustfs_audit::{audit_system, start_audit_system as start_global_audit_system, system::AuditSystemState}; use rustfs_config::DEFAULT_DELIMITER; use rustfs_config::server_config::Config; diff --git a/rustfs/src/admin/handlers/bucket_meta.rs b/rustfs/src/admin/handlers/bucket_meta.rs index a45095d3e..f3c9d7aa9 100644 --- a/rustfs/src/admin/handlers/bucket_meta.rs +++ b/rustfs/src/admin/handlers/bucket_meta.rs @@ -12,8 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::ecstore_utils::{deserialize, serialize}; -use super::super::{ +use crate::admin::storage_api::ecstore_utils::{deserialize, serialize}; +use crate::admin::storage_api::{ StorageError, metadata::{ BUCKET_LIFECYCLE_CONFIG, BUCKET_NOTIFICATION_CONFIG, BUCKET_POLICY_CONFIG, BUCKET_QUOTA_CONFIG_FILE, diff --git a/rustfs/src/admin/handlers/cluster_snapshot.rs b/rustfs/src/admin/handlers/cluster_snapshot.rs index cd6974934..7771bdce4 100644 --- a/rustfs/src/admin/handlers/cluster_snapshot.rs +++ b/rustfs/src/admin/handlers/cluster_snapshot.rs @@ -14,12 +14,12 @@ use crate::admin::{ auth::validate_admin_request, - ecstore_cluster::{ + router::{AdminOperation, Operation, S3Router}, + storage_api::ecstore_cluster::{ ClusterDriveMembership, ClusterEndpointType, ClusterLocalNodeStorage, ClusterLocalNodeStorageSnapshot, ClusterMembershipSnapshot, ClusterNodeMembership, ClusterPeerHealth, ClusterPeerHealthSnapshot, ClusterPoolState, ClusterPoolStateSnapshot, }, - router::{AdminOperation, Operation, S3Router}, system, }; use crate::app::admin_usecase::DefaultAdminUsecase; @@ -590,7 +590,7 @@ fn summarize_named_capability_statuses( #[cfg(test)] mod tests { use super::{ClusterSnapshotResponse, ClusterSnapshotSummary, ClusterSnapshotView}; - use crate::admin::ecstore_cluster::{ + use crate::admin::storage_api::ecstore_cluster::{ ClusterDriveMembership, ClusterEndpointType, ClusterLocalNodeStorage, ClusterLocalNodeStorageSnapshot, ClusterMembershipSnapshot, ClusterNodeMembership, ClusterPeerHealth, ClusterPeerHealthSnapshot, ClusterPoolState, ClusterPoolStateSnapshot, diff --git a/rustfs/src/admin/handlers/config_admin.rs b/rustfs/src/admin/handlers/config_admin.rs index 7be19b9f0..751ce1a0c 100644 --- a/rustfs/src/admin/handlers/config_admin.rs +++ b/rustfs/src/admin/handlers/config_admin.rs @@ -12,11 +12,6 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::storageclass::{INLINE_BLOCK_ENV, OPTIMIZE_ENV, RRS_ENV, STANDARD_ENV}; -use super::super::{ - RUSTFS_META_BUCKET, STORAGE_CLASS_SUB_SYS, delete_admin_config, read_admin_config, read_admin_config_without_migrate, - save_admin_config, save_admin_server_config, -}; use crate::admin::auth::validate_admin_request; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::{publish_server_config, resolve_object_store_handle, resolve_server_config}; @@ -24,6 +19,11 @@ use crate::admin::service::config::{ apply_dynamic_config_for_subsystem, is_dynamic_config_subsystem, signal_config_snapshot_reload, signal_dynamic_config_reload, validate_server_config, }; +use crate::admin::storage_api::storageclass::{INLINE_BLOCK_ENV, OPTIMIZE_ENV, RRS_ENV, STANDARD_ENV}; +use crate::admin::storage_api::{ + RUSTFS_META_BUCKET, STORAGE_CLASS_SUB_SYS, delete_admin_config, read_admin_config, read_admin_config_without_migrate, + save_admin_config, save_admin_server_config, +}; use crate::admin::utils::{encode_compatible_admin_payload, is_compat_admin_request, read_compatible_admin_body}; use crate::auth::{check_key_valid, get_session_token}; use crate::error::ApiError; @@ -708,7 +708,7 @@ fn success_response(config_applied: bool) -> S3Result S3Result> { +fn object_store() -> S3Result> { resolve_object_store_handle().ok_or_else(|| s3_error!(InternalError, "server storage not initialized")) } @@ -753,7 +753,7 @@ fn config_update_sub_system(directives: &[ConfigDirective]) -> S3Result S3Result<()> { if DEFAULT_KVS.get().is_none() { - super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); } let Some(defaults) = DEFAULT_KVS.get() else { return Err(s3_error!(InternalError, "config defaults are not initialized")); @@ -1407,7 +1407,7 @@ fn env_help_key(sub_system: &str, key: &str) -> String { fn default_help_postfix(sub_system: &str, key: &str) -> String { if DEFAULT_KVS.get().is_none() { - super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); } DEFAULT_KVS @@ -1894,7 +1894,7 @@ mod tests { #[test] fn full_config_export_can_be_reapplied() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); let mut original = ServerConfig::new(); apply_set_directives( &mut original, @@ -1941,7 +1941,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#, #[test] fn build_help_response_appends_default_value_postfix() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); let response = build_help_response(Some("identity_openid"), Some("scopes"), false).expect("help response"); assert_eq!(response.keys_help.len(), 2); @@ -2054,7 +2054,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#, #[test] fn render_selected_config_includes_env_override_lines() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); temp_env::with_vars( [ ("RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY", Some("http://env.example")), @@ -2090,7 +2090,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#, #[test] fn render_selected_config_lists_env_only_targets() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); temp_env::with_vars([("RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY", Some("http://env.example"))], || { let config = ServerConfig::new(); let rendered = String::from_utf8( @@ -2113,7 +2113,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#, #[test] fn render_selected_config_supports_specific_env_only_target_queries() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); temp_env::with_vars([("RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY", Some("http://env.example"))], || { let config = ServerConfig::new(); let rendered = String::from_utf8( @@ -2136,7 +2136,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#, #[test] fn render_selected_config_orders_default_before_named_targets() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); temp_env::with_vars([("RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_ALPHA", Some("http://alpha.example"))], || { let mut config = ServerConfig::new(); apply_set_directives( @@ -2317,7 +2317,7 @@ identity_openid client_id="existing-client""#, #[test] fn storage_class_get_target_none_matches_full_export() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); let mut config = ServerConfig::new(); apply_set_directives( &mut config, diff --git a/rustfs/src/admin/handlers/heal.rs b/rustfs/src/admin/handlers/heal.rs index 072ee615c..03ccdd532 100644 --- a/rustfs/src/admin/handlers/heal.rs +++ b/rustfs/src/admin/handlers/heal.rs @@ -12,11 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::ecstore_utils::is_valid_object_prefix; -use super::super::is_reserved_or_invalid_bucket; use crate::admin::auth::{authenticate_request, validate_admin_request}; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::resolve_object_store_handle; +use crate::admin::storage_api::ecstore_utils::is_valid_object_prefix; +use crate::admin::storage_api::is_reserved_or_invalid_bucket; use crate::admin::storage_api::spawn_traced; use crate::server::ADMIN_PREFIX; use crate::server::RemoteAddr; @@ -386,10 +386,10 @@ fn should_handle_root_heal_directly(_hip: &HealInitParams) -> bool { false } -fn map_root_heal_status(heal_err: Option) -> S3Result<()> { +fn map_root_heal_status(heal_err: Option) -> S3Result<()> { match heal_err { None => Ok(()), - Some(super::super::StorageError::NoHealRequired) => { + Some(crate::admin::storage_api::StorageError::NoHealRequired) => { info!( event = EVENT_ADMIN_RESPONSE_EMITTED, component = LOG_COMPONENT_ADMIN_API, @@ -743,7 +743,6 @@ impl Operation for BackgroundHealStatusHandler { #[cfg(test)] mod tests { - use super::super::super::StorageError; use super::extract_heal_init_params; use super::{ HealInitParams, HealResp, build_heal_channel_request, encode_background_heal_status, encode_heal_start_success, @@ -751,6 +750,7 @@ mod tests { json_response, map_heal_response, map_root_heal_status, should_handle_root_heal_directly, validate_heal_request_mode, validate_heal_target, }; + use crate::admin::storage_api::StorageError; use bytes::Bytes; use http::StatusCode; use http::Uri; diff --git a/rustfs/src/admin/handlers/kms_dynamic.rs b/rustfs/src/admin/handlers/kms_dynamic.rs index 93862ad41..9faef0e5d 100644 --- a/rustfs/src/admin/handlers/kms_dynamic.rs +++ b/rustfs/src/admin/handlers/kms_dynamic.rs @@ -14,12 +14,12 @@ //! KMS dynamic configuration admin API handlers -use super::super::{read_admin_config, save_admin_config}; use crate::admin::auth::validate_admin_request; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::{ resolve_kms_runtime_service_manager, resolve_object_store_handle, resolve_or_init_kms_runtime_service_manager, }; +use crate::admin::storage_api::{read_admin_config, save_admin_config}; use crate::auth::{check_key_valid, get_session_token}; use crate::server::{ADMIN_PREFIX, RemoteAddr}; use hyper::{Method, StatusCode}; diff --git a/rustfs/src/admin/handlers/metrics.rs b/rustfs/src/admin/handlers/metrics.rs index 7bc818945..1f91a2e36 100644 --- a/rustfs/src/admin/handlers/metrics.rs +++ b/rustfs/src/admin/handlers/metrics.rs @@ -18,10 +18,10 @@ //! keeping the response format explicitly NDJSON. It is not a Prometheus text //! exposition endpoint. -use super::super::{CollectMetricsOpts, MetricType, collect_local_metrics}; use crate::admin::auth::validate_admin_request; use crate::admin::router::Operation; use crate::admin::storage_api::spawn_traced; +use crate::admin::storage_api::{CollectMetricsOpts, MetricType, collect_local_metrics}; use crate::auth::{check_key_valid, get_session_token}; use crate::server::RemoteAddr; use bytes::Bytes; diff --git a/rustfs/src/admin/handlers/object_zip_download.rs b/rustfs/src/admin/handlers/object_zip_download.rs index 756d6e3f8..7c1166d1d 100644 --- a/rustfs/src/admin/handlers/object_zip_download.rs +++ b/rustfs/src/admin/handlers/object_zip_download.rs @@ -645,7 +645,7 @@ async fn preflight_zip_items(request: &CreateObjectZipDownloadRequest, items: &[ Ok(()) } -fn storage_error_to_s3(err: super::super::Error) -> s3s::S3Error { +fn storage_error_to_s3(err: crate::admin::storage_api::Error) -> s3s::S3Error { ApiError::from(err).into() } diff --git a/rustfs/src/admin/handlers/oidc.rs b/rustfs/src/admin/handlers/oidc.rs index ac7bcb9bf..ec03e0ba7 100644 --- a/rustfs/src/admin/handlers/oidc.rs +++ b/rustfs/src/admin/handlers/oidc.rs @@ -12,11 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::{read_admin_config_without_migrate, save_admin_server_config}; use super::sts::create_oidc_sts_credentials; use crate::admin::auth::validate_admin_request; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::{resolve_object_store_handle, resolve_oidc_handle, resolve_server_config}; +use crate::admin::storage_api::{read_admin_config_without_migrate, save_admin_server_config}; use crate::auth::{check_key_valid, get_session_token}; use crate::server::{ADMIN_PREFIX, MINIO_ADMIN_PREFIX, RemoteAddr}; use http::StatusCode; diff --git a/rustfs/src/admin/handlers/pools.rs b/rustfs/src/admin/handlers/pools.rs index 1b6a598d5..b85dcd36c 100644 --- a/rustfs/src/admin/handlers/pools.rs +++ b/rustfs/src/admin/handlers/pools.rs @@ -27,9 +27,9 @@ use tracing::{error, info, warn}; use crate::{ admin::runtime_sources::{resolve_endpoints_handle, resolve_notification_system, resolve_object_store_handle}, admin::{ - EndpointServerPools, PeerRestClient, auth::validate_admin_request, router::{AdminOperation, Operation, S3Router}, + storage_api::{EndpointServerPools, PeerRestClient}, }, app::admin_usecase::{DefaultAdminUsecase, QueryPoolStatusRequest}, auth::{check_key_valid, get_session_token}, @@ -211,7 +211,7 @@ macro_rules! log_pool_response_emitted { }; } -fn endpoints_from_context() -> Option { +fn endpoints_from_context() -> Option { resolve_endpoints_handle() } @@ -1078,7 +1078,7 @@ mod pools_handler_tests { pool_admin_pool_parse_error_with_audit, pool_admin_query_parse_error, pool_admin_query_parse_error_with_audit, validate_pool_mutation_leader, validate_start_decommission_guards, }; - use crate::admin::{Endpoint, EndpointServerPools, Endpoints, PoolEndpoints}; + use crate::admin::storage_api::{Endpoint, EndpointServerPools, Endpoints, PoolEndpoints}; fn test_pool_endpoints(is_local: bool) -> EndpointServerPools { let mut endpoint = Endpoint::try_from("http://127.0.0.1:9000/disk").expect("test endpoint should parse"); diff --git a/rustfs/src/admin/handlers/quota.rs b/rustfs/src/admin/handlers/quota.rs index 12f3471ed..c5f81b938 100644 --- a/rustfs/src/admin/handlers/quota.rs +++ b/rustfs/src/admin/handlers/quota.rs @@ -14,12 +14,12 @@ //! Quota admin handlers for HTTP API -use super::super::metadata_sys::BucketMetadataSys; -use super::super::quota::checker::QuotaChecker; -use super::super::quota::{BucketQuota, QuotaError, QuotaOperation}; use crate::admin::auth::{validate_admin_request, validate_admin_request_with_bucket}; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::{resolve_bucket_metadata_handle, resolve_object_store_handle}; +use crate::admin::storage_api::metadata_sys::BucketMetadataSys; +use crate::admin::storage_api::quota::checker::QuotaChecker; +use crate::admin::storage_api::quota::{BucketQuota, QuotaError, QuotaOperation}; use crate::auth::{check_key_valid, get_session_token}; use crate::server::ADMIN_PREFIX; use hyper::{Method, StatusCode}; @@ -175,7 +175,7 @@ async fn current_usage_from_context(bucket: &str) -> u64 { return 0; }; - match super::super::storage_api::data_usage::load_data_usage_from_backend(store).await { + match crate::admin::storage_api::data_usage::load_data_usage_from_backend(store).await { Ok(data_usage_info) => data_usage_info .buckets_usage .get(bucket) diff --git a/rustfs/src/admin/handlers/rebalance.rs b/rustfs/src/admin/handlers/rebalance.rs index 0dfe99c01..0484da879 100644 --- a/rustfs/src/admin/handlers/rebalance.rs +++ b/rustfs/src/admin/handlers/rebalance.rs @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::{ +use crate::admin::storage_api::{ DiskStat, ECStore, NotificationSys, RebalSaveOpt, RebalanceCleanupWarnings, RebalanceMeta, RebalanceStopPropagationRecord, StorageError, decode_rebalance_stop_propagation_record, }; @@ -296,7 +296,7 @@ fn build_rebalance_pool_progress( now: OffsetDateTime, stop_time: Option, percent_free_goal: f64, - ps: &super::super::RebalanceStats, + ps: &crate::admin::storage_api::RebalanceStats, ) -> Option { let total_bytes_to_rebal = ps.init_capacity as f64 * percent_free_goal - ps.init_free_space as f64; let terminal_time = ps.info.end_time.or(stop_time); @@ -339,7 +339,7 @@ fn build_rebalance_pool_statuses( now: OffsetDateTime, stop_time: Option, percent_free_goal: f64, - pool_stats: &[super::super::RebalanceStats], + pool_stats: &[crate::admin::storage_api::RebalanceStats], disk_stats: &[DiskStat], ) -> Vec { pool_stats @@ -895,10 +895,6 @@ mod offsetdatetime_rfc3339 { #[cfg(test)] mod rebalance_handler_tests { - use super::super::super::{ - DiskStat, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta, - RebalanceStats, RebalanceStopPropagationRecord, encode_rebalance_stop_propagation_record, - }; use super::build_rebalance_pool_progress; use super::calculate_rebalance_progress; use super::{ @@ -907,6 +903,10 @@ mod rebalance_handler_tests { rebalance_pool_used, rebalance_query_present, rebalance_remaining_buckets, rebalance_rollback_failure_message, rebalance_rollback_stop_failure_message, rebalance_start_rollback_error, rebalance_used_pct, rollback_result_label, }; + use crate::admin::storage_api::{ + DiskStat, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta, + RebalanceStats, RebalanceStopPropagationRecord, encode_rebalance_stop_propagation_record, + }; use time::OffsetDateTime; #[test] diff --git a/rustfs/src/admin/handlers/replication.rs b/rustfs/src/admin/handlers/replication.rs index 354fa383f..53c5a8fd0 100644 --- a/rustfs/src/admin/handlers/replication.rs +++ b/rustfs/src/admin/handlers/replication.rs @@ -12,17 +12,17 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::StorageError; -use super::super::bucket_target_sys::{BucketTargetError, BucketTargetSys}; -use super::super::metadata::BUCKET_TARGETS_FILE; -use super::super::metadata_sys; -use super::super::metadata_sys::get_replication_config; -use super::super::replication::BucketStats; -use super::super::target::BucketTarget; use crate::admin::auth::validate_admin_request; use crate::admin::handlers::site_replication::site_replication_peer_deployment_id_for_endpoint; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::{resolve_object_store_handle, resolve_replication_stats_handle, resolve_runtime_port}; +use crate::admin::storage_api::StorageError; +use crate::admin::storage_api::bucket_target_sys::{BucketTargetError, BucketTargetSys}; +use crate::admin::storage_api::metadata::BUCKET_TARGETS_FILE; +use crate::admin::storage_api::metadata_sys; +use crate::admin::storage_api::metadata_sys::get_replication_config; +use crate::admin::storage_api::replication::BucketStats; +use crate::admin::storage_api::target::BucketTarget; use crate::admin::utils::read_compatible_admin_body; use crate::auth::{check_key_valid, get_session_token}; use crate::error::ApiError; @@ -448,7 +448,7 @@ impl Operation for RemoveRemoteTargetHandler { #[cfg(test)] mod tests { use super::{extract_query_params, validate_remote_target_tls_settings}; - use crate::admin::target::BucketTarget; + use crate::admin::storage_api::target::BucketTarget; use http::Uri; #[test] diff --git a/rustfs/src/admin/handlers/site_replication.rs b/rustfs/src/admin/handlers/site_replication.rs index 564838053..0da7461a9 100644 --- a/rustfs/src/admin/handlers/site_replication.rs +++ b/rustfs/src/admin/handlers/site_replication.rs @@ -12,18 +12,6 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::Error as StorageError; -use super::super::bucket_target_sys::BucketTargetSys; -use super::super::ecstore_utils::{deserialize, serialize}; -use super::super::metadata::{ - BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_POLICY_CONFIG, BUCKET_QUOTA_CONFIG_FILE, BUCKET_REPLICATION_CONFIG, - BUCKET_SSECONFIG, BUCKET_TAGGING_CONFIG, BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG, OBJECT_LOCK_CONFIG, -}; -use super::super::metadata_sys; -use super::super::replication::ResyncOpts; -use super::super::target::{ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials}; -use super::super::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _}; -use super::super::{delete_admin_config, read_admin_config, save_admin_config}; use crate::admin::auth::validate_admin_request; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::{ @@ -35,6 +23,18 @@ use crate::admin::site_replication_identity::{ canonical_endpoint, deployment_id_for_endpoint, normalize_peer_map_by_identity_with, same_identity_endpoint, site_identity_key, }; +use crate::admin::storage_api::Error as StorageError; +use crate::admin::storage_api::bucket_target_sys::BucketTargetSys; +use crate::admin::storage_api::ecstore_utils::{deserialize, serialize}; +use crate::admin::storage_api::metadata::{ + BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_POLICY_CONFIG, BUCKET_QUOTA_CONFIG_FILE, BUCKET_REPLICATION_CONFIG, + BUCKET_SSECONFIG, BUCKET_TAGGING_CONFIG, BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG, OBJECT_LOCK_CONFIG, +}; +use crate::admin::storage_api::metadata_sys; +use crate::admin::storage_api::replication::ResyncOpts; +use crate::admin::storage_api::target::{ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials}; +use crate::admin::storage_api::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _}; +use crate::admin::storage_api::{delete_admin_config, read_admin_config, save_admin_config}; use crate::admin::utils::{encode_compatible_admin_payload, read_compatible_admin_body}; use crate::auth::{check_key_valid, get_session_token}; use crate::config::get_config_snapshot; @@ -652,7 +652,7 @@ async fn site_replication_peer_client() -> S3Result { built } -fn runtime_tls_enabled_with(endpoints: Option<&super::super::EndpointServerPools>) -> bool { +fn runtime_tls_enabled_with(endpoints: Option<&crate::admin::storage_api::EndpointServerPools>) -> bool { if !rustfs_utils::get_env_str(ENV_RUSTFS_TLS_PATH, DEFAULT_RUSTFS_TLS_PATH).is_empty() { return true; } @@ -3357,7 +3357,10 @@ fn is_stale_update(local_updated_at: OffsetDateTime, incoming_updated_at: Option incoming_updated_at.is_some_and(|incoming_updated_at| incoming_updated_at < local_updated_at) } -fn bucket_meta_local_updated_at(bucket_meta: &super::super::metadata::BucketMetadata, config_file: &str) -> OffsetDateTime { +fn bucket_meta_local_updated_at( + bucket_meta: &crate::admin::storage_api::metadata::BucketMetadata, + config_file: &str, +) -> OffsetDateTime { match config_file { BUCKET_POLICY_CONFIG => bucket_meta.policy_config_updated_at, BUCKET_TAGGING_CONFIG => bucket_meta.tagging_config_updated_at, @@ -4600,10 +4603,10 @@ impl Operation for SRRotateServiceAccountHandler { #[cfg(test)] mod tests { - use super::super::super::Endpoint; - use super::super::super::{EndpointServerPools, Endpoints, PoolEndpoints}; use super::*; use crate::admin::runtime_sources::{resolve_outbound_tls_generation, set_test_outbound_tls_generation}; + use crate::admin::storage_api::Endpoint; + use crate::admin::storage_api::{EndpointServerPools, Endpoints, PoolEndpoints}; use http::{HeaderMap, HeaderValue, Uri}; use rustfs_policy::policy::action::S3Action; use serial_test::serial; diff --git a/rustfs/src/admin/handlers/sts.rs b/rustfs/src/admin/handlers/sts.rs index eea179f54..30e92633c 100644 --- a/rustfs/src/admin/handlers/sts.rs +++ b/rustfs/src/admin/handlers/sts.rs @@ -12,8 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::ecstore_utils::serialize; use super::is_admin::IsAdminHandler; +use crate::admin::storage_api::ecstore_utils::serialize; use crate::{ admin::runtime_sources::{resolve_action_credentials, resolve_oidc_handle, resolve_token_signing_key}, admin::{ diff --git a/rustfs/src/admin/handlers/table_catalog.rs b/rustfs/src/admin/handlers/table_catalog.rs index 6453d275c..dd3d03bd6 100644 --- a/rustfs/src/admin/handlers/table_catalog.rs +++ b/rustfs/src/admin/handlers/table_catalog.rs @@ -12,8 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::{ECStore, metadata::table_catalog_path_hash, metadata_sys}; use crate::admin::runtime_sources::{resolve_object_store_handle, resolve_token_signing_key}; +use crate::admin::storage_api::{ECStore, metadata::table_catalog_path_hash, metadata_sys}; use crate::admin::{ auth::{AdminResourceScope, validate_admin_request, validate_admin_request_with_bucket_object}, router::{AdminOperation, Operation, S3Router}, diff --git a/rustfs/src/admin/handlers/tier.rs b/rustfs/src/admin/handlers/tier.rs index c3c9ef8dc..ac33a941f 100644 --- a/rustfs/src/admin/handlers/tier.rs +++ b/rustfs/src/admin/handlers/tier.rs @@ -13,7 +13,7 @@ // limitations under the License. #![allow(unused_variables, unused_mut, unused_must_use)] -use super::super::{ +use crate::admin::storage_api::{ AdminError, DailyAllTierStats, ERR_TIER_ALREADY_EXISTS, ERR_TIER_BACKEND_IN_USE, ERR_TIER_BACKEND_NOT_EMPTY, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_MISSING_CREDENTIALS, ERR_TIER_NAME_NOT_UPPERCASE, ERR_TIER_NOT_FOUND, TierConfig, TierCreds, TierType, storageclass, @@ -931,8 +931,8 @@ impl Operation for ClearTier { #[cfg(test)] mod tests { - use super::super::super::lifecycle::tier_last_day_stats::LastDayTierStats; use super::*; + use crate::admin::storage_api::lifecycle::tier_last_day_stats::LastDayTierStats; use http::Uri; use matchit::Router; diff --git a/rustfs/src/admin/handlers/trace.rs b/rustfs/src/admin/handlers/trace.rs index 45fabd80e..db0a8ae9e 100644 --- a/rustfs/src/admin/handlers/trace.rs +++ b/rustfs/src/admin/handlers/trace.rs @@ -12,9 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::PeerRestClient; use crate::admin::router::Operation; use crate::admin::runtime_sources::resolve_endpoints_handle; +use crate::admin::storage_api::PeerRestClient; use http::StatusCode; use hyper::Uri; use matchit::Params; diff --git a/rustfs/src/admin/mod.rs b/rustfs/src/admin/mod.rs index 21c396dde..f8126222d 100644 --- a/rustfs/src/admin/mod.rs +++ b/rustfs/src/admin/mod.rs @@ -88,413 +88,3 @@ fn register_admin_routes(r: &mut S3Router) -> std::io::Result<() Ok(()) } - -// Copyright 2024 RustFS Team -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -use std::ops::Deref; -use std::sync::Arc; - -mod ecstore_bucket { - pub(crate) use crate::storage::ecstore_bucket::{ - bandwidth, bucket_target_sys, lifecycle, metadata, metadata_sys, quota, replication, target, utils, versioning, - versioning_sys, - }; -} - -mod ecstore_capacity { - pub(crate) use crate::storage::ecstore_capacity::is_reserved_or_invalid_bucket; -} - -mod ecstore_client { - pub(crate) use crate::storage::ecstore_client::admin_handler_utils; -} - -mod ecstore_cluster { - pub(crate) use crate::storage::ecstore_cluster::{ - ClusterDriveMembership, ClusterEndpointType, ClusterLocalNodeStorage, ClusterLocalNodeStorageSnapshot, - ClusterMembershipSnapshot, ClusterNodeMembership, ClusterPeerHealth, ClusterPeerHealthSnapshot, ClusterPoolState, - ClusterPoolStateSnapshot, - }; -} - -mod ecstore_config { - pub(crate) use crate::storage::ecstore_config::{com, init, storageclass}; -} - -#[allow(unused_imports)] -mod ecstore_disk { - pub(crate) use crate::storage::ecstore_disk::{RUSTFS_META_BUCKET, endpoint}; -} - -mod ecstore_error { - pub(crate) use crate::storage::ecstore_error::StorageError; -} - -#[allow(unused_imports)] -mod ecstore_layout { - pub(crate) use crate::storage::ecstore_layout::{EndpointServerPools, Endpoints, PoolEndpoints}; -} - -mod ecstore_metrics { - pub(crate) use crate::storage::ecstore_metrics::{CollectMetricsOpts, MetricType, collect_local_metrics}; -} - -mod ecstore_notification { - pub(crate) use crate::storage::ecstore_notification::NotificationSys; -} - -#[allow(unused_imports)] -mod ecstore_rebalance { - pub(crate) use crate::storage::ecstore_rebalance::{ - DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, - RebalanceMeta, RebalanceStats, RebalanceStopPropagationRecord, decode_rebalance_stop_propagation_record, - encode_rebalance_stop_propagation_record, - }; -} - -mod ecstore_rpc { - pub(crate) use crate::storage::ecstore_rpc::PeerRestClient; -} - -mod ecstore_storage { - pub(crate) use crate::storage::ecstore_storage::ECStore; -} - -mod ecstore_tier { - pub(crate) use crate::storage::ecstore_tier::{tier, tier_admin, tier_config, tier_handlers}; -} - -pub(crate) const RUSTFS_META_BUCKET: &str = ecstore_disk::RUSTFS_META_BUCKET; -pub(crate) const STORAGE_CLASS_SUB_SYS: &str = ecstore_config::com::STORAGE_CLASS_SUB_SYS; - -pub(crate) type AdminError = ecstore_client::admin_handler_utils::AdminError; -pub(crate) type CollectMetricsOpts = ecstore_metrics::CollectMetricsOpts; -pub(crate) type DiskStat = ecstore_rebalance::DiskStat; -pub(crate) type ECStore = ecstore_storage::ECStore; -pub(crate) type EndpointServerPools = ecstore_layout::EndpointServerPools; -pub(crate) type MetricType = ecstore_metrics::MetricType; -pub(crate) type NotificationSys = ecstore_notification::NotificationSys; -pub(crate) type PeerRestClient = ecstore_rpc::PeerRestClient; -pub(crate) type RebalSaveOpt = ecstore_rebalance::RebalSaveOpt; -pub(crate) type RebalanceCleanupWarnings = ecstore_rebalance::RebalanceCleanupWarnings; -pub(crate) type RebalanceMeta = ecstore_rebalance::RebalanceMeta; -pub(crate) type RebalanceStats = ecstore_rebalance::RebalanceStats; -pub(crate) type RebalanceStopPropagationRecord = ecstore_rebalance::RebalanceStopPropagationRecord; -pub(crate) type StorageError = ecstore_error::StorageError; -pub(crate) type Error = StorageError; -pub(crate) type Result = core::result::Result; -pub(crate) type TierConfig = ecstore_tier::tier_config::TierConfig; -pub(crate) type TierCreds = ecstore_tier::tier_admin::TierCreds; -pub(crate) type TierType = ecstore_tier::tier_config::TierType; - -#[cfg(test)] -pub(crate) type Endpoint = ecstore_disk::endpoint::Endpoint; -#[cfg(test)] -pub(crate) type Endpoints = ecstore_layout::Endpoints; -#[cfg(test)] -pub(crate) type PoolEndpoints = ecstore_layout::PoolEndpoints; -#[cfg(test)] -pub(crate) type RebalStatus = ecstore_rebalance::RebalStatus; -#[cfg(test)] -pub(crate) type RebalanceCleanupWarningEntry = ecstore_rebalance::RebalanceCleanupWarningEntry; -#[cfg(test)] -pub(crate) type RebalanceInfo = ecstore_rebalance::RebalanceInfo; - -pub(crate) fn decode_rebalance_stop_propagation_record(message: &str) -> Option { - ecstore_rebalance::decode_rebalance_stop_propagation_record(message) -} - -#[cfg(test)] -pub(crate) fn encode_rebalance_stop_propagation_record(record: &RebalanceStopPropagationRecord) -> String { - ecstore_rebalance::encode_rebalance_stop_propagation_record(record) -} - -pub(crate) trait AdminReplicationConfigExt { - fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec; - fn has_existing_object_replication(&self, arn: &str) -> (bool, bool); -} - -impl AdminReplicationConfigExt for s3s::dto::ReplicationConfiguration { - fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec { - ::filter_target_arns( - self, obj, - ) - } - - fn has_existing_object_replication(&self, arn: &str) -> (bool, bool) { - ::has_existing_object_replication( - self, arn, - ) - } -} - -pub(crate) trait AdminVersioningConfigExt { - fn enabled(&self) -> bool; -} - -impl AdminVersioningConfigExt for s3s::dto::VersioningConfiguration { - fn enabled(&self) -> bool { - ::enabled(self) - } -} - -pub(crate) mod bandwidth { - pub(crate) mod monitor { - pub(crate) type BandwidthDetails = super::super::ecstore_bucket::bandwidth::monitor::BandwidthDetails; - } -} - -pub(crate) mod bucket_target_sys { - pub(crate) type AdvancedPutOptions = super::ecstore_bucket::bucket_target_sys::AdvancedPutOptions; - pub(crate) type BucketTargetError = super::ecstore_bucket::bucket_target_sys::BucketTargetError; - pub(crate) type BucketTargetSys = super::ecstore_bucket::bucket_target_sys::BucketTargetSys; - pub(crate) type PutObjectOptions = super::ecstore_bucket::bucket_target_sys::PutObjectOptions; - pub(crate) type RemoveObjectOptions = super::ecstore_bucket::bucket_target_sys::RemoveObjectOptions; - pub(crate) type S3ClientError = super::ecstore_bucket::bucket_target_sys::S3ClientError; - pub(crate) type TargetClient = super::ecstore_bucket::bucket_target_sys::TargetClient; -} - -pub(crate) mod lifecycle { - pub(crate) mod tier_last_day_stats { - #[cfg(test)] - pub(crate) type LastDayTierStats = super::super::ecstore_bucket::lifecycle::tier_last_day_stats::LastDayTierStats; - } -} - -pub(crate) mod metadata { - pub(crate) const BUCKET_CORS_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_CORS_CONFIG; - pub(crate) const BUCKET_LIFECYCLE_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_LIFECYCLE_CONFIG; - pub(crate) const BUCKET_NOTIFICATION_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_NOTIFICATION_CONFIG; - pub(crate) const BUCKET_POLICY_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_POLICY_CONFIG; - pub(crate) const BUCKET_QUOTA_CONFIG_FILE: &str = super::ecstore_bucket::metadata::BUCKET_QUOTA_CONFIG_FILE; - pub(crate) const BUCKET_REPLICATION_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_REPLICATION_CONFIG; - pub(crate) const BUCKET_SSECONFIG: &str = super::ecstore_bucket::metadata::BUCKET_SSECONFIG; - pub(crate) const BUCKET_TAGGING_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_TAGGING_CONFIG; - pub(crate) const BUCKET_TARGETS_FILE: &str = super::ecstore_bucket::metadata::BUCKET_TARGETS_FILE; - pub(crate) const BUCKET_VERSIONING_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_VERSIONING_CONFIG; - pub(crate) const OBJECT_LOCK_CONFIG: &str = super::ecstore_bucket::metadata::OBJECT_LOCK_CONFIG; - - pub(crate) type BucketMetadata = super::ecstore_bucket::metadata::BucketMetadata; - - pub(crate) fn table_catalog_path_hash(value: &str) -> String { - super::ecstore_bucket::metadata::table_catalog_path_hash(value) - } -} - -pub(crate) mod metadata_sys { - use std::sync::Arc; - - use rustfs_policy::policy::BucketPolicy; - use s3s::dto::{ - BucketLifecycleConfiguration, NotificationConfiguration, ObjectLockConfiguration, ServerSideEncryptionConfiguration, - Tagging, VersioningConfiguration, - }; - use time::OffsetDateTime; - - use super::Result; - use super::metadata::BucketMetadata; - use super::quota::BucketQuota; - use super::target::BucketTargets; - - pub(crate) type BucketMetadataSys = super::ecstore_bucket::metadata_sys::BucketMetadataSys; - - pub(crate) async fn get(bucket: &str) -> Result> { - super::ecstore_bucket::metadata_sys::get(bucket).await - } - - pub(crate) async fn update(bucket: &str, config_file: &str, data: Vec) -> Result { - super::ecstore_bucket::metadata_sys::update(bucket, config_file, data).await - } - - pub(crate) async fn delete(bucket: &str, config_file: &str) -> Result { - super::ecstore_bucket::metadata_sys::delete(bucket, config_file).await - } - - pub(crate) async fn get_bucket_policy(bucket: &str) -> Result<(BucketPolicy, OffsetDateTime)> { - super::ecstore_bucket::metadata_sys::get_bucket_policy(bucket).await - } - - pub(crate) async fn get_bucket_targets_config(bucket: &str) -> Result { - super::ecstore_bucket::metadata_sys::get_bucket_targets_config(bucket).await - } - - pub(crate) async fn get_config_from_disk(bucket: &str) -> Result { - super::ecstore_bucket::metadata_sys::get_config_from_disk(bucket).await - } - - pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> { - super::ecstore_bucket::metadata_sys::get_lifecycle_config(bucket).await - } - - pub(crate) async fn get_notification_config(bucket: &str) -> Result> { - super::ecstore_bucket::metadata_sys::get_notification_config(bucket).await - } - - pub(crate) async fn get_object_lock_config(bucket: &str) -> Result<(ObjectLockConfiguration, OffsetDateTime)> { - super::ecstore_bucket::metadata_sys::get_object_lock_config(bucket).await - } - - pub(crate) async fn get_quota_config(bucket: &str) -> Result<(BucketQuota, OffsetDateTime)> { - super::ecstore_bucket::metadata_sys::get_quota_config(bucket).await - } - - pub(crate) async fn get_replication_config(bucket: &str) -> Result<(s3s::dto::ReplicationConfiguration, OffsetDateTime)> { - super::ecstore_bucket::metadata_sys::get_replication_config(bucket).await - } - - pub(crate) async fn get_sse_config(bucket: &str) -> Result<(ServerSideEncryptionConfiguration, OffsetDateTime)> { - super::ecstore_bucket::metadata_sys::get_sse_config(bucket).await - } - - pub(crate) async fn get_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime)> { - super::ecstore_bucket::metadata_sys::get_tagging_config(bucket).await - } - - pub(crate) async fn get_versioning_config(bucket: &str) -> Result<(VersioningConfiguration, OffsetDateTime)> { - super::ecstore_bucket::metadata_sys::get_versioning_config(bucket).await - } - - pub(crate) async fn list_bucket_targets(bucket: &str) -> Result { - super::ecstore_bucket::metadata_sys::list_bucket_targets(bucket).await - } -} - -pub(crate) mod quota { - pub(crate) mod checker { - pub(crate) type QuotaChecker = super::super::ecstore_bucket::quota::checker::QuotaChecker; - } - - pub(crate) type BucketQuota = super::ecstore_bucket::quota::BucketQuota; - pub(crate) type QuotaError = super::ecstore_bucket::quota::QuotaError; - pub(crate) type QuotaOperation = super::ecstore_bucket::quota::QuotaOperation; -} - -pub(crate) mod replication { - pub(crate) type BucketReplicationResyncStatus = super::ecstore_bucket::replication::BucketReplicationResyncStatus; - pub(crate) type BucketStats = super::ecstore_bucket::replication::BucketStats; - pub(crate) type ObjectOpts = super::ecstore_bucket::replication::ObjectOpts; - pub(crate) type ResyncOpts = super::ecstore_bucket::replication::ResyncOpts; - #[cfg(test)] - pub(crate) type ResyncStatusType = super::ecstore_bucket::replication::ResyncStatusType; - #[cfg(test)] - pub(crate) type TargetReplicationResyncStatus = super::ecstore_bucket::replication::TargetReplicationResyncStatus; -} - -pub(crate) mod target { - #[allow(clippy::upper_case_acronyms)] - pub(crate) type ARN = super::ecstore_bucket::target::ARN; - pub(crate) type BucketTarget = super::ecstore_bucket::target::BucketTarget; - pub(crate) type BucketTargetType = super::ecstore_bucket::target::BucketTargetType; - pub(crate) type BucketTargets = super::ecstore_bucket::target::BucketTargets; - pub(crate) type Credentials = super::ecstore_bucket::target::Credentials; -} - -pub(crate) mod ecstore_utils { - pub(crate) fn deserialize(input: &[u8]) -> s3s::xml::DeResult - where - T: for<'xml> s3s::xml::Deserialize<'xml>, - { - super::ecstore_bucket::utils::deserialize(input) - } - - pub(crate) fn is_valid_object_prefix(object: &str) -> bool { - super::ecstore_bucket::utils::is_valid_object_prefix(object) - } - - pub(crate) fn serialize(val: &T) -> s3s::xml::SerResult> { - super::ecstore_bucket::utils::serialize(val) - } -} - -pub(crate) mod versioning {} - -pub(crate) mod versioning_sys { - pub(crate) type BucketVersioningSys = super::ecstore_bucket::versioning_sys::BucketVersioningSys; -} - -pub(crate) mod storageclass { - pub(crate) const INLINE_BLOCK_ENV: &str = super::ecstore_config::storageclass::INLINE_BLOCK_ENV; - pub(crate) const OPTIMIZE_ENV: &str = super::ecstore_config::storageclass::OPTIMIZE_ENV; - pub(crate) const RRS: &str = super::ecstore_config::storageclass::RRS; - pub(crate) const RRS_ENV: &str = super::ecstore_config::storageclass::RRS_ENV; - pub(crate) const STANDARD: &str = super::ecstore_config::storageclass::STANDARD; - pub(crate) const STANDARD_ENV: &str = super::ecstore_config::storageclass::STANDARD_ENV; - - pub(crate) type Config = super::ecstore_config::storageclass::Config; - - pub(crate) fn lookup_config(kvs: &rustfs_config::server_config::KVS, set_drive_count: usize) -> super::Result { - super::ecstore_config::storageclass::lookup_config(kvs, set_drive_count) - } -} - -pub(crate) type DailyAllTierStats = ecstore_bucket::lifecycle::tier_last_day_stats::DailyAllTierStats; - -pub(crate) fn is_reserved_or_invalid_bucket(bucket_entry: &str, strict: bool) -> bool { - ecstore_capacity::is_reserved_or_invalid_bucket(bucket_entry, strict) -} - -pub(crate) async fn read_admin_config(api: Arc, file: &str) -> Result> { - ecstore_config::com::read_config(api, file).await -} - -pub(crate) async fn read_admin_config_without_migrate(api: Arc) -> Result { - ecstore_config::com::read_config_without_migrate(api).await -} - -pub(crate) async fn save_admin_config(api: Arc, file: &str, data: Vec) -> Result<()> { - ecstore_config::com::save_config(api, file, data).await -} - -pub(crate) async fn delete_admin_config(api: Arc, file: &str) -> Result<()> { - ecstore_config::com::delete_config(api, file).await -} - -pub(crate) async fn save_admin_server_config(api: Arc, cfg: &rustfs_config::server_config::Config) -> Result<()> { - ecstore_config::com::save_server_config(api, cfg).await -} - -pub(crate) fn init_admin_config_defaults() { - ecstore_config::init(); -} - -pub(crate) async fn collect_local_metrics( - types: MetricType, - opts: &CollectMetricsOpts, -) -> rustfs_madmin::metrics::RealtimeMetrics { - ecstore_metrics::collect_local_metrics(types, opts).await -} - -pub(crate) struct AdminErrorRef(fn() -> &'static AdminError); - -impl Deref for AdminErrorRef { - type Target = AdminError; - - fn deref(&self) -> &Self::Target { - (self.0)() - } -} - -pub(crate) static ERR_TIER_BACKEND_IN_USE: AdminErrorRef = AdminErrorRef(|| &ecstore_tier::tier::ERR_TIER_BACKEND_IN_USE); -pub(crate) static ERR_TIER_BACKEND_NOT_EMPTY: AdminErrorRef = AdminErrorRef(|| &ecstore_tier::tier::ERR_TIER_BACKEND_NOT_EMPTY); -pub(crate) static ERR_TIER_MISSING_CREDENTIALS: AdminErrorRef = - AdminErrorRef(|| &ecstore_tier::tier::ERR_TIER_MISSING_CREDENTIALS); -pub(crate) static ERR_TIER_ALREADY_EXISTS: AdminErrorRef = - AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_ALREADY_EXISTS); -pub(crate) static ERR_TIER_CONNECT_ERR: AdminErrorRef = AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_CONNECT_ERR); -pub(crate) static ERR_TIER_INVALID_CREDENTIALS: AdminErrorRef = - AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_INVALID_CREDENTIALS); -pub(crate) static ERR_TIER_NAME_NOT_UPPERCASE: AdminErrorRef = - AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_NAME_NOT_UPPERCASE); -pub(crate) static ERR_TIER_NOT_FOUND: AdminErrorRef = AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_NOT_FOUND); diff --git a/rustfs/src/admin/router.rs b/rustfs/src/admin/router.rs index 4058364e7..c923cb52e 100644 --- a/rustfs/src/admin/router.rs +++ b/rustfs/src/admin/router.rs @@ -12,16 +12,18 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::PeerRestClient; -use super::bandwidth::monitor::BandwidthDetails; -use super::bucket_target_sys::{BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient}; -use super::metadata::BUCKET_TARGETS_FILE; -use super::metadata_sys; -use super::read_admin_config_without_migrate; -use super::replication::{BucketReplicationResyncStatus, BucketStats, ObjectOpts, ResyncOpts}; -use super::target::{BucketTarget, BucketTargetType, BucketTargets}; -use super::versioning_sys::BucketVersioningSys; -use super::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _}; +use super::storage_api::PeerRestClient; +use super::storage_api::bandwidth::monitor::BandwidthDetails; +use super::storage_api::bucket_target_sys::{ + BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient, +}; +use super::storage_api::metadata::BUCKET_TARGETS_FILE; +use super::storage_api::metadata_sys; +use super::storage_api::read_admin_config_without_migrate; +use super::storage_api::replication::{BucketReplicationResyncStatus, BucketStats, ObjectOpts, ResyncOpts}; +use super::storage_api::target::{BucketTarget, BucketTargetType, BucketTargets}; +use super::storage_api::versioning_sys::BucketVersioningSys; +use super::storage_api::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _, StorageError}; use crate::admin::console::{is_console_path, make_console_server}; use crate::admin::handlers::oidc::is_oidc_path; use crate::admin::runtime_sources::{ @@ -1414,7 +1416,7 @@ async fn ensure_replication_bucket_exists(bucket: &str) -> S3Result<()> { async fn ensure_replication_config_exists(bucket: &str) -> S3Result<()> { match metadata_sys::get_replication_config(bucket).await { Ok(_) => Ok(()), - Err(super::StorageError::ConfigNotFound) => Err(s3_error!(ReplicationConfigurationNotFoundError)), + Err(StorageError::ConfigNotFound) => Err(s3_error!(ReplicationConfigurationNotFoundError)), Err(err) => Err(ApiError::from(err).into()), } } @@ -1938,7 +1940,7 @@ async fn resolve_replication_target_client(bucket: &str, target: &BucketTarget) fn build_replication_probe_put_options(now: OffsetDateTime) -> PutObjectOptions { PutObjectOptions { - internal: super::bucket_target_sys::AdvancedPutOptions { + internal: super::storage_api::bucket_target_sys::AdvancedPutOptions { source_version_id: Uuid::new_v4().to_string(), replication_status: ReplicationStatusType::Replica, source_mtime: now, @@ -2053,7 +2055,7 @@ async fn source_bucket_requires_object_lock(bucket: &str) -> S3Result { .object_lock_enabled .as_ref() .is_some_and(|state| state.as_str() == s3s::dto::ObjectLockEnabled::ENABLED)), - Err(super::StorageError::ConfigNotFound) => Ok(false), + Err(StorageError::ConfigNotFound) => Ok(false), Err(err) => Err(ApiError::from(err).into()), } } @@ -2767,7 +2769,7 @@ mod tests { #[test] fn apply_replication_reset_to_targets_updates_matching_target() { let mut targets = BucketTargets { - targets: vec![super::super::target::BucketTarget { + targets: vec![crate::admin::storage_api::target::BucketTarget { arn: "arn:target".to_string(), ..Default::default() }], @@ -2789,10 +2791,10 @@ mod tests { let mut status = BucketReplicationResyncStatus::new(); status.targets_map.insert( "arn:z".to_string(), - super::super::replication::TargetReplicationResyncStatus { + crate::admin::storage_api::replication::TargetReplicationResyncStatus { resync_id: "rid-z".to_string(), last_update: Some(datetime!(2025-01-03 00:00 UTC)), - resync_status: super::super::replication::ResyncStatusType::ResyncFailed, + resync_status: crate::admin::storage_api::replication::ResyncStatusType::ResyncFailed, failed_count: 2, failed_size: 4, bucket: "bucket-z".to_string(), @@ -2802,10 +2804,10 @@ mod tests { ); status.targets_map.insert( "arn:a".to_string(), - super::super::replication::TargetReplicationResyncStatus { + crate::admin::storage_api::replication::TargetReplicationResyncStatus { resync_id: "rid-a".to_string(), last_update: Some(datetime!(2025-01-02 00:00 UTC)), - resync_status: super::super::replication::ResyncStatusType::ResyncCompleted, + resync_status: crate::admin::storage_api::replication::ResyncStatusType::ResyncCompleted, replicated_count: 3, replicated_size: 9, bucket: "bucket-a".to_string(), @@ -2835,10 +2837,10 @@ mod tests { let mut status = BucketReplicationResyncStatus::new(); status.targets_map.insert( "arn:z".to_string(), - super::super::replication::TargetReplicationResyncStatus { + crate::admin::storage_api::replication::TargetReplicationResyncStatus { resync_id: "rid-z".to_string(), last_update: Some(datetime!(2025-02-03 00:00 UTC)), - resync_status: super::super::replication::ResyncStatusType::ResyncFailed, + resync_status: crate::admin::storage_api::replication::ResyncStatusType::ResyncFailed, failed_count: 2, failed_size: 4, bucket: "bucket-z".to_string(), @@ -2848,10 +2850,10 @@ mod tests { ); status.targets_map.insert( "arn:a".to_string(), - super::super::replication::TargetReplicationResyncStatus { + crate::admin::storage_api::replication::TargetReplicationResyncStatus { resync_id: "rid-a".to_string(), last_update: Some(datetime!(2025-02-02 00:00 UTC)), - resync_status: super::super::replication::ResyncStatusType::ResyncCompleted, + resync_status: crate::admin::storage_api::replication::ResyncStatusType::ResyncCompleted, replicated_count: 3, replicated_size: 9, bucket: "bucket-a".to_string(), diff --git a/rustfs/src/admin/service/config.rs b/rustfs/src/admin/service/config.rs index 43eba801a..6123178e7 100644 --- a/rustfs/src/admin/service/config.rs +++ b/rustfs/src/admin/service/config.rs @@ -12,12 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::storageclass; -use super::super::{STORAGE_CLASS_SUB_SYS, read_admin_config_without_migrate}; use crate::admin::runtime_sources::{ AppContext, get_global_app_context, publish_server_config, publish_storage_class_config, resolve_notification_system, resolve_object_store_handle, resolve_object_store_handle_for_context, }; +use crate::admin::storage_api::storageclass; +use crate::admin::storage_api::{STORAGE_CLASS_SUB_SYS, read_admin_config_without_migrate}; use rustfs_audit::reload_audit_config; use rustfs_config::audit::{AUDIT_MQTT_SUB_SYS, AUDIT_REDIS_DEFAULT_CHANNEL, AUDIT_WEBHOOK_SUB_SYS}; use rustfs_config::notify::{NOTIFY_MQTT_SUB_SYS, NOTIFY_REDIS_DEFAULT_CHANNEL, NOTIFY_WEBHOOK_SUB_SYS}; @@ -383,8 +383,8 @@ pub async fn signal_config_snapshot_reload() { #[cfg(test)] mod tests { - use super::super::super::metadata::{BUCKET_LIFECYCLE_CONFIG, BUCKET_REPLICATION_CONFIG}; use super::*; + use crate::admin::storage_api::metadata::{BUCKET_LIFECYCLE_CONFIG, BUCKET_REPLICATION_CONFIG}; use rustfs_config::notify::NOTIFY_WEBHOOK_SUB_SYS; use rustfs_config::oidc::{OIDC_CLIENT_ID, OIDC_CONFIG_URL, OIDC_SCOPES}; use rustfs_config::{HEAL_SUB_SYS, SCANNER_SUB_SYS}; @@ -440,7 +440,7 @@ mod tests { #[test] fn validate_notify_subsystem_config_rejects_invalid_webhook_endpoint() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); let mut config = ServerConfig::new(); let targets = config.0.get_mut(NOTIFY_WEBHOOK_SUB_SYS).expect("notify webhook defaults"); let kvs = targets.get_mut(DEFAULT_DELIMITER).expect("default target"); @@ -454,7 +454,7 @@ mod tests { #[test] fn validate_audit_subsystem_config_rejects_relative_queue_dir() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); let mut config = ServerConfig::new(); let targets = config.0.get_mut(AUDIT_MQTT_SUB_SYS).expect("audit mqtt defaults"); let kvs = targets.get_mut(DEFAULT_DELIMITER).expect("default target"); @@ -469,7 +469,7 @@ mod tests { #[test] fn validate_identity_openid_config_rejects_missing_openid_scope() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); let mut config = ServerConfig::new(); let targets = config.0.get_mut(IDENTITY_OPENID_SUB_SYS).expect("openid defaults"); let kvs = targets.get_mut(DEFAULT_DELIMITER).expect("default target"); @@ -486,7 +486,7 @@ mod tests { #[test] fn validate_identity_openid_config_rejects_invalid_named_provider_id() { - super::super::super::init_admin_config_defaults(); + crate::admin::storage_api::init_admin_config_defaults(); let mut config = ServerConfig::new(); let targets = config.0.get_mut(IDENTITY_OPENID_SUB_SYS).expect("openid defaults"); let default_kvs = targets.get(DEFAULT_DELIMITER).cloned().expect("default target"); diff --git a/rustfs/src/admin/service/site_replication.rs b/rustfs/src/admin/service/site_replication.rs index b8f152fa7..68cf8cd35 100644 --- a/rustfs/src/admin/service/site_replication.rs +++ b/rustfs/src/admin/service/site_replication.rs @@ -12,10 +12,10 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::super::Error as StorageError; -use super::super::{read_admin_config, save_admin_config}; use crate::admin::runtime_sources::{AppContext, get_global_app_context, resolve_object_store_handle_for_context}; use crate::admin::site_replication_identity::{deployment_id_for_endpoint, normalize_peer_map_by_identity_with}; +use crate::admin::storage_api::Error as StorageError; +use crate::admin::storage_api::{read_admin_config, save_admin_config}; use rustfs_madmin::PeerInfo; use s3s::{S3Error, S3ErrorCode, S3Result}; use serde_json::{Map, Value}; diff --git a/rustfs/src/admin/storage_api.rs b/rustfs/src/admin/storage_api.rs index efc4e7d31..462ba74cb 100644 --- a/rustfs/src/admin/storage_api.rs +++ b/rustfs/src/admin/storage_api.rs @@ -12,6 +12,402 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::ops::Deref; +use std::sync::Arc; + +mod ecstore_bucket { + pub(crate) use crate::storage::ecstore_bucket::{ + bandwidth, bucket_target_sys, lifecycle, metadata, metadata_sys, quota, replication, target, utils, versioning, + versioning_sys, + }; +} + +mod ecstore_capacity { + pub(crate) use crate::storage::ecstore_capacity::is_reserved_or_invalid_bucket; +} + +mod ecstore_client { + pub(crate) use crate::storage::ecstore_client::admin_handler_utils; +} + +pub(crate) mod ecstore_cluster { + pub(crate) use crate::storage::ecstore_cluster::{ + ClusterDriveMembership, ClusterEndpointType, ClusterLocalNodeStorage, ClusterLocalNodeStorageSnapshot, + ClusterMembershipSnapshot, ClusterNodeMembership, ClusterPeerHealth, ClusterPeerHealthSnapshot, ClusterPoolState, + ClusterPoolStateSnapshot, + }; +} + +mod ecstore_config { + pub(crate) use crate::storage::ecstore_config::{com, init, storageclass}; +} + +#[allow(unused_imports)] +mod ecstore_disk { + pub(crate) use crate::storage::ecstore_disk::{RUSTFS_META_BUCKET, endpoint}; +} + +mod ecstore_error { + pub(crate) use crate::storage::ecstore_error::StorageError; +} + +#[allow(unused_imports)] +mod ecstore_layout { + pub(crate) use crate::storage::ecstore_layout::{EndpointServerPools, Endpoints, PoolEndpoints}; +} + +mod ecstore_metrics { + pub(crate) use crate::storage::ecstore_metrics::{CollectMetricsOpts, MetricType, collect_local_metrics}; +} + +mod ecstore_notification { + pub(crate) use crate::storage::ecstore_notification::NotificationSys; +} + +#[allow(unused_imports)] +mod ecstore_rebalance { + pub(crate) use crate::storage::ecstore_rebalance::{ + DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, + RebalanceMeta, RebalanceStats, RebalanceStopPropagationRecord, decode_rebalance_stop_propagation_record, + encode_rebalance_stop_propagation_record, + }; +} + +mod ecstore_rpc { + pub(crate) use crate::storage::ecstore_rpc::PeerRestClient; +} + +mod ecstore_storage { + pub(crate) use crate::storage::ecstore_storage::ECStore; +} + +mod ecstore_tier { + pub(crate) use crate::storage::ecstore_tier::{tier, tier_admin, tier_config, tier_handlers}; +} + +pub(crate) const RUSTFS_META_BUCKET: &str = ecstore_disk::RUSTFS_META_BUCKET; +pub(crate) const STORAGE_CLASS_SUB_SYS: &str = ecstore_config::com::STORAGE_CLASS_SUB_SYS; + +pub(crate) type AdminError = ecstore_client::admin_handler_utils::AdminError; +pub(crate) type CollectMetricsOpts = ecstore_metrics::CollectMetricsOpts; +pub(crate) type DiskStat = ecstore_rebalance::DiskStat; +pub(crate) type ECStore = ecstore_storage::ECStore; +pub(crate) type EndpointServerPools = ecstore_layout::EndpointServerPools; +pub(crate) type MetricType = ecstore_metrics::MetricType; +pub(crate) type NotificationSys = ecstore_notification::NotificationSys; +pub(crate) type PeerRestClient = ecstore_rpc::PeerRestClient; +pub(crate) type RebalSaveOpt = ecstore_rebalance::RebalSaveOpt; +pub(crate) type RebalanceCleanupWarnings = ecstore_rebalance::RebalanceCleanupWarnings; +pub(crate) type RebalanceMeta = ecstore_rebalance::RebalanceMeta; +pub(crate) type RebalanceStats = ecstore_rebalance::RebalanceStats; +pub(crate) type RebalanceStopPropagationRecord = ecstore_rebalance::RebalanceStopPropagationRecord; +pub(crate) type StorageError = ecstore_error::StorageError; +pub(crate) type Error = StorageError; +pub(crate) type Result = core::result::Result; +pub(crate) type TierConfig = ecstore_tier::tier_config::TierConfig; +pub(crate) type TierCreds = ecstore_tier::tier_admin::TierCreds; +pub(crate) type TierType = ecstore_tier::tier_config::TierType; + +#[cfg(test)] +pub(crate) type Endpoint = ecstore_disk::endpoint::Endpoint; +#[cfg(test)] +pub(crate) type Endpoints = ecstore_layout::Endpoints; +#[cfg(test)] +pub(crate) type PoolEndpoints = ecstore_layout::PoolEndpoints; +#[cfg(test)] +pub(crate) type RebalStatus = ecstore_rebalance::RebalStatus; +#[cfg(test)] +pub(crate) type RebalanceCleanupWarningEntry = ecstore_rebalance::RebalanceCleanupWarningEntry; +#[cfg(test)] +pub(crate) type RebalanceInfo = ecstore_rebalance::RebalanceInfo; + +pub(crate) fn decode_rebalance_stop_propagation_record(message: &str) -> Option { + ecstore_rebalance::decode_rebalance_stop_propagation_record(message) +} + +#[cfg(test)] +pub(crate) fn encode_rebalance_stop_propagation_record(record: &RebalanceStopPropagationRecord) -> String { + ecstore_rebalance::encode_rebalance_stop_propagation_record(record) +} + +pub(crate) trait AdminReplicationConfigExt { + fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec; + fn has_existing_object_replication(&self, arn: &str) -> (bool, bool); +} + +impl AdminReplicationConfigExt for s3s::dto::ReplicationConfiguration { + fn filter_target_arns(&self, obj: &replication::ObjectOpts) -> Vec { + ::filter_target_arns( + self, obj, + ) + } + + fn has_existing_object_replication(&self, arn: &str) -> (bool, bool) { + ::has_existing_object_replication( + self, arn, + ) + } +} + +pub(crate) trait AdminVersioningConfigExt { + fn enabled(&self) -> bool; +} + +impl AdminVersioningConfigExt for s3s::dto::VersioningConfiguration { + fn enabled(&self) -> bool { + ::enabled(self) + } +} + +pub(crate) mod bandwidth { + pub(crate) mod monitor { + pub(crate) type BandwidthDetails = super::super::ecstore_bucket::bandwidth::monitor::BandwidthDetails; + } +} + +pub(crate) mod bucket_target_sys { + pub(crate) type AdvancedPutOptions = super::ecstore_bucket::bucket_target_sys::AdvancedPutOptions; + pub(crate) type BucketTargetError = super::ecstore_bucket::bucket_target_sys::BucketTargetError; + pub(crate) type BucketTargetSys = super::ecstore_bucket::bucket_target_sys::BucketTargetSys; + pub(crate) type PutObjectOptions = super::ecstore_bucket::bucket_target_sys::PutObjectOptions; + pub(crate) type RemoveObjectOptions = super::ecstore_bucket::bucket_target_sys::RemoveObjectOptions; + pub(crate) type S3ClientError = super::ecstore_bucket::bucket_target_sys::S3ClientError; + pub(crate) type TargetClient = super::ecstore_bucket::bucket_target_sys::TargetClient; +} + +pub(crate) mod lifecycle { + pub(crate) mod tier_last_day_stats { + #[cfg(test)] + pub(crate) type LastDayTierStats = super::super::ecstore_bucket::lifecycle::tier_last_day_stats::LastDayTierStats; + } +} + +pub(crate) mod metadata { + pub(crate) const BUCKET_CORS_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_CORS_CONFIG; + pub(crate) const BUCKET_LIFECYCLE_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_LIFECYCLE_CONFIG; + pub(crate) const BUCKET_NOTIFICATION_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_NOTIFICATION_CONFIG; + pub(crate) const BUCKET_POLICY_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_POLICY_CONFIG; + pub(crate) const BUCKET_QUOTA_CONFIG_FILE: &str = super::ecstore_bucket::metadata::BUCKET_QUOTA_CONFIG_FILE; + pub(crate) const BUCKET_REPLICATION_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_REPLICATION_CONFIG; + pub(crate) const BUCKET_SSECONFIG: &str = super::ecstore_bucket::metadata::BUCKET_SSECONFIG; + pub(crate) const BUCKET_TAGGING_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_TAGGING_CONFIG; + pub(crate) const BUCKET_TARGETS_FILE: &str = super::ecstore_bucket::metadata::BUCKET_TARGETS_FILE; + pub(crate) const BUCKET_VERSIONING_CONFIG: &str = super::ecstore_bucket::metadata::BUCKET_VERSIONING_CONFIG; + pub(crate) const OBJECT_LOCK_CONFIG: &str = super::ecstore_bucket::metadata::OBJECT_LOCK_CONFIG; + + pub(crate) type BucketMetadata = super::ecstore_bucket::metadata::BucketMetadata; + + pub(crate) fn table_catalog_path_hash(value: &str) -> String { + super::ecstore_bucket::metadata::table_catalog_path_hash(value) + } +} + +pub(crate) mod metadata_sys { + use std::sync::Arc; + + use rustfs_policy::policy::BucketPolicy; + use s3s::dto::{ + BucketLifecycleConfiguration, NotificationConfiguration, ObjectLockConfiguration, ServerSideEncryptionConfiguration, + Tagging, VersioningConfiguration, + }; + use time::OffsetDateTime; + + use super::Result; + use super::metadata::BucketMetadata; + use super::quota::BucketQuota; + use super::target::BucketTargets; + + pub(crate) type BucketMetadataSys = super::ecstore_bucket::metadata_sys::BucketMetadataSys; + + pub(crate) async fn get(bucket: &str) -> Result> { + super::ecstore_bucket::metadata_sys::get(bucket).await + } + + pub(crate) async fn update(bucket: &str, config_file: &str, data: Vec) -> Result { + super::ecstore_bucket::metadata_sys::update(bucket, config_file, data).await + } + + pub(crate) async fn delete(bucket: &str, config_file: &str) -> Result { + super::ecstore_bucket::metadata_sys::delete(bucket, config_file).await + } + + pub(crate) async fn get_bucket_policy(bucket: &str) -> Result<(BucketPolicy, OffsetDateTime)> { + super::ecstore_bucket::metadata_sys::get_bucket_policy(bucket).await + } + + pub(crate) async fn get_bucket_targets_config(bucket: &str) -> Result { + super::ecstore_bucket::metadata_sys::get_bucket_targets_config(bucket).await + } + + pub(crate) async fn get_config_from_disk(bucket: &str) -> Result { + super::ecstore_bucket::metadata_sys::get_config_from_disk(bucket).await + } + + pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> { + super::ecstore_bucket::metadata_sys::get_lifecycle_config(bucket).await + } + + pub(crate) async fn get_notification_config(bucket: &str) -> Result> { + super::ecstore_bucket::metadata_sys::get_notification_config(bucket).await + } + + pub(crate) async fn get_object_lock_config(bucket: &str) -> Result<(ObjectLockConfiguration, OffsetDateTime)> { + super::ecstore_bucket::metadata_sys::get_object_lock_config(bucket).await + } + + pub(crate) async fn get_quota_config(bucket: &str) -> Result<(BucketQuota, OffsetDateTime)> { + super::ecstore_bucket::metadata_sys::get_quota_config(bucket).await + } + + pub(crate) async fn get_replication_config(bucket: &str) -> Result<(s3s::dto::ReplicationConfiguration, OffsetDateTime)> { + super::ecstore_bucket::metadata_sys::get_replication_config(bucket).await + } + + pub(crate) async fn get_sse_config(bucket: &str) -> Result<(ServerSideEncryptionConfiguration, OffsetDateTime)> { + super::ecstore_bucket::metadata_sys::get_sse_config(bucket).await + } + + pub(crate) async fn get_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime)> { + super::ecstore_bucket::metadata_sys::get_tagging_config(bucket).await + } + + pub(crate) async fn get_versioning_config(bucket: &str) -> Result<(VersioningConfiguration, OffsetDateTime)> { + super::ecstore_bucket::metadata_sys::get_versioning_config(bucket).await + } + + pub(crate) async fn list_bucket_targets(bucket: &str) -> Result { + super::ecstore_bucket::metadata_sys::list_bucket_targets(bucket).await + } +} + +pub(crate) mod quota { + pub(crate) mod checker { + pub(crate) type QuotaChecker = super::super::ecstore_bucket::quota::checker::QuotaChecker; + } + + pub(crate) type BucketQuota = super::ecstore_bucket::quota::BucketQuota; + pub(crate) type QuotaError = super::ecstore_bucket::quota::QuotaError; + pub(crate) type QuotaOperation = super::ecstore_bucket::quota::QuotaOperation; +} + +pub(crate) mod replication { + pub(crate) type BucketReplicationResyncStatus = super::ecstore_bucket::replication::BucketReplicationResyncStatus; + pub(crate) type BucketStats = super::ecstore_bucket::replication::BucketStats; + pub(crate) type ObjectOpts = super::ecstore_bucket::replication::ObjectOpts; + pub(crate) type ResyncOpts = super::ecstore_bucket::replication::ResyncOpts; + #[cfg(test)] + pub(crate) type ResyncStatusType = super::ecstore_bucket::replication::ResyncStatusType; + #[cfg(test)] + pub(crate) type TargetReplicationResyncStatus = super::ecstore_bucket::replication::TargetReplicationResyncStatus; +} + +pub(crate) mod target { + #[allow(clippy::upper_case_acronyms)] + pub(crate) type ARN = super::ecstore_bucket::target::ARN; + pub(crate) type BucketTarget = super::ecstore_bucket::target::BucketTarget; + pub(crate) type BucketTargetType = super::ecstore_bucket::target::BucketTargetType; + pub(crate) type BucketTargets = super::ecstore_bucket::target::BucketTargets; + pub(crate) type Credentials = super::ecstore_bucket::target::Credentials; +} + +pub(crate) mod ecstore_utils { + pub(crate) fn deserialize(input: &[u8]) -> s3s::xml::DeResult + where + T: for<'xml> s3s::xml::Deserialize<'xml>, + { + super::ecstore_bucket::utils::deserialize(input) + } + + pub(crate) fn is_valid_object_prefix(object: &str) -> bool { + super::ecstore_bucket::utils::is_valid_object_prefix(object) + } + + pub(crate) fn serialize(val: &T) -> s3s::xml::SerResult> { + super::ecstore_bucket::utils::serialize(val) + } +} + +pub(crate) mod versioning {} + +pub(crate) mod versioning_sys { + pub(crate) type BucketVersioningSys = super::ecstore_bucket::versioning_sys::BucketVersioningSys; +} + +pub(crate) mod storageclass { + pub(crate) const INLINE_BLOCK_ENV: &str = super::ecstore_config::storageclass::INLINE_BLOCK_ENV; + pub(crate) const OPTIMIZE_ENV: &str = super::ecstore_config::storageclass::OPTIMIZE_ENV; + pub(crate) const RRS: &str = super::ecstore_config::storageclass::RRS; + pub(crate) const RRS_ENV: &str = super::ecstore_config::storageclass::RRS_ENV; + pub(crate) const STANDARD: &str = super::ecstore_config::storageclass::STANDARD; + pub(crate) const STANDARD_ENV: &str = super::ecstore_config::storageclass::STANDARD_ENV; + + pub(crate) type Config = super::ecstore_config::storageclass::Config; + + pub(crate) fn lookup_config(kvs: &rustfs_config::server_config::KVS, set_drive_count: usize) -> super::Result { + super::ecstore_config::storageclass::lookup_config(kvs, set_drive_count) + } +} + +pub(crate) type DailyAllTierStats = ecstore_bucket::lifecycle::tier_last_day_stats::DailyAllTierStats; + +pub(crate) fn is_reserved_or_invalid_bucket(bucket_entry: &str, strict: bool) -> bool { + ecstore_capacity::is_reserved_or_invalid_bucket(bucket_entry, strict) +} + +pub(crate) async fn read_admin_config(api: Arc, file: &str) -> Result> { + ecstore_config::com::read_config(api, file).await +} + +pub(crate) async fn read_admin_config_without_migrate(api: Arc) -> Result { + ecstore_config::com::read_config_without_migrate(api).await +} + +pub(crate) async fn save_admin_config(api: Arc, file: &str, data: Vec) -> Result<()> { + ecstore_config::com::save_config(api, file, data).await +} + +pub(crate) async fn delete_admin_config(api: Arc, file: &str) -> Result<()> { + ecstore_config::com::delete_config(api, file).await +} + +pub(crate) async fn save_admin_server_config(api: Arc, cfg: &rustfs_config::server_config::Config) -> Result<()> { + ecstore_config::com::save_server_config(api, cfg).await +} + +pub(crate) fn init_admin_config_defaults() { + ecstore_config::init(); +} + +pub(crate) async fn collect_local_metrics( + types: MetricType, + opts: &CollectMetricsOpts, +) -> rustfs_madmin::metrics::RealtimeMetrics { + ecstore_metrics::collect_local_metrics(types, opts).await +} + +pub(crate) struct AdminErrorRef(fn() -> &'static AdminError); + +impl Deref for AdminErrorRef { + type Target = AdminError; + + fn deref(&self) -> &Self::Target { + (self.0)() + } +} + +pub(crate) static ERR_TIER_BACKEND_IN_USE: AdminErrorRef = AdminErrorRef(|| &ecstore_tier::tier::ERR_TIER_BACKEND_IN_USE); +pub(crate) static ERR_TIER_BACKEND_NOT_EMPTY: AdminErrorRef = AdminErrorRef(|| &ecstore_tier::tier::ERR_TIER_BACKEND_NOT_EMPTY); +pub(crate) static ERR_TIER_MISSING_CREDENTIALS: AdminErrorRef = + AdminErrorRef(|| &ecstore_tier::tier::ERR_TIER_MISSING_CREDENTIALS); +pub(crate) static ERR_TIER_ALREADY_EXISTS: AdminErrorRef = + AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_ALREADY_EXISTS); +pub(crate) static ERR_TIER_CONNECT_ERR: AdminErrorRef = AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_CONNECT_ERR); +pub(crate) static ERR_TIER_INVALID_CREDENTIALS: AdminErrorRef = + AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_INVALID_CREDENTIALS); +pub(crate) static ERR_TIER_NAME_NOT_UPPERCASE: AdminErrorRef = + AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_NAME_NOT_UPPERCASE); +pub(crate) static ERR_TIER_NOT_FOUND: AdminErrorRef = AdminErrorRef(|| &ecstore_tier::tier_handlers::ERR_TIER_NOT_FOUND); + pub(crate) use crate::storage::StorageObjectOptions; pub(crate) use crate::storage::access::{ReqInfo, authorize_request}; pub(crate) use crate::storage::request_context::{RequestContext, spawn_traced}; diff --git a/scripts/check_architecture_migration_rules.sh b/scripts/check_architecture_migration_rules.sh index 519ffe383..8cb260bc5 100755 --- a/scripts/check_architecture_migration_rules.sh +++ b/scripts/check_architecture_migration_rules.sh @@ -165,6 +165,7 @@ RUSTFS_APP_WILDCARD_IMPORT_HITS_FILE="${TMP_DIR}/rustfs_app_wildcard_import_hits RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_usecase_s3_api_bypass_hits.txt" RUSTFS_APP_USECASE_STORAGE_API_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_usecase_storage_api_bypass_hits.txt" RUSTFS_APP_ADMIN_STORAGE_API_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_admin_storage_api_bypass_hits.txt" +RUSTFS_ADMIN_STORAGE_API_ROOT_FACADE_HITS_FILE="${TMP_DIR}/rustfs_admin_storage_api_root_facade_hits.txt" RUSTFS_STORAGE_DIRECT_APP_CONTEXT_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_storage_direct_app_context_bypass_hits.txt" awk ' @@ -1500,6 +1501,24 @@ if [[ -s "$RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE" ]]; then report_failure "RustFS app/admin runtime and data-usage facades must stay behind local storage_api boundaries: $(paste -sd '; ' "$RUSTFS_APP_RUNTIME_STORAGE_API_BYPASS_HITS_FILE")" fi +( + cd "$ROOT_DIR" + { + rg -n --with-filename \ + '\b(?:ecstore_(?:bucket|capacity|client|cluster|config|disk|error|layout|metrics|notification|rebalance|rpc|storage|tier)|AdminReplicationConfigExt|AdminVersioningConfigExt|RUSTFS_META_BUCKET|STORAGE_CLASS_SUB_SYS|read_admin_config|read_admin_config_without_migrate|save_admin_config|save_admin_server_config|delete_admin_config|init_admin_config_defaults|StorageError|ECStore|Endpoint|Endpoints|EndpointServerPools|PoolEndpoints|PeerRestClient|RebalanceStats|RebalanceMeta|RebalanceStopPropagationRecord)\b' \ + rustfs/src/admin/mod.rs || true + rg -n --with-filename \ + '(?:crate::admin|super(?::super)+)::(?:bandwidth|bucket_target_sys|lifecycle|metadata|metadata_sys|quota|replication|target|versioning|versioning_sys|storageclass|tier|AdminReplicationConfigExt|AdminVersioningConfigExt|RUSTFS_META_BUCKET|STORAGE_CLASS_SUB_SYS|read_admin_config|read_admin_config_without_migrate|save_admin_config|save_admin_server_config|delete_admin_config|init_admin_config_defaults|StorageError|Error|ECStore|Endpoint|Endpoints|EndpointServerPools|PoolEndpoints|PeerRestClient|RebalanceStats|RebalanceMeta|RebalanceStopPropagationRecord|collect_local_metrics|is_reserved_or_invalid_bucket)\b' \ + rustfs/src/admin \ + -g '*.rs' \ + -g '!storage_api.rs' || true + } +) >"$RUSTFS_ADMIN_STORAGE_API_ROOT_FACADE_HITS_FILE" + +if [[ -s "$RUSTFS_ADMIN_STORAGE_API_ROOT_FACADE_HITS_FILE" ]]; then + report_failure "RustFS admin storage facades must stay behind rustfs/src/admin/storage_api.rs: $(paste -sd '; ' "$RUSTFS_ADMIN_STORAGE_API_ROOT_FACADE_HITS_FILE")" +fi + ( cd "$ROOT_DIR" rg -n --with-filename 'crate::app::context::|use crate::app::context|app::context::' rustfs/src/storage --glob '*.rs' |