gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 16:28:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor: clean app storage admin runtime boundaries (#3572)

Browse Source
This commit is contained in:
安正超
2026-06-18 16:42:51 +08:00
committed by GitHub
parent bdb2461b55
commit 8313767f75
49 changed files with 491 additions and 360 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/migration-progress.md
+49 -33
View File
@@ -5,19 +5,17 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
## Current Context
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
- Branch: `overtrue/arch-storage-dto-consumer-boundaries`
- Baseline: `main` at `99941f7e7c5e0c88532a93cc175a3ea4111d7098`
after `rustfs/rustfs#3565` merged.
- Branch: `overtrue/arch-app-storage-admin-runtime-boundaries`
- Baseline: rebased onto `origin/main` at
`bdb2461b5594f736cbbcd3788f12d39966c2ec78` after
`rustfs/rustfs#3571` merged.
- PR type for this branch: `consumer-migration`
- Runtime behavior changes: no migration behavior change expected; CI follow-up
preserves empty-object erasure recovery by avoiding zero-byte SIMD decode.
- Rust code changes: add crate-local semantic aliases for ECStore-owned object
metadata, options, readers, and delete DTOs in scanner, heal, notify, Swift,
S3 Select, and RustFS storage/app consumers so external call sites stop
importing raw `rustfs_ecstore::store_api` DTOs outside deliberate boundary
files. ECStore-owned DTO definitions and runtime behavior stay in ECStore.
CI follow-up handles zero-length shards in erasure reconstruction without
changing non-empty shard behavior.
- Rust code changes: add app-, storage-, and admin-local ECStore compatibility
boundaries and route existing direct runtime imports through those deliberate
boundary modules. ECStore-owned definitions and runtime behavior stay in
ECStore.
- CI/script changes: none.
- Docs changes: record the larger DTO consumer-boundary cleanup slice and the
empty-object erasure recovery CI follow-up.
@@ -864,6 +862,26 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
migration/layer guards, formatting, diff hygiene, Rust risk scan, full
pre-commit, and required three-expert review passed.
- [x] `API-030` Clean app, storage, and admin ECStore runtime boundaries.
- Current branch: `overtrue/arch-app-storage-admin-runtime-boundaries`.
- Completed slice: add crate-local app, storage, and admin compatibility
boundary modules for ECStore-owned runtime contracts, then migrate direct
`rustfs_ecstore` imports in `rustfs/src/app`, `rustfs/src/storage`, and
`rustfs/src/admin` through those boundary modules.
- Acceptance: direct `rustfs_ecstore` references in app/storage/admin source
are limited to the local compatibility boundary modules; app, storage, and
admin business/test modules consume local compatibility names.
- Must preserve: app object/bucket/multipart/admin usecase behavior, storage
ECFS/access/SSE/RPC behavior, admin route/handler/service behavior,
metadata serialization, encryption handling, authorization, and existing
ECStore-owned concrete type ownership.
- Risk defense: this slice changes import ownership only; it does not move
ECStore definitions, alter runtime control flow, adjust route registration,
change storage I/O, mutate metadata formats, or alter admin authorization.
- Verification: direct app/storage/admin import scan, RustFS test compile
check, migration/layer guards, formatting, diff hygiene, Rust risk scan,
full pre-commit, and required three-expert review passed.
## Phase 8 Background Controller Tasks
- [x] `BGC-001` Inventory background services.
@@ -1133,47 +1151,45 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
## Next PRs
1. `pure-move`/`consumer-migration`: continue larger cleanup slices with the
loss-prevention guards active for remaining storage compatibility contracts
around app/storage/admin runtime boundaries.
loss-prevention guards active for remaining ECStore compatibility contracts
outside the app/storage/admin scanner/heal/Swift boundaries already cleaned.
## Pre-Push Review Log
| Expert | Status | Notes |
|---|---|---|
| Quality/architecture | passed | Scanner and heal source modules now use crate-local compatibility boundaries for ECStore runtime, disk, metadata, lifecycle, replication, config, and error imports. |
| Migration preservation | passed | ECStore remains the owner of scanner/heal backing types and behavior; scanner cache/lifecycle/replication scans and heal object/bucket/format/resume semantics are preserved. |
| Testing/verification | passed | Focused scanner/heal compile/tests, direct source import scans, migration/layer guards, formatting, diff hygiene, Rust risk scan, and full `make pre-commit` passed. |
| Quality/architecture | passed | App, storage, and admin source modules now use local compatibility boundaries for ECStore runtime imports. |
| Migration preservation | passed | ECStore remains the owner of backing types and behavior; app usecases, storage ECFS/SSE/RPC, and admin handlers/services keep existing semantics. |
| Testing/verification | passed | Direct app/storage/admin import scan, RustFS test compile check, migration/layer guards, formatting, diff hygiene, Rust risk scan, and full `make pre-commit` passed. |
## Verification Notes
Passed before push:
- `cargo check --tests -p rustfs-scanner -p rustfs-heal`: passed.
- `cargo test -p rustfs-scanner -p rustfs-heal`: passed.
- `rg -n 'rustfs_ecstore' crates/scanner/src --glob '*.rs'`: remaining matches
are deliberate scanner compatibility boundary definitions.
- `rg -n 'rustfs_ecstore' crates/heal/src --glob '*.rs'`: remaining matches are
deliberate heal compatibility boundary definitions.
- `cargo check --tests -p rustfs`: passed.
- `rg -n 'rustfs_ecstore' rustfs/src/app rustfs/src/storage rustfs/src/admin --glob '*.rs'`:
remaining matches are deliberate app/storage/admin compatibility boundary
definitions.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- Rust risk scan: no new `unwrap`/`expect`, panic/todo markers, `unsafe`,
process-spawning calls, println/eprintln, relaxed ordering, or numeric casts
in added Rust lines.
- Rust risk scan: reviewed mechanical path-only hits on existing
`unwrap`/`expect` and numeric-cast lines; no new risky logic was introduced.
- `make pre-commit`: passed.
Notes:
- This slice builds on the merged API-028 compatibility cleanup.
- Direct scanner/heal ECStore imports now remain only in their compatibility
boundary modules.
- The slice does not alter scanner runtime behavior, heal runtime behavior,
object I/O behavior, disk operations, metadata serialization, or ECStore
definitions.
- This slice was prepared on the API-029 boundary and rebased onto main after
`rustfs/rustfs#3571` merged.
- Direct app/storage/admin ECStore imports now remain only in their
compatibility boundary modules.
- The slice does not alter app usecase behavior, storage behavior, admin
behavior, object I/O behavior, metadata serialization, or ECStore definitions.
## Handoff Notes
- Continue with larger consumer-migration batches around app/storage/admin
runtime boundaries; keep ECStore-owned behavior in ECStore until concrete
behavior is isolated enough for a pure-move slice.
- Continue with larger consumer-migration batches outside the cleaned
app/storage/admin/scanner/heal/Swift runtime boundaries; keep ECStore-owned
behavior in ECStore until concrete behavior is isolated enough for a
pure-move slice.
rustfs/src/admin/handlers/account_info.rs
+1 -1
View File
@@ -14,6 +14,7 @@
use crate::admin::auth::authenticate_request;
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::storage_compat::ecstore::bucket::versioning_sys::BucketVersioningSys;
use crate::app::context::resolve_object_store_handle;
use crate::auth::get_condition_values;
use crate::server::{ADMIN_PREFIX, RemoteAddr};
@@ -21,7 +22,6 @@ use http::{HeaderMap, HeaderValue};
use hyper::{Method, StatusCode};
use matchit::Params;
use rustfs_credentials::get_global_action_cred;
use rustfs_ecstore::bucket::versioning_sys::BucketVersioningSys;
use rustfs_policy::policy::BucketPolicy;
use rustfs_policy::policy::default::DEFAULT_POLICIES;
use rustfs_policy::policy::{Args, action::Action, action::S3Action};
rustfs/src/admin/handlers/audit_runtime_config.rs
+3 -3
View File
@@ -24,7 +24,7 @@ pub(crate) async fn load_server_config_from_store() -> S3Result<Config> {
return Ok(Config::new());
};
rustfs_ecstore::config::com::read_config_without_migrate(store)
crate::admin::storage_compat::ecstore::config::com::read_config_without_migrate(store)
.await
.map_err(|e| s3_error!(InternalError, "failed to read server config: {}", e))
}
@@ -82,7 +82,7 @@ where
return Err(s3_error!(InternalError, "server storage not initialized"));
};
let mut config = rustfs_ecstore::config::com::read_config_without_migrate(store.clone())
let mut config = crate::admin::storage_compat::ecstore::config::com::read_config_without_migrate(store.clone())
.await
.map_err(|e| s3_error!(InternalError, "failed to read server config: {}", e))?;
@@ -90,7 +90,7 @@ where
return Ok(());
}
rustfs_ecstore::config::com::save_server_config(store, &config)
crate::admin::storage_compat::ecstore::config::com::save_server_config(store, &config)
.await
.map_err(|e| s3_error!(InternalError, "failed to save audit config: {}", e))?;
rustfs/src/admin/handlers/bucket_meta.rs
+14 -14
View File
@@ -12,6 +12,20 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::admin::storage_compat::ecstore::bucket::utils::{deserialize, serialize};
use crate::admin::storage_compat::ecstore::{
bucket::{
metadata::{
BUCKET_LIFECYCLE_CONFIG, BUCKET_NOTIFICATION_CONFIG, BUCKET_POLICY_CONFIG, BUCKET_QUOTA_CONFIG_FILE,
BUCKET_REPLICATION_CONFIG, BUCKET_SSECONFIG, BUCKET_TAGGING_CONFIG, BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG,
BucketMetadata, OBJECT_LOCK_CONFIG,
},
metadata_sys,
quota::BucketQuota,
target::BucketTargets,
},
error::StorageError,
};
use crate::{
admin::{
auth::validate_admin_request,
@@ -25,20 +39,6 @@ use http::{HeaderMap, StatusCode};
use hyper::Method;
use matchit::Params;
use rustfs_config::MAX_BUCKET_METADATA_IMPORT_SIZE;
use rustfs_ecstore::bucket::utils::{deserialize, serialize};
use rustfs_ecstore::{
bucket::{
metadata::{
BUCKET_LIFECYCLE_CONFIG, BUCKET_NOTIFICATION_CONFIG, BUCKET_POLICY_CONFIG, BUCKET_QUOTA_CONFIG_FILE,
BUCKET_REPLICATION_CONFIG, BUCKET_SSECONFIG, BUCKET_TAGGING_CONFIG, BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG,
BucketMetadata, OBJECT_LOCK_CONFIG,
},
metadata_sys,
quota::BucketQuota,
target::BucketTargets,
},
error::StorageError,
};
use rustfs_policy::policy::{
BucketPolicy,
action::{Action, AdminAction},
rustfs/src/admin/handlers/config_admin.rs
+16 -14
View File
@@ -18,6 +18,12 @@ use crate::admin::service::config::{
apply_dynamic_config_for_subsystem, is_dynamic_config_subsystem, signal_config_snapshot_reload, signal_dynamic_config_reload,
validate_server_config,
};
use crate::admin::storage_compat::ecstore::config::com::STORAGE_CLASS_SUB_SYS;
use crate::admin::storage_compat::ecstore::config::com::{
delete_config, read_config, read_config_without_migrate, save_config, save_server_config,
};
use crate::admin::storage_compat::ecstore::config::storageclass::{INLINE_BLOCK_ENV, OPTIMIZE_ENV, RRS_ENV, STANDARD_ENV};
use crate::admin::storage_compat::ecstore::disk::RUSTFS_META_BUCKET;
use crate::admin::utils::{encode_compatible_admin_payload, is_compat_admin_request, read_compatible_admin_body};
use crate::app::context::resolve_object_store_handle;
use crate::auth::{check_key_valid, get_session_token};
@@ -69,10 +75,6 @@ use rustfs_config::{
WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
};
use rustfs_credentials::Credentials;
use rustfs_ecstore::config::com::STORAGE_CLASS_SUB_SYS;
use rustfs_ecstore::config::com::{delete_config, read_config, read_config_without_migrate, save_config, save_server_config};
use rustfs_ecstore::config::storageclass::{INLINE_BLOCK_ENV, OPTIMIZE_ENV, RRS_ENV, STANDARD_ENV};
use rustfs_ecstore::disk::RUSTFS_META_BUCKET;
use rustfs_policy::policy::action::{Action, AdminAction};
use rustfs_storage_api::ListOperations as _;
use s3s::header::CONTENT_TYPE;
@@ -709,7 +711,7 @@ fn success_response(config_applied: bool) -> S3Result<S3Response<(StatusCode, Bo
Ok(S3Response::with_headers((StatusCode::OK, Body::default()), headers))
}
fn object_store() -> S3Result<std::sync::Arc<rustfs_ecstore::store::ECStore>> {
fn object_store() -> S3Result<std::sync::Arc<crate::admin::storage_compat::ecstore::store::ECStore>> {
resolve_object_store_handle().ok_or_else(|| s3_error!(InternalError, "server storage not initialized"))
}
@@ -754,7 +756,7 @@ fn config_update_sub_system(directives: &[ConfigDirective]) -> S3Result<Option<&
fn validate_config_directives(directives: &[ConfigDirective]) -> S3Result<()> {
if DEFAULT_KVS.get().is_none() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
}
let Some(defaults) = DEFAULT_KVS.get() else {
return Err(s3_error!(InternalError, "config defaults are not initialized"));
@@ -1408,7 +1410,7 @@ fn env_help_key(sub_system: &str, key: &str) -> String {
fn default_help_postfix(sub_system: &str, key: &str) -> String {
if DEFAULT_KVS.get().is_none() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
}
DEFAULT_KVS
@@ -1895,7 +1897,7 @@ mod tests {
#[test]
fn full_config_export_can_be_reapplied() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
let mut original = ServerConfig::new();
apply_set_directives(
&mut original,
@@ -1942,7 +1944,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#,
#[test]
fn build_help_response_appends_default_value_postfix() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
let response = build_help_response(Some("identity_openid"), Some("scopes"), false).expect("help response");
assert_eq!(response.keys_help.len(), 2);
@@ -2055,7 +2057,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#,
#[test]
fn render_selected_config_includes_env_override_lines() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
temp_env::with_vars(
[
("RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY", Some("http://env.example")),
@@ -2091,7 +2093,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#,
#[test]
fn render_selected_config_lists_env_only_targets() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
temp_env::with_vars([("RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY", Some("http://env.example"))], || {
let config = ServerConfig::new();
let rendered = String::from_utf8(
@@ -2114,7 +2116,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#,
#[test]
fn render_selected_config_supports_specific_env_only_target_queries() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
temp_env::with_vars([("RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY", Some("http://env.example"))], || {
let config = ServerConfig::new();
let rendered = String::from_utf8(
@@ -2137,7 +2139,7 @@ identity_openid config_url="https://issuer.example" client_id="console""#,
#[test]
fn render_selected_config_orders_default_before_named_targets() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
temp_env::with_vars([("RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_ALPHA", Some("http://alpha.example"))], || {
let mut config = ServerConfig::new();
apply_set_directives(
@@ -2318,7 +2320,7 @@ identity_openid client_id="existing-client""#,
#[test]
fn storage_class_get_target_none_matches_full_export() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
let mut config = ServerConfig::new();
apply_set_directives(
&mut config,
rustfs/src/admin/handlers/heal.rs
+5 -5
View File
@@ -14,6 +14,8 @@
use crate::admin::auth::{authenticate_request, validate_admin_request};
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::storage_compat::ecstore::bucket::utils::is_valid_object_prefix;
use crate::admin::storage_compat::ecstore::store_utils::is_reserved_or_invalid_bucket;
use crate::app::context::resolve_object_store_handle;
use crate::server::ADMIN_PREFIX;
use crate::server::RemoteAddr;
@@ -24,8 +26,6 @@ use hyper::{Method, StatusCode};
use matchit::Params;
use rustfs_common::heal_channel::{HealChannelPriority, HealChannelRequest, HealOpts, HealRequestSource};
use rustfs_config::MAX_HEAL_REQUEST_SIZE;
use rustfs_ecstore::bucket::utils::is_valid_object_prefix;
use rustfs_ecstore::store_utils::is_reserved_or_invalid_bucket;
use rustfs_policy::policy::action::{Action, AdminAction};
use rustfs_scanner::scanner::{BackgroundHealInfo, read_background_heal_info};
use rustfs_storage_api::HealOperations as _;
@@ -345,10 +345,10 @@ fn should_handle_root_heal_directly(hip: &HealInitParams) -> bool {
hip.bucket.is_empty() && hip.obj_prefix.is_empty() && hip.client_token.is_empty() && !hip.force_stop
}
fn map_root_heal_status(heal_err: Option<rustfs_ecstore::error::Error>) -> S3Result<()> {
fn map_root_heal_status(heal_err: Option<crate::admin::storage_compat::ecstore::error::Error>) -> S3Result<()> {
match heal_err {
None => Ok(()),
Some(rustfs_ecstore::error::StorageError::NoHealRequired) => {
Some(crate::admin::storage_compat::ecstore::error::StorageError::NoHealRequired) => {
info!(
event = EVENT_ADMIN_RESPONSE_EMITTED,
component = LOG_COMPONENT_ADMIN_API,
@@ -703,12 +703,12 @@ mod tests {
encode_heal_task_status, heal_channel_response_items, heal_channel_response_summary, json_response, map_heal_response,
map_root_heal_status, should_handle_root_heal_directly, validate_heal_request_mode, validate_heal_target,
};
use crate::admin::storage_compat::ecstore::error::StorageError;
use bytes::Bytes;
use http::StatusCode;
use http::Uri;
use matchit::Router;
use rustfs_common::heal_channel::{HealChannelPriority, HealOpts, HealRequestSource, HealScanMode};
use rustfs_ecstore::error::StorageError;
use rustfs_scanner::scanner::BackgroundHealInfo;
use s3s::{
S3ErrorCode,
rustfs/src/admin/handlers/kms_dynamic.rs
+1 -1
View File
@@ -16,13 +16,13 @@
use crate::admin::auth::validate_admin_request;
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::storage_compat::ecstore::config::com::{read_config, save_config};
use crate::app::context::{resolve_kms_runtime_service_manager, resolve_object_store_handle};
use crate::auth::{check_key_valid, get_session_token};
use crate::server::{ADMIN_PREFIX, RemoteAddr};
use hyper::{Method, StatusCode};
use matchit::Params;
use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
use rustfs_ecstore::config::com::{read_config, save_config};
use rustfs_kms::{
ConfigureKmsRequest, ConfigureKmsResponse, KmsConfig, KmsConfigSummary, KmsServiceStatus, KmsStatusResponse, StartKmsRequest,
StartKmsResponse, StopKmsResponse,
rustfs/src/admin/handlers/metrics.rs
+1 -1
View File
@@ -20,6 +20,7 @@
use crate::admin::auth::validate_admin_request;
use crate::admin::router::Operation;
use crate::admin::storage_compat::ecstore::metrics_realtime::{CollectMetricsOpts, MetricType, collect_local_metrics};
use crate::auth::{check_key_valid, get_session_token};
use crate::server::RemoteAddr;
use crate::storage::request_context::spawn_traced;
@@ -28,7 +29,6 @@ use futures::{Stream, StreamExt};
use http::{HeaderMap, HeaderValue, Uri};
use hyper::StatusCode;
use matchit::Params;
use rustfs_ecstore::metrics_realtime::{CollectMetricsOpts, MetricType, collect_local_metrics};
use rustfs_madmin::metrics::RealtimeMetrics;
use rustfs_madmin::utils::parse_duration;
use rustfs_policy::policy::action::{Action, AdminAction};
rustfs/src/admin/handlers/object_zip_download.rs
+2 -2
View File
@@ -13,6 +13,7 @@
// limitations under the License.
use crate::admin::router::{ADMIN_OBJECT_ZIP_DOWNLOADS_PATH, AdminOperation, Operation, S3Router};
use crate::admin::storage_compat::ecstore::global::get_global_region;
use crate::app::context::resolve_object_store_handle;
use crate::auth::{check_key_valid, get_session_token};
use crate::error::ApiError;
@@ -31,7 +32,6 @@ use matchit::Params;
use rand::RngExt;
use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
use rustfs_credentials::get_global_action_cred;
use rustfs_ecstore::global::get_global_region;
use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_storage_api::{BucketOperations, ListOperations as _, ObjectIO as _, ObjectOperations as _, bucket::BucketOptions};
use rustfs_trusted_proxies::{ClientInfo, ValidationMode};
@@ -649,7 +649,7 @@ async fn preflight_zip_items(request: &CreateObjectZipDownloadRequest, items: &[
Ok(())
}
fn storage_error_to_s3(err: rustfs_ecstore::error::Error) -> s3s::S3Error {
fn storage_error_to_s3(err: crate::admin::storage_compat::ecstore::error::Error) -> s3s::S3Error {
ApiError::from(err).into()
}
rustfs/src/admin/handlers/oidc.rs
+1 -1
View File
@@ -15,6 +15,7 @@
use super::sts::create_oidc_sts_credentials;
use crate::admin::auth::validate_admin_request;
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::storage_compat::ecstore::config::com::{read_config_without_migrate, save_server_config};
use crate::app::context::resolve_object_store_handle;
use crate::auth::{check_key_valid, get_session_token};
use crate::server::{ADMIN_PREFIX, MINIO_ADMIN_PREFIX, RemoteAddr};
@@ -30,7 +31,6 @@ use rustfs_config::oidc::{
use rustfs_config::server_config::Config as ServerConfig;
use rustfs_config::server_config::get_global_server_config;
use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState, MAX_ADMIN_REQUEST_BODY_SIZE};
use rustfs_ecstore::config::com::{read_config_without_migrate, save_server_config};
use rustfs_policy::policy::action::{Action, AdminAction};
use rustfs_utils::egress::validate_outbound_url;
use s3s::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
rustfs/src/admin/handlers/pools.rs
+1 -1
View File
@@ -99,7 +99,7 @@ macro_rules! log_pool_response_emitted {
};
}
fn endpoints_from_context() -> Option<rustfs_ecstore::endpoints::EndpointServerPools> {
fn endpoints_from_context() -> Option<crate::admin::storage_compat::ecstore::endpoints::EndpointServerPools> {
resolve_endpoints_handle()
}
rustfs/src/admin/handlers/quota.rs
+4 -4
View File
@@ -16,14 +16,14 @@
use crate::admin::auth::{validate_admin_request, validate_admin_request_with_bucket};
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::storage_compat::ecstore::bucket::metadata_sys::BucketMetadataSys;
use crate::admin::storage_compat::ecstore::bucket::quota::checker::QuotaChecker;
use crate::admin::storage_compat::ecstore::bucket::quota::{BucketQuota, QuotaError, QuotaOperation};
use crate::app::context::{resolve_bucket_metadata_handle, resolve_object_store_handle};
use crate::auth::{check_key_valid, get_session_token};
use crate::server::ADMIN_PREFIX;
use hyper::{Method, StatusCode};
use matchit::Params;
use rustfs_ecstore::bucket::metadata_sys::BucketMetadataSys;
use rustfs_ecstore::bucket::quota::checker::QuotaChecker;
use rustfs_ecstore::bucket::quota::{BucketQuota, QuotaError, QuotaOperation};
use rustfs_policy::policy::action::{Action, AdminAction, S3Action};
use s3s::{Body, S3Request, S3Response, S3Result, s3_error};
use serde::{Deserialize, Serialize};
@@ -175,7 +175,7 @@ async fn current_usage_from_context(bucket: &str) -> u64 {
return 0;
};
match rustfs_ecstore::data_usage::load_data_usage_from_backend(store).await {
match crate::admin::storage_compat::ecstore::data_usage::load_data_usage_from_backend(store).await {
Ok(data_usage_info) => data_usage_info
.buckets_usage
.get(bucket)
rustfs/src/admin/handlers/rebalance.rs
+10 -8
View File
@@ -12,6 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::admin::storage_compat::ecstore::{
error::StorageError,
notification_sys::get_global_notification_sys,
rebalance::{DiskStat, RebalSaveOpt, RebalanceCleanupWarnings, RebalanceMeta},
};
use crate::{
admin::{
auth::validate_admin_request,
@@ -24,11 +29,6 @@ use crate::{
use http::{HeaderMap, HeaderValue, StatusCode};
use hyper::Method;
use matchit::Params;
use rustfs_ecstore::{
error::StorageError,
notification_sys::get_global_notification_sys,
rebalance::{DiskStat, RebalSaveOpt, RebalanceCleanupWarnings, RebalanceMeta},
};
use rustfs_policy::policy::action::{Action, AdminAction};
use rustfs_storage_api::{BucketOperations, BucketOptions, StorageAdminApi};
use s3s::{
@@ -150,7 +150,7 @@ fn build_rebalance_pool_progress(
now: OffsetDateTime,
stop_time: Option<OffsetDateTime>,
percent_free_goal: f64,
ps: &rustfs_ecstore::rebalance::RebalanceStats,
ps: &crate::admin::storage_compat::ecstore::rebalance::RebalanceStats,
) -> Option<RebalPoolProgress> {
let total_bytes_to_rebal = ps.init_capacity as f64 * percent_free_goal - ps.init_free_space as f64;
let terminal_time = ps.info.end_time.or(stop_time);
@@ -193,7 +193,7 @@ fn build_rebalance_pool_statuses(
now: OffsetDateTime,
stop_time: Option<OffsetDateTime>,
percent_free_goal: f64,
pool_stats: &[rustfs_ecstore::rebalance::RebalanceStats],
pool_stats: &[crate::admin::storage_compat::ecstore::rebalance::RebalanceStats],
disk_stats: &[DiskStat],
) -> Vec<RebalancePoolStatus> {
pool_stats
@@ -563,7 +563,9 @@ mod rebalance_handler_tests {
RebalPoolProgress, RebalanceAdminStatus, RebalancePoolStatus, build_rebalance_pool_statuses, rebalance_pool_used,
rebalance_remaining_buckets, rebalance_used_pct,
};
use rustfs_ecstore::rebalance::{DiskStat, RebalStatus, RebalanceCleanupWarnings, RebalanceInfo, RebalanceStats};
use crate::admin::storage_compat::ecstore::rebalance::{
DiskStat, RebalStatus, RebalanceCleanupWarnings, RebalanceInfo, RebalanceStats,
};
use time::OffsetDateTime;
#[test]
rustfs/src/admin/handlers/replication.rs
+9 -9
View File
@@ -15,6 +15,15 @@
use crate::admin::auth::validate_admin_request;
use crate::admin::handlers::site_replication::site_replication_peer_deployment_id_for_endpoint;
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::storage_compat::ecstore::bucket::bucket_target_sys::{BucketTargetError, BucketTargetSys};
use crate::admin::storage_compat::ecstore::bucket::metadata::BUCKET_TARGETS_FILE;
use crate::admin::storage_compat::ecstore::bucket::metadata_sys;
use crate::admin::storage_compat::ecstore::bucket::metadata_sys::get_replication_config;
use crate::admin::storage_compat::ecstore::bucket::replication::BucketStats;
use crate::admin::storage_compat::ecstore::bucket::replication::GLOBAL_REPLICATION_STATS;
use crate::admin::storage_compat::ecstore::bucket::target::BucketTarget;
use crate::admin::storage_compat::ecstore::error::StorageError;
use crate::admin::storage_compat::ecstore::global::global_rustfs_port;
use crate::admin::utils::read_compatible_admin_body;
use crate::app::context::resolve_object_store_handle;
use crate::auth::{check_key_valid, get_session_token};
@@ -25,15 +34,6 @@ use hyper::{Method, StatusCode};
use matchit::Params;
use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
use rustfs_credentials::Credentials;
use rustfs_ecstore::bucket::bucket_target_sys::{BucketTargetError, BucketTargetSys};
use rustfs_ecstore::bucket::metadata::BUCKET_TARGETS_FILE;
use rustfs_ecstore::bucket::metadata_sys;
use rustfs_ecstore::bucket::metadata_sys::get_replication_config;
use rustfs_ecstore::bucket::replication::BucketStats;
use rustfs_ecstore::bucket::replication::GLOBAL_REPLICATION_STATS;
use rustfs_ecstore::bucket::target::BucketTarget;
use rustfs_ecstore::error::StorageError;
use rustfs_ecstore::global::global_rustfs_port;
use rustfs_policy::policy::action::{Action, AdminAction};
use rustfs_storage_api::{BucketOperations, BucketOptions};
use s3s::header::CONTENT_TYPE;
rustfs/src/admin/handlers/site_replication.rs
+22 -18
View File
@@ -18,6 +18,24 @@ use crate::admin::site_replication_identity::{
canonical_endpoint, deployment_id_for_endpoint, normalize_peer_map_by_identity_with, same_identity_endpoint,
site_identity_key,
};
use crate::admin::storage_compat::ecstore::bucket::bucket_target_sys::BucketTargetSys;
use crate::admin::storage_compat::ecstore::bucket::metadata::{
BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_POLICY_CONFIG, BUCKET_QUOTA_CONFIG_FILE, BUCKET_REPLICATION_CONFIG,
BUCKET_SSECONFIG, BUCKET_TAGGING_CONFIG, BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG, OBJECT_LOCK_CONFIG,
};
use crate::admin::storage_compat::ecstore::bucket::metadata_sys;
use crate::admin::storage_compat::ecstore::bucket::replication::GLOBAL_REPLICATION_STATS;
use crate::admin::storage_compat::ecstore::bucket::replication::{
ReplicationConfigurationExt, ResyncOpts, get_global_replication_pool,
};
use crate::admin::storage_compat::ecstore::bucket::target::{ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials};
use crate::admin::storage_compat::ecstore::bucket::utils::{deserialize, serialize};
use crate::admin::storage_compat::ecstore::bucket::versioning::VersioningApi;
use crate::admin::storage_compat::ecstore::config::com::{delete_config, read_config, save_config};
use crate::admin::storage_compat::ecstore::error::Error as StorageError;
use crate::admin::storage_compat::ecstore::global::{
get_global_deployment_id, get_global_endpoints_opt, get_global_region, global_rustfs_port,
};
use crate::admin::utils::{encode_compatible_admin_payload, read_compatible_admin_body};
use crate::app::context::resolve_object_store_handle;
use crate::auth::{check_key_valid, get_session_token};
@@ -36,20 +54,6 @@ use rustfs_config::{
DEFAULT_CONSOLE_ADDRESS, DEFAULT_DELIMITER, DEFAULT_RUSTFS_TLS_PATH, ENV_RUSTFS_CONSOLE_ADDRESS, ENV_RUSTFS_TLS_PATH,
MAX_ADMIN_REQUEST_BODY_SIZE,
};
use rustfs_ecstore::bucket::bucket_target_sys::BucketTargetSys;
use rustfs_ecstore::bucket::metadata::{
BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_POLICY_CONFIG, BUCKET_QUOTA_CONFIG_FILE, BUCKET_REPLICATION_CONFIG,
BUCKET_SSECONFIG, BUCKET_TAGGING_CONFIG, BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG, OBJECT_LOCK_CONFIG,
};
use rustfs_ecstore::bucket::metadata_sys;
use rustfs_ecstore::bucket::replication::GLOBAL_REPLICATION_STATS;
use rustfs_ecstore::bucket::replication::{ReplicationConfigurationExt, ResyncOpts, get_global_replication_pool};
use rustfs_ecstore::bucket::target::{ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials};
use rustfs_ecstore::bucket::utils::{deserialize, serialize};
use rustfs_ecstore::bucket::versioning::VersioningApi;
use rustfs_ecstore::config::com::{delete_config, read_config, save_config};
use rustfs_ecstore::error::Error as StorageError;
use rustfs_ecstore::global::{get_global_deployment_id, get_global_endpoints_opt, get_global_region, global_rustfs_port};
use rustfs_iam::error::is_err_no_such_service_account;
use rustfs_iam::store::{MappedPolicy, UserType};
use rustfs_iam::sys::{
@@ -652,7 +656,7 @@ async fn site_replication_peer_client() -> S3Result<reqwest::Client> {
built
}
fn runtime_tls_enabled_with(endpoints: Option<&rustfs_ecstore::endpoints::EndpointServerPools>) -> bool {
fn runtime_tls_enabled_with(endpoints: Option<&crate::admin::storage_compat::ecstore::endpoints::EndpointServerPools>) -> bool {
if !rustfs_utils::get_env_str(ENV_RUSTFS_TLS_PATH, DEFAULT_RUSTFS_TLS_PATH).is_empty() {
return true;
}
@@ -3353,7 +3357,7 @@ fn is_stale_update(local_updated_at: OffsetDateTime, incoming_updated_at: Option
}
fn bucket_meta_local_updated_at(
bucket_meta: &rustfs_ecstore::bucket::metadata::BucketMetadata,
bucket_meta: &crate::admin::storage_compat::ecstore::bucket::metadata::BucketMetadata,
config_file: &str,
) -> OffsetDateTime {
match config_file {
@@ -4574,10 +4578,10 @@ impl Operation for SRRotateServiceAccountHandler {
#[cfg(test)]
mod tests {
use super::*;
use crate::admin::storage_compat::ecstore::disk::endpoint::Endpoint;
use crate::admin::storage_compat::ecstore::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints};
use http::{HeaderMap, HeaderValue, Uri};
use rustfs_common::{get_global_outbound_tls_generation, set_global_outbound_tls_generation};
use rustfs_ecstore::disk::endpoint::Endpoint;
use rustfs_ecstore::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints};
use rustfs_policy::policy::action::S3Action;
use serial_test::serial;
use temp_env::with_var;
rustfs/src/admin/handlers/sts.rs
+1 -1
View File
@@ -13,6 +13,7 @@
// limitations under the License.
use super::is_admin::IsAdminHandler;
use crate::admin::storage_compat::ecstore::bucket::utils::serialize;
use crate::{
admin::{
handlers::site_replication::site_replication_iam_change_hook,
@@ -28,7 +29,6 @@ use hyper::Method;
use matchit::Params;
use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
use rustfs_credentials::get_global_action_cred;
use rustfs_ecstore::bucket::utils::serialize;
use rustfs_iam::{manager::get_token_signing_key, oidc::OidcClaims, sys::SESSION_POLICY_NAME};
use rustfs_madmin::{SITE_REPL_API_VERSION, SRIAMItem, SRSTSCredential};
use rustfs_policy::{
rustfs/src/admin/handlers/table_catalog.rs
+4 -4
View File
@@ -12,6 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::admin::storage_compat::ecstore::{
bucket::{metadata::table_catalog_path_hash, metadata_sys},
store::ECStore,
};
use crate::admin::{
auth::{AdminResourceScope, validate_admin_request, validate_admin_request_with_bucket_object},
router::{AdminOperation, Operation, S3Router},
@@ -25,10 +29,6 @@ use hyper::Method;
use matchit::Params;
use metrics::{counter, histogram};
use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
use rustfs_ecstore::{
bucket::{metadata::table_catalog_path_hash, metadata_sys},
store::ECStore,
};
use rustfs_iam::{manager::get_token_signing_key, sys::SESSION_POLICY_NAME};
use rustfs_policy::{
auth::get_new_credentials_with_metadata,
rustfs/src/admin/handlers/tier.rs
+17 -17
View File
@@ -13,6 +13,22 @@
// limitations under the License.
#![allow(unused_variables, unused_mut, unused_must_use)]
use crate::admin::storage_compat::ecstore::bucket::lifecycle::bucket_lifecycle_ops::GLOBAL_TransitionState;
use crate::admin::storage_compat::ecstore::{
bucket::lifecycle::tier_last_day_stats::DailyAllTierStats,
client::admin_handler_utils::AdminError,
config::storageclass,
notification_sys::get_global_notification_sys,
tier::{
tier::{ERR_TIER_BACKEND_IN_USE, ERR_TIER_BACKEND_NOT_EMPTY, ERR_TIER_MISSING_CREDENTIALS},
tier_admin::TierCreds,
tier_config::{TierConfig, TierType},
tier_handlers::{
ERR_TIER_ALREADY_EXISTS, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_NAME_NOT_UPPERCASE,
ERR_TIER_NOT_FOUND,
},
},
};
use crate::{
admin::{
auth::validate_admin_request,
@@ -30,22 +46,6 @@ use matchit::Params;
use percent_encoding::percent_decode_str;
use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
use rustfs_data_usage::TierStats;
use rustfs_ecstore::bucket::lifecycle::bucket_lifecycle_ops::GLOBAL_TransitionState;
use rustfs_ecstore::{
bucket::lifecycle::tier_last_day_stats::DailyAllTierStats,
client::admin_handler_utils::AdminError,
config::storageclass,
notification_sys::get_global_notification_sys,
tier::{
tier::{ERR_TIER_BACKEND_IN_USE, ERR_TIER_BACKEND_NOT_EMPTY, ERR_TIER_MISSING_CREDENTIALS},
tier_admin::TierCreds,
tier_config::{TierConfig, TierType},
tier_handlers::{
ERR_TIER_ALREADY_EXISTS, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_NAME_NOT_UPPERCASE,
ERR_TIER_NOT_FOUND,
},
},
};
use rustfs_policy::policy::action::{Action, AdminAction};
use s3s::{
Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result,
@@ -916,9 +916,9 @@ impl Operation for ClearTier {
#[cfg(test)]
mod tests {
use super::*;
use crate::admin::storage_compat::ecstore::bucket::lifecycle::tier_last_day_stats::LastDayTierStats;
use http::Uri;
use matchit::Router;
use rustfs_ecstore::bucket::lifecycle::tier_last_day_stats::LastDayTierStats;
#[test]
fn resolve_tier_name_prefers_path_parameter() {
rustfs/src/admin/handlers/trace.rs
+1 -1
View File
@@ -13,11 +13,11 @@
// limitations under the License.
use crate::admin::router::Operation;
use crate::admin::storage_compat::ecstore::rpc::PeerRestClient;
use crate::app::context::resolve_endpoints_handle;
use http::StatusCode;
use hyper::Uri;
use matchit::Params;
use rustfs_ecstore::rpc::PeerRestClient;
use rustfs_madmin::service_commands::ServiceTraceOpts;
use s3s::{Body, S3Request, S3Response, S3Result, s3_error};
rustfs/src/admin/mod.rs
+1
View File
@@ -23,6 +23,7 @@ pub(crate) mod route_policy;
pub mod router;
pub mod service;
pub mod site_replication_identity;
pub(crate) mod storage_compat;
pub mod utils;
#[cfg(test)]
rustfs/src/admin/router.rs
+32 -30
View File
@@ -14,6 +14,24 @@
use crate::admin::console::{is_console_path, make_console_server};
use crate::admin::handlers::oidc::is_oidc_path;
use crate::admin::storage_compat::ecstore::bucket::bandwidth::monitor::BandwidthDetails;
use crate::admin::storage_compat::ecstore::bucket::bucket_target_sys::{
BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient,
};
use crate::admin::storage_compat::ecstore::bucket::metadata::BUCKET_TARGETS_FILE;
use crate::admin::storage_compat::ecstore::bucket::metadata_sys;
use crate::admin::storage_compat::ecstore::bucket::replication::{
BucketReplicationResyncStatus, BucketStats, GLOBAL_REPLICATION_STATS, ObjectOpts, ReplicationConfigurationExt, ResyncOpts,
get_global_replication_pool,
};
use crate::admin::storage_compat::ecstore::bucket::target::{BucketTarget, BucketTargetType, BucketTargets};
use crate::admin::storage_compat::ecstore::bucket::versioning::VersioningApi;
use crate::admin::storage_compat::ecstore::bucket::versioning_sys::BucketVersioningSys;
use crate::admin::storage_compat::ecstore::config::com::read_config_without_migrate;
use crate::admin::storage_compat::ecstore::global::GLOBAL_BOOT_TIME;
use crate::admin::storage_compat::ecstore::global::{get_global_bucket_monitor, get_global_deployment_id, get_global_region};
use crate::admin::storage_compat::ecstore::notification_sys::get_global_notification_sys;
use crate::admin::storage_compat::ecstore::rpc::PeerRestClient;
use crate::app::context::resolve_object_store_handle;
use crate::app::object_usecase::DefaultObjectUsecase;
use crate::auth::{check_key_valid, get_session_token};
@@ -44,24 +62,6 @@ use rustfs_config::{
ENABLE_KEY, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT,
WEBHOOK_SKIP_TLS_VERIFY,
};
use rustfs_ecstore::bucket::bandwidth::monitor::BandwidthDetails;
use rustfs_ecstore::bucket::bucket_target_sys::{
BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient,
};
use rustfs_ecstore::bucket::metadata::BUCKET_TARGETS_FILE;
use rustfs_ecstore::bucket::metadata_sys;
use rustfs_ecstore::bucket::replication::{
BucketReplicationResyncStatus, BucketStats, GLOBAL_REPLICATION_STATS, ObjectOpts, ReplicationConfigurationExt, ResyncOpts,
get_global_replication_pool,
};
use rustfs_ecstore::bucket::target::{BucketTarget, BucketTargetType, BucketTargets};
use rustfs_ecstore::bucket::versioning::VersioningApi;
use rustfs_ecstore::bucket::versioning_sys::BucketVersioningSys;
use rustfs_ecstore::config::com::read_config_without_migrate;
use rustfs_ecstore::global::GLOBAL_BOOT_TIME;
use rustfs_ecstore::global::{get_global_bucket_monitor, get_global_deployment_id, get_global_region};
use rustfs_ecstore::notification_sys::get_global_notification_sys;
use rustfs_ecstore::rpc::PeerRestClient;
use rustfs_filemeta::{ReplicationStatusType, ReplicationType};
use rustfs_madmin::utils::parse_duration;
use rustfs_notify::{Event as NotificationEvent, notification_system};
@@ -1420,7 +1420,9 @@ async fn ensure_replication_bucket_exists(bucket: &str) -> S3Result<()> {
async fn ensure_replication_config_exists(bucket: &str) -> S3Result<()> {
match metadata_sys::get_replication_config(bucket).await {
Ok(_) => Ok(()),
Err(rustfs_ecstore::error::StorageError::ConfigNotFound) => Err(s3_error!(ReplicationConfigurationNotFoundError)),
Err(crate::admin::storage_compat::ecstore::error::StorageError::ConfigNotFound) => {
Err(s3_error!(ReplicationConfigurationNotFoundError))
}
Err(err) => Err(ApiError::from(err).into()),
}
}
@@ -1945,7 +1947,7 @@ async fn resolve_replication_target_client(bucket: &str, target: &BucketTarget)
fn build_replication_probe_put_options(now: OffsetDateTime) -> PutObjectOptions {
PutObjectOptions {
internal: rustfs_ecstore::bucket::bucket_target_sys::AdvancedPutOptions {
internal: crate::admin::storage_compat::ecstore::bucket::bucket_target_sys::AdvancedPutOptions {
source_version_id: Uuid::new_v4().to_string(),
replication_status: ReplicationStatusType::Replica,
source_mtime: now,
@@ -2060,7 +2062,7 @@ async fn source_bucket_requires_object_lock(bucket: &str) -> S3Result<bool> {
.object_lock_enabled
.as_ref()
.is_some_and(|state| state.as_str() == s3s::dto::ObjectLockEnabled::ENABLED)),
Err(rustfs_ecstore::error::StorageError::ConfigNotFound) => Ok(false),
Err(crate::admin::storage_compat::ecstore::error::StorageError::ConfigNotFound) => Ok(false),
Err(err) => Err(ApiError::from(err).into()),
}
}
@@ -2774,7 +2776,7 @@ mod tests {
#[test]
fn apply_replication_reset_to_targets_updates_matching_target() {
let mut targets = BucketTargets {
targets: vec![rustfs_ecstore::bucket::target::BucketTarget {
targets: vec![crate::admin::storage_compat::ecstore::bucket::target::BucketTarget {
arn: "arn:target".to_string(),
..Default::default()
}],
@@ -2796,10 +2798,10 @@ mod tests {
let mut status = BucketReplicationResyncStatus::new();
status.targets_map.insert(
"arn:z".to_string(),
rustfs_ecstore::bucket::replication::TargetReplicationResyncStatus {
crate::admin::storage_compat::ecstore::bucket::replication::TargetReplicationResyncStatus {
resync_id: "rid-z".to_string(),
last_update: Some(datetime!(2025-01-03 00:00 UTC)),
resync_status: rustfs_ecstore::bucket::replication::ResyncStatusType::ResyncFailed,
resync_status: crate::admin::storage_compat::ecstore::bucket::replication::ResyncStatusType::ResyncFailed,
failed_count: 2,
failed_size: 4,
bucket: "bucket-z".to_string(),
@@ -2809,10 +2811,10 @@ mod tests {
);
status.targets_map.insert(
"arn:a".to_string(),
rustfs_ecstore::bucket::replication::TargetReplicationResyncStatus {
crate::admin::storage_compat::ecstore::bucket::replication::TargetReplicationResyncStatus {
resync_id: "rid-a".to_string(),
last_update: Some(datetime!(2025-01-02 00:00 UTC)),
resync_status: rustfs_ecstore::bucket::replication::ResyncStatusType::ResyncCompleted,
resync_status: crate::admin::storage_compat::ecstore::bucket::replication::ResyncStatusType::ResyncCompleted,
replicated_count: 3,
replicated_size: 9,
bucket: "bucket-a".to_string(),
@@ -2842,10 +2844,10 @@ mod tests {
let mut status = BucketReplicationResyncStatus::new();
status.targets_map.insert(
"arn:z".to_string(),
rustfs_ecstore::bucket::replication::TargetReplicationResyncStatus {
crate::admin::storage_compat::ecstore::bucket::replication::TargetReplicationResyncStatus {
resync_id: "rid-z".to_string(),
last_update: Some(datetime!(2025-02-03 00:00 UTC)),
resync_status: rustfs_ecstore::bucket::replication::ResyncStatusType::ResyncFailed,
resync_status: crate::admin::storage_compat::ecstore::bucket::replication::ResyncStatusType::ResyncFailed,
failed_count: 2,
failed_size: 4,
bucket: "bucket-z".to_string(),
@@ -2855,10 +2857,10 @@ mod tests {
);
status.targets_map.insert(
"arn:a".to_string(),
rustfs_ecstore::bucket::replication::TargetReplicationResyncStatus {
crate::admin::storage_compat::ecstore::bucket::replication::TargetReplicationResyncStatus {
resync_id: "rid-a".to_string(),
last_update: Some(datetime!(2025-02-02 00:00 UTC)),
resync_status: rustfs_ecstore::bucket::replication::ResyncStatusType::ResyncCompleted,
resync_status: crate::admin::storage_compat::ecstore::bucket::replication::ResyncStatusType::ResyncCompleted,
replicated_count: 3,
replicated_size: 9,
bucket: "bucket-a".to_string(),
rustfs/src/admin/service/config.rs
+9 -9
View File
@@ -12,6 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::admin::storage_compat::ecstore::config::com::{STORAGE_CLASS_SUB_SYS, read_config_without_migrate};
use crate::admin::storage_compat::ecstore::config::set_global_storage_class;
use crate::admin::storage_compat::ecstore::config::storageclass;
use crate::admin::storage_compat::ecstore::notification_sys::get_global_notification_sys;
use crate::app::context::resolve_object_store_handle;
use rustfs_audit::reload_audit_config;
use rustfs_config::audit::{AUDIT_MQTT_SUB_SYS, AUDIT_REDIS_DEFAULT_CHANNEL, AUDIT_WEBHOOK_SUB_SYS};
@@ -21,10 +25,6 @@ use rustfs_config::server_config::{Config as ServerConfig, KVS, set_global_serve
use rustfs_config::{AUDIT_DEFAULT_DIR, EVENT_DEFAULT_DIR};
use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState};
use rustfs_config::{HEAL_SUB_SYS, SCANNER_SUB_SYS};
use rustfs_ecstore::config::com::{STORAGE_CLASS_SUB_SYS, read_config_without_migrate};
use rustfs_ecstore::config::set_global_storage_class;
use rustfs_ecstore::config::storageclass;
use rustfs_ecstore::notification_sys::get_global_notification_sys;
use rustfs_iam::oidc::load_oidc_provider_configs_from_server_config;
use rustfs_storage_api::StorageAdminApi;
use rustfs_targets::config::{
@@ -371,11 +371,11 @@ pub async fn signal_config_snapshot_reload() {
#[cfg(test)]
mod tests {
use super::*;
use crate::admin::storage_compat::ecstore::bucket::metadata::{BUCKET_LIFECYCLE_CONFIG, BUCKET_REPLICATION_CONFIG};
use rustfs_config::notify::NOTIFY_WEBHOOK_SUB_SYS;
use rustfs_config::oidc::{OIDC_CLIENT_ID, OIDC_CONFIG_URL, OIDC_SCOPES};
use rustfs_config::{HEAL_SUB_SYS, SCANNER_SUB_SYS};
use rustfs_config::{MQTT_BROKER, MQTT_QUEUE_DIR, MQTT_TOPIC, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR};
use rustfs_ecstore::bucket::metadata::{BUCKET_LIFECYCLE_CONFIG, BUCKET_REPLICATION_CONFIG};
const LIFECYCLE_RELOAD_LABEL: &str = "lifecycle";
const REPLICATION_RELOAD_LABEL: &str = "replication";
@@ -427,7 +427,7 @@ mod tests {
#[test]
fn validate_notify_subsystem_config_rejects_invalid_webhook_endpoint() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
let mut config = ServerConfig::new();
let targets = config.0.get_mut(NOTIFY_WEBHOOK_SUB_SYS).expect("notify webhook defaults");
let kvs = targets.get_mut(DEFAULT_DELIMITER).expect("default target");
@@ -441,7 +441,7 @@ mod tests {
#[test]
fn validate_audit_subsystem_config_rejects_relative_queue_dir() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
let mut config = ServerConfig::new();
let targets = config.0.get_mut(AUDIT_MQTT_SUB_SYS).expect("audit mqtt defaults");
let kvs = targets.get_mut(DEFAULT_DELIMITER).expect("default target");
@@ -456,7 +456,7 @@ mod tests {
#[test]
fn validate_identity_openid_config_rejects_missing_openid_scope() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
let mut config = ServerConfig::new();
let targets = config.0.get_mut(IDENTITY_OPENID_SUB_SYS).expect("openid defaults");
let kvs = targets.get_mut(DEFAULT_DELIMITER).expect("default target");
@@ -473,7 +473,7 @@ mod tests {
#[test]
fn validate_identity_openid_config_rejects_invalid_named_provider_id() {
rustfs_ecstore::config::init();
crate::admin::storage_compat::ecstore::config::init();
let mut config = ServerConfig::new();
let targets = config.0.get_mut(IDENTITY_OPENID_SUB_SYS).expect("openid defaults");
let default_kvs = targets.get(DEFAULT_DELIMITER).cloned().expect("default target");
rustfs/src/admin/service/site_replication.rs
+2 -2
View File
@@ -13,9 +13,9 @@
// limitations under the License.
use crate::admin::site_replication_identity::{deployment_id_for_endpoint, normalize_peer_map_by_identity_with};
use crate::admin::storage_compat::ecstore::config::com::{read_config, save_config};
use crate::admin::storage_compat::ecstore::error::Error as StorageError;
use crate::app::context::resolve_object_store_handle;
use rustfs_ecstore::config::com::{read_config, save_config};
use rustfs_ecstore::error::Error as StorageError;
use rustfs_madmin::PeerInfo;
use s3s::{S3Error, S3ErrorCode, S3Result};
use serde_json::{Map, Value};
rustfs/src/admin/storage_compat.rs
+15
View File
@@ -0,0 +1,15 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) use rustfs_ecstore as ecstore;
rustfs/src/app/admin_usecase.rs
+8 -6
View File
@@ -15,15 +15,17 @@
//! Admin application use-case contracts.
use crate::app::context::{AppContext, get_global_app_context, resolve_object_store_handle_for_context};
use crate::app::storage_compat::ecstore::admin_server_info::get_server_info;
use crate::app::storage_compat::ecstore::data_usage::{apply_bucket_usage_memory_overlay, load_data_usage_from_backend};
use crate::app::storage_compat::ecstore::endpoints::EndpointServerPools;
use crate::app::storage_compat::ecstore::pools::{
PoolDecommissionInfo, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free,
};
use crate::app::storage_compat::ecstore::store::ECStore;
use crate::capacity::resolve_admin_used_capacity;
use crate::error::ApiError;
use crate::server::{DependencyReadiness, collect_dependency_readiness as collect_runtime_dependency_readiness};
use rustfs_data_usage::DataUsageInfo;
use rustfs_ecstore::admin_server_info::get_server_info;
use rustfs_ecstore::data_usage::{apply_bucket_usage_memory_overlay, load_data_usage_from_backend};
use rustfs_ecstore::endpoints::EndpointServerPools;
use rustfs_ecstore::pools::{PoolDecommissionInfo, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free};
use rustfs_ecstore::store::ECStore;
use rustfs_madmin::{InfoMessage, StorageInfo};
use rustfs_storage_api::StorageAdminApi;
use s3s::S3ErrorCode;
@@ -316,7 +318,7 @@ impl DefaultAdminUsecase {
#[cfg(test)]
mod tests {
use super::*;
use rustfs_ecstore::pools::{PoolDecommissionInfo, PoolStatus};
use crate::app::storage_compat::ecstore::pools::{PoolDecommissionInfo, PoolStatus};
use time::OffsetDateTime;
#[tokio::test]
rustfs/src/app/bucket_usecase.rs
+25 -23
View File
@@ -20,23 +20,7 @@ use crate::admin::handlers::site_replication::{
use crate::app::context::{
AppContext, default_notify_interface, get_global_app_context, resolve_object_store_handle_for_context,
};
use crate::auth::get_condition_values_with_client_info;
use crate::error::ApiError;
use crate::server::RemoteAddr;
use crate::storage::StorageObjectInfo as ObjectInfo;
use crate::storage::access::{ReqInfo, authorize_request, req_info_ref};
use crate::storage::helper::{OperationHelper, spawn_background_with_context};
use crate::storage::s3_api::bucket::{
ListObjectVersionsParams, ListObjectsV2Params, build_list_buckets_output, build_list_object_versions_output,
build_list_objects_output, build_list_objects_v2_output, parse_list_object_versions_params, parse_list_objects_v2_params,
};
use crate::storage::s3_api::common::rustfs_owner;
use crate::storage::*;
use futures::StreamExt;
use http::StatusCode;
use metrics::counter;
use rustfs_config::RUSTFS_REGION;
use rustfs_ecstore::bucket::{
use crate::app::storage_compat::ecstore::bucket::{
bucket_target_sys::BucketTargetSys,
lifecycle::bucket_lifecycle_ops::{
enqueue_expiry_for_existing_objects, enqueue_transition_for_existing_objects, validate_transition_tier,
@@ -54,10 +38,26 @@ use rustfs_ecstore::bucket::{
versioning::VersioningApi,
versioning_sys::BucketVersioningSys,
};
use rustfs_ecstore::client::object_api_utils::to_s3s_etag;
use rustfs_ecstore::error::StorageError;
use rustfs_ecstore::notification_sys::get_global_notification_sys;
use rustfs_ecstore::store::ECStore;
use crate::app::storage_compat::ecstore::client::object_api_utils::to_s3s_etag;
use crate::app::storage_compat::ecstore::error::StorageError;
use crate::app::storage_compat::ecstore::notification_sys::get_global_notification_sys;
use crate::app::storage_compat::ecstore::store::ECStore;
use crate::auth::get_condition_values_with_client_info;
use crate::error::ApiError;
use crate::server::RemoteAddr;
use crate::storage::StorageObjectInfo as ObjectInfo;
use crate::storage::access::{ReqInfo, authorize_request, req_info_ref};
use crate::storage::helper::{OperationHelper, spawn_background_with_context};
use crate::storage::s3_api::bucket::{
ListObjectVersionsParams, ListObjectsV2Params, build_list_buckets_output, build_list_object_versions_output,
build_list_objects_output, build_list_objects_v2_output, parse_list_object_versions_params, parse_list_objects_v2_params,
};
use crate::storage::s3_api::common::rustfs_owner;
use crate::storage::*;
use futures::StreamExt;
use http::StatusCode;
use metrics::counter;
use rustfs_config::RUSTFS_REGION;
use rustfs_madmin::{SITE_REPL_API_VERSION, SRBucketMeta};
use rustfs_policy::policy::{
action::{Action, S3Action},
@@ -1628,7 +1628,9 @@ impl DefaultBucketUsecase {
}
};
if let Err(err) = rustfs_ecstore::bucket::lifecycle::lifecycle::Lifecycle::validate(&input_cfg, &rcfg).await {
if let Err(err) =
crate::app::storage_compat::ecstore::bucket::lifecycle::lifecycle::Lifecycle::validate(&input_cfg, &rcfg).await
{
return Err(s3_error!(InvalidArgument, "{err}"));
}
@@ -2278,7 +2280,7 @@ mod tests {
BucketTargets {
targets: arns
.iter()
.map(|arn| rustfs_ecstore::bucket::target::BucketTarget {
.map(|arn| crate::app::storage_compat::ecstore::bucket::target::BucketTarget {
arn: (*arn).to_string(),
target_type: BucketTargetType::ReplicationService,
..Default::default()
rustfs/src/app/capacity_dirty_scope_test.rs
+5 -3
View File
@@ -12,13 +12,13 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_common::heal_channel::{HealOpts, HealScanMode};
use rustfs_ecstore::{
use crate::app::storage_compat::ecstore::{
bucket::metadata_sys,
disk::endpoint::Endpoint,
endpoints::{EndpointServerPools, Endpoints, PoolEndpoints},
store::ECStore,
};
use rustfs_common::heal_channel::{HealOpts, HealScanMode};
use rustfs_object_capacity::capacity_manager::{HybridStrategyConfig, create_isolated_manager};
use rustfs_storage_api::{BucketOperations, BucketOptions, HealOperations as _, MakeBucketOptions, ObjectIO as _};
use serial_test::serial;
@@ -82,7 +82,9 @@ async fn setup_capacity_dirty_scope_env() -> (Vec<PathBuf>, Arc<ECStore>) {
};
let endpoint_pools = EndpointServerPools(vec![pool_endpoints]);
rustfs_ecstore::store::init_local_disks(endpoint_pools.clone()).await.unwrap();
crate::app::storage_compat::ecstore::store::init_local_disks(endpoint_pools.clone())
.await
.unwrap();
let server_addr: std::net::SocketAddr = "127.0.0.1:0".parse().unwrap();
let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
rustfs/src/app/context/compat.rs
+9 -9
View File
@@ -17,14 +17,14 @@ use super::handles::{
default_bucket_metadata_interface, default_endpoints_interface, default_kms_runtime_interface,
default_server_config_interface, default_tier_config_interface,
};
use crate::app::storage_compat::ecstore::bucket::metadata_sys::BucketMetadataSys;
use crate::app::storage_compat::ecstore::endpoints::EndpointServerPools;
use crate::app::storage_compat::ecstore::new_object_layer_fn;
use crate::app::storage_compat::ecstore::store::ECStore;
use crate::app::storage_compat::ecstore::tier::tier::TierConfigMgr;
#[cfg(test)]
use crate::config::RustFSBufferConfig;
use rustfs_config::server_config::Config;
use rustfs_ecstore::bucket::metadata_sys::BucketMetadataSys;
use rustfs_ecstore::endpoints::EndpointServerPools;
use rustfs_ecstore::new_object_layer_fn;
use rustfs_ecstore::store::ECStore;
use rustfs_ecstore::tier::tier::TierConfigMgr;
use rustfs_kms::KmsServiceManager;
use std::sync::Arc;
use tokio::sync::RwLock;
@@ -126,11 +126,11 @@ mod tests {
BucketMetadataInterface, BufferConfigInterface, EndpointsInterface, IamInterface, KmsInterface, KmsRuntimeInterface,
ServerConfigInterface, TierConfigInterface,
};
use crate::app::storage_compat::ecstore::disk::endpoint::Endpoint;
use crate::app::storage_compat::ecstore::endpoints::{Endpoints, PoolEndpoints};
use crate::app::storage_compat::ecstore::new_object_layer_fn;
use crate::app::storage_compat::ecstore::store::init_local_disks;
use crate::config::{RustFSBufferConfig, WorkloadProfile};
use rustfs_ecstore::disk::endpoint::Endpoint;
use rustfs_ecstore::endpoints::{Endpoints, PoolEndpoints};
use rustfs_ecstore::new_object_layer_fn;
use rustfs_ecstore::store::init_local_disks;
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
use std::path::PathBuf;
use tempfile::TempDir;
rustfs/src/app/context/global.rs
+1 -1
View File
@@ -21,7 +21,7 @@ use super::interfaces::{
BucketMetadataInterface, BufferConfigInterface, EndpointsInterface, IamInterface, KmsInterface, KmsRuntimeInterface,
NotifyInterface, RegionInterface, ServerConfigInterface, TierConfigInterface,
};
use rustfs_ecstore::{set_object_store_resolver, store::ECStore};
use crate::app::storage_compat::ecstore::{set_object_store_resolver, store::ECStore};
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
use rustfs_kms::KmsServiceManager;
use std::sync::{Arc, OnceLock};
rustfs/src/app/context/handles.rs
+4 -4
View File
@@ -16,14 +16,14 @@ use super::interfaces::{
BucketMetadataInterface, BufferConfigInterface, EndpointsInterface, IamInterface, KmsInterface, KmsRuntimeInterface,
NotifyInterface, RegionInterface, ServerConfigInterface, TierConfigInterface,
};
use crate::app::storage_compat::ecstore::bucket::metadata_sys::{BucketMetadataSys, get_global_bucket_metadata_sys};
use crate::app::storage_compat::ecstore::endpoints::EndpointServerPools;
use crate::app::storage_compat::ecstore::global::{get_global_endpoints_opt, get_global_region, get_global_tier_config_mgr};
use crate::app::storage_compat::ecstore::tier::tier::TierConfigMgr;
use crate::config::{RustFSBufferConfig, get_global_buffer_config};
use async_trait::async_trait;
use rustfs_config::server_config::Config;
use rustfs_config::server_config::get_global_server_config;
use rustfs_ecstore::bucket::metadata_sys::{BucketMetadataSys, get_global_bucket_metadata_sys};
use rustfs_ecstore::endpoints::EndpointServerPools;
use rustfs_ecstore::global::{get_global_endpoints_opt, get_global_region, get_global_tier_config_mgr};
use rustfs_ecstore::tier::tier::TierConfigMgr;
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
use rustfs_kms::{KmsServiceManager, get_global_kms_service_manager};
use rustfs_notify::{EventArgs, NotificationError, notifier_global};
rustfs/src/app/context/interfaces.rs
+3 -3
View File
@@ -12,12 +12,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::app::storage_compat::ecstore::bucket::metadata_sys::BucketMetadataSys;
use crate::app::storage_compat::ecstore::endpoints::EndpointServerPools;
use crate::app::storage_compat::ecstore::tier::tier::TierConfigMgr;
use crate::config::RustFSBufferConfig;
use async_trait::async_trait;
use rustfs_config::server_config::Config;
use rustfs_ecstore::bucket::metadata_sys::BucketMetadataSys;
use rustfs_ecstore::endpoints::EndpointServerPools;
use rustfs_ecstore::tier::tier::TierConfigMgr;
use rustfs_iam::{store::object::ObjectStore, sys::IamSys};
use rustfs_kms::KmsServiceManager;
use rustfs_notify::{EventArgs, NotificationError};
rustfs/src/app/lifecycle_transition_api_test.rs
+16 -14
View File
@@ -14,16 +14,7 @@
use super::{multipart_usecase::DefaultMultipartUsecase, object_usecase::DefaultObjectUsecase};
use crate::app::bucket_usecase::DefaultBucketUsecase;
use crate::storage::ecfs::FS;
use crate::storage::{
StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
};
use bytes::Bytes;
use futures::FutureExt;
use futures::stream;
use http::{Extensions, HeaderMap, HeaderValue, Method, Uri, header::IF_NONE_MATCH};
use rustfs_config::{ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE, ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT};
use rustfs_ecstore::{
use crate::app::storage_compat::ecstore::{
bucket::metadata::{BUCKET_LIFECYCLE_CONFIG, OBJECT_LOCK_CONFIG},
bucket::metadata_sys,
client::object_api_utils::to_s3s_etag,
@@ -37,6 +28,15 @@ use rustfs_ecstore::{
warm_backend::{WarmBackend, WarmBackendGetOpts},
},
};
use crate::storage::ecfs::FS;
use crate::storage::{
StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
};
use bytes::Bytes;
use futures::FutureExt;
use futures::stream;
use http::{Extensions, HeaderMap, HeaderValue, Method, Uri, header::IF_NONE_MATCH};
use rustfs_config::{ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE, ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT};
use rustfs_object_capacity::capacity_manager::{HybridStrategyConfig, create_isolated_manager};
use rustfs_storage_api::{
BucketOperations, BucketOptions, ListOperations as _, MakeBucketOptions, MultipartOperations as _, ObjectIO as _,
@@ -113,7 +113,9 @@ async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>) {
let endpoint_pools = EndpointServerPools(vec![pool_endpoints]);
rustfs_ecstore::store::init_local_disks(endpoint_pools.clone()).await.unwrap();
crate::app::storage_compat::ecstore::store::init_local_disks(endpoint_pools.clone())
.await
.unwrap();
let server_addr: std::net::SocketAddr = "127.0.0.1:9003".parse().unwrap();
let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
@@ -130,7 +132,7 @@ async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>) {
let buckets = buckets_list.into_iter().map(|v| v.name).collect();
metadata_sys::init_bucket_metadata_sys(ecstore.clone(), buckets).await;
rustfs_ecstore::bucket::lifecycle::bucket_lifecycle_ops::init_background_expiry(ecstore.clone()).await;
crate::app::storage_compat::ecstore::bucket::lifecycle::bucket_lifecycle_ops::init_background_expiry(ecstore.clone()).await;
let _ = GLOBAL_ENV.set((disk_paths.clone(), ecstore.clone()));
@@ -930,7 +932,7 @@ async fn delete_transitioned_object_removes_remote_tier_copy_via_usecase() {
.expect("Failed to set lifecycle configuration");
let _ = upload_test_object(&ecstore, bucket.as_str(), object, payload).await;
rustfs_ecstore::bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(
crate::app::storage_compat::ecstore::bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(
ecstore.clone(),
bucket.as_str(),
)
@@ -990,7 +992,7 @@ async fn lifecycle_transition_marks_dirty_disks_for_capacity_manager() {
.expect("Failed to set lifecycle configuration");
let _ = upload_test_object(&ecstore, bucket.as_str(), object, payload).await;
rustfs_ecstore::bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(
crate::app::storage_compat::ecstore::bucket::lifecycle::bucket_lifecycle_ops::enqueue_transition_for_existing_objects(
ecstore.clone(),
bucket.as_str(),
)
rustfs/src/app/mod.rs
+1
View File
@@ -21,6 +21,7 @@ pub mod context;
pub mod multipart_usecase;
pub mod object_usecase;
mod select_object;
pub(crate) mod storage_compat;
#[cfg(test)]
mod capacity_dirty_scope_test;
rustfs/src/app/multipart_usecase.rs
+58 -34
View File
@@ -16,6 +16,22 @@
use crate::app::context::{AppContext, get_global_app_context, resolve_object_store_handle_for_context};
use crate::app::object_usecase::{build_put_like_object_lock_metadata, validate_existing_object_lock_for_write};
use crate::app::storage_compat::ecstore::bucket::quota::checker::QuotaChecker;
use crate::app::storage_compat::ecstore::bucket::{
lifecycle::{bucket_lifecycle_audit::LcEventSrc, bucket_lifecycle_ops::enqueue_transition_immediate},
metadata_sys,
quota::QuotaOperation,
replication::{get_must_replicate_options, must_replicate, schedule_replication},
versioning_sys::BucketVersioningSys,
};
use crate::app::storage_compat::ecstore::client::object_api_utils::to_s3s_etag;
use crate::app::storage_compat::ecstore::compress::is_disk_compressible;
use crate::app::storage_compat::ecstore::error::{StorageError, is_err_object_not_found, is_err_version_not_found};
#[cfg(test)]
use crate::app::storage_compat::ecstore::rio::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader, wrap_reader};
use crate::app::storage_compat::ecstore::rio::{HashReader, WritePlan};
use crate::app::storage_compat::ecstore::set_disk::is_valid_storage_class;
use crate::app::storage_compat::ecstore::store::ECStore;
use crate::capacity::record_capacity_write;
use crate::error::ApiError;
use crate::storage::access::has_bypass_governance_header;
@@ -38,22 +54,6 @@ use crate::table_catalog;
use bytes::Bytes;
use futures::StreamExt;
use http::{HeaderMap, Uri};
use rustfs_ecstore::bucket::quota::checker::QuotaChecker;
use rustfs_ecstore::bucket::{
lifecycle::{bucket_lifecycle_audit::LcEventSrc, bucket_lifecycle_ops::enqueue_transition_immediate},
metadata_sys,
quota::QuotaOperation,
replication::{get_must_replicate_options, must_replicate, schedule_replication},
versioning_sys::BucketVersioningSys,
};
use rustfs_ecstore::client::object_api_utils::to_s3s_etag;
use rustfs_ecstore::compress::is_disk_compressible;
use rustfs_ecstore::error::{StorageError, is_err_object_not_found, is_err_version_not_found};
#[cfg(test)]
use rustfs_ecstore::rio::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader, wrap_reader};
use rustfs_ecstore::rio::{HashReader, WritePlan};
use rustfs_ecstore::set_disk::is_valid_storage_class;
use rustfs_ecstore::store::ECStore;
use rustfs_filemeta::{ReplicationStatusType, ReplicationType};
use rustfs_s3_ops::S3Operation;
#[cfg(test)]
@@ -479,7 +479,7 @@ impl DefaultMultipartUsecase {
));
}
// Update quota tracking after successful multipart upload
rustfs_ecstore::data_usage::record_bucket_object_write_memory(
crate::app::storage_compat::ecstore::data_usage::record_bucket_object_write_memory(
&bucket,
previous_current_size,
obj_info.size.max(0) as u64,
@@ -672,7 +672,7 @@ impl DefaultMultipartUsecase {
rustfs_utils::http::insert_str(
&mut metadata,
rustfs_utils::http::SUFFIX_COMPRESSION,
rustfs_ecstore::rio::compression_metadata_value(CompressionAlgorithm::default()),
crate::app::storage_compat::ecstore::rio::compression_metadata_value(CompressionAlgorithm::default()),
);
}
@@ -891,10 +891,17 @@ impl DefaultMultipartUsecase {
.ok_or_else(|| ApiError::from(StorageError::other("Missing SSE-C session material")))?;
let ssec_write = match ssec_material.key_kind {
crate::storage::sse::EncryptionKeyKind::Object => {
rustfs_ecstore::rio::WriteEncryption::multipart_object_key(ssec_material.key_bytes, part_id as u32)
crate::app::storage_compat::ecstore::rio::WriteEncryption::multipart_object_key(
ssec_material.key_bytes,
part_id as u32,
)
}
crate::storage::sse::EncryptionKeyKind::Direct => {
rustfs_ecstore::rio::WriteEncryption::multipart(ssec_material.key_bytes, ssec_material.base_nonce, part_id)
crate::app::storage_compat::ecstore::rio::WriteEncryption::multipart(
ssec_material.key_bytes,
ssec_material.base_nonce,
part_id,
)
}
};
write_plan = write_plan.with_encryption(ssec_write);
@@ -911,13 +918,18 @@ impl DefaultMultipartUsecase {
.ok_or_else(|| ApiError::from(StorageError::other("Missing managed SSE session material")))?;
let managed_write = match managed_material.key_kind {
crate::storage::sse::EncryptionKeyKind::Object => {
rustfs_ecstore::rio::WriteEncryption::multipart_object_key(managed_material.key_bytes, part_id as u32)
crate::app::storage_compat::ecstore::rio::WriteEncryption::multipart_object_key(
managed_material.key_bytes,
part_id as u32,
)
}
crate::storage::sse::EncryptionKeyKind::Direct => {
crate::app::storage_compat::ecstore::rio::WriteEncryption::multipart(
managed_material.key_bytes,
managed_material.base_nonce,
part_id,
)
}
crate::storage::sse::EncryptionKeyKind::Direct => rustfs_ecstore::rio::WriteEncryption::multipart(
managed_material.key_bytes,
managed_material.base_nonce,
part_id,
),
};
write_plan = write_plan.with_encryption(managed_write);
(Some(server_side_encryption), ssekms_key_id)
@@ -1232,10 +1244,17 @@ impl DefaultMultipartUsecase {
.ok_or_else(|| ApiError::from(StorageError::other("Missing SSE-C session material")))?;
let ssec_write = match ssec_material.key_kind {
crate::storage::sse::EncryptionKeyKind::Object => {
rustfs_ecstore::rio::WriteEncryption::multipart_object_key(ssec_material.key_bytes, part_id as u32)
crate::app::storage_compat::ecstore::rio::WriteEncryption::multipart_object_key(
ssec_material.key_bytes,
part_id as u32,
)
}
crate::storage::sse::EncryptionKeyKind::Direct => {
rustfs_ecstore::rio::WriteEncryption::multipart(ssec_material.key_bytes, ssec_material.base_nonce, part_id)
crate::app::storage_compat::ecstore::rio::WriteEncryption::multipart(
ssec_material.key_bytes,
ssec_material.base_nonce,
part_id,
)
}
};
write_plan = write_plan.with_encryption(ssec_write);
@@ -1256,13 +1275,18 @@ impl DefaultMultipartUsecase {
.ok_or_else(|| ApiError::from(StorageError::other("Missing managed SSE session material")))?;
let managed_write = match managed_material.key_kind {
crate::storage::sse::EncryptionKeyKind::Object => {
rustfs_ecstore::rio::WriteEncryption::multipart_object_key(managed_material.key_bytes, part_id as u32)
crate::app::storage_compat::ecstore::rio::WriteEncryption::multipart_object_key(
managed_material.key_bytes,
part_id as u32,
)
}
crate::storage::sse::EncryptionKeyKind::Direct => {
crate::app::storage_compat::ecstore::rio::WriteEncryption::multipart(
managed_material.key_bytes,
managed_material.base_nonce,
part_id,
)
}
crate::storage::sse::EncryptionKeyKind::Direct => rustfs_ecstore::rio::WriteEncryption::multipart(
managed_material.key_bytes,
managed_material.base_nonce,
part_id,
),
};
write_plan = write_plan.with_encryption(managed_write);
(Some(server_side_encryption), ssekms_key_id, mp_info.user_defined.clone())
rustfs/src/app/object_usecase.rs
+32 -25
View File
@@ -48,9 +48,8 @@ use metrics::{counter, histogram};
use pin_project_lite::pin_project;
use rustfs_object_capacity::capacity_manager::get_capacity_manager;
// Performance metrics recording (with zero-copy-metrics integration)
use rustfs_concurrency::GetObjectQueueSnapshot;
use rustfs_ecstore::bucket::quota::checker::QuotaChecker;
use rustfs_ecstore::bucket::{
use crate::app::storage_compat::ecstore::bucket::quota::checker::QuotaChecker;
use crate::app::storage_compat::ecstore::bucket::{
lifecycle::{
bucket_lifecycle_audit::LcEventSrc,
bucket_lifecycle_ops::{RestoreRequestOps, enqueue_transition_immediate, post_restore_opts},
@@ -70,16 +69,17 @@ use rustfs_ecstore::bucket::{
versioning::VersioningApi,
versioning_sys::BucketVersioningSys,
};
use rustfs_ecstore::client::object_api_utils::to_s3s_etag;
use rustfs_ecstore::compress::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible};
use rustfs_ecstore::config::storageclass;
use rustfs_ecstore::disk::{error::DiskError, error_reduce::is_all_buckets_not_found};
use rustfs_ecstore::error::{
use crate::app::storage_compat::ecstore::client::object_api_utils::to_s3s_etag;
use crate::app::storage_compat::ecstore::compress::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible};
use crate::app::storage_compat::ecstore::config::storageclass;
use crate::app::storage_compat::ecstore::disk::{error::DiskError, error_reduce::is_all_buckets_not_found};
use crate::app::storage_compat::ecstore::error::{
Error as EcstoreError, StorageError, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found,
};
use rustfs_ecstore::rio::{DynReader, HashReader, WritePlan, wrap_reader};
use rustfs_ecstore::set_disk::{get_lock_acquire_timeout, is_valid_storage_class};
use rustfs_ecstore::store::ECStore;
use crate::app::storage_compat::ecstore::rio::{DynReader, HashReader, WritePlan, wrap_reader};
use crate::app::storage_compat::ecstore::set_disk::{get_lock_acquire_timeout, is_valid_storage_class};
use crate::app::storage_compat::ecstore::store::ECStore;
use rustfs_concurrency::GetObjectQueueSnapshot;
use rustfs_filemeta::{
REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateTargetDecision, ReplicationState, ReplicationStatusType,
ReplicationType, RestoreStatusOps, VersionPurgeStatusType, parse_restore_obj_status, replication_statuses_map,
@@ -276,10 +276,12 @@ async fn enqueue_transitioned_delete_cleanup(
let _activity_guard = DeleteTailActivityGuard::new(DeleteTailStage::Cleanup);
let je = if opts.delete_prefix {
rustfs_ecstore::bucket::lifecycle::tier_sweeper::transitioned_force_delete_journal_entry(&existing.transitioned_object)
crate::app::storage_compat::ecstore::bucket::lifecycle::tier_sweeper::transitioned_force_delete_journal_entry(
&existing.transitioned_object,
)
} else {
let version_id = opts.version_id.as_ref().and_then(|v| Uuid::parse_str(v).ok());
rustfs_ecstore::bucket::lifecycle::tier_sweeper::transitioned_delete_journal_entry(
crate::app::storage_compat::ecstore::bucket::lifecycle::tier_sweeper::transitioned_delete_journal_entry(
version_id,
opts.versioned,
opts.version_suspended,
@@ -290,9 +292,10 @@ async fn enqueue_transitioned_delete_cleanup(
return Ok(());
};
rustfs_ecstore::bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry(store, &je).await?;
crate::app::storage_compat::ecstore::bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry(store, &je)
.await?;
let mut expiry_state = rustfs_ecstore::bucket::lifecycle::bucket_lifecycle_ops::GLOBAL_ExpiryState
let mut expiry_state = crate::app::storage_compat::ecstore::bucket::lifecycle::bucket_lifecycle_ops::GLOBAL_ExpiryState
.write()
.await;
if let Err(err) = expiry_state.enqueue_tier_journal_entry(&je).await {
@@ -1537,7 +1540,7 @@ impl DefaultObjectUsecase {
#[allow(clippy::too_many_arguments)]
async fn prepare_get_object_read(
req: &S3Request<GetObjectInput>,
store: &rustfs_ecstore::store::ECStore,
store: &crate::app::storage_compat::ecstore::store::ECStore,
manager: &ConcurrencyManager,
bucket: &str,
key: &str,
@@ -2131,7 +2134,7 @@ impl DefaultObjectUsecase {
insert_str(
&mut metadata,
SUFFIX_COMPRESSION,
rustfs_ecstore::rio::compression_metadata_value(algorithm),
crate::app::storage_compat::ecstore::rio::compression_metadata_value(algorithm),
);
insert_str(&mut metadata, SUFFIX_ACTUAL_SIZE, size.to_string());
@@ -2146,7 +2149,7 @@ impl DefaultObjectUsecase {
insert_str(
&mut opts.user_defined,
SUFFIX_COMPRESSION,
rustfs_ecstore::rio::compression_metadata_value(algorithm),
crate::app::storage_compat::ecstore::rio::compression_metadata_value(algorithm),
);
insert_str(&mut opts.user_defined, SUFFIX_ACTUAL_SIZE, size.to_string());
@@ -2338,7 +2341,7 @@ impl DefaultObjectUsecase {
maybe_enqueue_transition_immediate(&obj_info, LcEventSrc::S3PutObject).await;
// Fast in-memory update for immediate quota and admin usage consistency
rustfs_ecstore::data_usage::record_bucket_object_write_memory(
crate::app::storage_compat::ecstore::data_usage::record_bucket_object_write_memory(
&bucket,
previous_current_size,
obj_info.size.max(0) as u64,
@@ -3208,7 +3211,7 @@ impl DefaultObjectUsecase {
insert_str(
&mut compress_metadata,
SUFFIX_COMPRESSION,
rustfs_ecstore::rio::compression_metadata_value(CompressionAlgorithm::default()),
crate::app::storage_compat::ecstore::rio::compression_metadata_value(CompressionAlgorithm::default()),
);
insert_str(&mut compress_metadata, SUFFIX_ACTUAL_SIZE, actual_size.to_string());
} else {
@@ -3301,8 +3304,12 @@ impl DefaultObjectUsecase {
// Update quota tracking after successful copy
if has_bucket_metadata {
rustfs_ecstore::data_usage::record_bucket_object_write_memory(&bucket, previous_current_size, oi.size.max(0) as u64)
.await;
crate::app::storage_compat::ecstore::data_usage::record_bucket_object_write_memory(
&bucket,
previous_current_size,
oi.size.max(0) as u64,
)
.await;
}
let raw_dest_version = oi.version_id.map(|v| v.to_string());
@@ -3579,7 +3586,7 @@ impl DefaultObjectUsecase {
);
}
let size = object_sizes[i].max(0) as u64;
rustfs_ecstore::data_usage::record_bucket_object_delete_memory(
crate::app::storage_compat::ecstore::data_usage::record_bucket_object_delete_memory(
&bucket,
size,
existing_object_infos[i].is_some() && object_to_delete[i].version_id.is_none(),
@@ -3817,7 +3824,7 @@ impl DefaultObjectUsecase {
}
// Fast in-memory update for immediate quota and admin usage consistency
rustfs_ecstore::data_usage::record_bucket_object_delete_memory(
crate::app::storage_compat::ecstore::data_usage::record_bucket_object_delete_memory(
&bucket,
obj_info.size.max(0) as u64,
opts.version_id.is_none(),
@@ -4750,7 +4757,7 @@ impl DefaultObjectUsecase {
insert_str(
&mut metadata,
SUFFIX_COMPRESSION,
rustfs_ecstore::rio::compression_metadata_value(algorithm),
crate::app::storage_compat::ecstore::rio::compression_metadata_value(algorithm),
);
insert_str(&mut metadata, SUFFIX_ACTUAL_SIZE, size.to_string());
rustfs/src/app/storage_compat.rs
+15
View File
@@ -0,0 +1,15 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) use rustfs_ecstore as ecstore;
rustfs/src/storage/access.rs
+6 -6
View File
@@ -18,12 +18,12 @@ use crate::error::ApiError;
use crate::license::license_check;
use crate::server::RemoteAddr;
use crate::storage::request_context::RequestContext;
use crate::storage::storage_compat::ecstore::bucket::metadata_sys;
use crate::storage::storage_compat::ecstore::bucket::policy_sys::PolicySys;
use crate::storage::storage_compat::ecstore::error::{StorageError, is_err_bucket_not_found};
use crate::storage::storage_compat::ecstore::resolve_object_store_handle;
use crate::storage::storage_compat::ecstore::store::ECStore;
use metrics::counter;
use rustfs_ecstore::bucket::metadata_sys;
use rustfs_ecstore::bucket::policy_sys::PolicySys;
use rustfs_ecstore::error::{StorageError, is_err_bucket_not_found};
use rustfs_ecstore::resolve_object_store_handle;
use rustfs_ecstore::store::ECStore;
use rustfs_iam::error::Error as IamError;
use rustfs_policy::policy::action::{Action, AdminAction, S3Action};
use rustfs_policy::policy::{
@@ -930,7 +930,7 @@ impl S3Access for FS {
let req_info = ReqInfo {
cred,
is_owner,
region: rustfs_ecstore::global::get_global_region(),
region: crate::storage::storage_compat::ecstore::global::get_global_region(),
request_context,
..Default::default()
};
rustfs/src/storage/ecfs.rs
+5 -5
View File
@@ -21,11 +21,7 @@ use crate::storage::access::has_bypass_governance_header;
use crate::storage::helper::OperationHelper;
use crate::storage::options::get_opts;
use crate::storage::s3_api::acl;
use crate::storage::{parse_object_lock_legal_hold, parse_object_lock_retention, validate_bucket_object_lock_enabled};
use crate::table_catalog;
use http::StatusCode;
use metrics::{counter, histogram};
use rustfs_ecstore::{
use crate::storage::storage_compat::ecstore::{
bucket::{
metadata::{
BUCKET_ACCELERATE_CONFIG, BUCKET_LOGGING_CONFIG, BUCKET_REQUEST_PAYMENT_CONFIG, BUCKET_VERSIONING_CONFIG,
@@ -41,6 +37,10 @@ use rustfs_ecstore::{
},
error::{StorageError, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found},
};
use crate::storage::{parse_object_lock_legal_hold, parse_object_lock_retention, validate_bucket_object_lock_enabled};
use crate::table_catalog;
use http::StatusCode;
use metrics::{counter, histogram};
use rustfs_io_metrics::record_s3_op;
use rustfs_s3_ops::S3Operation;
use rustfs_storage_api::{BucketOperations, BucketOptions, ObjectLockRetentionOptions, ObjectOperations as _};
rustfs/src/storage/ecfs_extend.rs
+7 -7
View File
@@ -16,15 +16,15 @@ use crate::config::{RustFSBufferConfig, WorkloadProfile, get_global_buffer_confi
use crate::error::ApiError;
use crate::server::cors;
use crate::storage::ecfs::ListObjectUnorderedQuery;
use crate::storage::storage_compat::ecstore::bucket::metadata_sys;
use crate::storage::storage_compat::ecstore::bucket::metadata_sys::get_replication_config;
use crate::storage::storage_compat::ecstore::bucket::object_lock::objectlock_sys;
use crate::storage::storage_compat::ecstore::bucket::replication::ReplicationConfigurationExt;
use crate::storage::storage_compat::ecstore::error::StorageError;
use crate::storage::storage_compat::ecstore::resolve_object_store_handle;
use http::header::{IF_MATCH, IF_MODIFIED_SINCE, IF_NONE_MATCH, IF_UNMODIFIED_SINCE};
use http::{HeaderMap, HeaderValue, StatusCode};
use metrics::counter;
use rustfs_ecstore::bucket::metadata_sys;
use rustfs_ecstore::bucket::metadata_sys::get_replication_config;
use rustfs_ecstore::bucket::object_lock::objectlock_sys;
use rustfs_ecstore::bucket::replication::ReplicationConfigurationExt;
use rustfs_ecstore::error::StorageError;
use rustfs_ecstore::resolve_object_store_handle;
use rustfs_storage_api::{BucketOperations, BucketOptions};
use rustfs_targets::EventName;
use rustfs_targets::arn::{TargetID, TargetIDError};
@@ -739,7 +739,7 @@ pub(crate) async fn has_replication_rules(bucket: &str, objects: &[ObjectToDelet
}
/// Helper function to get store and validate bucket exists
pub(crate) async fn get_validated_store(bucket: &str) -> S3Result<Arc<rustfs_ecstore::store::ECStore>> {
pub(crate) async fn get_validated_store(bucket: &str) -> S3Result<Arc<crate::storage::storage_compat::ecstore::store::ECStore>> {
let Some(store) = resolve_object_store_handle() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
rustfs/src/storage/ecfs_test.rs
+9 -6
View File
@@ -18,6 +18,8 @@ mod tests {
use crate::server::cors;
use crate::storage::ecfs::{FS, validate_object_lock_configuration_input};
use crate::storage::s3_api::common::{rustfs_initiator, rustfs_owner};
use crate::storage::storage_compat::ecstore::bucket::{metadata::BucketMetadata, metadata_sys};
use crate::storage::storage_compat::ecstore::set_disk::DEFAULT_READ_BUFFER_SIZE;
use crate::storage::{
StorageObjectInfo as ObjectInfo, apply_cors_headers, apply_default_lock_retention_metadata, check_preconditions,
get_adaptive_buffer_size_with_profile, get_buffer_size_opt_in, is_etag_equal, matches_origin_pattern, parse_etag,
@@ -27,8 +29,6 @@ mod tests {
};
use http::{Extensions, HeaderMap, HeaderValue, Method, StatusCode, Uri};
use rustfs_config::MI_B;
use rustfs_ecstore::bucket::{metadata::BucketMetadata, metadata_sys};
use rustfs_ecstore::set_disk::DEFAULT_READ_BUFFER_SIZE;
use rustfs_utils::http::{
AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER,
SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, contains_key_str, get_str, insert_str,
@@ -940,12 +940,15 @@ mod tests {
#[tokio::test]
async fn test_validate_bucket_object_lock_enabled() {
use rustfs_ecstore::bucket::metadata::BucketMetadata;
use rustfs_ecstore::bucket::metadata_sys::set_bucket_metadata;
use crate::storage::storage_compat::ecstore::bucket::metadata::BucketMetadata;
use crate::storage::storage_compat::ecstore::bucket::metadata_sys::set_bucket_metadata;
use s3s::dto::{ObjectLockConfiguration, ObjectLockEnabled};
use time::OffsetDateTime;
if rustfs_ecstore::bucket::metadata_sys::GLOBAL_BucketMetadataSys.get().is_none() {
if crate::storage::storage_compat::ecstore::bucket::metadata_sys::GLOBAL_BucketMetadataSys
.get()
.is_none()
{
eprintln!("Skipping test: GLOBAL_BucketMetadataSys not initialized");
return;
}
@@ -1780,7 +1783,7 @@ mod tests {
/// with a single-element vec value, matching the format expected by policy evaluation.
#[test]
fn test_object_tag_condition_key_format() {
use rustfs_ecstore::bucket::tagging::decode_tags_to_map;
use crate::storage::storage_compat::ecstore::bucket::tagging::decode_tags_to_map;
use std::collections::HashMap;
let tags_str = "security=public&project=webapp&env=prod";
rustfs/src/storage/head_prefix.rs
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_ecstore::store::ECStore;
use crate::storage::storage_compat::ecstore::store::ECStore;
use rustfs_storage_api::ListOperations as _;
use std::sync::Arc;
rustfs/src/storage/mod.rs
+7 -6
View File
@@ -24,15 +24,16 @@ pub mod request_context;
pub mod rpc;
pub(crate) mod s3_api;
pub(crate) mod sse;
pub(crate) mod storage_compat;
pub mod timeout_wrapper;
pub mod tonic_service;
pub(crate) type StorageDeletedObject = rustfs_ecstore::store_api::DeletedObject;
pub(crate) type StorageGetObjectReader = rustfs_ecstore::store_api::GetObjectReader;
pub(crate) type StorageObjectInfo = rustfs_ecstore::store_api::ObjectInfo;
pub(crate) type StorageObjectOptions = rustfs_ecstore::store_api::ObjectOptions;
pub(crate) type StorageObjectToDelete = rustfs_ecstore::store_api::ObjectToDelete;
pub(crate) type StoragePutObjReader = rustfs_ecstore::store_api::PutObjReader;
pub(crate) type StorageDeletedObject = crate::storage::storage_compat::ecstore::store_api::DeletedObject;
pub(crate) type StorageGetObjectReader = crate::storage::storage_compat::ecstore::store_api::GetObjectReader;
pub(crate) type StorageObjectInfo = crate::storage::storage_compat::ecstore::store_api::ObjectInfo;
pub(crate) type StorageObjectOptions = crate::storage::storage_compat::ecstore::store_api::ObjectOptions;
pub(crate) type StorageObjectToDelete = crate::storage::storage_compat::ecstore::store_api::ObjectToDelete;
pub(crate) type StoragePutObjReader = crate::storage::storage_compat::ecstore::store_api::PutObjReader;
#[cfg(test)]
mod concurrent_fix_test;
rustfs/src/storage/options.rs
+3 -3
View File
@@ -12,11 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage::storage_compat::ecstore::bucket::versioning_sys::BucketVersioningSys;
use crate::storage::storage_compat::ecstore::error::Result;
use crate::storage::storage_compat::ecstore::error::StorageError;
use http::header::{IF_MATCH, IF_NONE_MATCH};
use http::{HeaderMap, HeaderValue};
use rustfs_ecstore::bucket::versioning_sys::BucketVersioningSys;
use rustfs_ecstore::error::Result;
use rustfs_ecstore::error::StorageError;
use rustfs_utils::http::{
AMZ_META_UNENCRYPTED_CONTENT_LENGTH, AMZ_META_UNENCRYPTED_CONTENT_MD5, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER,
AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER,
rustfs/src/storage/rpc/http_service.rs
+4 -4
View File
@@ -14,16 +14,16 @@
use crate::server::RPC_PREFIX;
use crate::storage::request_context::spawn_traced;
use crate::storage::storage_compat::ecstore::disk::{DiskAPI, WalkDirOptions};
use crate::storage::storage_compat::ecstore::rpc::verify_rpc_signature;
use crate::storage::storage_compat::ecstore::set_disk::DEFAULT_READ_BUFFER_SIZE;
use crate::storage::storage_compat::ecstore::store::find_local_disk_by_ref;
use bytes::{Bytes, BytesMut};
use futures_util::TryStreamExt;
use http::{HeaderMap, Method, Request, Response, StatusCode, Uri};
use http_body_util::{BodyExt, Limited};
use hyper::body::Incoming;
use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
use rustfs_ecstore::disk::{DiskAPI, WalkDirOptions};
use rustfs_ecstore::rpc::verify_rpc_signature;
use rustfs_ecstore::set_disk::DEFAULT_READ_BUFFER_SIZE;
use rustfs_ecstore::store::find_local_disk_by_ref;
use rustfs_io_metrics::internode_metrics::{
INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_OPERATION_WALK_DIR,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics,
rustfs/src/storage/rpc/node_service.rs
+13 -10
View File
@@ -16,12 +16,7 @@ use crate::admin::service::{
config::{reload_dynamic_config_runtime_state, reload_runtime_config_snapshot},
site_replication::reload_site_replication_runtime_state,
};
use bytes::Bytes;
use futures::Stream;
use futures_util::future::join_all;
use rmp_serde::Deserializer;
use rustfs_common::{get_global_local_node_name, heal_channel::HealOpts};
use rustfs_ecstore::{
use crate::storage::storage_compat::ecstore::{
admin_server_info::get_local_server_property,
bucket::{metadata::load_bucket_metadata, metadata_sys},
disk::{
@@ -38,6 +33,11 @@ use rustfs_ecstore::{
},
store::{all_local_disk_path, find_local_disk_by_ref},
};
use bytes::Bytes;
use futures::Stream;
use futures_util::future::join_all;
use rmp_serde::Deserializer;
use rustfs_common::{get_global_local_node_name, heal_channel::HealOpts};
use rustfs_filemeta::{FileInfo, MetacacheReader};
use rustfs_iam::{get_global_iam_sys, store::UserType};
use rustfs_lock::{LockClient, LockRequest};
@@ -132,7 +132,9 @@ fn unimplemented_rpc(method: &str) -> Status {
Status::unimplemented(format!("{method} is not implemented"))
}
fn background_rebalance_start_error_message(result: rustfs_ecstore::error::Result<()>) -> Option<String> {
fn background_rebalance_start_error_message(
result: crate::storage::storage_compat::ecstore::error::Result<()>,
) -> Option<String> {
result.err().map(|err| format!("start_rebalance failed: {err}"))
}
@@ -2376,8 +2378,9 @@ mod tests {
#[test]
fn test_background_rebalance_start_error_message_formats_error() {
let message = background_rebalance_start_error_message(Err(rustfs_ecstore::error::Error::other("boom")))
.expect("background rebalance start failure should be formatted");
let message =
background_rebalance_start_error_message(Err(crate::storage::storage_compat::ecstore::error::Error::other("boom")))
.expect("background rebalance start failure should be formatted");
assert!(message.contains("start_rebalance failed"));
assert!(message.contains("boom"));
@@ -2707,7 +2710,7 @@ mod tests {
vars.insert(PEER_RESTSIGNAL.to_string(), SERVICE_SIGNAL_RELOAD_DYNAMIC.to_string());
vars.insert(
PEER_RESTSUB_SYS.to_string(),
rustfs_ecstore::config::com::STORAGE_CLASS_SUB_SYS.to_string(),
crate::storage::storage_compat::ecstore::config::com::STORAGE_CLASS_SUB_SYS.to_string(),
);
let request = Request::new(SignalServiceRequest {
rustfs/src/storage/s3_api/bucket.rs
+1 -1
View File
@@ -13,8 +13,8 @@
// limitations under the License.
use crate::storage::s3_api::common::rustfs_owner;
use crate::storage::storage_compat::ecstore::client::object_api_utils::to_s3s_etag;
use percent_encoding::percent_decode_str;
use rustfs_ecstore::client::object_api_utils::to_s3s_etag;
use rustfs_storage_api::{
BucketInfo, ListObjectVersionsInfo as StorageListObjectVersionsInfo, ListObjectsV2Info as StorageListObjectsV2Info,
};
rustfs/src/storage/s3_api/multipart.rs
+2 -2
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use crate::storage::s3_api::common::{rustfs_initiator, rustfs_owner};
use rustfs_ecstore::client::object_api_utils::to_s3s_etag;
use crate::storage::storage_compat::ecstore::client::object_api_utils::to_s3s_etag;
use rustfs_storage_api::{ListMultipartsInfo, ListPartsInfo};
use s3s::dto::{CommonPrefix, ListMultipartUploadsOutput, ListPartsOutput, MultipartUpload, Part, Timestamp};
use s3s::{S3Error, S3ErrorCode};
@@ -191,7 +191,7 @@ mod tests {
parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number,
};
use crate::storage::s3_api::common::{rustfs_initiator, rustfs_owner};
use rustfs_ecstore::client::object_api_utils::to_s3s_etag;
use crate::storage::storage_compat::ecstore::client::object_api_utils::to_s3s_etag;
use rustfs_storage_api::{ListMultipartsInfo, ListPartsInfo, MultipartInfo, PartInfo};
use s3s::S3ErrorCode;
use s3s::dto::Timestamp;
rustfs/src/storage/sse.rs
+20 -8
View File
@@ -69,6 +69,7 @@
//! }
//! ```
use crate::storage::storage_compat::ecstore::error::StorageError;
use aes_gcm::{
Aes256Gcm, Key, Nonce,
aead::{Aead, KeyInit},
@@ -81,7 +82,6 @@ use http::{HeaderMap, HeaderValue};
use rand::Rng;
#[cfg(feature = "rio-v2")]
use rand::RngExt;
use rustfs_ecstore::error::StorageError;
use rustfs_kms::{DataKey, service_manager::get_global_encryption_service, types::ObjectEncryptionContext};
use rustfs_utils::get_env_opt_str;
use s3s::S3ErrorCode;
@@ -128,8 +128,8 @@ const SEALED_KEY_SIZE: usize = DARE_HEADER_SIZE + 32 + DARE_TAG_SIZE;
const OBJECT_KEY_DERIVATION_CONTEXT: &[u8] = b"object-encryption-key generation";
use crate::error::ApiError;
use rustfs_ecstore::bucket::metadata_sys;
use rustfs_ecstore::error::Error;
use crate::storage::storage_compat::ecstore::bucket::metadata_sys;
use crate::storage::storage_compat::ecstore::error::Error;
use rustfs_utils::http::headers::{
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY,
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT,
@@ -745,17 +745,29 @@ pub struct ManagedSealedKey {
}
impl EncryptionMaterial {
pub fn write_encryption(&self, multipart_part_number: Option<usize>) -> rustfs_ecstore::rio::WriteEncryption {
pub fn write_encryption(
&self,
multipart_part_number: Option<usize>,
) -> crate::storage::storage_compat::ecstore::rio::WriteEncryption {
match (self.key_kind, multipart_part_number) {
(EncryptionKeyKind::Object, Some(part_number)) => {
rustfs_ecstore::rio::WriteEncryption::multipart_object_key(self.key_bytes, part_number as u32)
crate::storage::storage_compat::ecstore::rio::WriteEncryption::multipart_object_key(
self.key_bytes,
part_number as u32,
)
}
(EncryptionKeyKind::Object, None) => {
crate::storage::storage_compat::ecstore::rio::WriteEncryption::singlepart_object_key(self.key_bytes)
}
(EncryptionKeyKind::Object, None) => rustfs_ecstore::rio::WriteEncryption::singlepart_object_key(self.key_bytes),
(EncryptionKeyKind::Direct, Some(part_number)) => {
rustfs_ecstore::rio::WriteEncryption::multipart(self.key_bytes, self.base_nonce, part_number)
crate::storage::storage_compat::ecstore::rio::WriteEncryption::multipart(
self.key_bytes,
self.base_nonce,
part_number,
)
}
(EncryptionKeyKind::Direct, None) => {
rustfs_ecstore::rio::WriteEncryption::singlepart(self.key_bytes, self.base_nonce)
crate::storage::storage_compat::ecstore::rio::WriteEncryption::singlepart(self.key_bytes, self.base_nonce)
}
}
}
rustfs/src/storage/storage_compat.rs
+15
View File
@@ -0,0 +1,15 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) use rustfs_ecstore as ecstore;