refactor: segment storage owner contracts by domain (#3911)

This commit is contained in:
Zhengchao An
2026-06-26 18:34:32 +08:00
committed by GitHub
parent a8327f8901
commit a010dce93c
11 changed files with 140 additions and 45 deletions
+49 -9
View File
@@ -5,15 +5,18 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
## Current Context
- Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
- Branch: `overtrue/arch-storage-contract-boundary-batch`
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213/API-214/API-215/API-216/API-217/API-218/API-219/API-220/API-221/API-222/API-223/API-224/API-225/API-226/API-227/API-228/API-229/API-230/API-231/API-232/API-233/API-234/API-235/API-236/API-237/API-238/API-239/API-240/API-241/API-242/API-243/CTX-002`.
- Based on: rebased onto current `origin/main` after PR #3909 merged.
- Branch: `overtrue/arch-storage-owner-contract-domain-batch`
- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210/API-211/API-212/API-213/API-214/API-215/API-216/API-217/API-218/API-219/API-220/API-221/API-222/API-223/API-224/API-225/API-226/API-227/API-228/API-229/API-230/API-231/API-232/API-233/API-234/API-235/API-236/API-237/API-238/API-239/API-240/API-241/API-242/API-243/API-244/CTX-002`.
- Based on: rebased onto current `origin/main` after PR #3910 merged.
- PR type for this branch: `consumer-migration`
- Runtime behavior changes: none expected for API-244; ECStore still uses the
same storage contracts, now exposed from `storage_api_contracts` domain
modules instead of its flat root facade.
- Rust code changes: route replication pool, outbound TLS generation, runtime
region, KMS encryption service, runtime support handles, S3 Select DB,
- Runtime behavior changes: none expected for API-245; storage owner code still
uses the same storage contracts, now exposed from `contract` domain modules
instead of its flat root facade.
- Rust code changes: segment the storage-owner local `contract` facade into
domain modules, migrate storage-owner consumers to domain contract imports,
reject flat storage-owner contract consumers in migration guardrails, route
replication pool, outbound TLS generation, runtime region, KMS encryption
service, runtime support handles, S3 Select DB,
internode RPC metrics, IAM authorization/handler reads, notification
rules/event dispatch, admin OIDC/token-signing reads, IAM root credential
consumers, IAM OIDC config reads, scanner runtime-config reads, OBS metrics
@@ -5603,14 +5606,32 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
guards, ECStore root contract re-export/root consumer scans, diff hygiene,
Rust risk scan, and full PR gate passed before PR.
- [x] `API-245` Segment storage-owner contract facade by domain module.
- Do: split the storage-owner local `contract` facade into admin, bucket,
list, multipart, object, range, and topology modules, then migrate storage
owner consumers to those modules.
- Acceptance: `rustfs/src/storage/storage_api.rs` no longer uses raw
storage contract paths internally, storage owner consumers no longer import
contract symbols from the flat `contract` root, and migration rules reject
both regressions.
- Must preserve: ECFS bucket/object calls, storage RPC bucket/admin calls,
S3 list/multipart DTO projections, request option parsing, access checks,
and topology snapshot construction.
- Verification: focused RustFS compile, formatting, migration/layer guards,
storage-owner raw contract/root consumer scans, diff hygiene, and Rust risk
scan passed; full PR gate is planned before PR.
## Next PRs
1. `consumer-migration`: continue larger owner boundary batches after API-244.
1. `consumer-migration`: continue larger owner boundary batches after API-245.
## Pre-Push Review Log
| Expert | Status | Notes |
|---|---|---|
| Quality/architecture | pass | API-245 segments the storage-owner local contract facade into domain modules instead of flat root contract exposure. |
| Migration preservation | pass | ECFS bucket/object calls, storage RPC bucket/admin calls, S3 list/multipart DTO projections, request option parsing, access checks, and topology snapshot construction keep the same underlying contracts. |
| Testing/verification | pass | Focused RustFS compile, formatting, migration/layer guards, storage-owner raw contract/root consumer scans, diff hygiene, and diff-added Rust risk scan passed; full PR gate is planned before PR. |
| Quality/architecture | pass | API-244 segments ECStore storage_api_contracts into domain modules instead of a flat root facade. |
| Migration preservation | pass | ECStore object/list/multipart/heal/admin trait impls, lifecycle/tier/replication DTOs, config storage, peer S3 client, range helpers, topology snapshots, and error mappings keep the same contracts. |
| Testing/verification | pass | Focused ECStore compile, formatting, migration/layer guards, ECStore root contract re-export/root consumer scans, diff hygiene, diff-added Rust risk scan, and full PR gate passed before PR. |
@@ -5884,6 +5905,25 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block
Passed before push:
- Issue #660 API-245 current slice:
- Branch freshness check: rebased onto current `origin/main` after PR #3910
merged.
- `cargo check -p rustfs --lib`: passed.
- `cargo fmt --all`: passed.
- `./scripts/check_architecture_migration_rules.sh`: passed.
- `./scripts/check_layer_dependencies.sh`: passed.
- Storage-owner raw contract path scan: passed; no raw
`rustfs_storage_api::Type` contract paths remain in the owner boundary.
- Storage-owner root contract consumer scan: passed; storage consumers now
import contract symbols through domain modules.
- Diff-added Rust risk scan: passed; no new production unwrap/expect,
numeric cast, String error, Box dyn Error, print macro, or relaxed atomic
ordering lines.
- `cargo fmt --all --check`: passed.
- `git diff --check`: passed.
- `make pre-pr`: passed; nextest ran 6776 tests with 6776 passed and
112 skipped, and doctests passed.
- Issue #660 API-244 current slice:
- Branch freshness check: rebased onto current `origin/main` after PR #3909
merged.
+1 -1
View File
@@ -22,7 +22,7 @@ use crate::auth::{check_key_valid, get_condition_values_with_query_and_client_in
use crate::error::ApiError;
use crate::license::license_check;
use crate::server::RemoteAddr;
use crate::storage::contract::BucketOperations;
use crate::storage::contract::bucket::BucketOperations;
use crate::storage::request_context::RequestContext;
use crate::storage::runtime_sources;
use metrics::counter;
+4 -1
View File
@@ -23,7 +23,10 @@ use super::{
use super::{StorageReplicationConfigExt as _, StorageVersioningConfigExt as _};
use crate::error::ApiError;
use crate::storage::access::has_bypass_governance_header;
use crate::storage::contract::{BucketOperations, BucketOptions, ObjectLockRetentionOptions, ObjectOperations as _};
use crate::storage::contract::{
bucket::{BucketOperations, BucketOptions},
object::{ObjectLockRetentionOptions, ObjectOperations as _},
};
use crate::storage::helper::OperationHelper;
use crate::storage::options::get_opts;
use crate::storage::runtime_sources;
+4 -1
View File
@@ -20,7 +20,10 @@ use super::{
use crate::config::{RustFSBufferConfig, WorkloadProfile, is_buffer_profile_enabled};
use crate::error::ApiError;
use crate::server::cors;
use crate::storage::contract::{BucketOperations, BucketOptions, ObjectToDelete};
use crate::storage::contract::{
bucket::{BucketOperations, BucketOptions},
object::ObjectToDelete,
};
use crate::storage::ecfs::ListObjectUnorderedQuery;
use http::header::{IF_MATCH, IF_MODIFIED_SINCE, IF_NONE_MATCH, IF_UNMODIFIED_SINCE};
use http::{HeaderMap, HeaderValue, StatusCode};
+1 -1
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use super::ECStore;
use crate::storage::contract::ListOperations as _;
use crate::storage::contract::list::ListOperations as _;
use std::sync::Arc;
/// Determines if the key "looks like a prefix" (ends with `/`).
+1 -1
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use super::{BucketVersioningSys, Result, StorageError};
use crate::storage::contract::{HTTPPreconditions, HTTPRangeSpec};
use crate::storage::contract::{object::HTTPPreconditions, range::HTTPRangeSpec};
use http::header::{IF_MATCH, IF_NONE_MATCH};
use http::{HeaderMap, HeaderValue};
use rustfs_utils::http::{
+4 -1
View File
@@ -23,7 +23,10 @@ use crate::admin::service::{
config::{reload_dynamic_config_runtime_state, reload_runtime_config_snapshot},
site_replication::reload_site_replication_runtime_state,
};
use crate::storage::contract::{BucketOptions, DeleteBucketOptions, MakeBucketOptions, StorageAdminApi};
use crate::storage::contract::{
admin::StorageAdminApi,
bucket::{BucketOptions, DeleteBucketOptions, MakeBucketOptions},
};
use crate::storage::runtime_sources;
use bytes::Bytes;
use futures::Stream;
+3 -2
View File
@@ -13,7 +13,8 @@
// limitations under the License.
use crate::storage::contract::{
BucketInfo, ListObjectVersionsInfo as StorageListObjectVersionsInfo, ListObjectsV2Info as StorageListObjectsV2Info,
bucket::BucketInfo,
list::{ListObjectVersionsInfo as StorageListObjectVersionsInfo, ListObjectsV2Info as StorageListObjectsV2Info},
};
use crate::storage::s3_api::common::rustfs_owner;
use crate::storage::to_s3s_etag;
@@ -395,7 +396,7 @@ mod tests {
parse_list_object_versions_params, parse_list_objects_v2_params,
};
use crate::storage::StorageObjectInfo as ObjectInfo;
use crate::storage::contract::BucketInfo;
use crate::storage::contract::bucket::BucketInfo;
use crate::storage::s3_api::common::rustfs_owner;
use s3s::S3ErrorCode;
use s3s::dto::{CommonPrefix, EncodingType, ListObjectsV2Output, Object};
+2 -2
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::storage::contract::{ListMultipartsInfo, ListPartsInfo};
use crate::storage::contract::multipart::{ListMultipartsInfo, ListPartsInfo};
use crate::storage::s3_api::common::{rustfs_initiator, rustfs_owner};
use crate::storage::to_s3s_etag;
use s3s::dto::{CommonPrefix, ListMultipartUploadsOutput, ListPartsOutput, MultipartUpload, Part, Timestamp};
@@ -195,7 +195,7 @@ mod tests {
MAX_MULTIPART_UPLOADS_LIST, build_list_multipart_uploads_output, build_list_parts_output,
parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number,
};
use crate::storage::contract::{ListMultipartsInfo, ListPartsInfo, MultipartInfo, PartInfo};
use crate::storage::contract::multipart::{ListMultipartsInfo, ListPartsInfo, MultipartInfo, PartInfo};
use crate::storage::s3_api::common::{rustfs_initiator, rustfs_owner};
use crate::storage::to_s3s_etag;
use s3s::S3ErrorCode;
+52 -26
View File
@@ -19,20 +19,46 @@ use std::sync::Arc;
use rustfs_storage_api as storage_contracts;
pub(crate) mod contract {
pub(crate) use rustfs_storage_api::{
BucketInfo, BucketOperations, BucketOptions, DeleteBucketOptions, HTTPPreconditions, HTTPRangeSpec, ListMultipartsInfo,
ListObjectVersionsInfo, ListObjectsV2Info, ListOperations, ListPartsInfo, MakeBucketOptions, ObjectLockRetentionOptions,
ObjectOperations, ObjectToDelete, StorageAdminApi,
};
#[cfg(test)]
pub(crate) use rustfs_storage_api::{MultipartInfo, PartInfo};
pub(crate) mod admin {
pub(crate) use super::super::storage_contracts::StorageAdminApi;
}
pub(crate) mod bucket {
pub(crate) use super::super::storage_contracts::{
BucketInfo, BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions,
};
}
pub(crate) mod list {
pub(crate) use super::super::storage_contracts::{ListObjectVersionsInfo, ListObjectsV2Info, ListOperations};
}
pub(crate) mod multipart {
pub(crate) use super::super::storage_contracts::{ListMultipartsInfo, ListPartsInfo};
#[cfg(test)]
pub(crate) use super::super::storage_contracts::{MultipartInfo, PartInfo};
}
pub(crate) mod object {
pub(crate) use super::super::storage_contracts::{
DeletedObject, HTTPPreconditions, ObjectIO, ObjectLockRetentionOptions, ObjectOperations, ObjectToDelete,
};
}
pub(crate) mod range {
pub(crate) use super::super::storage_contracts::HTTPRangeSpec;
}
pub(crate) mod topology {
pub(crate) use super::super::storage_contracts::{DiskCapabilities, TopologyCapabilities, TopologySnapshot};
}
}
pub(crate) type StorageDeletedObject = storage_contracts::DeletedObject;
pub(crate) type StorageDeletedObject = contract::object::DeletedObject;
pub(crate) type StorageGetObjectReader = super::GetObjectReader;
pub(crate) type StorageObjectInfo = super::ObjectInfo;
pub(crate) type StorageObjectOptions = super::ObjectOptions;
pub(crate) type StorageObjectToDelete = storage_contracts::ObjectToDelete;
pub(crate) type StorageObjectToDelete = contract::object::ObjectToDelete;
pub(crate) type StoragePutObjReader = super::PutObjReader;
pub(crate) mod ecstore_admin {
@@ -630,14 +656,14 @@ pub(crate) trait StoragePeerS3ClientExt {
bucket: &str,
opts: &rustfs_common::heal_channel::HealOpts,
) -> DiskResult<rustfs_madmin::heal_commands::HealResultItem>;
async fn make_bucket(&self, bucket: &str, opts: &storage_contracts::MakeBucketOptions) -> DiskResult<()>;
async fn list_bucket(&self, opts: &storage_contracts::BucketOptions) -> DiskResult<Vec<storage_contracts::BucketInfo>>;
async fn delete_bucket(&self, bucket: &str, opts: &storage_contracts::DeleteBucketOptions) -> DiskResult<()>;
async fn make_bucket(&self, bucket: &str, opts: &contract::bucket::MakeBucketOptions) -> DiskResult<()>;
async fn list_bucket(&self, opts: &contract::bucket::BucketOptions) -> DiskResult<Vec<contract::bucket::BucketInfo>>;
async fn delete_bucket(&self, bucket: &str, opts: &contract::bucket::DeleteBucketOptions) -> DiskResult<()>;
async fn get_bucket_info(
&self,
bucket: &str,
opts: &storage_contracts::BucketOptions,
) -> DiskResult<storage_contracts::BucketInfo>;
opts: &contract::bucket::BucketOptions,
) -> DiskResult<contract::bucket::BucketInfo>;
}
impl StoragePeerS3ClientExt for LocalPeerS3Client {
@@ -649,23 +675,23 @@ impl StoragePeerS3ClientExt for LocalPeerS3Client {
ecstore_rpc::PeerS3Client::heal_bucket(self, bucket, opts).await
}
async fn make_bucket(&self, bucket: &str, opts: &storage_contracts::MakeBucketOptions) -> DiskResult<()> {
async fn make_bucket(&self, bucket: &str, opts: &contract::bucket::MakeBucketOptions) -> DiskResult<()> {
ecstore_rpc::PeerS3Client::make_bucket(self, bucket, opts).await
}
async fn list_bucket(&self, opts: &storage_contracts::BucketOptions) -> DiskResult<Vec<storage_contracts::BucketInfo>> {
async fn list_bucket(&self, opts: &contract::bucket::BucketOptions) -> DiskResult<Vec<contract::bucket::BucketInfo>> {
ecstore_rpc::PeerS3Client::list_bucket(self, opts).await
}
async fn delete_bucket(&self, bucket: &str, opts: &storage_contracts::DeleteBucketOptions) -> DiskResult<()> {
async fn delete_bucket(&self, bucket: &str, opts: &contract::bucket::DeleteBucketOptions) -> DiskResult<()> {
ecstore_rpc::PeerS3Client::delete_bucket(self, bucket, opts).await
}
async fn get_bucket_info(
&self,
bucket: &str,
opts: &storage_contracts::BucketOptions,
) -> DiskResult<storage_contracts::BucketInfo> {
opts: &contract::bucket::BucketOptions,
) -> DiskResult<contract::bucket::BucketInfo> {
ecstore_rpc::PeerS3Client::get_bucket_info(self, bucket, opts).await
}
}
@@ -926,9 +952,9 @@ where
pub(crate) fn topology_snapshot_from_endpoint_pools_with_capabilities(
endpoint_pools: &EndpointServerPools,
capabilities: storage_contracts::TopologyCapabilities,
disk_capabilities: storage_contracts::DiskCapabilities,
) -> storage_contracts::TopologySnapshot {
capabilities: contract::topology::TopologyCapabilities,
disk_capabilities: contract::topology::DiskCapabilities,
) -> contract::topology::TopologySnapshot {
ecstore_cluster::topology_snapshot_from_endpoint_pools_with_capabilities(endpoint_pools, capabilities, disk_capabilities)
}
@@ -966,7 +992,7 @@ impl StorageVersioningConfigExt for s3s::dto::VersioningConfiguration {
}
}
pub(crate) type GetObjectReader = <ECStore as storage_contracts::ObjectIO>::GetObjectReader;
pub(crate) type ObjectInfo = <ECStore as storage_contracts::ObjectOperations>::ObjectInfo;
pub(crate) type ObjectOptions = <ECStore as storage_contracts::ObjectOperations>::ObjectOptions;
pub(crate) type PutObjReader = <ECStore as storage_contracts::ObjectIO>::PutObjectReader;
pub(crate) type GetObjectReader = <ECStore as contract::object::ObjectIO>::GetObjectReader;
pub(crate) type ObjectInfo = <ECStore as contract::object::ObjectOperations>::ObjectInfo;
pub(crate) type ObjectOptions = <ECStore as contract::object::ObjectOperations>::ObjectOptions;
pub(crate) type PutObjReader = <ECStore as contract::object::ObjectIO>::PutObjectReader;
@@ -94,6 +94,7 @@ RUSTFS_ADMIN_CONFIG_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_admin_con
RUSTFS_STORAGE_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_storage_bucket_storage_compat_module_hits.txt"
RUSTFS_STORAGE_OWNER_COMPAT_REEXPORT_HITS_FILE="${TMP_DIR}/rustfs_storage_owner_compat_reexport_hits.txt"
RUSTFS_STORAGE_OWNER_DIRECT_STORAGE_SOURCE_HITS_FILE="${TMP_DIR}/rustfs_storage_owner_direct_storage_source_hits.txt"
RUSTFS_STORAGE_OWNER_CONTRACT_ROOT_CONSUMER_HITS_FILE="${TMP_DIR}/rustfs_storage_owner_contract_root_consumer_hits.txt"
RUSTFS_ADMIN_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_admin_bucket_storage_compat_module_hits.txt"
RUSTFS_APP_BUCKET_STORAGE_COMPAT_MODULE_HITS_FILE="${TMP_DIR}/rustfs_app_bucket_storage_compat_module_hits.txt"
RUSTFS_OUTER_COMPAT_FACADE_ALIAS_HITS_FILE="${TMP_DIR}/rustfs_outer_compat_facade_alias_hits.txt"
@@ -1063,6 +1064,24 @@ if [[ -s "$RUSTFS_STORAGE_OWNER_DIRECT_STORAGE_SOURCE_HITS_FILE" ]]; then
report_failure "RustFS storage owner modules must route ECStore and storage-api symbols through rustfs/src/storage/storage_api.rs: $(paste -sd '; ' "$RUSTFS_STORAGE_OWNER_DIRECT_STORAGE_SOURCE_HITS_FILE")"
fi
(
cd "$ROOT_DIR"
{
rg -n -U --with-filename 'crate::storage::contract::\{\s*[A-Z]' \
rustfs/src/storage \
--glob '*.rs' \
--glob '!storage_api.rs' || true
rg -n --with-filename 'crate::storage::contract::[A-Z]' \
rustfs/src/storage \
--glob '*.rs' \
--glob '!storage_api.rs' || true
}
) >"$RUSTFS_STORAGE_OWNER_CONTRACT_ROOT_CONSUMER_HITS_FILE"
if [[ -s "$RUSTFS_STORAGE_OWNER_CONTRACT_ROOT_CONSUMER_HITS_FILE" ]]; then
report_failure "RustFS storage owner modules must import storage contracts from domain modules, not the root contract facade: $(paste -sd '; ' "$RUSTFS_STORAGE_OWNER_CONTRACT_ROOT_CONSUMER_HITS_FILE")"
fi
(
cd "$ROOT_DIR"
rg -n --with-filename 'rustfs_storage_api::[A-Za-z_]' \