diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index c40b17666..77cf04819 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,9 +5,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-app-usecase-context-import-batch` -- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208`. -- Based on: stacked on PR #3848 head while PR #3848 is pending. +- Branch: `overtrue/arch-app-explicit-dto-imports` +- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209`. +- Based on: stacked on API-208 while PR #3848 is pending. - PR type for this branch: `consumer-migration` - Runtime behavior changes: none. - Rust code changes: route replication pool, outbound TLS generation, runtime @@ -45,7 +45,7 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block readiness-time, observability, metrics, buffer, and TLS runtime source helpers, plus server readiness/audit/event/module-switch runtime source helpers, storage request/RPC/SSE runtime source helpers, and admin - handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, and app bucket/object/multipart usecase explicit storage imports, + handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, app bucket/object/multipart usecase explicit storage imports, and app select/bucket/object/multipart explicit S3 DTO plus ECFS owner imports, through AppContext-first or owner-crate resolver boundaries. - CI/script changes: lock completed owner and test/fuzz boundaries against bare/glob imports, scattered raw ECStore facade subpaths, and startup @@ -56,8 +56,8 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block event-bridge thin module regressions, plus IAM runtime-source bypasses; accept the reviewed AppContext resolver reverse dependencies in the layer baseline, and block direct admin AppContext resolver consumers outside the - admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, and reject app usecase storage wildcard imports. -- Docs changes: record the API-136 through API-208 owner facade and lifecycle + admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, and narrow the object-usecase ECFS layer baseline entry to `FS`. +- Docs changes: record the API-136 through API-209 owner facade and lifecycle runtime-source cleanup. ## Phase 0 Tasks @@ -4996,6 +4996,21 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block layer guards, diff hygiene, residual wildcard scan, Rust risk scan, fast PR gate, and full PR gate before PR. +- [x] `API-209` Remove app DTO and ECFS wildcard imports. + - Do: replace broad `s3s::dto::*` imports in app select, bucket, object, and + multipart usecases, replace the object usecase ECFS wildcard with the + explicit owner type, and narrow the matching layer baseline entry. + - Acceptance: app consumers no longer import S3 DTOs or ECFS owner APIs + through wildcards, and migration rules reject reintroducing those wildcard + imports in app-layer files. + - Must preserve: select-object request validation and event streaming, bucket + DTO mapping/list output behavior, object CRUD/copy/delete/restore DTO + mapping, multipart upload/copy checksum behavior, and existing ECFS FS test + harness usage. + - Verification: focused RustFS app compile/tests, formatting, migration and + layer guards, diff hygiene, residual wildcard scan, Rust risk scan, fast PR + gate, and full PR gate before PR. + ## Next PRs 1. `consumer-migration`: continue reducing direct global reads behind AppContext resolver boundaries. @@ -5004,6 +5019,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block | Expert | Status | Notes | |---|---|---| +| Quality/architecture | pass | API-209 removes app-layer S3 DTO and ECFS wildcard imports and adds a regression guard for the completed boundary. | +| Migration preservation | pass | Select-object streaming, bucket DTO mapping, object CRUD/copy/delete/restore DTO handling, multipart checksum/copy behavior, and ECFS FS test harness usage keep the same call paths. | +| Testing/verification | pass | Focused RustFS app compile/tests, formatting, migration/layer guards, residual wildcard scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. | | Quality/architecture | pass | API-208 removes broad storage wildcard imports from app bucket, object, and multipart usecases and adds a regression guard. | | Migration preservation | pass | Bucket listing/notification, object precondition/SSE/object-lock/CORS, and multipart SSE/object-lock behavior keep the same storage owner call sites. | | Testing/verification | pass | Focused app compile/tests, formatting, migration/layer guards, residual wildcard scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. | diff --git a/rustfs/src/app/bucket_usecase.rs b/rustfs/src/app/bucket_usecase.rs index 893ae2df0..412c6c2c7 100644 --- a/rustfs/src/app/bucket_usecase.rs +++ b/rustfs/src/app/bucket_usecase.rs @@ -79,7 +79,30 @@ use rustfs_trusted_proxies::ClientInfo; use rustfs_utils::http::{SUFFIX_FORCE_DELETE, get_header}; use rustfs_utils::obj::extract_user_defined_metadata; use rustfs_utils::string::parse_bool; -use s3s::dto::*; +use s3s::dto::{ + BucketLifecycleConfiguration, BucketLocationConstraint, CommonPrefix, CreateBucketInput, CreateBucketOutput, + DeleteBucketCorsInput, DeleteBucketCorsOutput, DeleteBucketEncryptionInput, DeleteBucketEncryptionOutput, DeleteBucketInput, + DeleteBucketLifecycleInput, DeleteBucketLifecycleOutput, DeleteBucketOutput, DeleteBucketPolicyInput, + DeleteBucketPolicyOutput, DeleteBucketReplicationInput, DeleteBucketReplicationOutput, DeleteBucketTaggingInput, + DeleteBucketTaggingOutput, DeleteMarkerM, DeletePublicAccessBlockInput, DeletePublicAccessBlockOutput, EncodingType, + ExpirationStatus, GetBucketCorsInput, GetBucketCorsOutput, GetBucketEncryptionInput, GetBucketEncryptionOutput, + GetBucketLifecycleConfigurationInput, GetBucketLifecycleConfigurationOutput, GetBucketLocationInput, GetBucketLocationOutput, + GetBucketNotificationConfigurationInput, GetBucketNotificationConfigurationOutput, GetBucketPolicyInput, + GetBucketPolicyOutput, GetBucketPolicyStatusInput, GetBucketPolicyStatusOutput, GetBucketReplicationInput, + GetBucketReplicationOutput, GetBucketTaggingInput, GetBucketTaggingOutput, GetBucketVersioningInput, + GetBucketVersioningOutput, GetPublicAccessBlockInput, GetPublicAccessBlockOutput, HeadBucketInput, HeadBucketOutput, + LifecycleRule, ListBucketsInput, ListBucketsOutput, ListObjectVersionMEntry, ListObjectVersionsInput, + ListObjectVersionsMOutput, ListObjectVersionsOutput, ListObjectsInput, ListObjectsOutput, ListObjectsV2Input, + ListObjectsV2MOutput, ListObjectsV2Output, NotificationConfiguration, NotificationConfigurationFilter, ObjectInternalInfo, + ObjectLockConfiguration, ObjectM, ObjectStorageClass, ObjectVersionM, ObjectVersionStorageClass, PolicyStatus, + PutBucketCorsInput, PutBucketCorsOutput, PutBucketEncryptionInput, PutBucketEncryptionOutput, + PutBucketLifecycleConfigurationInput, PutBucketLifecycleConfigurationOutput, PutBucketNotificationConfigurationInput, + PutBucketNotificationConfigurationOutput, PutBucketPolicyInput, PutBucketPolicyOutput, PutBucketReplicationInput, + PutBucketReplicationOutput, PutBucketTaggingInput, PutBucketTaggingOutput, PutBucketVersioningInput, + PutBucketVersioningOutput, PutPublicAccessBlockInput, PutPublicAccessBlockOutput, ReplicationConfiguration, + ReplicationRuleStatus, ServerSideEncryption, Tagging, Timestamp, UserMetadataCollection, UserMetadataEntry, + VersioningConfiguration, +}; use s3s::region::Region; use s3s::xml; use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error}; @@ -2186,6 +2209,11 @@ impl DefaultBucketUsecase { mod tests { use super::*; use http::{Extensions, HeaderMap, Method, Uri}; + use s3s::dto::{ + BucketVersioningStatus, CORSConfiguration, Destination, ExcludedPrefix, FilterRule, FilterRuleName, LifecycleExpiration, + NoncurrentVersionTransition, PublicAccessBlockConfiguration, QueueConfiguration, ReplicationRule, S3KeyFilter, + ServerSideEncryptionConfiguration, Tag, Transition, TransitionStorageClass, + }; use std::sync::Arc; fn build_request(input: T, method: Method) -> S3Request { diff --git a/rustfs/src/app/multipart_usecase.rs b/rustfs/src/app/multipart_usecase.rs index 67fd982fc..2bd64c7f9 100644 --- a/rustfs/src/app/multipart_usecase.rs +++ b/rustfs/src/app/multipart_usecase.rs @@ -71,7 +71,12 @@ use rustfs_utils::http::{ headers::{AMZ_DECODED_CONTENT_LENGTH, AMZ_OBJECT_TAGGING}, insert_str, }; -use s3s::dto::*; +use s3s::dto::{ + AbortMultipartUploadInput, AbortMultipartUploadOutput, ChecksumAlgorithm, ChecksumType, CompleteMultipartUploadInput, + CompleteMultipartUploadOutput, CompletedPart, CopyPartResult, CopySource, CreateMultipartUploadInput, + CreateMultipartUploadOutput, ETag, ListMultipartUploadsInput, ListMultipartUploadsOutput, ListPartsInput, ListPartsOutput, + ServerSideEncryption, StreamingBlob, Timestamp, UploadPartCopyInput, UploadPartCopyOutput, UploadPartInput, UploadPartOutput, +}; use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE}; use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error}; #[cfg(test)] @@ -1338,6 +1343,7 @@ mod tests { use rustfs_utils::http::{ AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, }; + use s3s::dto::{CompletedMultipartUpload, StorageClass}; use std::{collections::HashMap, io::Cursor}; use temp_env::async_with_vars; use tokio::io::AsyncReadExt; diff --git a/rustfs/src/app/object_usecase.rs b/rustfs/src/app/object_usecase.rs index 1f6942b31..70facc70d 100644 --- a/rustfs/src/app/object_usecase.rs +++ b/rustfs/src/app/object_usecase.rs @@ -57,7 +57,7 @@ use crate::storage::concurrency::{ ConcurrencyManager, GetObjectGuard, PutObjectGuard, get_concurrency_aware_buffer_size, get_concurrency_manager, get_put_concurrency_aware_buffer_size, }; -use crate::storage::ecfs::*; +use crate::storage::ecfs::FS; use crate::storage::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children}; use crate::storage::helper::{OperationHelper, spawn_background_with_context}; use crate::storage::options::{ @@ -123,7 +123,16 @@ use rustfs_utils::http::{ }; use rustfs_utils::path::{encode_dir_object, is_dir_object, path_join_buf}; use rustfs_zip::{ArchiveLimits, CompressionFormat}; -use s3s::dto::*; +use s3s::dto::{ + CacheControl, Checksum, ChecksumAlgorithm, ChecksumType, ContentDisposition, ContentEncoding, ContentLanguage, ContentType, + CopyObjectInput, CopyObjectOutput, CopyObjectResult, CopySource, DeleteObjectInput, DeleteObjectOutput, DeleteObjectsInput, + DeleteObjectsOutput, DeletedObject, ETag, GetObjectAttributesInput, GetObjectAttributesOutput, GetObjectAttributesParts, + GetObjectInput, GetObjectOutput, HeadObjectInput, HeadObjectOutput, MetadataDirective, ObjectAttributes, ObjectLockLegalHold, + ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, ObjectPart, PutObjectInput, + PutObjectOutput, Range, ReplicationConfiguration, RequestCharged, RestoreObjectInput, RestoreObjectOutput, RestoreStatus, + SSECustomerAlgorithm, SSECustomerKeyMD5, SSEKMSKeyId, SelectObjectContentInput, SelectObjectContentOutput, + ServerSideEncryption, StorageClass, StreamingBlob, TaggingHeader, Timestamp, TimestampFormat, WebsiteRedirectLocation, +}; use s3s::header::{X_AMZ_RESTORE, X_AMZ_RESTORE_OUTPUT_PATH}; use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error}; @@ -4997,8 +5006,9 @@ mod tests { use super::*; use http::{Extensions, HeaderMap, HeaderName, HeaderValue, Method, Uri}; use s3s::dto::{ - DeleteMarkerReplication, DeleteMarkerReplicationStatus, Destination, ExistingObjectReplication, - ExistingObjectReplicationStatus, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus, + Delete, DeleteMarkerReplication, DeleteMarkerReplicationStatus, Destination, ExistingObjectReplication, + ExistingObjectReplicationStatus, ObjectIdentifier, ReplicaModifications, ReplicaModificationsStatus, + ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus, RestoreRequest, SourceSelectionCriteria, }; use std::pin::Pin; use std::sync::Arc; diff --git a/rustfs/src/app/select_object.rs b/rustfs/src/app/select_object.rs index f99045105..f6eff9bf8 100644 --- a/rustfs/src/app/select_object.rs +++ b/rustfs/src/app/select_object.rs @@ -17,7 +17,12 @@ use rustfs_s3select_api::{ query::{Context, Query}, }; use rustfs_storage_api::ObjectOperations as _; -use s3s::dto::*; +use s3s::dto::{ + CSVOutput, CompressionType, ContinuationEvent, EndEvent, ExpressionType, FileHeaderInfo, InputSerialization, JSONInput, + JSONOutput, JSONType, OutputSerialization, Progress, ProgressEvent, QuoteFields, RecordsEvent, SelectObjectContentEvent, + SelectObjectContentEventStream, SelectObjectContentInput, SelectObjectContentOutput, SelectObjectContentRequest, Stats, + StatsEvent, +}; use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error}; use std::sync::Arc; use tokio::sync::mpsc; @@ -545,6 +550,7 @@ mod tests { use super::*; use datafusion::sql::sqlparser::parser::ParserError; use http::HeaderMap; + use s3s::dto::{CSVInput, ParquetInput, ScanRange}; fn base_input() -> SelectObjectContentInput { SelectObjectContentInput { diff --git a/rustfs/src/table_catalog.rs b/rustfs/src/table_catalog.rs index 7f2b04e70..58cde21d5 100644 --- a/rustfs/src/table_catalog.rs +++ b/rustfs/src/table_catalog.rs @@ -9190,10 +9190,14 @@ mod tests { #[derive(Clone, Default)] struct TestCatalogObjectBackend { - state: std::sync::Arc>, - locks: std::sync::Arc>>>>, + state: Arc>, + locks: TestCatalogObjectLocks, } + type TestCatalogObjectLockKey = (String, String); + type TestCatalogObjectLock = Arc>; + type TestCatalogObjectLocks = Arc>>; + #[derive(Default)] struct TestCatalogObjectState { objects: BTreeMap<(String, String), TestCatalogObjectRecord>, diff --git a/scripts/check_architecture_migration_rules.sh b/scripts/check_architecture_migration_rules.sh index 05c6e157d..91c79cff0 100755 --- a/scripts/check_architecture_migration_rules.sh +++ b/scripts/check_architecture_migration_rules.sh @@ -160,6 +160,7 @@ RUSTFS_ADMIN_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_admin_runtime_so RUSTFS_APP_CONTEXT_DIRECT_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_context_direct_bypass_hits.txt" RUSTFS_APP_USECASE_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_usecase_runtime_source_bypass_hits.txt" RUSTFS_APP_USECASE_STORAGE_WILDCARD_HITS_FILE="${TMP_DIR}/rustfs_app_usecase_storage_wildcard_hits.txt" +RUSTFS_APP_WILDCARD_IMPORT_HITS_FILE="${TMP_DIR}/rustfs_app_wildcard_import_hits.txt" RUSTFS_STORAGE_DIRECT_APP_CONTEXT_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_storage_direct_app_context_bypass_hits.txt" awk ' @@ -1418,6 +1419,15 @@ if [[ -s "$RUSTFS_APP_USECASE_STORAGE_WILDCARD_HITS_FILE" ]]; then report_failure "RustFS app usecase consumers must import storage owner APIs explicitly instead of crate::storage::*: $(paste -sd '; ' "$RUSTFS_APP_USECASE_STORAGE_WILDCARD_HITS_FILE")" fi +( + cd "$ROOT_DIR" + rg -n --with-filename 'use (s3s::dto|crate::storage::ecfs)::\*;' rustfs/src/app --glob '*.rs' || true +) >"$RUSTFS_APP_WILDCARD_IMPORT_HITS_FILE" + +if [[ -s "$RUSTFS_APP_WILDCARD_IMPORT_HITS_FILE" ]]; then + report_failure "RustFS app consumers must import S3 DTO and ECFS owner APIs explicitly instead of wildcard imports: $(paste -sd '; ' "$RUSTFS_APP_WILDCARD_IMPORT_HITS_FILE")" +fi + ( cd "$ROOT_DIR" rg -n --with-filename 'crate::app::context::|use crate::app::context|app::context::' rustfs/src/storage --glob '*.rs' | diff --git a/scripts/layer-dependency-baseline.txt b/scripts/layer-dependency-baseline.txt index 1b3505301..d3ba2176f 100644 --- a/scripts/layer-dependency-baseline.txt +++ b/scripts/layer-dependency-baseline.txt @@ -37,7 +37,7 @@ dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::m dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_list_multipart_uploads_params dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_list_parts_params dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_upload_part_number -dep|rustfs/src/app/object_usecase.rs|app->interface|crate::storage::ecfs +dep|rustfs/src/app/object_usecase.rs|app->interface|crate::storage::ecfs::FS dep|rustfs/src/app/object_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_list_parts_params dep|rustfs/src/init.rs|infra->interface|crate::admin dep|rustfs/src/protocols/client.rs|infra->interface|crate::storage::ecfs::FS