diff --git a/docs/architecture/migration-progress.md b/docs/architecture/migration-progress.md index 77cf04819..54075f113 100644 --- a/docs/architecture/migration-progress.md +++ b/docs/architecture/migration-progress.md @@ -5,9 +5,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block ## Current Context - Issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660) -- Branch: `overtrue/arch-app-explicit-dto-imports` -- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209`. -- Based on: stacked on API-208 while PR #3848 is pending. +- Branch: `overtrue/arch-app-s3-api-boundary` +- Baseline: completed `C-011/C-012/C-013/API-055/API-059/API-079/API-080/API-081/API-082/API-083/API-084/API-085/API-086/API-087/API-088/API-089/API-090/API-091/API-092/API-093/API-094/API-095/API-096/API-097/API-098/API-099/API-100/API-101/API-102/API-103/API-104/API-105/API-106/API-107/API-108/API-109/API-110/API-111/API-112/API-113/API-114/API-115/API-116/API-117/API-118/API-119/API-120/API-121/API-122/API-123/API-124/API-125/API-126/API-127/API-128/API-129/API-130/API-131/API-132/API-133/API-134/API-135/API-136/API-137/API-138/API-139/API-140/API-141/API-142/API-143/API-144/API-145/API-146/API-147/API-148/API-149/API-150/API-151/API-152/API-153/API-154/API-155/API-156/API-157/API-158/API-159/API-160/API-161/API-162/API-163/API-164/API-165/API-166/API-167/API-168/API-169/API-170/API-171/API-172/API-173/API-174/API-175/API-176/API-177/API-178/API-179/API-180/API-181/API-182/API-183/API-184/API-185/API-186/API-187/API-188/API-189/API-190/API-191/API-192/API-193/API-194/API-195/API-196/API-197/API-198/API-199/API-200/API-201/API-202/API-203/API-204/API-205/API-206/API-207/API-208/API-209/API-210`. +- Based on: latest main after PR #3853 merged. - PR type for this branch: `consumer-migration` - Runtime behavior changes: none. - Rust code changes: route replication pool, outbound TLS generation, runtime @@ -45,7 +45,7 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block readiness-time, observability, metrics, buffer, and TLS runtime source helpers, plus server readiness/audit/event/module-switch runtime source helpers, storage request/RPC/SSE runtime source helpers, and admin - handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, app bucket/object/multipart usecase explicit storage imports, and app select/bucket/object/multipart explicit S3 DTO plus ECFS owner imports, + handler/service/router runtime source helpers, plus root auth/init/config/protocol/workload, app usecase, storage node-service, remaining admin grouped context import runtime source helpers, app bucket/object/multipart usecase explicit storage imports, app select/bucket/object/multipart explicit S3 DTO plus ECFS owner imports, and app-local S3 API response helper imports for bucket/object/multipart usecases, through AppContext-first or owner-crate resolver boundaries. - CI/script changes: lock completed owner and test/fuzz boundaries against bare/glob imports, scattered raw ECStore facade subpaths, and startup @@ -56,8 +56,8 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block event-bridge thin module regressions, plus IAM runtime-source bypasses; accept the reviewed AppContext resolver reverse dependencies in the layer baseline, and block direct admin AppContext resolver consumers outside the - admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, and narrow the object-usecase ECFS layer baseline entry to `FS`. -- Docs changes: record the API-136 through API-209 owner facade and lifecycle + admin runtime-source boundary, block root, app usecase, and storage direct AppContext resolver consumers outside their runtime-source boundaries, catch grouped AppContext imports, reject app usecase storage wildcard imports, reject app-layer S3 DTO and ECFS wildcard imports, narrow the object-usecase ECFS layer baseline entry to `FS`, and reject direct storage S3 API helper imports from app usecase files. +- Docs changes: record the API-136 through API-210 owner facade and lifecycle runtime-source cleanup. ## Phase 0 Tasks @@ -5011,6 +5011,20 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block layer guards, diff hygiene, residual wildcard scan, Rust risk scan, fast PR gate, and full PR gate before PR. +- [x] `API-210` Route app S3 API helper imports through app boundary. + - Do: add an app-local S3 API helper boundary and route bucket, object, and + multipart usecases through it instead of direct `crate::storage::s3_api` + imports. + - Acceptance: app usecase files no longer import storage S3 API helpers + directly, layer baseline records the reviewed app boundary, and migration + rules reject direct usecase bypasses. + - Must preserve: bucket list response mapping, multipart list/part-number + parsing, object list-parts parameter parsing, owner/initiator response + metadata, and existing storage S3 API helper behavior. + - Verification: focused RustFS app compile/tests, formatting, migration and + layer guards, diff hygiene, residual direct S3 API scan, Rust risk scan, + fast PR gate, and full PR gate before PR. + ## Next PRs 1. `consumer-migration`: continue reducing direct global reads behind AppContext resolver boundaries. @@ -5019,6 +5033,9 @@ Status values: `[ ]` not started, `[~]` in progress, `[x]` complete, `[!]` block | Expert | Status | Notes | |---|---|---| +| Quality/architecture | pass | API-210 routes app usecase S3 API helper imports through an app-local boundary and adds a direct-bypass guard. | +| Migration preservation | pass | Bucket list response mapping, multipart list/part-number parsing, object list-parts parsing, and S3 owner metadata keep the same storage helper implementations. | +| Testing/verification | pass | Focused RustFS app compile/tests, formatting, migration/layer guards, residual direct S3 API scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. | | Quality/architecture | pass | API-209 removes app-layer S3 DTO and ECFS wildcard imports and adds a regression guard for the completed boundary. | | Migration preservation | pass | Select-object streaming, bucket DTO mapping, object CRUD/copy/delete/restore DTO handling, multipart checksum/copy behavior, and ECFS FS test harness usage keep the same call paths. | | Testing/verification | pass | Focused RustFS app compile/tests, formatting, migration/layer guards, residual wildcard scan, Rust risk scan, fast PR gate, and full PR gate are planned before PR. | diff --git a/rustfs/src/app/bucket_usecase.rs b/rustfs/src/app/bucket_usecase.rs index 412c6c2c7..8304b2186 100644 --- a/rustfs/src/app/bucket_usecase.rs +++ b/rustfs/src/app/bucket_usecase.rs @@ -17,6 +17,11 @@ use super::ECStore; use super::StorageError; use super::object_api_utils::to_s3s_etag; +use super::s3_api::bucket::{ + ListObjectVersionsParams, ListObjectsV2Params, build_list_buckets_output, build_list_object_versions_output, + build_list_objects_output, build_list_objects_v2_output, parse_list_object_versions_params, parse_list_objects_v2_params, + rustfs_owner, +}; use super::{AppObjectLockConfigExt as _, AppVersioningConfigExt as _}; use super::{ bucket_target_sys::BucketTargetSys, @@ -48,11 +53,6 @@ use crate::server::RemoteAddr; use crate::storage::StorageObjectInfo as ObjectInfo; use crate::storage::access::{ReqInfo, authorize_request, req_info_ref}; use crate::storage::helper::{OperationHelper, spawn_background_with_context}; -use crate::storage::s3_api::bucket::{ - ListObjectVersionsParams, ListObjectsV2Params, build_list_buckets_output, build_list_object_versions_output, - build_list_objects_output, build_list_objects_v2_output, parse_list_object_versions_params, parse_list_objects_v2_params, -}; -use crate::storage::s3_api::common::rustfs_owner; use crate::storage::{ get_validated_store, process_lambda_configurations, process_queue_configurations, process_topic_configurations, request_context, validate_list_object_unordered_with_delimiter, diff --git a/rustfs/src/app/mod.rs b/rustfs/src/app/mod.rs index f4bbe6a20..eb0e0795c 100644 --- a/rustfs/src/app/mod.rs +++ b/rustfs/src/app/mod.rs @@ -21,6 +21,7 @@ pub mod context; pub mod multipart_usecase; pub mod object_usecase; pub(crate) mod runtime_sources; +pub(crate) mod s3_api; mod select_object; #[cfg(test)] diff --git a/rustfs/src/app/multipart_usecase.rs b/rustfs/src/app/multipart_usecase.rs index 2bd64c7f9..2eceddece 100644 --- a/rustfs/src/app/multipart_usecase.rs +++ b/rustfs/src/app/multipart_usecase.rs @@ -19,6 +19,10 @@ use super::is_disk_compressible; use super::is_valid_storage_class; use super::object_api_utils::to_s3s_etag; use super::quota::checker::QuotaChecker; +use super::s3_api::multipart::{ + ListMultipartUploadsParams, build_list_multipart_uploads_output, build_list_parts_output, + parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number, +}; #[cfg(test)] use super::{DecryptReader, EncryptReader, HardLimitReader, boxed_reader, wrap_reader}; use super::{HashReader, WritePlan}; @@ -40,10 +44,6 @@ use crate::storage::options::{ copy_src_opts, extract_metadata_from_mime, get_complete_multipart_upload_opts, get_content_sha256_with_query, get_opts, parse_copy_source_range, put_opts, validate_archive_content_encoding, }; -use crate::storage::s3_api::multipart::{ - ListMultipartUploadsParams, build_list_multipart_uploads_output, build_list_parts_output, - parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number, -}; use crate::storage::sse::{ build_ssec_read_headers, encryption_material_to_metadata, extract_ssec_params_from_headers, extract_ssekms_context_from_headers, map_get_object_reader_error, mark_encrypted_multipart_metadata, diff --git a/rustfs/src/app/object_usecase.rs b/rustfs/src/app/object_usecase.rs index 70facc70d..e42629174 100644 --- a/rustfs/src/app/object_usecase.rs +++ b/rustfs/src/app/object_usecase.rs @@ -19,6 +19,7 @@ use super::quota::checker::QuotaChecker; use super::storageclass; // Performance metrics recording (with zero-copy-metrics integration) use super::ECStore; +use super::s3_api::multipart::parse_list_parts_params; use super::{AppReplicationConfigExt as _, AppVersioningConfigExt as _, predict_lifecycle_expiration, validate_restore_request}; use super::{DiskError, is_all_buckets_not_found}; use super::{DynReader, HashReader, WritePlan, wrap_reader}; @@ -65,7 +66,6 @@ use crate::storage::options::{ filter_object_metadata, get_content_sha256_with_query, get_opts, normalize_content_encoding_for_storage, put_opts, }; use crate::storage::request_context::spawn_traced; -use crate::storage::s3_api::multipart::parse_list_parts_params; use crate::storage::sse::{ SSEType, build_ssec_read_headers, encryption_material_to_metadata, extract_ssekms_context_from_headers, map_get_object_reader_error, diff --git a/rustfs/src/app/s3_api.rs b/rustfs/src/app/s3_api.rs new file mode 100644 index 000000000..0f1bbc0f1 --- /dev/null +++ b/rustfs/src/app/s3_api.rs @@ -0,0 +1,30 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! App-local S3 API response helpers. + +pub(crate) mod bucket { + pub(crate) use crate::storage::s3_api::bucket::{ + ListObjectVersionsParams, ListObjectsV2Params, build_list_buckets_output, build_list_object_versions_output, + build_list_objects_output, build_list_objects_v2_output, parse_list_object_versions_params, parse_list_objects_v2_params, + }; + pub(crate) use crate::storage::s3_api::common::rustfs_owner; +} + +pub(crate) mod multipart { + pub(crate) use crate::storage::s3_api::multipart::{ + ListMultipartUploadsParams, build_list_multipart_uploads_output, build_list_parts_output, + parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number, + }; +} diff --git a/scripts/check_architecture_migration_rules.sh b/scripts/check_architecture_migration_rules.sh index 91c79cff0..1d214d60d 100755 --- a/scripts/check_architecture_migration_rules.sh +++ b/scripts/check_architecture_migration_rules.sh @@ -161,6 +161,7 @@ RUSTFS_APP_CONTEXT_DIRECT_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_context_direct RUSTFS_APP_USECASE_RUNTIME_SOURCE_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_usecase_runtime_source_bypass_hits.txt" RUSTFS_APP_USECASE_STORAGE_WILDCARD_HITS_FILE="${TMP_DIR}/rustfs_app_usecase_storage_wildcard_hits.txt" RUSTFS_APP_WILDCARD_IMPORT_HITS_FILE="${TMP_DIR}/rustfs_app_wildcard_import_hits.txt" +RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_app_usecase_s3_api_bypass_hits.txt" RUSTFS_STORAGE_DIRECT_APP_CONTEXT_BYPASS_HITS_FILE="${TMP_DIR}/rustfs_storage_direct_app_context_bypass_hits.txt" awk ' @@ -1428,6 +1429,15 @@ if [[ -s "$RUSTFS_APP_WILDCARD_IMPORT_HITS_FILE" ]]; then report_failure "RustFS app consumers must import S3 DTO and ECFS owner APIs explicitly instead of wildcard imports: $(paste -sd '; ' "$RUSTFS_APP_WILDCARD_IMPORT_HITS_FILE")" fi +( + cd "$ROOT_DIR" + rg -n --with-filename 'crate::storage::s3_api::|use crate::storage::s3_api' rustfs/src/app --glob '*_usecase.rs' || true +) >"$RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE" + +if [[ -s "$RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE" ]]; then + report_failure "RustFS app usecases must consume S3 API helpers through rustfs/src/app/s3_api.rs: $(paste -sd '; ' "$RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE")" +fi + ( cd "$ROOT_DIR" rg -n --with-filename 'crate::app::context::|use crate::app::context|app::context::' rustfs/src/storage --glob '*.rs' | diff --git a/scripts/layer-dependency-baseline.txt b/scripts/layer-dependency-baseline.txt index d3ba2176f..51f1f22cb 100644 --- a/scripts/layer-dependency-baseline.txt +++ b/scripts/layer-dependency-baseline.txt @@ -21,24 +21,23 @@ cycle|infra<->interface dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::admin::handlers::site_replication::site_replication_bucket_meta_hook dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::admin::handlers::site_replication::site_replication_delete_bucket_hook dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::admin::handlers::site_replication::site_replication_make_bucket_hook -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::bucket::ListObjectVersionsParams -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::bucket::ListObjectsV2Params -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::bucket::build_list_buckets_output -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::bucket::build_list_object_versions_output -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::bucket::build_list_objects_output -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::bucket::build_list_objects_v2_output -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::bucket::parse_list_object_versions_params -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::bucket::parse_list_objects_v2_params -dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::storage::s3_api::common::rustfs_owner dep|rustfs/src/app/lifecycle_transition_api_test.rs|app->interface|crate::storage::ecfs::FS -dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::ListMultipartUploadsParams -dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::build_list_multipart_uploads_output -dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::build_list_parts_output -dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_list_multipart_uploads_params -dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_list_parts_params -dep|rustfs/src/app/multipart_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_upload_part_number dep|rustfs/src/app/object_usecase.rs|app->interface|crate::storage::ecfs::FS -dep|rustfs/src/app/object_usecase.rs|app->interface|crate::storage::s3_api::multipart::parse_list_parts_params +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::bucket::ListObjectVersionsParams +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::bucket::ListObjectsV2Params +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::bucket::build_list_buckets_output +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::bucket::build_list_object_versions_output +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::bucket::build_list_objects_output +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::bucket::build_list_objects_v2_output +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::bucket::parse_list_object_versions_params +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::bucket::parse_list_objects_v2_params +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::common::rustfs_owner +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::multipart::ListMultipartUploadsParams +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::multipart::build_list_multipart_uploads_output +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::multipart::build_list_parts_output +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::multipart::parse_list_multipart_uploads_params +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::multipart::parse_list_parts_params +dep|rustfs/src/app/s3_api.rs|app->interface|crate::storage::s3_api::multipart::parse_upload_part_number dep|rustfs/src/init.rs|infra->interface|crate::admin dep|rustfs/src/protocols/client.rs|infra->interface|crate::storage::ecfs::FS dep|rustfs/src/runtime_sources.rs|infra->app|crate::app::context