gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-29 00:17:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(ecstore): extract the embedded S3 client into rustfs-s3-client (#6627)

Browse Source 
The storage engine embedded a ~8.4K-line hand-written S3 HTTP client under crates/ecstore/src/client (rustfs/backlog#1842). That client is a legitimate engine capability — it consumes remote S3-compatible endpoints for ILM tier warm backends and transition targets — but it was misfiled inside the engine, dragging s3s/hyper wire types into ecstore and blocking ARCHITECTURE.md invariant 4.

This PR is the pure-move step: 21 modules move verbatim to the new crates/s3-client crate (rustfs-s3-client), and crates/ecstore/src/client/mod.rs becomes a re-export shim so every in-crate crate::client:: path keeps working. The two server-side modules that were historically misfiled under client/ — object_api_utils.rs and object_handlers_common.rs — stay in ecstore.

Three reverse dependencies from the client into engine internals are severed so the move can be pure:

- transition_api::ReaderImpl::ObjectBody held ecstore's GetObjectReader; the client only ever reads the body, so the variant now holds an ObjectReader newtype over Box<dyn AsyncRead + Send + Sync + Unpin> with the same read_all() surface. The single production construction site (set_disk transition upload) and the two engine-side consumers were adjusted.
- api_list/api_remove used ecstore's storage_api_contracts / object_api types; api_list now imports BucketInfo from rustfs-storage-api directly, and api_remove uses the client's own transition_api::ObjectInfo (only .name/.version_id were read; the error-path bucket name is now threaded as a parameter instead of read from the deleted objects).
- the api_put_object_streaming regression tests built a GetObjectReader by hand; they now wrap the duplex stream in ObjectReader::new.

Guard updates: the s3s footprint ratchet gains an ecstore-scoped counter (42 files, shrink-only, per rustfs/backlog#1842), the ecstore module-lint-blanket register follows the moved files into crates/s3-client so the blanket ratchet keeps covering them, the logging guardrail path pin follows transition_api.rs, and the ::other(format!) baseline is regenerated (moved call sites left ecstore).

Verification: cargo check -p rustfs-s3-client -p rustfs-ecstore; cargo nextest run -p rustfs-s3-client (43 passed) and -p rustfs-ecstore (4515/4523; the 8 failures reproduce identically on pristine origin/main on the same machine); cargo clippy --all-targets; scripts/check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_s3s_footprint.sh, check_logging_guardrails.sh, check_error_other_format_ratchet.sh, check_doc_paths.sh, check_ci_paths_sync.sh all pass.
This commit is contained in:
Zhengchao An
2026-08-26 12:38:52 +08:00
committed by GitHub
parent 65a7cc9cd4
commit 8f0d4a20d1
35 changed files with 352 additions and 182 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/check_architecture_migration_rules.sh
+2 -2
View File
@@ -5494,7 +5494,7 @@ ECSTORE_LINT_EXPECTED="${TMP_DIR}/ecstore_lint_expected.txt"
(
cd "$ROOT_DIR"
rg -n '^#!\[allow\(dead_code\)\]' crates/ecstore/src/ 2>/dev/null || true
rg -n '^#!\[allow\(dead_code\)\]' crates/ecstore/src/ crates/s3-client/src/ 2>/dev/null || true
) >"$ECSTORE_DEAD_CODE_HITS"
if [[ -s "$ECSTORE_DEAD_CODE_HITS" ]]; then
@@ -5503,7 +5503,7 @@ fi
(
cd "$ROOT_DIR"
rg -n '^#!\[allow\((unused_variables|unused_must_use|clippy::all)\)\]' crates/ecstore/src/ 2>/dev/null |
rg -n '^#!\[allow\((unused_variables|unused_must_use|clippy::all)\)\]' crates/ecstore/src/ crates/s3-client/src/ 2>/dev/null |
sed -E 's#^([^:]+):[0-9]+:\#!\[allow\(([^)]+)\)\]#\1|\2#' | sort -u
) >"$ECSTORE_LINT_ACTUAL"
scripts/check_logging_guardrails.sh
+1 -1
View File
@@ -60,7 +60,7 @@ checked_files=(
"crates/targets/src/target/webhook.rs"
"crates/ecstore/src/store/peer.rs"
"crates/ecstore/src/store/init.rs"
"crates/ecstore/src/client/transition_api.rs"
"crates/s3-client/src/transition_api.rs"
"crates/ecstore/src/services/tier/tier.rs"
"crates/heal/src/heal/manager.rs"
"crates/heal/src/heal/storage.rs"
scripts/check_s3s_footprint.sh
+11 -1
View File
@@ -27,6 +27,12 @@ cd "$(dirname "$0")/.."
# to verify S3 behavior and does not widen the production s3s surface.
S3S_IMPORT_FILES_BASELINE=211
S3_ERROR_LINES_BASELINE=1620
# ecstore-scoped ratchet (rustfs/backlog#1842): the storage engine must not
# know S3 wire/DTO types (ARCHITECTURE.md invariant 4). The S3-*consuming*
# client was extracted to crates/s3-client, where s3s usage is legitimate;
# this counter ratchets the remaining serving-side s3s references out of
# crates/ecstore. Baseline verified on 2026-08-26.
S3S_ECSTORE_FILES_BASELINE=42
S3S_PATH_PATTERN='(^|[^"[:alnum:]_])s3s::'
E2E_TEST_GLOB='--glob=!crates/e2e_test/**'
@@ -47,11 +53,13 @@ run_rg_to() {
run_rg_to "$TMP_DIR/import_files" -l "$S3S_PATH_PATTERN" --type rust $E2E_TEST_GLOB
run_rg_to "$TMP_DIR/error_lines" -c 's3_error!' --type rust $E2E_TEST_GLOB
run_rg_to "$TMP_DIR/ecstore_files" -l "$S3S_PATH_PATTERN" --type rust crates/ecstore/src
s3s_import_files="$(grep -c . "$TMP_DIR/import_files" || true)"
s3_error_lines="$(awk -F: '{sum += $NF} END {print sum + 0}' "$TMP_DIR/error_lines")"
s3s_ecstore_files="$(grep -c . "$TMP_DIR/ecstore_files" || true)"
for value in "$s3s_import_files" "$s3_error_lines"; do
for value in "$s3s_import_files" "$s3_error_lines" "$s3s_ecstore_files"; do
if ! [[ "$value" =~ ^[0-9]+$ ]]; then
echo "error: could not compute s3s footprint counts (got: '$value')" >&2
exit 1
@@ -82,6 +90,8 @@ check_ratchet "files importing s3s" "$s3s_import_files" "$S3S_IMPORT_FILES_BASEL
"rg -l '$S3S_PATH_PATTERN' --type rust $E2E_TEST_GLOB"
check_ratchet "s3_error! invocation lines" "$s3_error_lines" "$S3_ERROR_LINES_BASELINE" \
"rg -c 's3_error!' --type rust $E2E_TEST_GLOB"
check_ratchet "ecstore files referencing s3s" "$s3s_ecstore_files" "$S3S_ECSTORE_FILES_BASELINE" \
"rg -l '$S3S_PATH_PATTERN' --type rust crates/ecstore/src"
if ((status != 0)); then
exit 1
scripts/ecstore-module-lint-register.txt
+47 -47
View File
@@ -23,56 +23,56 @@ crates/ecstore/src/bucket/lifecycle/tier_last_day_stats.rs|unused_variables
crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs|clippy::all
crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs|unused_must_use
crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs|unused_variables
crates/ecstore/src/client/api_error_response.rs|clippy::all
crates/ecstore/src/client/api_error_response.rs|unused_must_use
crates/ecstore/src/client/api_error_response.rs|unused_variables
crates/ecstore/src/client/api_get_object.rs|clippy::all
crates/ecstore/src/client/api_get_object.rs|unused_must_use
crates/ecstore/src/client/api_get_object.rs|unused_variables
crates/ecstore/src/client/api_get_options.rs|clippy::all
crates/ecstore/src/client/api_get_options.rs|unused_must_use
crates/ecstore/src/client/api_get_options.rs|unused_variables
crates/ecstore/src/client/api_list.rs|clippy::all
crates/ecstore/src/client/api_list.rs|unused_must_use
crates/ecstore/src/client/api_list.rs|unused_variables
crates/ecstore/src/client/api_put_object.rs|clippy::all
crates/ecstore/src/client/api_put_object.rs|unused_must_use
crates/ecstore/src/client/api_put_object.rs|unused_variables
crates/ecstore/src/client/api_put_object_common.rs|clippy::all
crates/ecstore/src/client/api_put_object_common.rs|unused_must_use
crates/ecstore/src/client/api_put_object_common.rs|unused_variables
crates/ecstore/src/client/api_put_object_multipart.rs|clippy::all
crates/ecstore/src/client/api_put_object_multipart.rs|unused_must_use
crates/ecstore/src/client/api_put_object_multipart.rs|unused_variables
crates/ecstore/src/client/api_put_object_streaming.rs|clippy::all
crates/ecstore/src/client/api_put_object_streaming.rs|unused_must_use
crates/ecstore/src/client/api_put_object_streaming.rs|unused_variables
crates/ecstore/src/client/api_remove.rs|clippy::all
crates/ecstore/src/client/api_remove.rs|unused_must_use
crates/ecstore/src/client/api_remove.rs|unused_variables
crates/ecstore/src/client/api_s3_datatypes.rs|clippy::all
crates/ecstore/src/client/api_s3_datatypes.rs|unused_must_use
crates/ecstore/src/client/api_s3_datatypes.rs|unused_variables
crates/ecstore/src/client/api_stat.rs|clippy::all
crates/ecstore/src/client/api_stat.rs|unused_must_use
crates/ecstore/src/client/api_stat.rs|unused_variables
crates/ecstore/src/client/bucket_cache.rs|clippy::all
crates/ecstore/src/client/bucket_cache.rs|unused_must_use
crates/ecstore/src/client/bucket_cache.rs|unused_variables
crates/ecstore/src/client/checksum.rs|clippy::all
crates/ecstore/src/client/checksum.rs|unused_must_use
crates/ecstore/src/client/checksum.rs|unused_variables
crates/ecstore/src/client/constants.rs|unused_must_use
crates/ecstore/src/client/constants.rs|unused_variables
crates/ecstore/src/client/credentials.rs|clippy::all
crates/ecstore/src/client/credentials.rs|unused_must_use
crates/ecstore/src/client/credentials.rs|unused_variables
crates/s3-client/src/api_error_response.rs|clippy::all
crates/s3-client/src/api_error_response.rs|unused_must_use
crates/s3-client/src/api_error_response.rs|unused_variables
crates/s3-client/src/api_get_object.rs|clippy::all
crates/s3-client/src/api_get_object.rs|unused_must_use
crates/s3-client/src/api_get_object.rs|unused_variables
crates/s3-client/src/api_get_options.rs|clippy::all
crates/s3-client/src/api_get_options.rs|unused_must_use
crates/s3-client/src/api_get_options.rs|unused_variables
crates/s3-client/src/api_list.rs|clippy::all
crates/s3-client/src/api_list.rs|unused_must_use
crates/s3-client/src/api_list.rs|unused_variables
crates/s3-client/src/api_put_object.rs|clippy::all
crates/s3-client/src/api_put_object.rs|unused_must_use
crates/s3-client/src/api_put_object.rs|unused_variables
crates/s3-client/src/api_put_object_common.rs|clippy::all
crates/s3-client/src/api_put_object_common.rs|unused_must_use
crates/s3-client/src/api_put_object_common.rs|unused_variables
crates/s3-client/src/api_put_object_multipart.rs|clippy::all
crates/s3-client/src/api_put_object_multipart.rs|unused_must_use
crates/s3-client/src/api_put_object_multipart.rs|unused_variables
crates/s3-client/src/api_put_object_streaming.rs|clippy::all
crates/s3-client/src/api_put_object_streaming.rs|unused_must_use
crates/s3-client/src/api_put_object_streaming.rs|unused_variables
crates/s3-client/src/api_remove.rs|clippy::all
crates/s3-client/src/api_remove.rs|unused_must_use
crates/s3-client/src/api_remove.rs|unused_variables
crates/s3-client/src/api_s3_datatypes.rs|clippy::all
crates/s3-client/src/api_s3_datatypes.rs|unused_must_use
crates/s3-client/src/api_s3_datatypes.rs|unused_variables
crates/s3-client/src/api_stat.rs|clippy::all
crates/s3-client/src/api_stat.rs|unused_must_use
crates/s3-client/src/api_stat.rs|unused_variables
crates/s3-client/src/bucket_cache.rs|clippy::all
crates/s3-client/src/bucket_cache.rs|unused_must_use
crates/s3-client/src/bucket_cache.rs|unused_variables
crates/s3-client/src/checksum.rs|clippy::all
crates/s3-client/src/checksum.rs|unused_must_use
crates/s3-client/src/checksum.rs|unused_variables
crates/s3-client/src/constants.rs|unused_must_use
crates/s3-client/src/constants.rs|unused_variables
crates/s3-client/src/credentials.rs|clippy::all
crates/s3-client/src/credentials.rs|unused_must_use
crates/s3-client/src/credentials.rs|unused_variables
crates/ecstore/src/client/object_api_utils.rs|clippy::all
crates/ecstore/src/client/object_api_utils.rs|unused_must_use
crates/ecstore/src/client/object_api_utils.rs|unused_variables
crates/ecstore/src/client/transition_api.rs|clippy::all
crates/ecstore/src/client/transition_api.rs|unused_must_use
crates/ecstore/src/client/transition_api.rs|unused_variables
crates/s3-client/src/transition_api.rs|clippy::all
crates/s3-client/src/transition_api.rs|unused_must_use
crates/s3-client/src/transition_api.rs|unused_variables
crates/ecstore/src/services/event_notification.rs|unused_variables
crates/ecstore/src/services/tier/tier.rs|clippy::all
crates/ecstore/src/services/tier/tier.rs|unused_must_use
scripts/error-other-format-baseline.txt
-5
View File
@@ -19,11 +19,6 @@
1|crates/ecstore/src/bucket/replication/replication_object_config.rs
2|crates/ecstore/src/bucket/replication/replication_pool.rs
2|crates/ecstore/src/bucket/replication/replication_target_boundary.rs
1|crates/ecstore/src/client/api_put_object_multipart.rs
2|crates/ecstore/src/client/api_put_object_streaming.rs
1|crates/ecstore/src/client/api_s3_datatypes.rs
1|crates/ecstore/src/client/signer_error.rs
5|crates/ecstore/src/client/transition_api.rs
3|crates/ecstore/src/cluster/rpc/http_auth.rs
2|crates/ecstore/src/cluster/rpc/internode_data_transport.rs
11|crates/ecstore/src/cluster/rpc/peer_rest_client.rs