mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-27 23:47:28 +00:00
Zhengchao An
8f0d4a20d1
The storage engine embedded a ~8.4K-line hand-written S3 HTTP client under crates/ecstore/src/client (rustfs/backlog#1842). That client is a legitimate engine capability — it consumes remote S3-compatible endpoints for ILM tier warm backends and transition targets — but it was misfiled inside the engine, dragging s3s/hyper wire types into ecstore and blocking ARCHITECTURE.md invariant 4. This PR is the pure-move step: 21 modules move verbatim to the new crates/s3-client crate (rustfs-s3-client), and crates/ecstore/src/client/mod.rs becomes a re-export shim so every in-crate crate::client:: path keeps working. The two server-side modules that were historically misfiled under client/ — object_api_utils.rs and object_handlers_common.rs — stay in ecstore. Three reverse dependencies from the client into engine internals are severed so the move can be pure: - transition_api::ReaderImpl::ObjectBody held ecstore's GetObjectReader; the client only ever reads the body, so the variant now holds an ObjectReader newtype over Box<dyn AsyncRead + Send + Sync + Unpin> with the same read_all() surface. The single production construction site (set_disk transition upload) and the two engine-side consumers were adjusted. - api_list/api_remove used ecstore's storage_api_contracts / object_api types; api_list now imports BucketInfo from rustfs-storage-api directly, and api_remove uses the client's own transition_api::ObjectInfo (only .name/.version_id were read; the error-path bucket name is now threaded as a parameter instead of read from the deleted objects). - the api_put_object_streaming regression tests built a GetObjectReader by hand; they now wrap the duplex stream in ObjectReader::new. Guard updates: the s3s footprint ratchet gains an ecstore-scoped counter (42 files, shrink-only, per rustfs/backlog#1842), the ecstore module-lint-blanket register follows the moved files into crates/s3-client so the blanket ratchet keeps covering them, the logging guardrail path pin follows transition_api.rs, and the ::other(format!) baseline is regenerated (moved call sites left ecstore). Verification: cargo check -p rustfs-s3-client -p rustfs-ecstore; cargo nextest run -p rustfs-s3-client (43 passed) and -p rustfs-ecstore (4515/4523; the 8 failures reproduce identically on pristine origin/main on the same machine); cargo clippy --all-targets; scripts/check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_s3s_footprint.sh, check_logging_guardrails.sh, check_error_other_format_ratchet.sh, check_doc_paths.sh, check_ci_paths_sync.sh all pass.
101 lines
4.2 KiB
Bash
Executable File
101 lines
4.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Ratchet guard freezing the s3s dependency footprint ahead of the
|
||
# s3gate/gateway migration (rustfs/backlog#1677, review finding F1;
|
||
# acceptance criteria recorded in rustfs/backlog#1733).
|
||
#
|
||
# The migration's goal is to shrink the direct s3s surface, so new code must
|
||
# not grow it. Two counters are ratcheted, baselines verified on 2026-08-05:
|
||
#
|
||
# - files referencing s3s paths: rg -l "$S3S_PATH_PATTERN" --type rust (files)
|
||
# - s3_error! invocation lines: rg -c 's3_error!' --type rust (summed)
|
||
#
|
||
# Either count exceeding its baseline fails the check with the offending
|
||
# delta. Baselines are LOWER-ONLY: when a PR shrinks the footprint, lower the
|
||
# matching baseline in the same PR so the ratchet stays tight. Never raise a
|
||
# baseline to get green (AGENTS.md, Verification Before PR) — route new S3
|
||
# API code through the gateway abstractions instead of importing s3s
|
||
# directly.
|
||
#
|
||
# Usage: scripts/check_s3s_footprint.sh
|
||
|
||
set -euo pipefail
|
||
|
||
cd "$(dirname "$0")/.."
|
||
|
||
# Baselines verified on 2026-08-11. Lower-only; see header.
|
||
# Excludes crates/e2e_test/ — test infrastructure legitimately uses s3s
|
||
# to verify S3 behavior and does not widen the production s3s surface.
|
||
S3S_IMPORT_FILES_BASELINE=211
|
||
S3_ERROR_LINES_BASELINE=1620
|
||
# ecstore-scoped ratchet (rustfs/backlog#1842): the storage engine must not
|
||
# know S3 wire/DTO types (ARCHITECTURE.md invariant 4). The S3-*consuming*
|
||
# client was extracted to crates/s3-client, where s3s usage is legitimate;
|
||
# this counter ratchets the remaining serving-side s3s references out of
|
||
# crates/ecstore. Baseline verified on 2026-08-26.
|
||
S3S_ECSTORE_FILES_BASELINE=42
|
||
S3S_PATH_PATTERN='(^|[^"[:alnum:]_])s3s::'
|
||
E2E_TEST_GLOB='--glob=!crates/e2e_test/**'
|
||
|
||
TMP_DIR="$(mktemp -d)"
|
||
trap 'rm -rf "$TMP_DIR"' EXIT
|
||
|
||
# rg exits 1 on zero matches (a legitimate count of 0 at the end of the
|
||
# migration) and >1 on real errors; only the latter may abort the check.
|
||
run_rg_to() {
|
||
local out="$1" rg_status=0
|
||
shift
|
||
rg "$@" >"$out" || rg_status=$?
|
||
if ((rg_status > 1)); then
|
||
echo "error: 'rg $*' failed with status $rg_status" >&2
|
||
exit 1
|
||
fi
|
||
}
|
||
|
||
run_rg_to "$TMP_DIR/import_files" -l "$S3S_PATH_PATTERN" --type rust $E2E_TEST_GLOB
|
||
run_rg_to "$TMP_DIR/error_lines" -c 's3_error!' --type rust $E2E_TEST_GLOB
|
||
run_rg_to "$TMP_DIR/ecstore_files" -l "$S3S_PATH_PATTERN" --type rust crates/ecstore/src
|
||
|
||
s3s_import_files="$(grep -c . "$TMP_DIR/import_files" || true)"
|
||
s3_error_lines="$(awk -F: '{sum += $NF} END {print sum + 0}' "$TMP_DIR/error_lines")"
|
||
s3s_ecstore_files="$(grep -c . "$TMP_DIR/ecstore_files" || true)"
|
||
|
||
for value in "$s3s_import_files" "$s3_error_lines" "$s3s_ecstore_files"; do
|
||
if ! [[ "$value" =~ ^[0-9]+$ ]]; then
|
||
echo "error: could not compute s3s footprint counts (got: '$value')" >&2
|
||
exit 1
|
||
fi
|
||
done
|
||
|
||
status=0
|
||
|
||
check_ratchet() {
|
||
local label="$1" count="$2" baseline="$3" inspect_cmd="$4"
|
||
|
||
if ((count > baseline)); then
|
||
echo "❌ s3s footprint ratchet violation: $label is $count, baseline is $baseline (+$((count - baseline)))" >&2
|
||
echo " New code must not widen the s3s surface being removed by the s3gate migration" >&2
|
||
echo " (rustfs/backlog#1677 F1, rustfs/backlog#1733). Use the gateway abstractions" >&2
|
||
echo " instead of importing s3s directly. To find the offenders, compare" >&2
|
||
echo " '$inspect_cmd' against origin/main." >&2
|
||
status=1
|
||
elif ((count < baseline)); then
|
||
echo "ℹ️ s3s footprint shrank: $label is $count, baseline is $baseline ($((count - baseline)))." >&2
|
||
echo " Lower the baseline in scripts/check_s3s_footprint.sh in this PR to keep the ratchet tight." >&2
|
||
else
|
||
echo "s3s footprint OK: $label is $count (baseline: $baseline)"
|
||
fi
|
||
}
|
||
|
||
check_ratchet "files importing s3s" "$s3s_import_files" "$S3S_IMPORT_FILES_BASELINE" \
|
||
"rg -l '$S3S_PATH_PATTERN' --type rust $E2E_TEST_GLOB"
|
||
check_ratchet "s3_error! invocation lines" "$s3_error_lines" "$S3_ERROR_LINES_BASELINE" \
|
||
"rg -c 's3_error!' --type rust $E2E_TEST_GLOB"
|
||
check_ratchet "ecstore files referencing s3s" "$s3s_ecstore_files" "$S3S_ECSTORE_FILES_BASELINE" \
|
||
"rg -l '$S3S_PATH_PATTERN' --type rust crates/ecstore/src"
|
||
|
||
if ((status != 0)); then
|
||
exit 1
|
||
fi
|
||
|
||
echo "✅ s3s footprint ratchet check passed"
|