* fix(ecstore): restore odm source contract tests
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ci): initialize replication evidence in chain test
Run the replication workflow's evidence initialization before the chain handoff self-test executes the suite step. This keeps the test model aligned with the workflow-provided LOG_FILE and TMPDIR values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(odm): distinguish missing GCS buckets from object misses (#7221)
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* fix(ci): preserve reported functional suite failures
* fix(ci): isolate functional evidence and preserve every result
* fix(ci): exclude sensitive scratch files from suite artifacts
* test(ecstore): require core invariant tests in existing CI lane
* test(ci): require a fresh core JUnit report
* test(ecstore): match sealed context fixture map type
test(e2e): add the outbound target matrix and the replication checksum postmortem
Defense work for rustfs#7082, the regression rustfs#6895 introduced while
fixing rustfs#6853: a fix for one target class changed a client default for
every target class and nothing in tree modeled the other classes.
- docs/postmortems: timeline, root cause, why four defense layers missed
it, and the SOP for changing any outbound client default; AGENTS.md and
the adversarial compatibility lens point at it; the two env knobs from
rustfs#6895 are documented in docs/operations.
- fake_s3_target: reject_aws_chunked_uploads, require_checksum_for_object_lock
(Content-MD5 always verified), create_bucket_with_object_lock with a
GetObjectLockConfiguration handler, and a TransportSnapshot on every
journal record.
- replication_target_matrix_test: six object shapes against four target
modes with an explicit expectation table; the two rustfs#7082 cells are
pinned KnownFailing and fail with an XPASS message once the fix lands.
Wired into e2e-repl-nightly, excluded from e2e-full.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
Update the workspace s3s pin and refresh the lockfile with cargo update/upgrade.
Remove the unused lifecycle url dependency reported by cargo shear.
Tighten the s3s footprint ratchet to the current observed baseline.
scripts/check_test_wiring.py and scripts/check_security_coverage.py import
tomllib, which landed in Python 3.11. macOS ships /usr/bin/python3 at 3.9, so
`make pre-commit` failed on a clean machine with `ModuleNotFoundError: No
module named 'tomllib'` in test-wiring-check, even though the checkers
themselves are fine.
Add scripts/python_bin.sh, which resolves an interpreter (explicit
RUSTFS_PYTHON, then python3.14..3.11/python3/python on PATH, then a
`uv run --python 3.12 --no-project` fallback) and execs it, failing with the
concrete remediation when nothing usable exists. Route the Make call sites
through RUSTFS_PYTHON_BIN. CI workflows keep calling python3 directly because
their runners already provide 3.11+.
* fix(release): normalize development package versions
* ci: build only the rustfs release binary
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Align object-capacity refresh defaults with the production-oriented environment values and keep docs, script examples, and tests in sync.
Co-authored-by: heihutu <heihutu@gmail.com>
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <heihutu@gmail.com>
refactor(ecstore): huaweicloud/tencent reuse the shared S3 constructor
Migrates the Huaweicloud and Tencent tier warm backends onto the shared
S3-compatible constructor (backlog#2040). Also makes the shared
constructor's outbound-URL validation injectable per provider
(S3CompatibleWarmBackendParams::validate_endpoint) so it can centralize
rustfs/rustfs#6764's SSRF check for the providers that don't need an
exception, while accommodating rustfs/rustfs#6773's RustFS-specific
debug-only loopback opt-in without weakening the other six providers.
Updates scripts/error-other-format-baseline.txt: the one ::other(format!)
call site moves from the two per-provider files into the new shared
call site in warm_backend.rs (net call-site count unchanged).
Refs rustfs/backlog#2042
warp is killed at the write threshold and can leave in-flight multipart
uploads behind. rc.4-preview.1's decommission post-check refuses to
finalize a pool that still contains one (data is already moved, then the
pool is marked failed with 'resolve it before retrying'). Abort any
multipart uploads in the test bucket before starting decommission
(ListMultipartUploads + AbortMultipartUpload via the admin API).
WarmBackendS3::new already rejects loopback, private, link-local, and
cloud metadata-service endpoints via validate_outbound_url, but the
Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, RustFS, and GCS warm
backend constructors built their transition clients directly from
conf.endpoint without the same check.
The endpoint comes from the AddTier admin API, gated only by
SetTierAction, which can be a narrower IAM grant than root. Any
principal holding it could point one of these eight tier types at an
internal address (loopback, RFC1918, link-local, or a cloud metadata
IP) and have the server issue authenticated outbound requests to it, a
server-side SSRF vector that the S3 and Wasabi tier types were already
closed against.
Apply the same validate_outbound_url check at construction time for
all eight providers, before any credentials or network client are
built, mirroring the existing WarmBackendS3 pattern. GCS keeps its
default-endpoint behavior when conf.endpoint is empty and only
validates an explicitly configured endpoint.
Add a regression test per provider asserting that a loopback endpoint
is rejected before any backend/network setup, matching the existing
WarmBackendS3 coverage.
Update the error(format!) ratchet baseline: these are one-shot admin
tier-configuration validation errors returned once per AddTier call,
not per-disk I/O errors that flow through reduce_errs quorum
aggregation (backlog#1845), so the new ::other(format!) call sites do
not introduce a quorum-bucketing hazard. They mirror the pre-existing,
already-baselined warm_backend_s3.rs call site.
houseme