* test(e2e): add distributed 4x4 validation
* ci(e2e): use tmpfs for distributed pool isolation
sm-standard-4 is an ARC pod without usable loop devices, so
mount -o loop fails with ENOENT before any pool filesystem is
attached. Sized tmpfs still gives each pool a distinct st_dev
and independent 1G statfs capacity.
Co-authored-by: RustFS <hello@rustfs.com>
* test(e2e): prove operations overlap data movement
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* test(odm): pin gcs source status-to-error-class mapping
The native GCS backend classifies every failure from the HTTP status
alone, because GCS states its error code in a body this backend never
reads. Only NotFound is negative-cached and only a retryable class may be
re-sent, so cover 401/403 -> AccessDenied, 429/503 -> Throttled,
500/502 -> ServerError and 404 -> NotFound over both HEAD and GET.
* ci(odm): require the source-backend contract tests in test-and-lint
The shared contract tests already run in ci/test-and-lint, but only
because gcs is a rustfs default feature; nothing failed if that
selection went away. Pin the S3, Azure and native GCS contracts in the
core required-test manifest so a lost selection fails the lane.
test(e2e): prove bucket config survives rc.5 upgrade and rollback
Add two upgrade-compatibility scenarios pinned to the on-demand-migration
series' on-disk surfaces: BucketMetadata's 44 -> 46 msgpack keys, the
fail-closed bucket-config reads of rustfs#7172, the encryption-gated PUT
path of rustfs#7183, and the default-on migration module of rustfs#7089.
The upgrade case writes versioning, SSE-S3 default encryption, a validated
replication target plus rule, lifecycle, tags, quota, a public access block,
a bucket policy and an object lock configuration with the pinned previous
release, then asserts each one reads back unchanged on the current build,
that list-remote-targets still reports the target, that writes to the
encrypted and plain buckets keep their encryption posture, that every
pre-upgrade object including a multipart one is byte-identical, and that an
unconfigured bucket reports no migration and still answers NoSuchKey.
The rollback case is the reverse: the current build writes the 46-key blob
and the previous release must decode it by skipping the two unknown keys.
* feat(ecstore): add a native azure blob odm source backend
* feat(ecstore): add a native gcs odm source backend and one backend contract
* fix(ecstore): refuse an empty azure account key at client build
* fix(ecstore): probe gcs sources with the listing permission
* fix(app): drop a redundant match guard on the sse config lookup
* fix(ecstore): drop stale rename commit duplicates from local.rs
* test(ecstore): use the sanctioned placeholder key in the gcs fixture
* test(ecstore): require core invariant tests in existing CI lane
* test(ci): require a fresh core JUnit report
* test(ecstore): match sealed context fixture map type
Replication targets, remote tiers and on-demand migration sources each store a
remote secret in clear text under .rustfs.sys today. Record one design for all
three before any of them invents its own: seal only the secret fields, into an
added field so an old reader finds the credential absent rather than a
ciphertext it would sign with, unseal at remote-client construction, and reach
KMS through an installed hook because ECStore does not depend on rustfs-kms.
Covers the envelope format and its encryption context, the mixed-version
compatibility matrix, the rollout gate, rotation ownership and the fail-closed
rules.
* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
`GET /v3/tier-stats` answered from whichever process received the
request, returning that node's rolling 24-hour transition counters as
if they were cluster totals, and the `TierRequestsSuccess` and
`TierRequestsFailure` metric names had no producer at all.
The body now separates the two quantities a tier carries. Stored
inventory comes from the persisted scanner usage snapshot, which is
already cluster-wide; rolling activity is summed over every member
through a new read-only `TierDailyStats` peer RPC. Rings are merged
rather than added, so an idle node's expired hours age out, and each
node counts only its own committed transitions, so a retry is counted
once. Coverage travels with the numbers: `activity.status` names the
reporting members and the ones that could not be asked, timed out, or
answered with a ring this build refuses to merge, and per-tier
inventory is absent rather than zero when the snapshot has no
accounting. The version 1 body stays reachable at `?format=legacy`.
Tier request counters are recorded at the two seams every remote
request passes through, so a new provider is counted by construction,
with a closed operation/outcome label set that can never grow a tier
name, endpoint or object key.
Closesrustfs/backlog#2207
Co-authored-by: cxymds <cxymds@gmail.com>
* feat(odm): merge the source listing into ListObjectsV2
Adds policy.list_through: ListObjectsV2 merges the local and source
listings into one ordered page so clients see the whole namespace during
an on-demand migration. Local entries win a key both sides hold,
CommonPrefixes are unioned under a delimiter, and the continuation token
is an opaque versioned envelope carrying both cursors.
A source listing failure or an open breaker follows policy.source_error:
propagate answers 424, not_found answers from local state and marks the
response x-rustfs-on-demand-migration-list: local_only. Source listings
are capped at 10 per second per bucket.
* test(odm): refresh the e2e-full darwin selection digest
The list-through e2e module adds seven cases to the merge lane.
* fix(odm): declare the remote client retry policy per consumer
The SDK retry policy was an inherited default: one logical call could cost
three wire requests, so the migration breaker counted logical calls on top
of a threefold amplification against a source that was already failing.
Make it an explicit RemoteS3EndpointSpec field. Replication targets declare
today's standard three attempts and keep their behaviour; the on-demand
migration source and its admin probe declare a disabled policy, so one
counted failure is exactly one source request and pull.rs owns the only
retry budget.
* fix(odm): count a stalled inline source as a source timeout
The inline tee wraps its source body in the idle guard, but the tee turns a
stalled source into an ordinary body read error, so the write-back reported
it as a local write failure. Hand commit_inline the guard so the pull is
counted under source_timeout instead.
The background pump now enforces the idle budget through the same guard
rather than a second copy of the timeout loop.
* test(odm): cover a stalled source body end to end
The fake target can now deliver a GetObject body in slices with a pause
between them, so the inline abort can be driven by a stalled source instead
of a truncated one. Two fault cases drop the workarounds they carried for
the SDK's retries: the scripted fault count and the observed source request
count now have to agree.
The operations guide records the retry and idle-timeout guarantees.
* fix(replication): send an integrity header on Object Lock replication PUTs
AWS S3, MinIO and most compatible targets reject a PutObject that carries
x-amz-object-lock-* headers unless it also carries Content-MD5 or an
x-amz-checksum-* header. Since rustfs#6895 the replication client sends
plain signed payloads with no SDK checksum, so every replicated object
with a retention period or legal hold failed against such targets.
TargetClient::put_object now decides per request through the pure
rustfs_replication::object_lock_put_integrity: a plaintext single-part
object whose source ETag is its MD5 gets Content-MD5 derived from the
ETag (no body pass, framing unchanged); a multipart-layout ETag, managed
SSE or SSE-C passthrough falls back to an SDK CRC32; a forwarded source
checksum or an unlocked PUT is left alone.
The outbound target matrix flips its two KnownFailing(rustfs#7082) cells
to Completed and every Completed cell now asserts that a locked
PutObject carried an integrity header.
Fixes rustfs#7082.
* test(e2e): keep the matrix expectation table clippy-clean under -D warnings
The CI lint runs cargo clippy --all-targets -- -D warnings. With every cell
green the single-arm match tripped match_single_binding and the unused
KnownFailing variant tripped dead_code, and the target-client tests tripped
field_reassign_with_default. Drive the expectation table from a
KNOWN_FAILING_CELLS constant (so the variant stays live and adding a red
cell is a one-line entry), build the test options as struct literals, and
refresh the e2e-repl-nightly selection digest for the renamed table test.
RustFS