* test(odm): pin gcs source status-to-error-class mapping
The native GCS backend classifies every failure from the HTTP status
alone, because GCS states its error code in a body this backend never
reads. Only NotFound is negative-cached and only a retryable class may be
re-sent, so cover 401/403 -> AccessDenied, 429/503 -> Throttled,
500/502 -> ServerError and 404 -> NotFound over both HEAD and GET.
* ci(odm): require the source-backend contract tests in test-and-lint
The shared contract tests already run in ci/test-and-lint, but only
because gcs is a rustfs default feature; nothing failed if that
selection went away. Pin the S3, Azure and native GCS contracts in the
core required-test manifest so a lost selection fails the lane.
test(e2e): prove bucket config survives rc.5 upgrade and rollback
Add two upgrade-compatibility scenarios pinned to the on-demand-migration
series' on-disk surfaces: BucketMetadata's 44 -> 46 msgpack keys, the
fail-closed bucket-config reads of rustfs#7172, the encryption-gated PUT
path of rustfs#7183, and the default-on migration module of rustfs#7089.
The upgrade case writes versioning, SSE-S3 default encryption, a validated
replication target plus rule, lifecycle, tags, quota, a public access block,
a bucket policy and an object lock configuration with the pinned previous
release, then asserts each one reads back unchanged on the current build,
that list-remote-targets still reports the target, that writes to the
encrypted and plain buckets keep their encryption posture, that every
pre-upgrade object including a multipart one is byte-identical, and that an
unconfigured bucket reports no migration and still answers NoSuchKey.
The rollback case is the reverse: the current build writes the 46-key blob
and the previous release must decode it by skipping the two unknown keys.
* feat(ecstore): add a native azure blob odm source backend
* feat(ecstore): add a native gcs odm source backend and one backend contract
* fix(ecstore): refuse an empty azure account key at client build
* fix(ecstore): probe gcs sources with the listing permission
* fix(app): drop a redundant match guard on the sse config lookup
* fix(ecstore): drop stale rename commit duplicates from local.rs
* test(ecstore): use the sanctioned placeholder key in the gcs fixture
* test(ecstore): require core invariant tests in existing CI lane
* test(ci): require a fresh core JUnit report
* test(ecstore): match sealed context fixture map type
Replication targets, remote tiers and on-demand migration sources each store a
remote secret in clear text under .rustfs.sys today. Record one design for all
three before any of them invents its own: seal only the secret fields, into an
added field so an old reader finds the credential absent rather than a
ciphertext it would sign with, unseal at remote-client construction, and reach
KMS through an installed hook because ECStore does not depend on rustfs-kms.
Covers the envelope format and its encryption context, the mixed-version
compatibility matrix, the rollout gate, rotation ownership and the fail-closed
rules.
* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
`GET /v3/tier-stats` answered from whichever process received the
request, returning that node's rolling 24-hour transition counters as
if they were cluster totals, and the `TierRequestsSuccess` and
`TierRequestsFailure` metric names had no producer at all.
The body now separates the two quantities a tier carries. Stored
inventory comes from the persisted scanner usage snapshot, which is
already cluster-wide; rolling activity is summed over every member
through a new read-only `TierDailyStats` peer RPC. Rings are merged
rather than added, so an idle node's expired hours age out, and each
node counts only its own committed transitions, so a retry is counted
once. Coverage travels with the numbers: `activity.status` names the
reporting members and the ones that could not be asked, timed out, or
answered with a ring this build refuses to merge, and per-tier
inventory is absent rather than zero when the snapshot has no
accounting. The version 1 body stays reachable at `?format=legacy`.
Tier request counters are recorded at the two seams every remote
request passes through, so a new provider is counted by construction,
with a closed operation/outcome label set that can never grow a tier
name, endpoint or object key.
Closesrustfs/backlog#2207
Co-authored-by: cxymds <cxymds@gmail.com>
* feat(odm): merge the source listing into ListObjectsV2
Adds policy.list_through: ListObjectsV2 merges the local and source
listings into one ordered page so clients see the whole namespace during
an on-demand migration. Local entries win a key both sides hold,
CommonPrefixes are unioned under a delimiter, and the continuation token
is an opaque versioned envelope carrying both cursors.
A source listing failure or an open breaker follows policy.source_error:
propagate answers 424, not_found answers from local state and marks the
response x-rustfs-on-demand-migration-list: local_only. Source listings
are capped at 10 per second per bucket.
* test(odm): refresh the e2e-full darwin selection digest
The list-through e2e module adds seven cases to the merge lane.
* fix(odm): declare the remote client retry policy per consumer
The SDK retry policy was an inherited default: one logical call could cost
three wire requests, so the migration breaker counted logical calls on top
of a threefold amplification against a source that was already failing.
Make it an explicit RemoteS3EndpointSpec field. Replication targets declare
today's standard three attempts and keep their behaviour; the on-demand
migration source and its admin probe declare a disabled policy, so one
counted failure is exactly one source request and pull.rs owns the only
retry budget.
* fix(odm): count a stalled inline source as a source timeout
The inline tee wraps its source body in the idle guard, but the tee turns a
stalled source into an ordinary body read error, so the write-back reported
it as a local write failure. Hand commit_inline the guard so the pull is
counted under source_timeout instead.
The background pump now enforces the idle budget through the same guard
rather than a second copy of the timeout loop.
* test(odm): cover a stalled source body end to end
The fake target can now deliver a GetObject body in slices with a pause
between them, so the inline abort can be driven by a stalled source instead
of a truncated one. Two fault cases drop the workarounds they carried for
the SDK's retries: the scripted fault count and the observed source request
count now have to agree.
The operations guide records the retry and idle-timeout guarantees.
* fix(replication): send an integrity header on Object Lock replication PUTs
AWS S3, MinIO and most compatible targets reject a PutObject that carries
x-amz-object-lock-* headers unless it also carries Content-MD5 or an
x-amz-checksum-* header. Since rustfs#6895 the replication client sends
plain signed payloads with no SDK checksum, so every replicated object
with a retention period or legal hold failed against such targets.
TargetClient::put_object now decides per request through the pure
rustfs_replication::object_lock_put_integrity: a plaintext single-part
object whose source ETag is its MD5 gets Content-MD5 derived from the
ETag (no body pass, framing unchanged); a multipart-layout ETag, managed
SSE or SSE-C passthrough falls back to an SDK CRC32; a forwarded source
checksum or an unlocked PUT is left alone.
The outbound target matrix flips its two KnownFailing(rustfs#7082) cells
to Completed and every Completed cell now asserts that a locked
PutObject carried an integrity header.
Fixes rustfs#7082.
* test(e2e): keep the matrix expectation table clippy-clean under -D warnings
The CI lint runs cargo clippy --all-targets -- -D warnings. With every cell
green the single-arm match tripped match_single_binding and the unused
KnownFailing variant tripped dead_code, and the target-client tests tripped
field_reassign_with_default. Drive the expectation table from a
KNOWN_FAILING_CELLS constant (so the variant stays live and adding a red
cell is a one-line entry), build the test options as struct literals, and
refresh the e2e-repl-nightly selection digest for the renamed table test.
test(e2e): add the outbound target matrix and the replication checksum postmortem
Defense work for rustfs#7082, the regression rustfs#6895 introduced while
fixing rustfs#6853: a fix for one target class changed a client default for
every target class and nothing in tree modeled the other classes.
- docs/postmortems: timeline, root cause, why four defense layers missed
it, and the SOP for changing any outbound client default; AGENTS.md and
the adversarial compatibility lens point at it; the two env knobs from
rustfs#6895 are documented in docs/operations.
- fake_s3_target: reject_aws_chunked_uploads, require_checksum_for_object_lock
(Content-MD5 always verified), create_bucket_with_object_lock with a
GetObjectLockConfiguration handler, and a TransportSnapshot on every
journal record.
- replication_target_matrix_test: six object shapes against four target
modes with an explicit expectation table; the two rustfs#7082 cells are
pinned KnownFailing and fail with an XPASS message once the fix lands.
Wired into e2e-repl-nightly, excluded from e2e-full.
* feat(odm): enable on-demand migration by default
The module switch RUSTFS_ON_DEMAND_MIGRATION_ENABLED now defaults to true,
so the feature is reachable without an opt-in; setting it to false still
keeps the module out of the read path entirely. A bucket without an
on-demand-migration.json is never resolved by the runtime and makes no
source call, so the flip changes nothing for unconfigured buckets.
The admin plane now reads the switch through the predicate published by
module_switches.rs instead of its own duplicated env constant; the
behaviour (an environment read per call) is unchanged.
* test(e2e): wire three on-demand migration cases into e2e-smoke
The PR smoke lane gains one case per user-visible contract: a GET miss
that pulls and persists, a HEAD miss that answers from the source and
stores nothing, and the admin config/status pair that must redact the
source secret. The HEAD case did not exist outside the nightly
real-source lane, so it is added to get_basic_test.
Measured on darwin: the lane goes from 168 tests in 101.98 s to 171
tests in 101.92 s, since the three cases overlap the lane's existing
work. The darwin selection digests for e2e-smoke and e2e-full are
regenerated; the e2e-full linux digest still needs a Linux runner.
* docs(changelog): record the on-demand migration feature
* feat(ecstore): add on-demand migration backfill job core
Add the background backfill job for on-demand migration
(rustfs/backlog#2159): a durable checkpoint under
buckets/<bucket>/on-demand-migration-backfill.json saved by If-Match
compare-and-set every 1000 keys or 10 s, a 60 s owner lease renewed by
every save, a recovery pass that takes over expired leases (or jobs this
node owned before a restart) and cancels jobs whose config changed, and a
main loop over the source ListObjectsV2 pages with the skip_existing
policy, dry runs, bounded outstanding pulls and wait-on-full enqueueing.
The pull queue gains per-job completion reports so the job can count
pulled/failed keys (hashes only), and pull permits become two-tier so an
online miss is never queued behind a backfill pull.
* feat(admin): expose on-demand migration backfill job
Wire the ODM-12 backfill job (rustfs/backlog#2159) to its operators:
POST /v3/on-demand-migration/{bucket}/backfill?op=start|cancel and
GET .../backfill return the checkpoint document, GET .../status gains a
backfill summary, and the recovery loop plus the process-wide runner are
installed at startup. Backfill control reuses
Set/GetBucketOnDemandMigrationAction and is recorded in the route policy,
the registration matrix and the admin route snapshot.
Add the rustfs-madmin wire types and client methods with golden fixtures
shared by the server tests, the backfill_* metric descriptors and their
collector, and three e2e scenarios: a full backfill across list pages,
cancellation, and resuming from the persisted continuation token after a
server restart.
* docs(odm): add the on-demand migration operations guide
Positioning against replication, site replication, tiering and the
replication read-proxy; the admin API walkthrough; the full config field
table with defaults and bounds; the semantics table with the test that
pins each row; provider presets and least-privilege source permissions;
integrity, ETag and metadata mapping; protections, error codes,
observability with PromQL, troubleshooting, known limitations and
security notes.
Adds docs/architecture/background-services-inventory.md with the
write-back pipeline, backfill job and backfill recovery loop rows, and
registers it in the architecture index.
Refs rustfs/backlog#2160, rustfs/backlog#2147.
* docs(odm): note migration sources on the replication egress guard
Refs rustfs/backlog#2160.
houseme