* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
`GET /v3/tier-stats` answered from whichever process received the
request, returning that node's rolling 24-hour transition counters as
if they were cluster totals, and the `TierRequestsSuccess` and
`TierRequestsFailure` metric names had no producer at all.
The body now separates the two quantities a tier carries. Stored
inventory comes from the persisted scanner usage snapshot, which is
already cluster-wide; rolling activity is summed over every member
through a new read-only `TierDailyStats` peer RPC. Rings are merged
rather than added, so an idle node's expired hours age out, and each
node counts only its own committed transitions, so a retry is counted
once. Coverage travels with the numbers: `activity.status` names the
reporting members and the ones that could not be asked, timed out, or
answered with a ring this build refuses to merge, and per-tier
inventory is absent rather than zero when the snapshot has no
accounting. The version 1 body stays reachable at `?format=legacy`.
Tier request counters are recorded at the two seams every remote
request passes through, so a new provider is counted by construction,
with a closed operation/outcome label set that can never grow a tier
name, endpoint or object key.
Closesrustfs/backlog#2207
Co-authored-by: cxymds <cxymds@gmail.com>
* feat(odm): merge the source listing into ListObjectsV2
Adds policy.list_through: ListObjectsV2 merges the local and source
listings into one ordered page so clients see the whole namespace during
an on-demand migration. Local entries win a key both sides hold,
CommonPrefixes are unioned under a delimiter, and the continuation token
is an opaque versioned envelope carrying both cursors.
A source listing failure or an open breaker follows policy.source_error:
propagate answers 424, not_found answers from local state and marks the
response x-rustfs-on-demand-migration-list: local_only. Source listings
are capped at 10 per second per bucket.
* test(odm): refresh the e2e-full darwin selection digest
The list-through e2e module adds seven cases to the merge lane.
* fix(odm): declare the remote client retry policy per consumer
The SDK retry policy was an inherited default: one logical call could cost
three wire requests, so the migration breaker counted logical calls on top
of a threefold amplification against a source that was already failing.
Make it an explicit RemoteS3EndpointSpec field. Replication targets declare
today's standard three attempts and keep their behaviour; the on-demand
migration source and its admin probe declare a disabled policy, so one
counted failure is exactly one source request and pull.rs owns the only
retry budget.
* fix(odm): count a stalled inline source as a source timeout
The inline tee wraps its source body in the idle guard, but the tee turns a
stalled source into an ordinary body read error, so the write-back reported
it as a local write failure. Hand commit_inline the guard so the pull is
counted under source_timeout instead.
The background pump now enforces the idle budget through the same guard
rather than a second copy of the timeout loop.
* test(odm): cover a stalled source body end to end
The fake target can now deliver a GetObject body in slices with a pause
between them, so the inline abort can be driven by a stalled source instead
of a truncated one. Two fault cases drop the workarounds they carried for
the SDK's retries: the scripted fault count and the observed source request
count now have to agree.
The operations guide records the retry and idle-timeout guarantees.
* fix(replication): send an integrity header on Object Lock replication PUTs
AWS S3, MinIO and most compatible targets reject a PutObject that carries
x-amz-object-lock-* headers unless it also carries Content-MD5 or an
x-amz-checksum-* header. Since rustfs#6895 the replication client sends
plain signed payloads with no SDK checksum, so every replicated object
with a retention period or legal hold failed against such targets.
TargetClient::put_object now decides per request through the pure
rustfs_replication::object_lock_put_integrity: a plaintext single-part
object whose source ETag is its MD5 gets Content-MD5 derived from the
ETag (no body pass, framing unchanged); a multipart-layout ETag, managed
SSE or SSE-C passthrough falls back to an SDK CRC32; a forwarded source
checksum or an unlocked PUT is left alone.
The outbound target matrix flips its two KnownFailing(rustfs#7082) cells
to Completed and every Completed cell now asserts that a locked
PutObject carried an integrity header.
Fixes rustfs#7082.
* test(e2e): keep the matrix expectation table clippy-clean under -D warnings
The CI lint runs cargo clippy --all-targets -- -D warnings. With every cell
green the single-arm match tripped match_single_binding and the unused
KnownFailing variant tripped dead_code, and the target-client tests tripped
field_reassign_with_default. Drive the expectation table from a
KNOWN_FAILING_CELLS constant (so the variant stays live and adding a red
cell is a one-line entry), build the test options as struct literals, and
refresh the e2e-repl-nightly selection digest for the renamed table test.
test(e2e): add the outbound target matrix and the replication checksum postmortem
Defense work for rustfs#7082, the regression rustfs#6895 introduced while
fixing rustfs#6853: a fix for one target class changed a client default for
every target class and nothing in tree modeled the other classes.
- docs/postmortems: timeline, root cause, why four defense layers missed
it, and the SOP for changing any outbound client default; AGENTS.md and
the adversarial compatibility lens point at it; the two env knobs from
rustfs#6895 are documented in docs/operations.
- fake_s3_target: reject_aws_chunked_uploads, require_checksum_for_object_lock
(Content-MD5 always verified), create_bucket_with_object_lock with a
GetObjectLockConfiguration handler, and a TransportSnapshot on every
journal record.
- replication_target_matrix_test: six object shapes against four target
modes with an explicit expectation table; the two rustfs#7082 cells are
pinned KnownFailing and fail with an XPASS message once the fix lands.
Wired into e2e-repl-nightly, excluded from e2e-full.
* feat(odm): enable on-demand migration by default
The module switch RUSTFS_ON_DEMAND_MIGRATION_ENABLED now defaults to true,
so the feature is reachable without an opt-in; setting it to false still
keeps the module out of the read path entirely. A bucket without an
on-demand-migration.json is never resolved by the runtime and makes no
source call, so the flip changes nothing for unconfigured buckets.
The admin plane now reads the switch through the predicate published by
module_switches.rs instead of its own duplicated env constant; the
behaviour (an environment read per call) is unchanged.
* test(e2e): wire three on-demand migration cases into e2e-smoke
The PR smoke lane gains one case per user-visible contract: a GET miss
that pulls and persists, a HEAD miss that answers from the source and
stores nothing, and the admin config/status pair that must redact the
source secret. The HEAD case did not exist outside the nightly
real-source lane, so it is added to get_basic_test.
Measured on darwin: the lane goes from 168 tests in 101.98 s to 171
tests in 101.92 s, since the three cases overlap the lane's existing
work. The darwin selection digests for e2e-smoke and e2e-full are
regenerated; the e2e-full linux digest still needs a Linux runner.
* docs(changelog): record the on-demand migration feature
* feat(ecstore): add on-demand migration backfill job core
Add the background backfill job for on-demand migration
(rustfs/backlog#2159): a durable checkpoint under
buckets/<bucket>/on-demand-migration-backfill.json saved by If-Match
compare-and-set every 1000 keys or 10 s, a 60 s owner lease renewed by
every save, a recovery pass that takes over expired leases (or jobs this
node owned before a restart) and cancels jobs whose config changed, and a
main loop over the source ListObjectsV2 pages with the skip_existing
policy, dry runs, bounded outstanding pulls and wait-on-full enqueueing.
The pull queue gains per-job completion reports so the job can count
pulled/failed keys (hashes only), and pull permits become two-tier so an
online miss is never queued behind a backfill pull.
* feat(admin): expose on-demand migration backfill job
Wire the ODM-12 backfill job (rustfs/backlog#2159) to its operators:
POST /v3/on-demand-migration/{bucket}/backfill?op=start|cancel and
GET .../backfill return the checkpoint document, GET .../status gains a
backfill summary, and the recovery loop plus the process-wide runner are
installed at startup. Backfill control reuses
Set/GetBucketOnDemandMigrationAction and is recorded in the route policy,
the registration matrix and the admin route snapshot.
Add the rustfs-madmin wire types and client methods with golden fixtures
shared by the server tests, the backfill_* metric descriptors and their
collector, and three e2e scenarios: a full backfill across list pages,
cancellation, and resuming from the persisted continuation token after a
server restart.
* test(odm): add the fault, concurrency, interaction and real-source e2e
Twenty-one scenarios for on-demand migration: source failures and the
circuit breaker, single-flight and the pull-concurrency limits, how a
pulled object composes with encryption, Object Lock, quota, events,
replication, delete markers and the disable switch, and two cases against
a real second RustFS as the source.
Every assertion carries a source-request-count dimension so a case cannot
pass by serving the client while quietly re-reading the source.
* ci(odm): route the slow on-demand-migration e2e to the nightly lane
The fault, concurrency and real-source modules wait out the 30 s breaker
window, drive 100-deep bursts, or start extra RustFS processes, so they
join e2e-nightly and are subtracted from the e2e-full merge budget. The
e2e-smoke selection is unchanged.
feat(rustfs): serve GET misses from the migration source
Wire the on-demand migration read-through into the GET path, after the
local read and the replication proxy have both missed (rustfs/backlog#2156).
A source HEAD supplies size, validators and metadata. Conditional headers
are evaluated locally against it and never forwarded, so a source 304/412
cannot be mistaken for a source failure. An object within inline_max_bytes
is teed: the primary streams to the client while the secondary commits the
local copy in a background task, so a client disconnect still stores the
whole object and a failed write-back never touches the client stream.
Range reads and larger objects stream straight through and queue a
background pull per policy. Concurrent misses of one key share the
singleflight slot: the leader tees, followers re-read local after it
commits or degrade to passthrough after first_byte_ms.
Version reads, partNumber reads, anti-loop marked requests and a respected
local delete marker keep their original 404. Source answers carry
x-rustfs-on-demand-migration: source; local hits are untouched, and the
local hit path gains no await or lock.
* fix(ecstore): persist merged checksum type for full-object multipart
complete_multipart_upload built the object-level checksum record from a
ChecksumType copied before the MULTIPART / INCLUDES_MULTIPART flags were
merged in. ChecksumType::merge takes &mut self, so the merge updated the
local variable while the copy already inside the Checksum struct stayed
behind. The composite branch rebuilt the Checksum from the merged type
and was unaffected; the full-object branch never rebuilt it, so those
flags never reached disk.
rustfs_rio::read_checksums only sets its multipart flag and only emits
the "x-amz-checksum-type" = "FULL_OBJECT" entry inside its MULTIPART
branch, so a full-object multipart object read back as non-multipart with
no type entry, and GetObject and HeadObject answered with no
x-amz-checksum-type header at all where AWS returns FULL_OBJECT.
Hand the full-object branch the merged type instead of rebuilding the
Checksum: the value must stay the running merge produced by add_part,
because hashing the concatenated part digests would yield the COMPOSITE
value, a different number than the one the client sent. The serialization
now lives in multipart_object_checksum_record so both shapes are covered
by unit tests.
Records written by earlier builds carry the bare algorithm type with no
MULTIPART flags and no trailing part block; they keep reading back to the
same checksum value, and the FULL_OBJECT reader arm predates this change
so older peers parse the new record shape correctly too.
Found while root-causing rustfs#6825.
* fix(s3): reject contradicting multipart checksum type as client error
A CompleteMultipartUpload declaring an x-amz-checksum-type that
contradicts the type recorded at CreateMultipartUpload answered 500
InternalError, telling the caller to retry a request that can only ever
fail. The storage layer does refuse the combination, but through a
generic error that maps to InternalError.
Validate the header against the recorded type in the usecase, where the
upload metadata returned by get_multipart_info is already in hand, and
answer InvalidRequest naming both types, matching AWS. The storage-layer
check stays as a backstop for non-HTTP callers.
Uploads created without a checksum algorithm record no type, so there is
nothing to contradict and the header is left alone rather than newly
rejected. Replication is unaffected: replication_put_object_options
already excludes x-amz-checksum-type from the metadata it forwards.
* test(e2e): cover full-object multipart checksum type round-trip
Adds an end-to-end test that a CRC32 FULL_OBJECT multipart upload reports
x-amz-checksum-type: FULL_OBJECT and the unsuffixed full-object value on
both GetObject and HeadObject, and one that a CompleteMultipartUpload
contradicting the recorded type is rejected as InvalidRequest while
leaving the upload intact. Extends the existing CRC64NVME multipart test
with the same checksum-type assertion.
* fix(s3): keep checksum-type validation off the s3s error macro
The s3s footprint ratchet (scripts/check_s3s_footprint.sh) counts
s3_error! invocation lines and is lower-only: new code must route
through the gateway abstractions rather than widen the direct s3s
surface the s3gate migration is shrinking.
Raise the contradiction through ApiError::invalid_request instead. The
response is byte-for-byte identical -- From<ApiError> for S3Error carries
the InvalidRequest code and the message through unchanged -- and the
usecase already returns ApiError elsewhere, so this is the idiomatic
path rather than a way around the counter.
The explanatory comment deliberately says "the s3s error macro" instead
of naming the macro: the ratchet counts raw matches, so spelling it out
in a comment tripped the same check.
* test(e2e): extend fake S3 target as an on-demand migration source
Add ListObjectsV2 paging, Range GET/HEAD, unversioned buckets, standard
and user metadata replay, ResponseStatus/TruncateBodyAt/Stall fault
actions, Range/User-Agent/prefix/continuation-token journal fields,
count_requests, direct seeding, and a configurable object cap to the
programmable fake S3 target, and add the on_demand_migration e2e
harness (OdmTestEnv, admin wrappers, source seeding, local-state
assertions, second RustFS source) with its self-test.
* test(ci): refresh darwin e2e-full selection for ODM harness
The mixed-version rolling upgrade suite asserted a single list_objects_v2
snapshot seconds after restarting a node. Peers keep a restarted node's
drive in Suspect/Returning for ~probe_interval(2s) x success_threshold(3),
and while one drive is excluded the strict listing quorum (write quorum,
3 of 4) drops objects that were themselves legally written at 3/4 during
an earlier node's identical post-restart window, under-counting the
listing (observed as 254 vs 258 in CI) even though every object still
GETs correctly. Replace the snapshot asserts with a bounded convergence
poll; a real upgrade data-loss regression still fails after the deadline.
* fix(kms): restore persisted configuration after restart
* docs(kms): cover the reload route and startup load states
The admin contract matrix pins every dynamic KMS route for the rc and
console handoff, so the new POST /kms/reload needs a row there, and the
reload response reuses the configure snapshot shape rather than adding a
wire type. The observability runbook gains the operator procedure the
reload exists for: telling a load_failed startup apart from a server
that was never configured, and recovering without resubmitting secrets.
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <heihutu@gmail.com>
- common.rs gains an AdminTransport knob (Signed | Awscurl) with admin_execute_at plus three family wrappers: admin_create_user_via, admin_add_canned_policy_via, admin_attach_user_policy_via; the existing admin_create_user now delegates over the Signed transport.
- Deleted the four signed admin request clones in admin_mfa_test, admin_auth_test, reliant/tiering, and inline_fast_path_cluster_test; each keeps a thin local wrapper over common::admin_request so call sites keep their Option<&str> body shape.
- Deduped the notification_webhook signer onto common::signed_request and the webdav_core signer plus its three admin helpers onto the shared _via helpers.
- Consolidated the S3-client-with-credentials builders: admin_auth s3_client_with, existing_object_tag user_client/sts_session_client, bucket_policy_check create_user_client, and the create_user_s3_client copies in group_delete_test and replication_extension_test now delegate to create_s3_client_with_credentials / build_test_s3_config; replication_extension admin_add_canned_policy and admin_attach_policy_to_user route through the _via helpers on the Signed transport.
- The awscurl-gated suites (existing_object_tag_policy, bucket_policy_check, policy/policy_variables) keep going through the external awscurl binary via AdminTransport::Awscurl, preserving their wire behavior.
Part of rustfs/backlog#1846 (cluster 2).
* test(ecstore): decouple server config snapshot test from global defaults
The final assertion of server_config_snapshot_serializes_read_modify_write_transactions
compared the second snapshot against a fresh Config::new(). Config::new()
reads the process-global DEFAULT_KVS OnceLock, which a sibling test in the
same process can register mid-run (crate::config::init()), so the in-process
run 'cargo test -p rustfs-ecstore --lib config::' failed while nextest's
process-per-test isolation hid the coupling. Assert on the snapshot's raw
bytes against the baseline blob instead, which is deterministic and matches
the invariant under test: the second transaction observes the store unchanged
by the first.
* test: deflake presigned tamper helper and relocated-pool resume staging
tamper_signature only remapped '0' and 'a', so a signature containing
neither (about 1 in 5000) left the URI unchanged and tripped the helper's
own guard assert in CI. Complement every hex digit (15 - v) instead: the
map has no fixed point, so the tamper always changes the value while
keeping length and hex shape.
execute_get_object_resumes_from_relocated_pool_without_splicing_body
staged the relocation by reading xl.meta from every source-pool disk, but
a write-quorum commit legitimately leaves a lagging minority disk without
the object directory (#6701) — the test already tolerates that gap when
normalizing the upload pool, and CI suite IO load hit the same gap in the
staging loop. Skip sourceless disks, carry the staged metadata path
explicitly, and assert a write-quorum majority was staged.
Henry Guo