put_object_part's commit phase held an exclusive write lock on the whole
upload_id_path namespace, so concurrent UploadPart commits for different
part numbers of one upload serialized behind a single lock and returned
503 once the 5s lock-acquire timeout elapsed.
Adopt MinIO's PutObjectPart lock scope: a shared read lock on the
uploadId namespace plus an exclusive write lock on
{upload_id_path}/part.{N}. Different part numbers now commit
concurrently; same-part retries still serialize (backlog#853);
complete/abort keep the uploadId write lock and still exclude every
in-flight part commit. The lock-loss fence covers both guards.
Fixes#5961
test_format_v1 (ecstore layout::format) only printed its results; the pinned v1 format.json literal never parsed at all because "this": null fails Uuid deserialization, and the Err was silently discarded. Fix the fixture to the real on-disk shape (MinIO and RustFS always write a concrete disk UUID there) and assert a serialize->parse roundtrip identity plus every pinned field of the literal.
test_console_cors_configuration discarded all four parse_cors_origins results; parse_cors_origins returns an opaque CorsLayer, so the test now drives real CORS preflight requests through an axum router and asserts the allow-origin outcomes: wildcard answers any origin with *, a configured list echoes listed origins and refuses unlisted ones, empty/unset configurations allow no cross-origin caller.
test_heal_channel_processor_new only constructed the processor; it now asserts the response channel accepts a send.
Ref rustfs/backlog#1836 (PR1).
select_object.rs re-declared six interop header names as SELECT_* locals (five X-Minio-Internal-Server-Side-Encryption-* markers plus x-rustfs-encryption-key-id). The canonical owners live in rustfs-utils' object_encryption_keys module, which the rustfs crate already depends on with the full feature set. Import them under their canonical names and drop the local copies; SELECT_KMS_ARN_PREFIX stays local because no canonical owner exists for the KMS ARN prefix.
Values are byte-identical, so no behavior change.
Ref rustfs/backlog#1833 (PR3).
kms/service.rs re-declared x-rustfs-encryption-key-id and x-rustfs-encryption-algorithm locally; the canonical owners live in rustfs-utils' object_encryption_keys module, which kms already transitively builds. Enable the http feature on the existing rustfs-utils dependency and import the two constants instead. The explanatory comment about why the algorithm header exists (SSE mode vs AEAD cipher round-trip) moves to the import site.
No dependency-graph change (cargo tree -p rustfs-kms is unchanged apart from the feature) and no behavior change: the imported values are byte-identical.
Ref rustfs/backlog#1833 (PR2).
backlog#1833 PR1 prescribed deduplicating crates/replication/src/http.rs onto the canonical rustfs-utils http modules via a re-export facade. That plan conflicts with a standing architecture guard the issue's review missed: check_architecture_migration_rules.sh rejects any rustfs-utils import or dependency from the replication crate ("replication crate HTTP/helper contracts must not import or depend on rustfs-utils"), the same way it bans rustfs-filemeta and rustfs-storage-api — the wire-contract crate deliberately has zero internal dependencies.
So this lands the issue's fallback shape instead (the same bidirectional do-not-merge pattern the issue itself prescribes for the policy path.rs cluster): a module doc on replication/http.rs naming the canonical owners and the guard that forces the local copy, mirror notes on utils' metadata_compat.rs and header_compat.rs, and a new test pinning every duplicated constant to its literal wire value so the two copies cannot drift silently.
No production code changed.
Ref rustfs/backlog#1833 (PR1).
Add a per-backend rotation-driver matrix to docs/operations/kms-backend-security.md: who performs the rotation on each backend (RustFS for Vault KV2, Vault's Transit engine for Transit, AWS RotateKeyOnDemand for AWS, nobody for Local/Static), how periodic rotation must be scheduled on each (external scheduler for KV2 by design, Vault auto_rotate_period for Transit, AWS-native automatic rotation for AWS since RotateKeyOnDemand carries a lifetime quota), the NIST SP 800-38D 2^32 random-nonce AES-GCM wrap ceiling that Local/Static can never reset, and a pre-rotation checklist referencing the existing upgrade-ordering hard constraint.
Add the KmsKeyRotationOverdue Prometheus rule on rustfs_kms_oldest_key_rotation_age_seconds (400-day conservative default, warning severity, no traffic guard because it is direct gauge state) and its runbook response procedure in docs/operations/kms-observability-runbook.md, following the existing per-alert format.
Sharpen the runbook's rotation-timestamp paragraph with verified per-backend behavior: only Vault KV2 persists rotated_at (stamped in the same check-and-set write that commits the rotation), while Transit and AWS key listings always report it absent, so on those backends the gauge measures key age and does not reset on rotation. Update Threshold calibration and Coverage gaps for the new rule.
Validated with promtool check rules (7 rules, SUCCESS) and scripts/check_doc_paths.sh.
Part of rustfs/backlog#1636 (PR-4).
Deletes three blocks of commented-out code that can never be revived: seven dead tokio::tests plus ~20 commented use statements at the tail of ecstore's store/list_objects.rs test module (all hardcoded to a developer's personal machine path), the commented test_extract_claims in policy/utils.rs, and the commented-out pre-strum AdminAction enum draft in policy/action.rs (the live enum below it is untouched).
Also rewords two doc comments on the bucket-metadata inline-data interop test to drop the personal attribution while keeping the technical content, so the corpse check (rg weisd) now returns zero across the repo.
Ref rustfs/backlog#1836 (PR2).
Delete five zero-caller client modules (api_bucket_policy,
api_get_object_acl, api_get_object_attributes, api_get_object_file,
api_restore), the orphaned TransitionCore get/put_bucket_policy
wrappers, and the two constants only they consumed. Also removes two
unwrap() panics on remote-controlled data in api_get_object_acl.rs
(non-UTF-8 body, missing Owner ID). Tier warm-backend APIs untouched.
Ref: rustfs/backlog#1822 (T1)
Delete the dead ecstore save_data_usage_cache and the thin
DataUsageCache::marshal_msg it was the only caller of, drop the
Serialize derive (and the dead DataUsageCacheStorage trait with its
save path) from the thin projection types so no write path can exist
outside the scanner's canonical map-encoded writer, and pin the
persisted .usage-cache.bin wire bytes with cross-crate fixture tests
on both the scanner writer and the thin reader.
Refs rustfs/backlog#1828 (T1-T3).
The StorageError -> DiskError conversion dropped seven variants with
exact DiskError counterparts (FaultyRemoteDisk, DiskAccessDenied,
DriveIsRoot, IsNotRegular, VolumeNotEmpty, VolumeAccessDenied,
FileAccessDenied) into the DiskError::other fallback, degrading them to
an opaque Io error. Two of them sit on the quorum ignore-lists in
disk/error_reduce.rs, so a degraded instance would stop matching the
ignore list and count toward the dominant error in reduce_errs.
Also mirror the StorageError-side io::Error downdrill in
From<io::Error> for DiskError: recover a StorageError boxed through
From<StorageError> for io::Error instead of wrapping it as Io.
Add a round-trip identity test over every DiskError variant
(DiskError -> StorageError -> DiskError) and a boxed-StorageError
recovery test.
* test(heal): strengthen replacement e2e evidence
* test(heal): fix replacement e2e barriers
Accept the real post-fault scanner failure-to-idle sequence as the live disk loss barrier, and preserve the first definitive completed status while only resampling the physical census for premature-completion confirmation.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
A key-marker that does not start with the request prefix is invalid input, not an unimplemented feature.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: houseme <housemecn@gmail.com>
fix(get): give UringBackend the only fd cache for its disk (#1801)
#1801 made `StdBackend::new` always build a descriptor cache. `UringBackend`
wraps a `StdBackend` (`inner`), so under io_uring a disk ended up with TWO
`FdCache`s: the wrapper's and the inner's. `UringBackend::pread_bytes`
delegates to `inner.pread_bytes` on four fallback paths (latch-off, O_DIRECT
unsupported / error, buffered-read error), which populated `inner.fd_cache` —
but `UringBackend`'s invalidation only touches its own cache, so the inner
cache was never invalidated. For up to `FD_CACHE_TTL` (5s) after a heal/rename/
delete, a fallback read could serve the pre-mutation inode: exactly the
stale-descriptor hazard `FdCache`'s generation guard exists to close
(rustfs/backlog#1176). It also double-counted `FD_CACHE_CAPACITY` (512 fds)
against `RLIMIT_NOFILE` per disk (backlog#1178).
Fix: `UringBackend` now constructs its inner `StdBackend` with the new
`StdBackend::new_without_fd_cache`, so the wrapper owns the only cache for the
disk. The inner backend opens per read on fallback, leaving nothing
unguarded. `StdBackend::new` (standalone default) is unchanged; a private
`build(root, build_fd_cache)` holds the shared construction.
- Default (non-io_uring) path: byte-for-byte unchanged.
- io_uring path: one cache per disk, fully covered by the wrapper's
invalidation; halves the per-disk fd budget under `RLIMIT_NOFILE`.
- `RUSTFS_IO_URING_FD_CACHE` / `RUSTFS_LOCAL_FD_CACHE` semantics preserved.
- Regression test pins `new_without_fd_cache` -> no cache.
Found by a post-merge re-review of the Wave 1 GET PRs. cargo check/clippy
clean; Linux compile + the io_uring fd-cache suite deferred to CI.
Co-authored-by: heihutu <heihutu@gmail.com>
docs(durability): document the new-bucket relaxed default (#1811)
#5971 seeds a `relaxed` durability override into newly created buckets'
metadata (rustfs/backlog#1811), but the operator guide still described the
default as `strict` with relaxed as opt-in, and never mentioned the new-bucket
seed or its `RUSTFS_NEW_BUCKET_DURABILITY_MODE` knob. Document the behavior:
- intro notes new buckets default to `relaxed` (gradual migration; the
process-wide default and pre-existing buckets stay `strict`);
- the Configuration block lists `RUSTFS_NEW_BUCKET_DURABILITY_MODE`
(relaxed|strict|none|inherit, default relaxed);
- a new "New-bucket default" subsection under per-bucket durability covers the
seed semantics, the opt-out (`inherit` / `=strict`), fail-closed invalid
values, and that existing buckets are never retroactively rewritten.
No code change.
Co-authored-by: heihutu <heihutu@gmail.com>
Every GET fans out a `read_version` across all disks to resolve xl.meta. Each
fanout allocated an `Arc<ReadOptions>` (3 bools) plus four `Arc<String>`
(`Arc::new(x.to_string())` = two allocations each) and cloned them into every
spawned task. This trims the per-fanout allocation footprint.
- `ReadOptions` is three bools, so it is now `Copy`. The fanout drops the
`Arc<ReadOptions>` and hands each spawned task a copy; the two pre-existing
`ReadOptions::clone()` sites (set_disk/read.rs, set_disk/ops/heal.rs) stop
cloning a `Copy` type.
- The four request strings use `Arc::<str>::from(&str)` (one allocation each)
instead of `Arc::new(..to_string())` (string buffer + Arc = two each) — four
fewer allocations per fanout, transparent to the `read_version(&str)` call.
Behavior is unchanged: the fanout still spawns one task per disk (the spawn is
deliberate — `read_version_call_counter_observes_spawned_fanout` verifies the
process-global counter observes every per-disk increment across workers), quorum
/ early-stop / full-wait semantics are untouched, and no result ordering or
error handling changed.
Two larger items from the audit are intentionally NOT in this PR:
- `tokio::spawn` -> `FuturesUnordered`: the spawn is a tested, deliberate
design (cross-worker counter observation for #1309/#1314), and converting
would also change panic isolation. Left as-is.
- `vec![FileInfo::default(); N]`: `FileInfo`'s empty containers (String /
HashMap / Vec) do not allocate, so this is one `Vec` allocation, not the
per-element allocation the audit implied — not a real hot spot.
`cargo fmt`, `cargo clippy -p rustfs-ecstore --lib` (0 warnings),
`cargo check --lib --tests`, and the 26 fanout / call-counter unit tests pass
on macOS (the change is fully cross-platform).
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A small object whose data shards are inlined in xl.meta can be reassembled
straight from the already-resolved metadata, skipping the Erasure reconstruct
pipeline. The fast path existed but was opt-in
(`RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY`, default off), so every deployment
paid the full shard-read fan-out for eligible small GETs by default.
- Flip `DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY` to `true`. The path is
correctness-neutral on a miss: `try_get_object_direct_data_shards_*` returns
None when the inline reassembly cannot satisfy the read, and the GET then
proceeds through the normal shard-read pipeline. The env stays as a kill
switch (`=false` restores the legacy path).
- Drop the bucket-level `versioned` / `version_suspended` exclusions. The
decision is made on `fi` — the already-resolved target version — so
reassembling its inlined data is correct on a versioned bucket too. An
explicit versionId GET still falls back (`opts.version_id`), and a
delete-marker latest is still rejected. The two now-unused fallback reasons
and their metric labels are removed.
- Tests updated: a versioned latest-version object is now eligible / `Use`
(covers the newly allowed path); the removed reasons' label assertions are
dropped.
cargo check --lib --tests and the direct_memory unit tests pass on macOS
(the change is fully cross-platform).
Co-authored-by: heihutu <heihutu@gmail.com>
StdBackend opened, stat'd, and access-checked the shard file on every
positioned read, so each small-object GET paid N x (open + access + fstat)
syscalls even for hot shards. UringBackend already caches descriptors
behind a generation-guarded, rlimit-bounded moka cache with full
rename/delete/heal invalidation; StdBackend had no equivalent.
Port that cache to the default backend:
- StdBackend gains an `fd_cache: Option<FdCache>` (Linux only, mirroring
UringBackend), built in `new()` behind `RUSTFS_LOCAL_FD_CACHE` (default
on) and the same `rlimit_allows_fd_cache` guard.
- pread_bytes consults the cache on the buffered path: a hit reuses the
descriptor via `dup` (one syscall, no path resolution or permission
re-check) and skips volume access; a miss opens as before, snapshots the
invalidation generation, and hands the freshly opened descriptor back
for `insert_if_fresh`, which refuses to cache if a heal/delete bumped the
generation mid-open (rustfs/backlog#1176). O_DIRECT reads keep opening
their own aligned descriptors.
- The DirectReadCopy branch switches from seek+read_exact to
`FileExt::read_exact_at`: a `dup`'d cached descriptor shares the source
descriptor's open-file offset, so a positioned read (like the mmap path's
offset argument) keeps concurrent cache hits on the same shard correct.
- The four `LocalIoBackend` invalidation methods now drop stale entries on
StdBackend. LocalDisk already calls them on rename_data/rename_file/
delete/delete_volume/close, so no new call sites are needed.
- Two tests mirror the io_uring ones: a heal rename must be hidden until
invalidate_cached_fds_under runs, and a repeated read caches exactly one
descriptor that prefix invalidation drops.
Behavior is byte-for-byte unchanged on a miss and on non-Linux; the cache
is auto-disabled only when RLIMIT_NOFILE is too low. macOS cargo check
--lib and --tests pass; Linux compile deferred to CI.
Co-authored-by: heihutu <heihutu@gmail.com>
Treat recovery-directory lookup on a replacement endpoint already deferred as replacement_path_unavailable as an expected debug diagnostic instead of a durable generation conflict.
Keep real survivor recovery conflicts and corrupt records on the existing warning/blocking path.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(heal): add privileged replacement rebuild e2e
Add ignored Linux-only 3x4 automatic replacement coverage for EC8+4 and EC6+6. The tests use real tmpfs mounts in an isolated mount namespace, wait for scanner-driven replacement recovery status, and verify the replacement target with per-version xl.meta and part.N physical census without invoking Admin deep heal.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(test): avoid unsafe in privileged replacement e2e
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): harden privileged replacement e2e
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): prove absent replacement recovery witness
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): prove absent replacement observation
Stop the target node before detaching the test mount so RustFS releases its mount lease instead of continuing to serve the old tmpfs through an open fd. Restart the node with the endpoint absent and wait for the scanner's real readiness rejection in that node's log.
Assert the absent window has no replacement intent, completion proof, checkpoint, healing marker, or Admin v4 durable record for the target before mounting the blank replacement and waiting for automatic recovery plus physical shard census.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): streamline cluster log capture
Move cluster-node log capture out of ClusterNode and into per-node cluster launch configuration so the privileged replacement E2E uses an explicit harness API instead of mutating node identity data.
Reuse the same stdout/stderr capture helper for single-node and cluster processes, and pin the per-node capture behavior with a focused common test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): harden privileged replacement proof
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* test(site-replication): pin retry-event lost-update against locked RMW (red)
P1-15 (rustfs/backlog#1675 B2): the site-replication retry-event writers
(enqueue/dequeue, which hang off every hook broadcast path) perform a
load -> mutate -> persist without taking SITE_REPLICATION_STATE_LOCK, so
a single process can lose a concurrent lock-holding writer's update; the
service-side reload path is equally unlocked, and no writer holds a
distributed lock across the read-modify-write, so multi-node RMW loses
updates even where the process lock is held.
Red evidence (current main): replaying enqueue's exact three steps around
a completed mark_pending_rotation_peer_acked commit wipes the rotation
ack — the final state holds the retry event but not the ack.
* fix(site-replication): route state RMW through one locked transaction
P1-15 PR1 (rustfs/backlog#1675 B2). The site-replication state object
(config/site-replication/state.json, which also carries the retry-event
queue) was mutated through read-modify-write sequences with inconsistent
locking: the retry-event writers on every hook broadcast path and the
RPC-driven service reload took no lock at all (single-process lost
updates, pinned by the red commit), and no writer held a distributed lock
across the whole RMW (cross-node lost updates everywhere).
- New admin/site_replication_state module: the state transaction boundary
`with_site_replication_state_lock[_on]` — process mutex plus the
distributed config-object write lock (the pattern proven by the repair
state), with the shared path constant. The process mutex is transitional
until PR2 migrates the remaining ~26 call sites.
- handlers: typed `update_site_replication_state` (no-lock load /
persist-or-clear inside the boundary; normalizes the peer map exactly
once, retiring the double-clone/double-normalize persist path, P2-22).
Migrated: retry-event enqueue (always-write), dequeue (lock-free probe,
transaction on hit), mark_pending_rotation/remove_peer_acked.
- service reload: the tolerant byte-level read->normalize->save now runs
inside the same boundary via no-lock IO — a cluster-wide reload fan-out
can no longer overwrite a concurrent state writer. Normalization
semantics untouched (all six service-side tests unchanged and green).
- Add/PeerJoin/Edit handlers release the state guard before their peer
fan-out: the transport helpers' retry-event bookkeeping now re-enters
the state transaction and must not nest inside the guard (the
adversarial review caught this as a re-entrancy deadlock; the fix
mirrors the Remove/Rotate handlers' existing scope). The Edit non-
refresh branch commits before fanning out — the old fanout-first order
recorded retry events pointing at a state the local site had not saved.
- ecstore: delete_config_no_lock (+ facade/bridge exports) so the clear
half of persist-or-clear works under the held object lock.
Red -> green: the red commit pinned the deterministic lost-update
interleaving (stale retry-event persist wiping a committed rotation ack);
the test now drives the real functions concurrently for 8 rounds and
asserts every retry event and every ack survives. Full
handlers/service site-replication unit suites green (171 + 6); dual-node
site-replication e2e (state edit fresh/stale, object replication) green;
fmt / clippy / logging guardrails clean.
Adversarial review: one blocking finding (the re-entrancy deadlock above)
fixed and re-verified by a full second pass over all 30 lock sites and
the Add/Join/Edit call graphs. Non-blocking notes recorded for PR2:
mark_* now persists on miss (persist-or-clear semantics; a miss-skip
return is a cheap follow-up), Add still holds the guard across the peer
join probe (pre-existing availability debt), and a timeout-guarded
unreachable-peer regression test for the fan-out paths.
* fix(site-replication): keep the state mutex behind an owner helper
CI's architecture migration guard lists SITE_REPLICATION_STATE_LOCK as an
owner-local static, so it may not be `pub(crate)`. Keep it private to the
new module and let the not-yet-migrated RMW call sites take it through
`site_replication_state_process_guard()` — the sanctioned owner-helper
pattern; the helper disappears with the mutex in PR2.
* fix(site-replication): keep peer-edit delivery under the state guard
Review follow-up (#5882).
Releasing the guard before the fan-out (my deadlock fix) traded the
ordering the guard used to provide: edit A could commit and stall while
edit B committed and reached a peer first, then A arrived last and won.
The peer edit handler applies whatever arrives — it has no generation or
updated-at fence — and a successful stale delivery is not repaired by the
retry queue, so the sites diverge silently.
The fan-out is back under the guard. What actually could not run there is
the retry-event bookkeeping, which re-enters the state transaction, so the
edit branch now delivers with the plain transport and settles the retry
queue after the guard is released: successes dequeue, the first failure
enqueues and is returned. Ordering and bookkeeping both preserved. The add
handler keeps its peer-edit finalize fan-out under the guard for the same
reason and releases only before bootstrap/back-fill, which send bucket-ops
(not peer edits) through retry-event transports.
The concurrency test could not tell the two guards apart — both writers
took both locks, so it passed with either removed. Replaced by two tests
that isolate one guard each, both verified by mutation:
- a process-only legacy writer (the shape the not-yet-migrated call sites
still use) racing the transaction: fails when the transaction stops
taking the process mutex;
- two writers that bypass the process mutex, as separate nodes do, driving
the production object-lock path (`with_site_replication_state_object_lock`
factored out for exactly this): fails when the distributed lock is
removed.
Verification: handlers 173 + service 6 unit tests green; site-replication
dual-node and three-node edit e2e green; arch/layer/logging guardrails,
fmt and clippy clean.
* fix(site-replication): fence peer-edit delivery by generation
Review follow-up on the two remaining holes in the edit path.
Ordering was only process-local. `SITE_REPLICATION_STATE_LOCK` is per
node, so holding it across the fan-out orders the edits ONE node accepts
and nothing else: two nodes of the same site can both commit and reach a
peer in the opposite order, and the peer edit handler applied whatever
arrived last. Each edit now takes a generation from
`SiteReplicationState::edit_generation`, allocated in the same commit as
the edit itself — i.e. under the distributed state-object lock, so two
nodes can never share one. The generation rides the peer-edit request as
query parameters and the receiver rejects (acks without applying) a
delivery at or below the mark it already applied for that origin site,
recording the mark in the same commit as the edit it fences. Peers that
predate the fence send no parameters and are applied as before.
Retry settlement could discard a newer failure. After the guard is
released, a success for edit A removed every retry event for
(peer, peer-edit): if edit B committed, failed its own delivery and
enqueued while A was in flight, A erased it — local state B, peer on A,
nothing queued to converge them. Settlement now only removes events whose
recorded generation is not newer than the one being settled, and a later
failure never lowers the fence. Broadcast paths carry no generation and
settle unconditionally as before; their events live under their own
paths and cannot collide with a peer-edit delivery.
A departed peer's mark is dropped on load: a site that leaves drops below
two peers, which clears its state object and restarts its counter at
zero, so a leftover mark would reject every edit it sends after it
rejoins.
Tests: two-node generation uniqueness (drop the object lock and the two
nodes collide), the receiver's staleness predicate and its wiring, the
settlement interleaving (drop the fence and B's retry is erased), and the
rejoin reset.
Refs: rustfs/backlog#1675 (P1-15)
P1-20 (rustfs/backlog#1675 B2, test-only). No prior test wrote objects
BEFORE the replication rule arrived, leaving the scanner's existing-object
resync pass — the only channel for such objects — without end-to-end
coverage, and the enqueue truth table partially unpinned at unit level.
e2e (both negative cells are contracts, asserted over multiple fast-scanner
cycles next to a replicated control key that proves the scanner and the
live path are running):
- test_scanner_compensates_existing_objects_across_write_paths: plain PUT,
CopyObject and Snowball auto-extract products written pre-rule all
converge via scanner compensation; a null-version object (PUT before the
bucket became versioned) is pinned as never compensated (the scanner heal
gate skips nil-version objects).
- test_scanner_never_compensates_when_existing_object_replication_disabled:
ExistingObjectReplication=Disabled is a contract, not a delay — existing
keys stay absent while post-rule writes replicate normally.
Unit truth-table pins (crates/replication):
- queue.rs: an empty replicate decision (Disabled existing-object, inbound
REPLICA) skips heal queueing for every status; Completed without a resync
decision skips.
- operation.rs: existing-object resync without a reset replicates exactly
the never-replicated (Empty) objects.
Helper: put_bucket_replication_with_statuses parameterizes the previously
hardcoded ExistingObjectReplication status; the nextest count comments are
refreshed to the post-rebase totals.
Increase the replay cache resource model so 16 CPU / 31-32 GiB field nodes auto-size to the 32M cap without an env override.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(replication): pin the version-fidelity probe contract (red)
P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.
Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.
test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.
Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).
* fix(replication): probe the version-identity contract in replication-check
P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.
- The probe PUT now carries the source version as `?versionId=` (the exact
shape live replication uses since P0-5, and the only shape MinIO
consumes; the internal source-version-id header alone would let the
probe pass against targets the real data path drifts on). Reuses
ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
equal the sent source id. On mismatch the phase fails with the
machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
(new optional Code field on phase statuses; Go decoders ignore unknown
keys), the overall target fails, the later version-addressed mutation
phases are skipped, and cleanup still removes the probe via the id the
target actually assigned (with the existing list-based sweep as backstop
when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
id (mirroring remove_object), and the replication PUT path audits it —
every drifting PUT increments
rustfs_replication_version_identity_drift_total and the first drift per
target ARN logs a structured warning pointing at ?replication-check.
The drift judgment is a pure function with an exemption-matrix test
(empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.
Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.
* fix(e2e): stop the fake target self-deadlocking on version-id minting
The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.
The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* fix(ecstore): publish multipart parts on Windows
* test(ecstore): pin Windows multipart durability
---------
Co-authored-by: houseme <housemecn@gmail.com>
Zhengchao An