* chore(obs): ReplicationStats -> ReplicationMetricsSnapshot, BucketReplicationStats -> BucketReplicationMetricsSnapshot
Rename in-obs-crate ReplicationStats and BucketReplicationStats to
ReplicationMetricsSnapshot and BucketReplicationMetricsSnapshot respectively.
No serde impact (these types are Prometheus metric collectors, not serialized).
No external consumers found outside the obs crate.
* cleanup: remove #[serial] annotations from e2e_test, scanner, lifecycle, and object-capacity crates
Remove no-op #[serial] attributes (nextest ignores serial_test) and the
serial_test dependency from four crates. All tests already use temp_env
for env-var isolation, making #[serial] purely redundant.
Crates cleaned:
- e2e_test (37 annotations, 9 imports, removed serial_test dep)
- rustfs-scanner (115 annotations across 7 files, removed serial_test dep)
- rustfs-lifecycle (46 annotations, removed serial_test dep)
- rustfs-object-capacity (38 annotations, removed serial_test dep)
Also converted scanner/tests/lifecycle_integration_test.rs
with_forced_immediate_enqueue_timeout helper from unsafe raw
env::set_var/remove_var to temp_env::async_with_vars for proper
isolation, and added async_closure feature to scanner's temp-env dep.
* fix(lifecycle): restore #[serial] on 2 tests that read env vars without temp_env
eval_inner_expires_latest_object_after_days_due and
eval_inner_does_not_panic_on_many_equal_due_events call eval_inner()
which reads ENV_ILM_PROCESS_TIME via std::env::var(). Without #[serial]
they race with other tests that set these vars via temp_env.
* style: cargo fmt
* feat(ecstore): batch small file fdatasync commits
Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): wait for compression S3 readiness
Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* chore: adjudicate the last 18 bare dead_code allows in the library crates
Finishes backlog#1823 step 10 outside `rustfs/src` and `protocols`: config, s3select-query, common, madmin, heal, ecstore, signer and notify. Stripped first, then clippy asked which the compiler actually missed — 8 of the 18 were inert.
Seven items are deleted, each checked by grep as well as by clippy:
- `common/last_minute.rs`'s private `TimedAction` (with its impl) and `SizeCategory` (with its `Display` impl). The file's public surface — `AccElem`, `LastMinuteLatency` — stays; ecstore consumes it.
- `s3select-query`'s three `with_*` builders. `DefaultLogicalOptimizer::with_optimizer_rules` looks used, but the call in the same file is `SessionStateBuilder::with_optimizer_rules` from DataFusion; the local methods have no callers.
- `heal/manager.rs`'s `contains_key`. Its six apparent references are all `HashMap::contains_key`.
Three keep their code:
- `heal/storage.rs`'s `Test` variant is constructed by the `#[cfg(test)] test()` helper, which the lib target cannot see, so it takes a reasoned allow.
- `signer`'s `STREAMING_PAYLOAD_HDR` and `try_build_chunk_string_to_sign` gain the `_` prefix instead. That file already marks deliberately-unheld code that way — `_STREAMING_TRAILER_HDR`, `_PAYLOAD_CHUNK_SIZE`, and `_try_build_chunk_signature`, which is the only caller of that function. Following the existing convention removes the allow without an attribute.
`protocols` keeps its four; that crate needs `--features swift,sftp` to compile fully and is verified differently. The four `#![allow(dead_code)]` in `e2e_test` are module-root blankets in test-support files, which belong to steps 1-5 rather than step 10.
Refs backlog#1823
* chore(e2e_test): adjudicate the two dead_code allows the lib test target still needs
`cargo clippy --all-targets` compiles e2e_test's lib test target, which the earlier pass did not cover, so these two removals only surfaced in CI.
test_large_multipart_upload's allow was load-bearing: its call site in test_local_kms_multipart_upload is commented out behind "TODO: Re-enable after fixing streaming encryption issues with large files". The allow comes back with the reason string this batch uses everywhere else, so the next reader sees why it is parked instead of deleting a test we intend to run again.
TestDefinition.category was the opposite: written at all six definitions, read nowhere, and its enum's impl block is empty. The live copy of that type is crates/e2e_test/src/kms/test_runner.rs, which has an as_str; the policy copy is a vestige of it. Dropping the field, the enum, and the constructor parameter leaves the runner unchanged — it dispatches on name and filters on is_critical.
Verification: cargo clippy --all-targets -- -D warnings (workspace, the CI command) and cargo fmt --all --check both pass.
---------
Co-authored-by: houseme <housemecn@gmail.com>
Byte-exactness tests stay green if the compressed range seek regresses into
decoding from byte zero: the returned bytes are still correct and only the
read amplification explodes. Assert the cost side as well.
The observation reuses rustfs_io_get_object_shard_read_observed_bytes_total,
already emitted per shard read by the erasure layer, so no production code is
instrumented. The OTLP collector learns to accumulate a second counter, keyed
by its path/role/outcome labels rather than by data-point position, which is
not stable across exports.
Two failure modes the assertions guard against:
- With RUSTFS_OBS_METER_INTERVAL=1, treating one unchanged sample as settled
measures a delta of zero, because the range read's counter has not been
exported yet. Settling now requires several consecutive equal samples.
- An upper bound alone passes vacuously on a zero delta, so a lower bound
turns "measured nothing" into a failure instead of a green run.
Refs rustfs/rustfs#5957, backlog#1848.
* refactor(replication): split four oversized hot-path functions into focused helpers
Pure-move decomposition of the four oversized functions flagged by the
replication compatibility review (P1-18), unblocking migration milestone
M2 which requires resyncer moves to stay mechanical:
- resync_bucket (522 lines -> 61-line step sequence): leader lock,
target resolution, walk/collector/worker spawning, and dispatch loop
extracted into focused helpers; pure decision helpers (DTO builders,
HEAD-result classification) separated from IO orchestration.
- replicate_all (411 lines -> 113-line main body): initial target-info
seeding, read/stat option builders, skip-path notes, target HEAD
action resolution, and the multipart/single-put payload transport
extracted as private free functions.
- start_mrf_processor (306 lines -> 46-line spawn body): recovery guard,
ledger load, per-entry replay (delete/object/metadata), and retained
entry resolution extracted; retry bookkeeping semantics preserved
exactly (inner continue-paths push inside helpers, outer Missed push
stays in the loop).
- apply_iam_item (255 lines -> match dispatch skeleton): one helper per
IAM item type.
No behavior change: log texts, error paths, event emissions, and metric
counts are byte-identical; existing tests unchanged and green (238
ecstore replication/mrf/resync + 232 rustfs site-replication).
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets
Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.
Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
value is never re-proxied. Outbound client proxy calls send the marker
as "true"; replication worker convergence HEADs send it as "false" so
a peer's proxy layer cannot answer a convergence check by proxying
back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
when the marker is set, versioning is suspended, or no replication
config; otherwise filter_target_arns -> TargetClient lookup, skipping
targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
the three tagging calls. Proxy calls never send the replication-check
SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
the target performs real decryption. Conditional (If-*) headers are
not forwarded (MinIO parity); Range and part_number are, with
parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
MinIO-aligned (one total per proxied request, one failed when no
target served it). The previous misattributed counters — replication
worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
removed; ReplProxyMetric now maps the tagging counters instead of
dropping them.
e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.
Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.
Refs rustfs/backlog#1675 (P1-5)
* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)
SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).
Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
BucketTargetSys per target ARN with a recording timestamp. Entries reset
whenever the target is rebuilt, edited, or removed (arn_remotes_map
lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
an expired verdict in either direction is re-earned through the audit, so
an Unsupported target recovers automatically after an upgrade (at most one
wasted PUT+HEAD audit per bad target per TTL window) and a Supported
verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
targets never receive the PUT — the attempt fails immediately into the
normal MRF retry channel with a "run ?replication-check to re-probe" hint.
Unknown or expired verdicts are audited: after the PUT the worker HEADs
the replica back through the replication-check channel (source version id
mapped through resolve_read_api_version_id, so null-version objects audit
correctly) and requires SSE-C evidence (the echoed customer-algorithm
header); missing evidence records Unsupported and fails the attempt.
Convergence HEADs are audited the same way, so a broken ciphertext replica
from an earlier attempt can never launder itself into COMPLETED via an
ETag match. The gate/evidence policy is pure (replication_target_boundary,
staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
carrying the live transport-header shape, HEAD-back for evidence, and a
machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
The probe verdict is synced into the runtime capability cache. Unlike
VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
overall — it is a capability limit, not a broken replication contract,
and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
transport headers, echoes SSE-C evidence); the new
drop_unlisted_replication_headers mode models MinIO. The journal records
whether a request carried transport headers.
Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.
Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.
Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
A hypothetical target that preserves that one header while dropping other
transport headers (partial-drop) would pass the audit; no known target
behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
to different target nodes until the rollout completes; the TTL bounds how
long each stale verdict persists.
New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180)
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2)
Pure-move milestone M2 of the ECStore replication split (backlog#1675
P1-17): relocate the resyncer's IO-free decision helpers, with their unit
tests, into the crates they already belong to by type ownership. No
behavior change.
Moved into crates/replication:
- resync.rs: resync_status_duration
- delete.rs: resync_existing_delete_replication_info,
replicate_delete_outcome, target_delete_version_id,
delete_marker_purge_version_id, delete_marker_purge_mrf_entry
- object.rs: version_identity_drifted, is_replication_target_offline_error,
SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate,
ssec_passthrough_evidence_present (param-demoted to the echoed
customer-algorithm string; ECStore keeps the HeadObjectOutput adapter)
- filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the
filemeta-independence contract)
ECStore rewiring (Rule #14: imports stay in *_boundary.rs):
- resync/object-decision/target boundaries re-export the moved symbols;
resyncer call sites are unchanged
- bucket_target_sys keeps only the verdict cache + TTL and re-exports the
capability enum so existing consumer paths keep compiling
Not moved (signatures carry ECStore or aws-sdk types):
verify_resync_head_result, resync_target_error_detail, the SdkError
classifiers, the replicate_all_* option/info builders, and the env-coupled
bounded_resync_max_jobs admission clamp. README milestone table updated.
* chore(replication): retire the datatypes.rs relay early
README sanctions retiring datatypes.rs ahead of M4. The module was a
pure relay (resync boundary -> datatypes -> mod.rs facade) with no
external consumer importing it directly, so the facade now re-exports
ResyncStatusType from replication_resync_boundary and the relay file is
deleted. Consumers stay behind the ECStore facade, keeping Migration
Rule #15 intact — the original retirement wording ("consumers import
through rustfs-replication directly") conflicted with that rule and is
corrected in the README.
* chore(arch): extend migration guards to the M2-moved decision contracts
The adversarial review of the M2 move found the per-symbol ratchet in
check_architecture_migration_rules.sh was not extended for the moved
symbols, leaving them free to be redefined in ECStore or imported past
their boundary without CI noticing:
- resync definition pin + boundary fences gain resync_status_duration;
- the object-decision boundary fences gain the five delete-family
helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
replicate_delete_outcome, resync_existing_delete_replication_info,
target_delete_version_id);
- the target-boundary fence gains the SSE-C gate family, the offline
classifier, and version_identity_drifted;
- a new definition pin rejects ECStore redefinitions of the M2-moved
fns/enums (ssec_passthrough_evidence_present deliberately excluded:
ECStore keeps a thin HeadObjectOutput adapter under that name).
Mutation-verified: a probe fn ssec_passthrough_gate under
crates/ecstore/src/bucket/replication trips the new pin.
Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal
from the filemeta side and tightens the M2 README note on
bounded_resync_max_jobs.
test(e2e): drop no-op serial markers from the three densest e2e suites
serial_test's #[serial] is an in-process mutex. cargo-nextest, this repo's
authoritative runner, executes every test in its own process, so the mutex is
never contended and the attribute is a documented no-op -- see the "Serial
execution & nextest profiles" section of docs/testing/README.md and the header
of .config/nextest.toml. Cross-process serialization is provided only by a
[test-groups] entry with max-threads = 1.
Remove 187 such markers (plus 3 now-unused imports) from the three
marker-densest modules of crates/e2e_test:
multipart_auth_test.rs 85
replication_extension_test.rs 68
object_lock/object_lock_test.rs 34
None of these modules is covered by any [test-groups] entry, so the markers
were carrying no isolation for any lane. Every test in all three files builds
its own server via RustFSTestEnvironment::new(), which gives a UUID temp dir
and a uniquely allocated port -- the .config/nextest.toml comment on
replication_extension_test already states this explicitly ("parallel-safe by
construction"). No test mutates process env, binds a fixed port, or touches
process-global state, so nothing here needed temp_env or a test-group instead.
Pure deletion: 190 lines removed, 0 added, no test renamed, no behaviour
changed. serial_test stays in Cargo.toml -- 338 markers across 90 other files
in the crate still use it.
Refs: backlog#1846 (T1).
* feat(heal): track erasure set progress baseline
Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available.
Keep usage-cache failures observational so heal execution continues without a baseline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): skip filtered erasure set versions
Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them.
Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): wire abandoned data-dir cleanup check
Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented.
Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): add heal scanner trace bus
Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior.
Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): stream heal trace events from admin API
Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit heal trace events
Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit scanner trace events
Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(heal): route data usage loader through storage api
Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(heal): avoid lifecycle snapshots on ordinary heal pages
Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): update bug-fix mocks for lifecycle snapshots
Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): sync heal storage mock signature
Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): allocate smoke ports across nextest processes
Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* test(replication): pin missing LWW timestamp header transport
Red-light tests for the replication timestamp three-header contract:
- put_object_headers_carry_replication_timestamp_headers pins that
PutObjectOptions::header() must emit the
x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp
headers when the internal timestamps are set (currently missing).
- test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization
and test_complete_multipart_opts_persist_replication_timestamps_when_authorized
pin that an authorized replication PUT / multipart complete must persist
the inbound timestamps into the internal metadata keys while unauthorized
requests must not (currently never persisted).
- fake_s3_target journals the three timestamp headers per request
(ReplicationTimestampHeaders on RequestRecord) so sender-side e2e
assertions can observe what a real target receives; self-test included.
* fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold
Active-active conflict resolution for concurrent tag/retention/legal-hold
edits needs the source's per-category modification times on both sides of
the wire; the three AdvancedPutOptions timestamp fields were dead and the
headers were neither sent nor parsed.
- Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-
timestamp from PutObjectOptions::header(); names and RFC3339 values
interoperate with MinIO (minio-go constants.go, object-api-options.go),
pinned by a header_compat wire-name test.
- Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip
epoch values in header(), so "never modified" is not sent as a
modification made now.
- Parse the headers only on authorized replication PUTs and multipart
completes, expose them as Option<OffsetDateTime> on ObjectOptions, and
persist them into the dual-prefix internal metadata keys so the
outbound pass (replication_target_boundary) reads the source's
timestamps instead of the mod_time fallback.
- Record the local tagging timestamp in the PutObjectTagging and
DeleteObjectTagging eval metadata, mirroring the object-lock handlers;
without it the sender only ever had the mod_time fallback to offer.
Receiver-side LWW comparison (keep newer stored category metadata over a
stale inbound copy) is left as a TODO at the parse site.
* fix(replication): load the stored tagging timestamp independently of remaining tags
Review: DeleteObjectTagging persists the tagging-timestamp internal key
but leaves the object tagless, and the outbound mapper only loaded the
key inside the user_tags-nonempty branch — the deletion's LWW timestamp
stayed at the epoch and the header was omitted, so the deletion could
never win conflict resolution on the replica. The stored key is now
loaded unconditionally; the mod_time fallback still applies only while
tags exist (MinIO parity), and a tagless object without the key keeps
the epoch default (no header). Deletion-path regression test added.
* fix(storage): reserve replication transport names at metadata ingest
Second review round: a client PUT of
x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized
the bare transport key as stored user metadata. The outbound
replication header builder forwards user metadata verbatim on a
server-authorized request, so the receiver would persist the
attacker-chosen value as trusted internal LWW state — and for a
tagless object nothing later overwrites it.
The ingest namespacing guard now reserves the whole
x-rustfs-source- / x-minio-source- families (the new timestamps and
their siblings: source-mtime/-etag/-version-id/-replication-request),
folding forged keys back under x-amz-meta-. Forged-ingress regression
covers both prefixes and a sibling.
* fix(replication): harden timestamp replay
* fix(app): route retention helper through facade
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(storage): restore multipart disk compression and make the legacy decompressor resumable
Multipart uploads have bypassed disk compression since #5169 removed the session marker as a stopgap for mid-stream GET failures. The actual root cause was never the multipart layout: the legacy DecompressReader reset its payload consumption state on every poll re-entry, so a Poll::Pending in the middle of a block payload (routine under the erasure duplex) desynchronized the block framing and surfaced as LZ4 frameType errors. This rewrites the decoder as a resumable state machine, restores the multipart session compression marker, reports logical part sizes in ListParts, and makes the rebalance migration read raw stored bytes so compressed and encrypted objects survive migration verbatim.
Fixes#5957. Internal tracking: backlog#1848, backlog#1850.
* feat(storage): stage multipart compression behind RUSTFS_COMPRESSION_MULTIPART_ENABLED
Review follow-up: a rolling-upgrade window must not create new compressed multipart objects while pre-fix nodes (whose decompressor is not resumable) may still serve reads. The session marker is now additionally gated on RUSTFS_COMPRESSION_MULTIPART_ENABLED, default off, so the restored capability stays dark until the operator confirms fleet convergence. The default flips per the multipart-compression-default-off-window entry in docs/architecture/compat-cleanup-register.md once the minimum supported direct-upgrade release ships the resumable decoder.
* chore(compat): satisfy the cleanup-register guard for the multipart compression switch
The architecture guard requires every backticked identifier in a register entry to carry a RUSTFS_COMPAT_TODO source marker: keep only the entry slug in backticks, and add the marker (with its literal Remove-after condition) at the switch definition.
* chore(rio): drop a dead store in the poison guard and note the end-block branch
Review follow-up: the poison gate re-assigned an already-true flag, and the COMPRESS_TYPE_END branch reads as dead without stating that the writer never emits an end block — that absence is exactly what lets concatenated per-part streams decode as one.
* fix(s3): report empty compressed multipart part size
* fix(s3): report empty encrypted multipart part size
* fix(tier): decrypt transitioned objects instead of serving their ciphertext
A GET on a managed-SSE object that lifecycle had transitioned to a remote tier returned the ciphertext with the plaintext's Content-Length and no error: silent corruption on read-through, and worse than a failed request because nothing signals it. Restore of the same object failed server-side with IncompleteBody while POST ?restore still answered 200, so the object simply never came back and HEAD never showed an x-amz-restore marker.
Both symptoms are one cause. The transitioned read path built its fetch through new_getobjectreader, which decides nothing about encryption: it derived the range from the parts table — whose sizes are PLAINTEXT sizes — then used that range to fetch the object's STORED bytes from the tier, and handed the stream to the caller without any decrypt transform. The GET therefore served the first plaintext-length bytes of ciphertext; the restore copy-back, which validates against the stored size, came up short by exactly the encryption overhead.
The path now builds the same ReadPlan the local read path uses, so a single place decides how stored bytes map to requested bytes. ReadPlan gains a two-phase API — build_for_request to learn the storage coordinates before issuing the tier fetch, into_object_reader to wrap the returned stream — because the tier fetch has to be positioned before a stream exists. The encryption resolver reaches the path from InstanceContext, the same source the local read uses.
A restore read additionally stops synthesizing a range from the part number. A restore serves the stored representation (restore_request_active already forces the Plain branch), so a plaintext-coordinate range would be reinterpreted as a storage range and truncate the payload by its encoding overhead. An explicit caller range is already in storage coordinates on that path and is still honored, which two existing tests pin.
crates/e2e_test/src/kms/kms_ilm_sse_kms_test.rs drops its #[ignore]: the transition test now runs and asserts the plaintext round-trips byte-identically through transition, read-through and restore. The same file had its enforcement switch stuck at false from a control experiment; it is back to true, so the test again exercises what its name and module docs claim.
Fixes#6025. Refs rustfs/backlog#1582, rustfs/backlog#1637.
* test(tier): pass resolver to transitioned reader tests
* test(e2e): fold seven identical POST-policy exact-mismatch tests into one table
The seven *_policy_mismatch tests in multipart_auth_test.rs were body-identical after literal normalization: the policy pins one field to an exact value, the form sends a different value, and the upload must be rejected with 400 InvalidPolicyDocument naming the field. Each test booted its own full server.
They fold into one table-driven test on the run_post_object_policy_case helper introduced by the PR1 fold. Every row keeps its original test's exact bucket, key, field name, policy value, mismatched form value, body bytes, and expected error strings — including the three rows that asserted the stronger <Code>InvalidPolicyDocument</Code> form. The pinned condition is built with an explicit serde_json::Map since the field name is now a table parameter.
cargo nextest list reports 97 tests for this module; the inventory row is updated in the same diff (103 -> 97).
Ref rustfs/backlog#1838 (PR2).
* test(e2e): fold the remaining POST-policy duplicate groups (15 tests) (#6018)
Completes the multipart_auth table-driven fold: the six remaining body-identical groups collapse onto the shared run_post_object_policy_case helper.
- Seven single-field exact-mismatch tests (cache-control, expires, tagging, storage-class, content-type, success_action_status, metadata-field-exact) join the existing exact-condition mismatch table as rows — same shape as the PR2 fold.
- The two object-lock mismatch tests become a two-row table (policy pins mode + retain-until-date, one form field mismatches).
- The three SSE-KMS parameter mismatch tests become a three-row table (policy pins the SSE mode plus one KMS parameter, form differs).
- The three SSE-KMS outside-policy tests become a three-row table pinning the distinct contract: an undeclared KMS parameter sails past policy validation and is rejected at runtime with 501 NotImplemented, not a policy error.
Every row keeps its original test's exact bucket, key, field names, values, body bytes, and expected status/code strings. cargo nextest list reports 85 tests for the module; the inventory row is updated in the same diff (97 -> 85).
Ref rustfs/backlog#1838 (PR3).
The peer-edit delivery fence from #5882 treated an equal applied generation as stale. One edit legitimately fans out one delivery per peer record under a single generation (the ILM-expiry edit sends every peer's record), so the receiver applied only the first body, raised its high-water mark, and silently acked-success while dropping the rest — enableILMExpiryReplication never converged on receiving sites and the three-node nightly e2e failed deterministically (issue #5767).
Only a strictly newer applied generation is stale now. Equal generation implies the same logical edit and re-applying a delivery is idempotent (update_peer overwrites the peer record; the mark is raised with max), while strictly older deliveries — the cross-node ordering case the fence exists for — stay rejected.
Adds a composed unit test driving three same-generation bodies through the receiver's fenced sequence, and widens the replication e2e's two site-replication wait helpers from a 10s polling ceiling to the 30s deadline the file's other waits use.
PutBucketVersioning with Status=Suspended on a bucket that carries a replication configuration now fails with InvalidBucketState, matching AWS S3 and MinIO. Suspension would start minting null versions that the versioned replication engine can never converge — the state is unreachable on AWS and MinIO, and the nightly acceptance-matrix e2e that tried to exercise it failed every night since it landed (issue #5767).
The acceptance-matrix test tail now pins the rejection contract (InvalidBucketState) and verifies a fresh matched PUT still replicates with a real version id after the rejected suspension.
The nine *_missing_from_policy_conditions tests in multipart_auth_test.rs were literal-for-literal identical after normalization: policy pins bucket + key + content-length-range, the form smuggles one extra field the policy never declared, and the upload must be rejected with 403 AccessDenied naming the field. Each one started its own full server.
This adds the run_post_object_policy_case helper (parameterized by bucket, key, policy conditions, extra form fields, file body, and expected status/code/mention, with a per-case assertion prefix) and folds the nine tests into one table-driven test with nine rows. Every row keeps its original test's exact bucket, key, field name/value, body bytes, and expected error strings — including the two rows that asserted the stronger <Code>AccessDenied</Code> form — so no poison value is lost. The helper's signature is general enough for the policy_mismatch and sse-kms groups planned as PR2/PR3.
cargo nextest list now reports 103 tests for this module; the inventory row said 109 while the file actually held 111 before this change (stale by two), so the inventory is set to the measured 103 in the same diff per the issue's hard constraint.
Ref rustfs/backlog#1838 (PR1).
* test(heal): strengthen replacement e2e evidence
* test(heal): fix replacement e2e barriers
Accept the real post-fault scanner failure-to-idle sequence as the live disk loss barrier, and preserve the first definitive completed status while only resampling the physical census for premature-completion confirmation.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* test(heal): add privileged replacement rebuild e2e
Add ignored Linux-only 3x4 automatic replacement coverage for EC8+4 and EC6+6. The tests use real tmpfs mounts in an isolated mount namespace, wait for scanner-driven replacement recovery status, and verify the replacement target with per-version xl.meta and part.N physical census without invoking Admin deep heal.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(test): avoid unsafe in privileged replacement e2e
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): harden privileged replacement e2e
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): prove absent replacement recovery witness
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): prove absent replacement observation
Stop the target node before detaching the test mount so RustFS releases its mount lease instead of continuing to serve the old tmpfs through an open fd. Restart the node with the endpoint absent and wait for the scanner's real readiness rejection in that node's log.
Assert the absent window has no replacement intent, completion proof, checkpoint, healing marker, or Admin v4 durable record for the target before mounting the blank replacement and waiting for automatic recovery plus physical shard census.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): streamline cluster log capture
Move cluster-node log capture out of ClusterNode and into per-node cluster launch configuration so the privileged replacement E2E uses an explicit harness API instead of mutating node identity data.
Reuse the same stdout/stderr capture helper for single-node and cluster processes, and pin the per-node capture behavior with a focused common test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): harden privileged replacement proof
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
P1-20 (rustfs/backlog#1675 B2, test-only). No prior test wrote objects
BEFORE the replication rule arrived, leaving the scanner's existing-object
resync pass — the only channel for such objects — without end-to-end
coverage, and the enqueue truth table partially unpinned at unit level.
e2e (both negative cells are contracts, asserted over multiple fast-scanner
cycles next to a replicated control key that proves the scanner and the
live path are running):
- test_scanner_compensates_existing_objects_across_write_paths: plain PUT,
CopyObject and Snowball auto-extract products written pre-rule all
converge via scanner compensation; a null-version object (PUT before the
bucket became versioned) is pinned as never compensated (the scanner heal
gate skips nil-version objects).
- test_scanner_never_compensates_when_existing_object_replication_disabled:
ExistingObjectReplication=Disabled is a contract, not a delay — existing
keys stay absent while post-rule writes replicate normally.
Unit truth-table pins (crates/replication):
- queue.rs: an empty replicate decision (Disabled existing-object, inbound
REPLICA) skips heal queueing for every status; Completed without a resync
decision skips.
- operation.rs: existing-object resync without a reset replicates exactly
the never-replicated (Empty) objects.
Helper: put_bucket_replication_with_statuses parameterizes the previously
hardcoded ExistingObjectReplication status; the nextest count comments are
refreshed to the post-rebase totals.
* test(replication): pin the version-fidelity probe contract (red)
P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.
Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.
test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.
Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).
* fix(replication): probe the version-identity contract in replication-check
P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.
- The probe PUT now carries the source version as `?versionId=` (the exact
shape live replication uses since P0-5, and the only shape MinIO
consumes; the internal source-version-id header alone would let the
probe pass against targets the real data path drifts on). Reuses
ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
equal the sent source id. On mismatch the phase fails with the
machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
(new optional Code field on phase statuses; Go decoders ignore unknown
keys), the overall target fails, the later version-addressed mutation
phases are skipped, and cleanup still removes the probe via the id the
target actually assigned (with the existing list-based sweep as backstop
when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
id (mirroring remove_object), and the replication PUT path audits it —
every drifting PUT increments
rustfs_replication_version_identity_drift_total and the first drift per
target ARN logs a structured warning pointing at ?replication-check.
The drift judgment is a pure function with an exemption-matrix test
(empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.
Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.
* fix(e2e): stop the fake target self-deadlocking on version-id minting
The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.
The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* test(replication): pin delayed delete-marker purge failure handling (red)
P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing
failure handling of the delayed delete-marker purge:
- test_delayed_delete_marker_purge_retries_after_transient_target_failure:
four scripted 503s outlast every existing channel (version-purge
replication + its in-process MRF fast retries + the watcher's single
attempt = 3 target DELETEs, all faulted in the recorded run); the
replicated marker is stranded on the target forever.
- test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart:
exhausted purge intents never reach the durable MRF journal, so a restart
replays nothing (recorded run: 3 faulted attempts, zero post-restart).
Red-light evidence (current main):
- Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean
attempt, target marker still present after 15s.
- Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after
restart, marker still present.
Test infra: FakeS3Target::stored_versions() exposes per-key version state so
purge tests assert target state instead of inferring it from the journal;
nextest count comments 36->38 nightly / 56->58 total.
* fix(replication): retry, persist and replay failed delete-marker purges
P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was
fire-and-forget: the target DELETE discarded its result (`let _ =`), a
missing target client was silently skipped, and nothing recorded the intent
— one transient target error stranded the replicated marker on the target
forever. Separately, `replicate_delete_with_outcome` held its outcome
hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry
to Missed so the durable backlog retained them permanently.
Changes:
- `replicate_delete_marker_purge_to_targets` now reports per-target
results (warn + metrics on failure, including `target_client_missing`),
supports retrying only the failed targets, and treats a target-side
NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not
retain the intent forever).
- The delayed watcher (`watch_and_purge_source_delete_marker`) retries
failed targets across its 5x1s watch window; on exhaustion it persists
the purge intent to the durable MRF journal via the new
`ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live
re-dispatch would loop unboundedly against a down target). Intent entries
are shaped as marker-creation deletes so replay funnels into the stale-
marker branch.
- The stale-marker branch (source marker already gone) now purges the
targets instead of silently returning success — closing a latent leak —
and reports the purge result as the replay outcome. Heal callers retry
for the full window (the startup MRF processor runs before target
clients initialize); live callers attempt once and fall back to a fresh
durable intent, so a down target cannot pin a replication worker.
- The outcome formula (extracted as `replicate_delete_outcome` and pinned
by a unit test) no longer includes the delayed purge, so successfully
replayed delete-marker entries are acknowledged instead of retained
forever.
Verification: red -> green e2e pair (transient-failure retry; exhaustion ->
durable MRF -> restart replay -> second-restart zero-replay ack) plus unit
tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test)
all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS
kubernetes endpoint failures reproduce without this change).
Adversarial validation (7 roles): no blocking findings after adding the
outcome-formula guard test. Known residuals recorded in the PR: watcher
shutdown window (intent not yet persisted), rolling-downgrade replay acks
without purging (equals pre-fix behavior), and replay falling back to the
source version id on targets that mint their own version ids (P1-19).
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* fix(replication): purge the marker version the target actually assigned
Review follow-up (#5864), two real defects:
- The delayed purge watcher was spawned with the pre-merge `dobj`, so the
per-target marker version ids this round recorded were invisible to it.
Against a target that mints its own ids the purge fell back to a
source-derived id, the target answered the versioned DELETE with an
idempotent 204, and that "success" cleared the retry set while the real
marker stayed behind. The watcher now receives the merged replication
state (`drs`), which folds this round's target-assigned ids in.
- A target whose recorded version metadata is inconsistent was skipped
without entering `failed_arns`, so an empty result made both the watcher
and the MRF replay treat a purge that issued no DELETE as successful and
drop the intent. The refusal is now a per-target failure (own metric
label): the leak stays visible and the intent is retained instead of
being acknowledged. The version decision also moved ahead of the client
lookup, so the refusal is decided from metadata alone.
Tests: a new e2e drives a fake target with `assign_own_version_ids`, which
ignores the forwarded source-version header for both objects and delete
markers, and asserts the replicated marker is really gone; a unit test
pins the corrupt-metadata refusal as a failed outcome without any target
client registered. The detached-watcher shutdown window is documented at
the watcher as a known non-durable window with the write-ahead follow-up
spelled out.
Add a replacement recovery peer RPC so Admin v4 can distinguish definitive cluster proofs from unsupported, unavailable, or conflicting peer state without extending the existing background heal v3/v1 status protocol.
Co-authored-by: heihutu <heihutu@gmail.com>
Complete the encrypted-object replication series (backlog#1783, PR-C of
3, after #5872 and #5885): SSE-C objects replicate as ciphertext
passthrough — the source holds no customer key, so the stored bytes and
their encryption metadata travel verbatim and the replica decrypts only
with the original customer key, single-part and multipart.
- Sender: SSE-C objects read raw (raw_data_movement_read), transfer at
ciphertext size, and range multipart parts over stored part sizes.
- Receiver: authorized replication PUTs restore the stored SSE-C keys
from the transport headers (exact lowercase forms - the read-path
check is case-sensitive), set ObjectOptions.preserve_ciphertext, and
skip compression, bucket-default SSE, and sse_encryption behind one
restore-derived gate. Multipart uses an internal session marker to
store parts verbatim and strips it on complete.
- Convergence: the replication HEAD sends
x-rustfs-source-replication-check; the target authorizes it as
ReplicateObjectAction and skips SSE-C read validation for that
request only, so keyless convergence HEADs see etag/size/mtime
instead of 400 and SSE-C replicas stop re-driving forever.
- e2e: SSE-C contract flips to a key-gated readable replica (no-key and
wrong-key GETs fail - the direct silent-plaintext detector); new
multipart passthrough contract with ETag/marker/stability assertions.
Open the managed-SSE replication gate (backlog#1783, PR-B of 3, after
#5872): the replication reader already decrypts through the injected
object-encryption resolver, so the source sends plaintext plus an
encryption intent header (AES256 / aws:kms, never the source key id) and
the target re-encrypts on its normal PUT path with its own KMS. No DEK
crosses sites.
- replication_put_object_options: fail closed only on Unsupported;
insert the SSE intent after the strip loop.
- TargetClient::create_multipart_upload sends the full opts.header()
set, fixing multipart replicas losing content-type/user metadata
(plaintext included).
- Preserve source ETag and mtime on replicas (authorized replication
only): receiver wires x-rustfs-source-etag into preserve_etag for PUT
and CompleteMultipartUpload, resolve_complete_etag consumes it, and
complete options carry source_etag/source_mtime (absent mtime
degrades to epoch, not now_utc). Without this every replication HEAD
comparison re-drives re-encrypted objects forever.
- e2e: managed SSE contracts flip to success on an independent-KMS
dual-process pair (byte-identical plain GET proves target-owned
envelopes; ETag/mtime preserved; version stable across scanner
cycles; resync converges; multipart keeps structure and metadata);
new target-without-KMS fail-closed contract; SSE-C stays FAILED.
Co-authored-by: houseme <housemecn@gmail.com>
Zhengchao An