* feat(replication): purge delete markers by the target's own version id
When a delete marker is replicated, the target assigns it a version id. The
purge that follows derived one from the *source* uuid instead, which is only
correct when the target mirrors source version ids. A generic S3 target does
not: the derived id addresses a version that does not exist there, so the
purge is a no-op and the replica keeps a marker the source has already
removed. Same failure class as #4401.
Record the id the target reports and address it directly on purge.
Data path, all of it driven by the object's internal metadata rather than the
`ReplicationState` wire form, which encodes positionally and cannot carry a
map:
- `rustfs-utils`: the `replication-delete-marker-version-<arn>` key family,
plus `strip_internal_prefix_preserving_case` — ARNs are case-sensitive and
the existing `strip_internal_prefix` lowercases.
- `ReplicationState` gains the map and a `..._corrupt` flag, both
`#[serde(skip)]`; `ReplicatedTargetInfo` carries the per-target id.
- `persist_target_delete_marker_versions` is merge-only. A delete arriving
over internode RPC has an empty map, so treating it as authoritative would
let a remote disk erase an id the local disk still holds.
- `delete_object_version` copies the map into `fi.metadata` before dispatch,
so the durable carrier crosses the wire even though the field does not.
- The keys are folded into the quorum hash through their normalized form:
the dual internal prefixes carrying one mapping share an identity, while a
genuine disagreement between disks still shows up as a quorum difference.
- `corrupt` (the prefixes disagreed) fails closed: skip the purge and warn
rather than guess an id and risk destroying a live version on the target.
Ported from the rc.1 branch, which cannot merge as a whole: its MRF replay
rewrite collides with #5659/#5671/#5672/#5673 and regressed
`MRF_PENDING_CAP`. main's MRF machinery is kept; only this capability moves
across. It touches no MRF code.
Two things did not survive the port, deliberately. The branch's
`missing_is_complete` purge regression does not exist here — it came from its
own HEAD-precheck rewrite, and main's simpler path never had it. And the
branch's `MrfReplicateEntry` ordering fields are MRF-redesign scope, left
behind.
Verification: cargo fmt --all --check, git diff --check,
cargo check --workspace --all-targets, and the suites for the four touched
crates — 4070 tests, 2 pre-existing failures unrelated to this change
(`system_resolver_negative_result_reaches_the_dns_allowlist`,
`test_resolve_domain_preserves_system_resolver_error_provenance`; both are
the sandbox DNS interception, they fail on a clean checkout too).
* fix(replication): keep the layer guard happy
scripts/check_architecture_migration_rules.sh matches on text, so the doc
comments naming `rustfs_filemeta::` read as a cross-layer dependency even
though nothing imports it. Reword them; the guard passes.
* fix(replication): make the target-version cap deterministic
Two defects in this PR, both found in review.
The cap was applied while iterating a `HashMap`, so *which* 1000 entries
survived depended on iteration order. Two disks decoding the same oversized
metadata could keep different subsets, hash differently, and lose quorum —
instead of both reporting the same corruption. Collect first, then truncate
in `BTreeMap` order, which is total and identical everywhere.
And `persist_target_delete_marker_versions` discarded the `corrupt` flag from
the RPC carrier, committing a delete-marker update that looked clean while the
exact remote marker identity was unknown. It now declines to merge a corrupt
carrier. Because the helper only ever inserts, declining leaves the durable
keys already on the object untouched, which is strictly safer than writing a
mapping we cannot trust.
Residual, stated rather than papered over: corruption confined to the RPC
carrier is not persisted as a sentinel, so a later reader of an object that
carried no durable keys still sees "legacy, no mapping" rather than "corrupt".
Persisting that would need a wire-format addition; the consumer already fails
closed on any corruption it can observe.
New test: `target_delete_marker_versions_cap_is_deterministic_across_decodes`
decodes the same 1050-entry map twice and asserts both the corrupt flag and
the retained subset agree.
* fix(replication): preserve multipart source mtime (#5669)
* fix(kms): repair unopenable ciphertext and cover the Vault backends (#5668)
* Add black-box behavior tests for KMS resilience and serialization
* fix(kms): repair unopenable ciphertext across backends
Black-box testing of the KMS crate surfaced several defects that make
encrypted data permanently unreadable.
Symmetric envelopes. The Local and Vault Transit backends returned raw
cipher output from `encrypt` while `decrypt` parsed a JSON envelope, so
anything sealed through the master-key path could never be opened again.
Local also discarded the AES-GCM nonce. Both now emit the same envelope
`decrypt` consumes, matching the Static backend.
Deterministic AAD. The object layer derived AEAD additional data by
serializing a `HashMap` directly. Iteration order differs per instance,
so a context rebuilt from storage produced different AAD bytes than the
one used to seal and the object stopped opening. Ordering by key removes
that dependency, matching the Static backend's existing `context_aad`.
Objects written with the default single-key context are unaffected,
since a one-entry map has only one serialization.
Cipher in the header projection. `metadata_to_headers` recorded the SSE
mode (`AES256` / `aws:kms`), which cannot represent ChaCha20-Poly1305,
so a ChaCha-sealed object came back claiming `aws:kms` and was opened
with the wrong cipher. The cipher now travels in
`x-rustfs-encryption-algorithm` — the header the storage layer already
reads but nothing ever wrote. Objects without it fall back as before.
Also: the Static backend ignored `key_spec` and always issued 256-bit
data keys; Local `list_keys` hardcoded `truncated: false`, ignored
`marker`, and paginated over unordered `read_dir`, so a paginating
client silently saw a partial key list; and Local and Vault KV2 reported
`key_id: "unknown"` from `decrypt` despite the envelope naming the
master key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(kms): cover both Vault backends and key rotation
The behavior suite ran only against Local and Static, and its own harness
documented the gap: the Vault backends had no business-capability
coverage at all. Setting `RUSTFS_KMS_VAULT_TOKEN` now adds Vault KV2 and
Vault Transit to every `for_each_backend` spec against a live server.
That lane is what surfaced the Transit envelope defect fixed in the
previous commit.
`rotate` and `versioning` are advertised only by the Vault backends, so
until now every capability-gated branch for them took the
`UnsupportedCapability` side and the working half was never asserted — a
rotation that dropped prior key versions would have gone green. The new
`behavior_rotation.rs` pins that half: material sealed before a rotation
still opens after it, repeated rotations accumulate versions rather than
overwriting a single spare, and the history survives a restart.
Two test defects fixed. `objects_round_trip_across_sizes_and_algorithms`
asserted a 1-byte object differs from its own ciphertext, which collides
once every 256 runs; the assertion now applies only where a collision is
not realistic, and small objects stay covered by the tag check and the
decrypt round-trip. `test_from_env_selects_token_file` depended on
`RUSTFS_KMS_VAULT_TOKEN` being absent from the caller's environment and
now clears it explicitly.
The snapshots directory was also removed from `.gitignore`: insta
snapshots are the assertions themselves, so leaving them untracked gives
CI nothing to compare against. Only `.snap.new` scratch files are
ignored now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(kms): adapt behavior suite to current key APIs
Rebasing onto main brought four API changes the suite predates.
`DeleteKeyRequest` gained `confirm_key_id`, and immediate deletion is now
gated on the server's `allow_immediate_deletion`. Scheduled deletions pass
`None`; the four specs that destroy a key outright echo the key id back
and opt the harness config in, which is what the gate asks of a real
caller.
`LocalBackupExportRequest` gained `sanitized_config`. These specs cover
the key-material path, so they seal no configuration and pass `None`.
`KmsCacheStats` became a named struct with real hit, miss, and eviction
counters. `cache_stats_returns_an_entry_count_and_no_hit_or_miss_data`
existed to pin the old placeholder behavior — that the second tuple
element was always zero — which main has since fixed, so it is now
`cache_stats_reports_hits_and_misses_separately` and asserts the counters
actually move.
Starting the service provisions the reserved probe key, so it shows up in
listings and backup bundles. Exact-set assertions filter it through a new
`without_probe_key` helper rather than naming it, keeping those specs
about the keys they seeded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(kms): bind the AAD to the stored context bytes
Review caught that canonicalizing the AAD on decrypt breaks objects sealed
before canonicalization existed, and it was right. The AAD is the
*serialization* of the encryption context, and `x-rustfs-encryption-context`
stores that exact byte sequence: `encrypt_object` fed one `HashMap` to the
AEAD and then moved the same map into the metadata the header is written
from, so the stored string is byte-identical to the AAD the object was
sealed under. Those objects are therefore recoverable — but only while
nothing round-trips the value through a `HashMap` and re-serializes it.
Recomputing sorted AAD on decrypt would have turned a readable object into
a permanently unreadable one. The previous behavior was worse than the
first analysis credited: it did not merely fail intermittently, it made
the failure deterministic.
`EncryptionMetadata` now carries `context_aad`, the bytes the object was
actually sealed with. Encryption records what it fed the AEAD, the header
projection stores those bytes verbatim (and preserves a legacy ordering
across a re-projection rather than rewriting it into sorted form), and
`headers_to_metadata` carries the stored string through untouched. Both
decrypt paths, SSE-KMS and SSE-C, prefer it and fall back to canonical
serialization only when no stored serialization exists. Canonicalization
still applies to everything newly sealed, so the original ordering bug
cannot recur.
Two tests pin this: a legacy record whose sealed bytes are non-canonical
must survive a full header round trip unchanged, and a context header
rewritten to an equivalent-but-reordered serialization must fail
authentication rather than silently re-deriving a working AAD. Both were
mutation-checked against the reinstated bug on each side.
Also from review: the lifecycle churn test asserted only that every
request was accounted for, which holds whether the state gate exists or
not, so both branches are now pinned deterministically after the churn
(asserting `refused > 0` on the concurrent phase would only trade the hole
for a scheduling flake). And the Local and Vault KV2 envelopes compare
`encryption_context` without authenticating it — `DekCrypto` seals only
the plaintext — which is now documented at both sites; closing it needs a
versioned envelope, since existing ciphertext was sealed without AAD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: ccccpj <ccccpj@outlook.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Preserve metadata replication operations in the durable MRF and route tagging, retention, and legal-hold updates through the existing full-object replication transport. Keep ACL propagation outside the contract because the current object model has no durable object ACL state.
Refs #1616
* refactor(sse): decouple encryption from ecstore
* feat(kms): enhance KMS service manager with runtime state and persistence support
* feat(kms): add local key export functionality for SSE-S3 migration tests
* fix(kms): keep local key export narrowly scoped
* fix(sse): validate copy source customer algorithm
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
BucketMetadataSys::get_config lazily fabricated a default BucketMetadata
(object-lock off) for any bucket whose .metadata.bin was ConfigNotFound and
cached it in the map that the map-only, fail-closed metadata_sys::get()
serves. The object-lock batch-delete gate (object_lock_delete_check_required,
backlog#929 / #4297) treats that map as authoritative, so a metadata miss
became a cached "no lock" answer: a versioning peek could poison the cache
and let delete_objects skip the per-object retention/legal-hold stat. The
same fabrication raced make_bucket (lost update overwriting freshly
persisted lock-enabled metadata) and let the 15-minute refresh loop replace
good cached metadata on a transient quorum dip.
Production changes:
- get_config caches only metadata actually read from disk; misses are
recorded in a bounded negative cache (30s TTL, 10k entries, invalidated by
set()) so repeated lookups for metadata-less names cost no extra
namespace-lock + erasure-set fanout (reachable pre-auth via CORS
preflight and per-key in DeleteObjects).
- concurrent_load never lets a fabricated default REPLACE an existing map
entry; startup insert-if-vacant behavior for legacy buckets is preserved.
- delete_objects and new_ns_lock resolve dist-erasure, versioning, and the
object-lock gate from the set's own instance context (backlog#1052)
instead of the ambient facade, so a second in-process instance (or, in
tests, another test's transient DistErasure window) cannot reroute
locking onto an empty dist locker list or answer with the wrong
instance's bucket state.
Test-isolation changes (the bug that surfaced all of the above: the
delete_objects lock-gating test failed deterministically when sharing a
process with the lifecycle env tests):
- The MinIO-migration test builds on an isolated InstanceContext instead of
registering soon-deleted disks in the shared bootstrap registry.
- The cached lifecycle env re-registers its disks on every use, surviving
other serial tests' reset_local_disk_test_state.
- Hermetic SetDisks helpers gain isolated-context variants pinned to plain
erasure; tier-free non-serial test modules use them, guard-based
SetupTypeGuard tests stay on the bootstrap context.
- Three deterministic pin tests (nextest-safe) cover the caching contract,
the delete gate resolution source, and the ns-lock resolution source.
Verification:
- cargo test -p rustfs-ecstore --lib -- --exact <4-test combo from the
report> (previously failing, now green)
- cargo test -p rustfs-ecstore --lib: 3169 passed / 0 failed across
repeated runs; cargo fmt --check and cargo clippy --lib --tests clean
- Adversarial validation (high-risk tier, all seven roles) run per
AGENTS.md; all findings fixed or rebutted with evidence
fix(ilm): fail closed on unsafe manual job recovery
Mark expired manual transition jobs Unknown when recovery cannot prove queued worker outcomes are durable. Enforce ETag-guarded admission release with exact delete preconditions so stale releasers cannot remove a replacement admission, while still allowing drained cancelled jobs to terminate.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep corrupted no-parity heal results on the integrity-failure path, preserve the object geometry in operator output, and document the recovery boundary for historical bad shards.
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(ecstore): seek encrypted multipart Range GETs to the covering part boundary on the Legacy rio v1 backend
Phase A of https://github.com/rustfs/backlog/issues/1316. Encrypted Range GETs on the default (non rio-v2) backend previously planned storage_offset=0, storage_length=oi.size and discarded the decrypted prefix, so a small Range on a large SSE object read, erasure-decoded, and decrypted the whole object. Eligible multipart objects now seek to the covering part boundary using only the per-part size/actual_size metadata facts; the multipart decrypt reader already handles streams starting at any part boundary, so rio and the on-disk format are untouched. Single-part objects, compressed payloads, defective parts tables, and zero-length ranges keep the previous full read, and RUSTFS_ENCRYPTED_RANGE_SEEK=false restores it globally. A new histogram rustfs_get_encrypted_range_read_amplification plus a full|part_seek path counter record the physical/plaintext amplification at the ReadPlan decision point.
* fix(ecstore): guard encrypted multipart range seeks
* fix(io-metrics): align encrypted range metric names with the rustfs_io_ prefix
Every other metric in rustfs-io-metrics uses the rustfs_io_ prefix; the
two encrypted-range-seek metrics were the only exception. Rename before
first release so dashboards never see the unprefixed names.
* fix(tier): gate exact remote version consumption
Reject non-empty remote tier versions before transitioned GET and remote delete backend I/O when the tier backend does not support exact version operations.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(tier): split mock remote version validation fault
Separate one-shot mock remote version validation failures from persistent unsupported-backend behavior so cleanup durability tests can still verify exact-version recovery while #1358 fail-closed gate tests keep asserting no backend I/O.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Advance transition transactions to UploadOutcomeUnknown before remote tier PUT so response-loss windows can be recovered through provider-authoritative probing.
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An