mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-10 15:16:56 +00:00
ab35681928fbab38ecf93e2d4a3d66575717082b
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 唐小鸭
|
3792fed827 |
fix(replication): madmin reset/diff wire compat and config validation (#5799)
* fix(admin): align replication-reset responses with madmin ResyncTargetsInfo shape
The replication-reset and replication-reset-status responses serialized
their shell as "Targets" and per-target fields in PascalCase, while
madmin-go ResyncTargetsInfo/ResyncTarget expect the "target" shell key
and lowercase field tags (arn/resetid/resyncStatus/replicationCount/
completedReplicationSize/failedReplicationCount/failedReplicationSize).
Go json decoding is case-insensitive per field, but Targets vs target,
Status vs resyncStatus and the size/count key names cannot match, so
mc replicate resync decoded empty results.
Rename the serde tags to the exact madmin wire shape, keep the
ResetBeforeDate/Error RustFS extension keys (unknown keys are ignored
by Go decoders), pin the shape with a snapshot unit test, and update
the e2e client DTO to decode the madmin shape.
* fix(admin): stream bare madmin DiffInfo documents from replication diff
POST /v3/replication/diff returned a single enveloped object
({Entries, IsTruncated, ScannedVersions}) while madmin-go
BucketReplicationDiff decodes the body with a json.Decoder loop over
bare DiffInfo documents. The envelope decoded as exactly one DiffInfo
with an empty object, so mc replicate diff printed a phantom empty row
instead of the real backlog.
Emit one DiffInfo JSON document per line by default, using the exact
madmin json tags (object/versionId/rStatus/deletemarker/lastModified;
Size stays as a RustFS extension key that Go decoders ignore). The
enveloped shape moves to the opt-in ?aggregate=true RustFS extension,
which remains the only carrier of scan-coverage metadata; a truncated
default-mode scan is surfaced via a warn tracing event instead of
in-stream. Pin both shapes with unit tests and tighten the e2e helper
to reject any envelope in the stream.
* feat(replication): validate replication config structure before persisting
PutBucketReplication accepted structurally invalid configurations that
MinIO's replication.Config.Validate rejects: empty or oversized rule
lists, duplicate or negative rule priorities, over-long rule IDs,
filters carrying more than one of Prefix/Tag/And, and delete marker
replication enabled on tag-filtered rules. Such configs persisted
silently and later produced undefined routing (e.g. ambiguous priority
ties) instead of failing the PUT.
Add validate_replication_config_structure as a pure function in
rustfs-replication (limits documented as constants), surface it through
the ecstore api facade, and run it first in the PUT capability gate so
defects are named before any metadata write. Missing Priority counts as
zero for the uniqueness check, matching Go's zero-value semantics. The
self-target rejection deliberately stays at set-remote-target, where the
endpoint is known; a config can never reference a self-pointing ARN.
Document the rule-level Destination.StorageClass contract (use the
remote target's storage_class instead) and renumber the acceptance
matrix e2e to unique priorities, which MinIO would also require.
* test(replication): pin duplicated wire types with boundary reconciliation tests
rustfs-filemeta (xl.meta disk format) and rustfs-replication (MRF/resync
persistence format) deliberately each own ReplicationStatusType,
VersionPurgeStatusType and ReplicationState; the boundary converts
between them via as_str(), whose From<&str> impls fall back to Empty on
unknown tokens — a variant added on one side silently degrades to Empty
on the other.
Add reconciliation tests in replication_filemeta_boundary: exhaustive
matches with no wildcard arm on both sides of both enums (a new variant
fails compilation until the mapping is reconsidered), string-token
round-trip asserts (a token the other side does not recognize fails
instead of quietly becoming Empty), and a full-field ReplicationState
round-trip. Cross-reference the tests from both type definitions.
Struct drift was already compile-guarded by the exhaustive struct
literals in the conversion functions.
* docs(replication): define split completion criteria and milestone sequence
The ecstore replication split plan had no completion measure — the
boundary scaffolding risked ossifying because nothing said when the
migration counts as done. Record the criteria in the module inventory:
done means the Required Contracts table's 'Current dependency to
remove' column is empty; the end state moves pool/resyncer/state into
crates/replication, with the boundary micro-files dissolving as code
crosses the crate line (batch-merging them beforehand is explicitly
rejected — the guard scripts anchor on their file names, so merging is
churn with zero functional gain; only datatypes.rs can retire early).
Sequence the remaining work as M2 (resyncer pure decision logic, after
the oversized function splits) → M3 (worker runtime, highest risk,
last) → M4 (retire boundaries and guard entries). Refresh the stale
first-step text — the event sink / runtime contracts already landed —
and update the split-plan status table accordingly.
* fix(replication): align structural validator with MinIO semantics after adversarial review
Three interop corrections found by adversarial review of the new
structural validator, plus review fallout fixes:
- Delete-marker replication is now rejected only for a direct Filter.Tag,
not for tags inside Filter.And — MinIO's validator only inspects the
direct tag, and mc replicate add --tags "k1=v1&k2=v2" (delete-marker
replication on by default) puts multiple tags into And.Tags, so the
stricter check rejected mc-generated configs MinIO accepts.
- Rule ID length is measured in bytes (Go len semantics), not chars —
a 255-char multibyte ID must not round-trip into a config MinIO
rejects.
- An empty <Tag/> element (no key) counts as absent, matching MinIO's
Tag.IsEmpty(); console form serializers emit empty tags, which would
otherwise trip the exactly-one-of and delete-marker checks.
Also: repair the store-uninitialized PUT test whose empty-rules fixture
now (correctly) fails structural validation before reaching the store
lookup; pin the previously untested startTime madmin key in the
reset-status shape test; and signal a truncated default-mode diff scan
via the x-rustfs-replication-diff-truncated response header — the bare
madmin stream has no envelope, so a truncated scan was otherwise
indistinguishable from a complete healthy one (madmin/mc ignore unknown
headers).
* test(e2e): activate SSE-S3 replication contract and pin resync fail-closed path
The SSE-S3 replication contract e2e was ignored under backlog#1291
(silent plaintext replication); the fail-closed gate in
replication_target_boundary.rs closed that hole, so the ignore reason
expired. Un-ignore the test — it now pins the current fail-closed
contract (FAILED status, failure event, readable encrypted source,
stable absence of all target versions), verified green.
Add test_bucket_replication_sse_s3_resync_stays_fail_closed: drives the
existing-object resync path (PUT ?replication-reset) over a FAILED
SSE-S3 object and asserts the resync generation reaches a terminal
state without ever materializing a target version, with the
stays-absent window also spanning fast-scanner heal cycles. The new
start_bucket_replication_reset helper doubles as the madmin
ResyncTargetsInfo shape assertion (target[0].arn/resetid) for the
reset-start response.
Refresh the stale nextest count commentary (the module is at 20 fast +
36 nightly = 56 tests by cargo nextest list; the SSE-S3-ignored note no
longer holds).
|
||
|
Zhengchao An
|
5237a4465d |
feat(replication): purge delete markers by the target's own version id (#5676)
* feat(replication): purge delete markers by the target's own version id When a delete marker is replicated, the target assigns it a version id. The purge that follows derived one from the *source* uuid instead, which is only correct when the target mirrors source version ids. A generic S3 target does not: the derived id addresses a version that does not exist there, so the purge is a no-op and the replica keeps a marker the source has already removed. Same failure class as #4401. Record the id the target reports and address it directly on purge. Data path, all of it driven by the object's internal metadata rather than the `ReplicationState` wire form, which encodes positionally and cannot carry a map: - `rustfs-utils`: the `replication-delete-marker-version-<arn>` key family, plus `strip_internal_prefix_preserving_case` — ARNs are case-sensitive and the existing `strip_internal_prefix` lowercases. - `ReplicationState` gains the map and a `..._corrupt` flag, both `#[serde(skip)]`; `ReplicatedTargetInfo` carries the per-target id. - `persist_target_delete_marker_versions` is merge-only. A delete arriving over internode RPC has an empty map, so treating it as authoritative would let a remote disk erase an id the local disk still holds. - `delete_object_version` copies the map into `fi.metadata` before dispatch, so the durable carrier crosses the wire even though the field does not. - The keys are folded into the quorum hash through their normalized form: the dual internal prefixes carrying one mapping share an identity, while a genuine disagreement between disks still shows up as a quorum difference. - `corrupt` (the prefixes disagreed) fails closed: skip the purge and warn rather than guess an id and risk destroying a live version on the target. Ported from the rc.1 branch, which cannot merge as a whole: its MRF replay rewrite collides with #5659/#5671/#5672/#5673 and regressed `MRF_PENDING_CAP`. main's MRF machinery is kept; only this capability moves across. It touches no MRF code. Two things did not survive the port, deliberately. The branch's `missing_is_complete` purge regression does not exist here — it came from its own HEAD-precheck rewrite, and main's simpler path never had it. And the branch's `MrfReplicateEntry` ordering fields are MRF-redesign scope, left behind. Verification: cargo fmt --all --check, git diff --check, cargo check --workspace --all-targets, and the suites for the four touched crates — 4070 tests, 2 pre-existing failures unrelated to this change (`system_resolver_negative_result_reaches_the_dns_allowlist`, `test_resolve_domain_preserves_system_resolver_error_provenance`; both are the sandbox DNS interception, they fail on a clean checkout too). * fix(replication): keep the layer guard happy scripts/check_architecture_migration_rules.sh matches on text, so the doc comments naming `rustfs_filemeta::` read as a cross-layer dependency even though nothing imports it. Reword them; the guard passes. * fix(replication): make the target-version cap deterministic Two defects in this PR, both found in review. The cap was applied while iterating a `HashMap`, so *which* 1000 entries survived depended on iteration order. Two disks decoding the same oversized metadata could keep different subsets, hash differently, and lose quorum — instead of both reporting the same corruption. Collect first, then truncate in `BTreeMap` order, which is total and identical everywhere. And `persist_target_delete_marker_versions` discarded the `corrupt` flag from the RPC carrier, committing a delete-marker update that looked clean while the exact remote marker identity was unknown. It now declines to merge a corrupt carrier. Because the helper only ever inserts, declining leaves the durable keys already on the object untouched, which is strictly safer than writing a mapping we cannot trust. Residual, stated rather than papered over: corruption confined to the RPC carrier is not persisted as a sentinel, so a later reader of an object that carried no durable keys still sees "legacy, no mapping" rather than "corrupt". Persisting that would need a wire-format addition; the consumer already fails closed on any corruption it can observe. New test: `target_delete_marker_versions_cap_is_deterministic_across_decodes` decodes the same 1050-entry map twice and asserts both the corrupt flag and the retained subset agree. * fix(replication): preserve multipart source mtime (#5669) * fix(kms): repair unopenable ciphertext and cover the Vault backends (#5668) * Add black-box behavior tests for KMS resilience and serialization * fix(kms): repair unopenable ciphertext across backends Black-box testing of the KMS crate surfaced several defects that make encrypted data permanently unreadable. Symmetric envelopes. The Local and Vault Transit backends returned raw cipher output from `encrypt` while `decrypt` parsed a JSON envelope, so anything sealed through the master-key path could never be opened again. Local also discarded the AES-GCM nonce. Both now emit the same envelope `decrypt` consumes, matching the Static backend. Deterministic AAD. The object layer derived AEAD additional data by serializing a `HashMap` directly. Iteration order differs per instance, so a context rebuilt from storage produced different AAD bytes than the one used to seal and the object stopped opening. Ordering by key removes that dependency, matching the Static backend's existing `context_aad`. Objects written with the default single-key context are unaffected, since a one-entry map has only one serialization. Cipher in the header projection. `metadata_to_headers` recorded the SSE mode (`AES256` / `aws:kms`), which cannot represent ChaCha20-Poly1305, so a ChaCha-sealed object came back claiming `aws:kms` and was opened with the wrong cipher. The cipher now travels in `x-rustfs-encryption-algorithm` — the header the storage layer already reads but nothing ever wrote. Objects without it fall back as before. Also: the Static backend ignored `key_spec` and always issued 256-bit data keys; Local `list_keys` hardcoded `truncated: false`, ignored `marker`, and paginated over unordered `read_dir`, so a paginating client silently saw a partial key list; and Local and Vault KV2 reported `key_id: "unknown"` from `decrypt` despite the envelope naming the master key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(kms): cover both Vault backends and key rotation The behavior suite ran only against Local and Static, and its own harness documented the gap: the Vault backends had no business-capability coverage at all. Setting `RUSTFS_KMS_VAULT_TOKEN` now adds Vault KV2 and Vault Transit to every `for_each_backend` spec against a live server. That lane is what surfaced the Transit envelope defect fixed in the previous commit. `rotate` and `versioning` are advertised only by the Vault backends, so until now every capability-gated branch for them took the `UnsupportedCapability` side and the working half was never asserted — a rotation that dropped prior key versions would have gone green. The new `behavior_rotation.rs` pins that half: material sealed before a rotation still opens after it, repeated rotations accumulate versions rather than overwriting a single spare, and the history survives a restart. Two test defects fixed. `objects_round_trip_across_sizes_and_algorithms` asserted a 1-byte object differs from its own ciphertext, which collides once every 256 runs; the assertion now applies only where a collision is not realistic, and small objects stay covered by the tag check and the decrypt round-trip. `test_from_env_selects_token_file` depended on `RUSTFS_KMS_VAULT_TOKEN` being absent from the caller's environment and now clears it explicitly. The snapshots directory was also removed from `.gitignore`: insta snapshots are the assertions themselves, so leaving them untracked gives CI nothing to compare against. Only `.snap.new` scratch files are ignored now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(kms): adapt behavior suite to current key APIs Rebasing onto main brought four API changes the suite predates. `DeleteKeyRequest` gained `confirm_key_id`, and immediate deletion is now gated on the server's `allow_immediate_deletion`. Scheduled deletions pass `None`; the four specs that destroy a key outright echo the key id back and opt the harness config in, which is what the gate asks of a real caller. `LocalBackupExportRequest` gained `sanitized_config`. These specs cover the key-material path, so they seal no configuration and pass `None`. `KmsCacheStats` became a named struct with real hit, miss, and eviction counters. `cache_stats_returns_an_entry_count_and_no_hit_or_miss_data` existed to pin the old placeholder behavior — that the second tuple element was always zero — which main has since fixed, so it is now `cache_stats_reports_hits_and_misses_separately` and asserts the counters actually move. Starting the service provisions the reserved probe key, so it shows up in listings and backup bundles. Exact-set assertions filter it through a new `without_probe_key` helper rather than naming it, keeping those specs about the keys they seeded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(kms): bind the AAD to the stored context bytes Review caught that canonicalizing the AAD on decrypt breaks objects sealed before canonicalization existed, and it was right. The AAD is the *serialization* of the encryption context, and `x-rustfs-encryption-context` stores that exact byte sequence: `encrypt_object` fed one `HashMap` to the AEAD and then moved the same map into the metadata the header is written from, so the stored string is byte-identical to the AAD the object was sealed under. Those objects are therefore recoverable — but only while nothing round-trips the value through a `HashMap` and re-serializes it. Recomputing sorted AAD on decrypt would have turned a readable object into a permanently unreadable one. The previous behavior was worse than the first analysis credited: it did not merely fail intermittently, it made the failure deterministic. `EncryptionMetadata` now carries `context_aad`, the bytes the object was actually sealed with. Encryption records what it fed the AEAD, the header projection stores those bytes verbatim (and preserves a legacy ordering across a re-projection rather than rewriting it into sorted form), and `headers_to_metadata` carries the stored string through untouched. Both decrypt paths, SSE-KMS and SSE-C, prefer it and fall back to canonical serialization only when no stored serialization exists. Canonicalization still applies to everything newly sealed, so the original ordering bug cannot recur. Two tests pin this: a legacy record whose sealed bytes are non-canonical must survive a full header round trip unchanged, and a context header rewritten to an equivalent-but-reordered serialization must fail authentication rather than silently re-deriving a working AAD. Both were mutation-checked against the reinstated bug on each side. Also from review: the lifecycle churn test asserted only that every request was accounted for, which holds whether the state gate exists or not, so both branches are now pinned deterministically after the churn (asserting `refused > 0` on the concurrent phase would only trade the hole for a scheduling flake). And the Local and Vault KV2 envelopes compare `encryption_context` without authenticating it — `DekCrypto` seals only the plaintext — which is now documented at both sites; closing it needs a versioned envelope, since existing ciphertext was sealed without AAD. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: ccccpj <ccccpj@outlook.com> Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Zhengchao An
|
68f048b8fe |
fix(replication): persist delete marker mtime in MRF entries (#4331)
fix(replication): persist original mtime in MRF entries (backlog#867) MRF delete entries did not persist the original delete-marker mtime, so after a restart the recovery replay path reconstructed the delete without a source timestamp. Downstream the replica delete-marker was stamped with the replay time (now()) instead of the source mtime, causing delete-marker timestamp divergence across clusters. Extend the MrfReplicateEntry disk format with an optional deleteMarkerMtime field (persisted as Unix nanoseconds) in both duplicate struct definitions (rustfs-replication and rustfs-filemeta). DeletedObjectReplicationInfo now persists delete_marker_mtime, and start_mrf_processor restores it onto the reconstructed delete so the replica keeps the source timestamp. Backward compatibility: the new key uses skip_serializing_if + serde default, so historical MRF files without it decode to None and replay falls back to the current time (pre-#867 behaviour). No panic or entry loss on old files. Closes rustfs/backlog#867 |
||
|
cxymds
|
6e5c58ca3f | fix(replication): harden bucket replication correctness (#4116) | ||
|
abdullahnah92
|
a19560da41 |
fix(bucket-repl): persist MRF retry queue to disk and reload on startup (#3456)
* fix(bucket-repl): persist MRF retry queue to disk and reload on startup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(bucket-repl): address three blocking MRF issues from review 1. MRF replay loses delete operations — add `MrfOpKind` discriminator to `MrfReplicateEntry` (Object | Delete, default=Object for backward compat). `DeletedObjectReplicationInfo::to_mrf_entry` now persists `op=Delete`, `version_id`, `delete_marker_version_id`, and `delete_marker`. `start_mrf_processor` branches on `op`: delete entries skip `get_object_info` and replay via `schedule_replication_delete` with `ReplicationType::Heal`; object entries follow the existing heal path. 2. `flush_mrf_to_disk` cleared the in-memory batch even on encode/write failure — changed return type to `bool` and callers now only `pending.clear()` on `true`, so a transient storage error retries on the next tick instead of silently dropping the batch. 3. Add focused tests: encode/decode roundtrips for object, delete-marker, versioned-delete, and mixed-batch entries; a routing test confirming op-kind propagates correctly and that the default is Object for legacy files; a legacy-compat test verifying old entries round-trip cleanly through the new format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix clippy redundant-clone in MRF tests Replace &[entry.clone()] with std::slice::from_ref(&entry) in two encode_mrf_file call sites flagged by clippy's redundant_clone lint under --all-targets --features rio-v2. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(bucket-repl): strengthen legacy MRF compat test with hand-built msgpack The previous mrf_legacy_file_without_op_field_decoded_as_object test round-tripped through encode_mrf_file, so it exercised the new format and never touched a truly-legacy payload. Replace it with a hand-built msgpack payload that genuinely omits the "op", "deleteMarker", and "deleteMarkerVersionID" keys — exactly what the old binary would have written before MrfOpKind existed. The test now fails if #[serde(default)] is removed from the op field, which proves real backward compatibility rather than round-trip stability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
weisd
|
b9b7d86ae4 | feat: improve legacy metadata and admin compatibility (#2202) | ||
|
weisd
|
4d19b069c3 | fix: replication delete (#1714) | ||
|
weisd
|
6631407416 | feat: Add RustFS Scanner Module and Multiple Bug Fixes (#1579) | ||
|
houseme
|
8d7cd4cb1b | chore: upgrade dependencies and migrate to aws-lc-rs (#1333) | ||
|
houseme
|
eb33e82b56 |
fix: Prevent panic in GetMetrics gRPC handler on invalid input (#1291)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> |
||
|
houseme
|
d934e3905b |
Refactor telemetry initialization for non-production environments (#789)
* add dep `scopeguard` * improve for tracing * fix * fix * improve code for import * add logger trace id * fix * fix * fix * fix * fix |
||
|
weisd
|
cd1e244c68 |
Refactor: Introduce content checksums and improve multipart/object metadata handling (#671)
* feat: adapt to s3s typed etag support * refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug * add head_object checksum, filter object metadata output * fix multipart checksum * fix multipart checksum * add content md5,sha256 check * fix test * fix cargo --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
weisd
|
90f21a9102 |
refactor: Reimplement bucket replication system with enhanced architecture (#590)
* feat:refactor replication * use aws sdk for replication client * refactor/replication * merge main * fix lifecycle test |