Brings in #8825, #8831, #8836 and #8839. Conflicts were import lists and test blocks (both sides kept), plus the new restore_copyback request flag next to main's data-movement field. Main replaced completed_multipart_object_part with prepare_complete_multipart_parts, so the stale import is dropped.
Main's xl.meta fast-path scanner (inspect_v2_object_slice) predates #8836 and classified a transitioned version with a pending v1 restore copyback as having no physical data dir, which made find_unshared_data_dir_for_version drop the copyback data dir. The scanner now reads the copyback marker with get_consistent_bytes semantics (both aliases, ASCII case-insensitive, empty or conflicting values count as absent), and uses_data_dir keeps main's allocation-free transition check plus #8836's copyback exclusion. find_unshared_data_dir_for_version_keeps_pending_copyback_data_dir failed on the plain merge and passes now.
FileMetaShallowVersion::into_fileinfo and to_fileinfo_with_part_checksums previously decoded an owned FileMetaVersion and then converted it via shared reference (&self), cloning meta_user, meta_sys, part_etags, and part_indices before dropping the temporary FileMetaVersion and scanning the metadata map up to 10 times for optional internal suffixes. Additionally, metadata_compat helpers (get_str, contains_key_str, remove_str, get_consistent_value) allocated two temporary heap Strings via both_keys(suffix) on every absent-key fallback.
- Replace both_keys(suffix) in case-insensitive fallback scans with allocation-free prefix+suffix slice matching.
- Add consuming into_fileinfo_with_part_checksums on FileMetaVersion and MetaObject to move decoded part vectors and metadata maps in place.
- Fold transition-version metadata formatting and optional suffix presence detection into the single meta_sys pass, and skip target_delete_marker_versions in get_internal_replication_state when no delete-marker version keys are present.
* fix(utils): NUL-terminate and extend paths passed to Win32 volume APIs
`get_volume_name` returned the volume root truncated at its NUL and the
result was then passed to `GetVolumeInformationW` as a PCWSTR, an
out-of-bounds read. The disk-space and volume APIs also received the
caller's raw path, so they failed beyond MAX_PATH, and a UNC share root
lacked the trailing backslash these APIs require.
- Wrap every Win32 path in a NUL-terminated `WidePath` that rejects
interior NULs; volume roots returned by `GetVolumePathNameW` keep
their terminator.
- Convert inputs to the extended-length `\\?\` / `\\?\UNC\` form after
`GetFullPathNameW` normalization, and give directory queries a
trailing backslash.
- Query `GetDiskFreeSpaceW` and `GetVolumeInformationW` on the volume
root as documented; size buffers to the documented MAX_PATH + 1 or
the input length.
- Report Win32 error codes instead of raw HRESULTs so `ErrorKind`
matches `std::fs`.
- Keep volume ids in their conventional `C:\` / `\\server\share\` form.
Add Windows-only `os::` tests for long directories, UNC share roots and
non-empty fstype, and run them in the Windows filesystem workflow.
* fix(utils): report Windows network volumes to the fs guard
Windows names the server's filesystem (usually NTFS) for SMB shares, so
the startup unsupported-filesystem guard, which matches fstype names
such as cifs or smb2, never fired for UNC paths or mapped drives.
- Add `DiskInfo::remote`. On Windows it is set when the directory handle
answers `FileRemoteProtocolInfo` (only network redirectors do, and the
handle follows symlinks and junctions to their target) or the volume
root's `GetDriveTypeW` is `DRIVE_REMOTE`. Unix keeps it false because
the client fstype already names network mounts.
- Treat remote volumes as unsupported in the startup guard and label
them "<fstype> on network share".
- Cover UNC shares, paths beyond MAX_PATH below a share, a directory
symlink to a share and a `net use` mapped drive with Windows-only
`os::` tests that skip when `\\localhost\<drive>$` is unreachable;
the Windows filesystem workflow maps the drive before running them.
* chore(deps): replace yanked inout 0.2.3 with 0.2.4
cargo-deny's advisories check rejects the yanked inout 0.2.3 that main's lockfile pins, failing every open PR. Lockfile-only patch bump.
* fix(utils): detect network targets from the final NT path
On the Windows runner, a directory symlink to \\localhost\C$ was reported as local: FileRemoteProtocolInfo did not identify it, and the volume root of the link is a fixed drive.
Also treat a handle whose final NT path (GetFinalPathNameByHandleW with VOLUME_NAME_NT) lies under \Device\Mup\ as remote; the Multiple UNC Provider fronts every UNC and mapped-drive target. The tests now assert the handle probe on its own for local and share paths and report both signals when it disagrees.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): share metadata cache identity across Windows key aliases
Windows disks treat `\` as a path separator and `path_join_buf` drops the
empty segments that leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all
address the xl.meta of `a/b`. The GetObject metadata cache keyed and
hashed the raw spelling, so DELETE `a\b` retired nothing cached for
`a/b`: GET/HEAD `a/b` kept returning the deleted object for up to the
2s cache TTL.
Fold the object name into the same segment form the disk resolves to
before computing the cache hash, key and lookup key. Off Windows the
name stays verbatim and borrowed, so `\` remains an ordinary key byte.
* fix(ecstore): serialize Windows backslash key aliases on one namespace lock
Windows disks treat `\` as a path separator and drop the empty segments
it leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all address the xl.meta
of `a/b`. Namespace locks keyed the raw spelling, so writes, deletes and
heals through different spellings ran concurrently on one xl.meta and a
read-modify-write in rename_data could drop a version.
Fold the object name into its disk identity (rustfs_utils::path::
object_name_disk_identity, a no-op off Windows) wherever a namespace
lock key is formed:
- SetDisks::new_ns_lock, which every object lock path goes through;
- the batch delete lock request and its dedup/lookups, which build
ObjectKeys directly.
Two consumers compared lock identities by spelling and follow along:
- WriteCommitGuard records the folded key from the lock resource, so
matching a borrowed guard now folds the target name too;
- CopyObject holds the source read guard until EOF on streaming reads.
`a\b\` -> `a/b` is not a same-object copy by path_join_buf, but shares
one lock, so it now takes the self-copy lock instead of waiting on
itself until the lock timeout.
Distributed lock keys are folded by the requesting node only; lock
servers compare them verbatim.
* refactor(ecstore): key the metadata cache by the shared disk identity
The metadata cache fix carried its own copy of the Windows backslash
folding rule. Use rustfs_utils::path::object_name_disk_identity, which
namespace locks already use, so cache and lock identities cannot drift.
* ci(windows): run filesystem tests when key identity rules change
The Windows backslash folding for object names lives in utils path.rs
and is consumed by the namespace lock and write commit guard; run the
windows_ tests when any of them changes.
* fix(ecstore): keep Windows disk roots out of verbatim form
The disk root comes from GetFinalPathNameByHandleW, which returns a
verbatim path. `simplified` only strips `\\?\C:` below MAX_PATH, so a
data directory past 260 characters (or one reached through a UNC share)
kept the prefix. Disk IO extends the root by joining `/`-separated
relative paths as strings (`read_raw` builds `<object>/xl.meta` with
`path_join`), and inside a verbatim path `/` is not a separator: the
first metadata read, pool.bin.identity, failed with ERROR_INVALID_NAME
until startup gave up.
Add `rustfs_utils::non_verbatim`, which strips the disk and UNC verbatim
prefixes without a length limit while still keeping paths whose
components Win32 would rewrite, and use it for the publication root.
std re-adds the extended-length prefix for long paths.
Cover a root past MAX_PATH with a disk-level read/write test and run all
`dunce::` tests in the Windows filesystem workflow.
* fix(ecstore): query Win32 volume APIs with extended long disk roots
A disk root past the legacy limit used to reach `rustfs_utils::os` in
verbatim form, which the Win32 volume APIs accept. Now that the root
stays non-verbatim, GetDiskFreeSpaceExW rejected it with
ERROR_PATH_NOT_FOUND and LocalDisk::new failed in the Windows filesystem
workflow. Pass long roots to `get_info` and `get_physical_device_ids` in
extended-length form; shorter roots stay unchanged so reported volume
identities keep their `C:\` form.
* fix(ecstore): admit CopyObject destinations like PutObject
Object-key platform checks were split across call sites with gaps:
CopyObject only ran check_bucket_and_object_names on its destination and
then reached the pools' put_object directly, so it admitted keys PutObject
rejects (over 1024 bytes, leading slash, and on Windows reserved characters
or unaddressable segments, the "writes but cannot be read" case of #3449).
- rustfs-utils: expose check_windows_file_name as the single per-segment
Win32 rule set; dunce's simplified() reuses it. Reserved device names now
include COM/LPT with superscript digits and CONIN$/CONOUT$.
- ecstore: replace the duplicated reserved-name table with that function;
Windows also rejects control characters 0x01-0x1F (NUL keeps its
platform-independent ObjectNameInvalid answer).
- ecstore: add check_new_object_args as the shared admission entry for
PutObject, cross-key CopyObject destinations and CreateMultipartUpload.
In-place copies (metadata replace, restore) keep the copy-only check so
existing keys stay rewritable. CreateMultipartUpload now also rejects
over-NAME_MAX segments up front instead of failing at completion.
- CI: run bucket::utils and dunce tests in the Windows filesystem job.
* fix(ecstore): reject new backslash keys on Windows hosts
A Windows disk treats '\' as a path separator, so a new key `a\b` lands on
the same xl.meta as `a/b` and the two keys overwrite each other.
Reject '\' only where a key enters the namespace (check_new_object_args:
PutObject, cross-key CopyObject destinations, CreateMultipartUpload) and
only on Windows. Reads, deletes, in-flight multipart operations, in-place
copies, data movement and internal meta buckets keep accepting it, so keys
already stored with '\' stay readable, deletable and movable.
Pending owner ruling: whether new '\' keys should be rejected at all, and
whether reads/deletes should keep accepting them.
* ci(windows): quote filtered test runs and drop duplicate path
tempfile's Windows `persist` passes the raw paths to MoveFileExW and
SetFileAttributesW, bypassing std's extended-length handling. Once
`<object>/<rollback>/xl.meta.bkp` reached 260 UTF-16 units every delete
of that key failed with 500, while PUT and GET kept working.
Add `rustfs_utils::extended_length_path`, which normalizes a path with
GetFullPathNameW and prefixes it with `\\?\` (or `\\?\UNC\`), and use it
for the staging directory and destination handed to tempfile. Cover the
single and batch delete paths with a deep-key regression test and run the
helper's Windows assertions in the Windows filesystem workflow.
Fixes#8545
* fix(iam): require an explicit permission for force-delete
A force-delete header no longer inherits s3:* or consoleAdmin. Bucket
force-delete requires s3:ForceDeleteBucket whenever the header is present,
and recursive object force-delete requires s3:ForceDeleteObject. A plain
delete keeps the existing checks.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* test(e2e): keep force-delete header names static
The bucket force-delete helper must pass a static header name into the
SDK request mutator.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* test(e2e): move the force-delete header into the request mutator
The SDK request customizer requires a static header name owned by the
closure.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* fix(iam): keep force-delete out of NotAction grants
NotAction now uses plain wildcard matching, so NotAction "s3:*" still
excludes force-delete. An Allow statement grants s3:ForceDeleteObject or
s3:ForceDeleteBucket only when its Action list names the action; a
NotAction-only Allow never does. The rule applies to both IAM and bucket
policy statements.
Also build the invalid-header errors with S3Error::with_message to keep
the s3s footprint at its baseline, and fix a clippy single_match.
---------
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Record the upload id on the completed object so a lost-response retry
returns that object's ETag instead of NoSuchUpload, while a different
part list or a replaced object keeps the existing errors.
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
This is particularly useful for integration testing of services that use
an S3 API, as it lets many different instances bind a port on different
IPs.
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(replication): resolve drifted replicas via a target version ledger
A replication target that mints its own version ids (Wasabi, AWS S3)
never answers to the source uuid, so every version-addressed mutation
after the initial PUT failed forever: permanent version deletes answered
NoSuchVersion every heal cycle, and tag / retention / legal-hold updates
re-PUT the object, minting one more target version per update
(rustfs/backlog#2340).
Record the id the target assigned as a per-target ledger on the source
version (replication-target-version-<arn>, written through the existing
status writeback) and resolve every later mutation through it: version
deletes DELETE the ledger id, metadata updates go through the
metadata-only Object Lock and tagging APIs. Replicas written before the
ledger existed are located by exact key and ETag, minus the candidates
other generations of the key already claim through their own ledgers; an
ambiguous remainder is refused with a backoff instead of guessed, since a
wrong pick would destroy a live generation. A fresh write never consults
content identity. NoSuchVersion on a version-addressed DELETE counts as
purged.
The fake target gains the Wasabi shape (404 NoSuchVersion on an unknown
id, per-version Object Lock APIs) and the matrix covers the three
mutation classes plus the same-bytes generation case.
* fix(scanner): drop the unused Digest import
Same one-line change as rustfs/rustfs#7366 (main is red with it under -D warnings); carried here so the stacked PRs' merge commits compile until that fix lands.
* fix(admin): probe replication-check mutations by the assigned version id (#7373)
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names (#7377)
* fix(admin): probe replication-check mutations by the assigned version id
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names
A permanent version delete whose replication keeps failing stays in
xl.meta as a PENDING purge, hidden from listings, until every target
confirms it. Once the operator removes the replication configuration or
the rule naming that target nothing ever confirms it: the heal path
derived its delete decision from the configuration (the decision string
is not persisted) and skipped the version forever, so DeleteBucket
answered BucketNotEmpty for a residue the client could neither list nor
remove (rustfs/backlog#2340).
Owe a version purge to the targets its purge state names, let the heal
path through without a configuration, and have the delete worker settle
a target the configuration no longer names as abandoned: the purge is
reported complete locally through the normal writeback, the replica on
the former target is left alone, and the event
replication_purge_abandoned plus a counter are the record.
* fix(admin): send replication-check marker creation without a version id
Running the DeleteMarker / VersionDelete phases on a target that mints
its own version ids exposed two probe-shape bugs on real Wasabi:
- the DeleteMarker phase put the assigned version id on its DELETE. A
RustFS peer reads the source-deletemarker header and creates a marker,
but a generic S3 target executes it as a permanent delete of the probe
version, so VersionDelete then answered NoSuchVersion. Use the same wire
shape as live delete replication: no versionId on a marker creation.
- cleanup treated NoSuchVersion on the version the VersionDelete phase had
already removed as a failure (RustFS/MinIO answer 204 there).
Also gate the no-configuration heal pass-through for pending purges on a
purge state that actually names targets, so a purge without a recorded
target keeps the ordinary skip (scanner unit test), and merge origin/main
(#7365 settles the pool-metadata probe test that failed in CI).
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(scanner): drop the unused Digest import
* fix(replication): carry the compression layout through SSE-C passthrough (#7372)
* fix(replication): queue an in-flight version only once (#7376)
* Revert "perf(ecstore): use AHashMap for FileInfo metadata fields (#6738)"
This reverts commit 13a2ae212e.
* fix(filemeta): restore standard HashMap metadata (#6742)
Remove the direct ahash dependency added for FileInfo metadata and revert the affected filemeta/ecstore call sites back to std::collections::HashMap.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1823 step 10, batch 1 of the repo-wide item-allow sweep. 227 bare #[allow(dead_code)] remain across 83 files; this takes the 19 in utils, notify, checksums, policy, keystone and trusted-proxies, which are small enough to verify end to end.
Removing all 19 first, before writing any reason, matters: 8 of them suppress nothing. Every allow in utils, one in policy and three in notify sit on items that are publicly reachable, so dead_code never applied to them — the same shape as the swift module and kms's dek.rs. Writing a reason onto a no-op allow would dress noise up as considered judgement, so those are simply deleted.
Three items are genuinely dead and go with their allows: notify's new_target_id_set, the AWS metadata fetcher's get_metadata_token, and policy's empty `pub struct Value;`, none of which is referenced anywhere in the tree.
The remaining eight keep an allow, now saying why the item survives rather than who calls it. Two are exercised only by their own crate's tests (checksums' MD5_HEADER_NAME, policy's is_match_as_pattern_prefix). Four are fields written but never read back: keystone's verify_ssl, parsed from config after the reqwest client is already built; keystone's client handle, which keeps the Keystone client alive for the mapper's lifetime; the AWS IMDS endpoint, kept beside the client while requests build their own URLs; and notify's rules_map, whose own comment retains it for snapshot-time judgements no code performs.
checksums' Md5 needed the most care. Crc32, Sha256 and seven others each have an arm in ChecksumAlgorithm::into_impl, and Md5 has none, which reads like a missing algorithm. It is not: ChecksumAlgorithm has no Md5 variant at all. S3 carries Content-MD5 as its own header, separate from the x-amz-checksum-* family, and this impl exists so both paths share the Checksum trait. The reason records that, so the next reader does not re-derive it.
One measurement note for anyone continuing this sweep: cargo does not re-emit warnings for cached compilations, so a per-crate loop of `cargo check -p <crate>` under-reports. checksums showed zero that way while actually carrying three. Touch the sources and check the crates in one invocation, then attribute by path.
Verification: the six crates are warning-free under cargo check --tests; clippy --lib --tests -D warnings clean; cargo nextest run 1096 passed; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 10).
* test(replication): pin missing LWW timestamp header transport
Red-light tests for the replication timestamp three-header contract:
- put_object_headers_carry_replication_timestamp_headers pins that
PutObjectOptions::header() must emit the
x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp
headers when the internal timestamps are set (currently missing).
- test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization
and test_complete_multipart_opts_persist_replication_timestamps_when_authorized
pin that an authorized replication PUT / multipart complete must persist
the inbound timestamps into the internal metadata keys while unauthorized
requests must not (currently never persisted).
- fake_s3_target journals the three timestamp headers per request
(ReplicationTimestampHeaders on RequestRecord) so sender-side e2e
assertions can observe what a real target receives; self-test included.
* fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold
Active-active conflict resolution for concurrent tag/retention/legal-hold
edits needs the source's per-category modification times on both sides of
the wire; the three AdvancedPutOptions timestamp fields were dead and the
headers were neither sent nor parsed.
- Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-
timestamp from PutObjectOptions::header(); names and RFC3339 values
interoperate with MinIO (minio-go constants.go, object-api-options.go),
pinned by a header_compat wire-name test.
- Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip
epoch values in header(), so "never modified" is not sent as a
modification made now.
- Parse the headers only on authorized replication PUTs and multipart
completes, expose them as Option<OffsetDateTime> on ObjectOptions, and
persist them into the dual-prefix internal metadata keys so the
outbound pass (replication_target_boundary) reads the source's
timestamps instead of the mod_time fallback.
- Record the local tagging timestamp in the PutObjectTagging and
DeleteObjectTagging eval metadata, mirroring the object-lock handlers;
without it the sender only ever had the mod_time fallback to offer.
Receiver-side LWW comparison (keep newer stored category metadata over a
stale inbound copy) is left as a TODO at the parse site.
* fix(replication): load the stored tagging timestamp independently of remaining tags
Review: DeleteObjectTagging persists the tagging-timestamp internal key
but leaves the object tagless, and the outbound mapper only loaded the
key inside the user_tags-nonempty branch — the deletion's LWW timestamp
stayed at the epoch and the header was omitted, so the deletion could
never win conflict resolution on the replica. The stored key is now
loaded unconditionally; the mod_time fallback still applies only while
tags exist (MinIO parity), and a tagless object without the key keeps
the epoch default (no header). Deletion-path regression test added.
* fix(storage): reserve replication transport names at metadata ingest
Second review round: a client PUT of
x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized
the bare transport key as stored user metadata. The outbound
replication header builder forwards user metadata verbatim on a
server-authorized request, so the receiver would persist the
attacker-chosen value as trusted internal LWW state — and for a
tagless object nothing later overwrites it.
The ingest namespacing guard now reserves the whole
x-rustfs-source- / x-minio-source- families (the new timestamps and
their siblings: source-mtime/-etag/-version-id/-replication-request),
folding forged keys back under x-amz-meta-. Forged-ingress regression
covers both prefixes and a sibling.
* fix(replication): harden timestamp replay
* fix(app): route retention helper through facade
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* refactor(sse): sink managed-SSE attribution into the shared encryption-keys module
Moves the managed-SSE classifier — stored_managed_encryption_key, contains_managed_encryption_metadata, normalize_managed_metadata — and the SSEType enum from rustfs/src/storage/sse.rs into crates/utils/src/http/object_encryption_keys.rs, the module that already owns every constant they read. This is PR-B0 of rustfs/backlog#1643: crates/scanner must never depend on the rustfs binary crate, so encryption attribution has to live in a shared lower layer before the scanner can report per-scheme coverage without growing a second classifier.
SSEType moves wholesale (option a): its only impl is the dependency-free audit_label(), so the enum relocates verbatim (audit_label becomes pub) and rustfs::storage::sse re-exports it, keeping every existing path compiling. The one piece that cannot move verbatim is normalize_managed_metadata's KMS-context branch, which needs base64 and serde_json — dependencies rustfs-utils does not have and does not gain here. The shared normalizer instead takes an injected Option<fn(&str) -> Option<String>> context recoder; sse.rs passes recode_minio_kms_context, the old inline chain verbatim including the silent skip on decode failure. stored_managed_encryption_key passes no recoder because the context mapping only ever inserts the context key, which the key-id lookup never reads, so its output is identical.
Every metadata lookup stays a case-sensitive exact match (lowercase x-amz-* stored forms, TitleCase MinIO-internal names) per the backlog#1775 trap; new shared-module tests pin that, and a source-scan test in sse.rs asserts the classifier has exactly one definition so a second copy cannot silently return.
* fix(utils): satisfy encryption key test clippy
---------
Co-authored-by: cxymds <cxymds@gmail.com>
The policy crate's Go path.Clean port and rustfs-utils' Windows-aware clean look like duplicates but are not interchangeable: S3 ARN/resource matching must treat backslashes as object-name data, never as separators, so adopting the utils version would change policy evaluation semantics on Windows — a security-adjacent behavior change. Record that judgment as bidirectional do-not-merge notes on both implementations, per the issue's adversarial ruling.
Comment-only change.
Ref rustfs/backlog#1833 (PR7).
The shared module rustfs_utils::http::object_encryption_keys is the single source of truth for encryption metadata key names, but three call sites still carried their own copies or bare literals: crates/kms/src/service.rs (two private constants plus four bare x-rustfs-encryption-* literals on both the write and read path), rustfs/src/app/select_object.rs (six SELECT_* copies), and rustfs/src/storage/options.rs (two private prefix copies now imported from header_compat). All values are unchanged, so the change is a compiler-verified rename.
The reader-only x-rustfs-internal-server-side-encryption- family gets a named constant with the verified judgment recorded on it: no writer emits these keys anywhere in the repo (the SSE writer persists the MinIO-branded keys verbatim for interop), the two comments claiming the dual-key invariant writes this twin were wrong and are corrected, and the defensive redaction/strip readers are kept because removing them is risk-asymmetric.
rustfs-kms's rustfs-utils dependency now declares the http feature it uses instead of relying on feature unification from sibling crates.
Refs rustfs/backlog#1775, rustfs/backlog#1562.
backlog#1833 PR1 prescribed deduplicating crates/replication/src/http.rs onto the canonical rustfs-utils http modules via a re-export facade. That plan conflicts with a standing architecture guard the issue's review missed: check_architecture_migration_rules.sh rejects any rustfs-utils import or dependency from the replication crate ("replication crate HTTP/helper contracts must not import or depend on rustfs-utils"), the same way it bans rustfs-filemeta and rustfs-storage-api — the wire-contract crate deliberately has zero internal dependencies.
So this lands the issue's fallback shape instead (the same bidirectional do-not-merge pattern the issue itself prescribes for the policy path.rs cluster): a module doc on replication/http.rs naming the canonical owners and the guard that forces the local copy, mirror notes on utils' metadata_compat.rs and header_compat.rs, and a new test pinning every duplicated constant to its literal wire value so the two copies cannot drift silently.
No production code changed.
Ref rustfs/backlog#1833 (PR1).
Complete the encrypted-object replication series (backlog#1783, PR-C of
3, after #5872 and #5885): SSE-C objects replicate as ciphertext
passthrough — the source holds no customer key, so the stored bytes and
their encryption metadata travel verbatim and the replica decrypts only
with the original customer key, single-part and multipart.
- Sender: SSE-C objects read raw (raw_data_movement_read), transfer at
ciphertext size, and range multipart parts over stored part sizes.
- Receiver: authorized replication PUTs restore the stored SSE-C keys
from the transport headers (exact lowercase forms - the read-path
check is case-sensitive), set ObjectOptions.preserve_ciphertext, and
skip compression, bucket-default SSE, and sse_encryption behind one
restore-derived gate. Multipart uses an internal session marker to
store parts verbatim and strips it on complete.
- Convergence: the replication HEAD sends
x-rustfs-source-replication-check; the target authorizes it as
ReplicateObjectAction and skips SSE-C read validation for that
request only, so keyless convergence HEADs see etag/size/mtime
instead of 400 and SSE-C replicas stop re-driving forever.
- e2e: SSE-C contract flips to a key-gated readable replica (no-key and
wrong-key GETs fail - the direct silent-plaintext detector); new
multipart passthrough contract with ETag/marker/stability assertions.
Groundwork for encrypted-object replication (backlog#1783, PR-A of 3):
- classify_replication_source_encryption: accept the AES256 marker that
every stored SSE-C object carries; the SseC arm was unreachable.
- Fail closed on sealed material without an SSE marker (MinIO-written
objects) instead of replicating ciphertext as plaintext.
- Replace the dead VALID_SSE_REPLICATION_HEADERS table with a transport
map keyed by the metadata keys the SSE writer actually persists, shared
via the new rustfs_utils::http::object_encryption_keys module.
- Structurally strip all encryption metadata from outbound replication
(x-rustfs-encryption-* envelopes previously passed the filters).
- Skip decrypt_checksums for encrypted objects at the boundary so its
is_multipart=false (a response-path contract) cannot misroute
encrypted multipart objects once managed replication opens.
- Redact X-Rustfs-Replication-* SSE transport values in FileInfo Debug.
A reconciliation test pins that every key encryption_material_to_metadata
produces is either transport-mapped or stripped. All four SSE replication
e2e contracts still assert FAILED unchanged.
overtrue