Second review round:
- Authorization: the default stream enumerates object names and version
ids, which a metrics-only principal must not see — it now requires
admin:ReplicationDiff (MinIO parity, route policy updated);
?aggregate=true carries no object identities and keeps
admin:GetReplicationMetrics.
- The nil UUID is RustFS's in-memory null-version sentinel and now
leaves as the S3 wire token 'null' instead of a zero UUID (a
pre-versioning object scanned after versioning + existing-object
replication can persist it into the ledger).
- The durable ledger is not bounded by the in-memory pending cap and
the body is buffered before send; the stream now stops at 10,000
documents and signals truncation via
x-rustfs-replication-mrf-truncated (mirroring the diff endpoint)
plus a warn event, instead of staging an unbounded body.
Review: madmin only decodes the body of a 200, so the out-of-band
unavailability header was invisible to it and an unreadable ledger read
as a clean zero-row backlog. Stream mode now returns 503; aggregate
mode keeps the availability fields.
The mrf endpoint returned a single aggregate envelope, which madmin's
json.Decoder loop decoded as one phantom row (empty object) in
'mc replicate backlog' (backlog#1675 P1-13, mrf half; the diff half was
fixed in #5799 and this mirrors its pattern).
- Default response is now a bare stream of ReplicationMRF documents
(exact madmin json tags; Size/TargetARNs as ignored extension keys)
built from the durable backlog ledger; an empty backlog renders an
empty body, so mc shows zero rows instead of a phantom row.
- The aggregate counter envelope moves behind ?aggregate=true (RustFS
extension) and now advertises PerObjectEntriesAvailable whenever the
durable backlog is readable.
- An unreadable backlog is signalled out-of-band via
x-rustfs-replication-mrf-backlog-unavailable (mirrors the diff
truncation header) plus a warn event, since the bare stream cannot
carry source health.
- The madmin node parameter is accepted but documented as a no-op: the
durable ledger is cluster-shared with no per-node attribution.
- Delete-marker purge entries fall back to the marker version id so
those rows keep a version identity.
Red-light evidence for backlog#1675 P1-13 (mrf half): madmin's
BucketReplicationMRF decodes the response one ReplicationMRF document at
a time, so the current aggregate envelope decodes as a single phantom
row with an empty object in 'mc replicate backlog'. The new contract
tests assert the desired bare-document stream (exact madmin json tags,
empty body for an empty backlog) and fail against the current
render_mrf_backlog extraction, which preserves the envelope-only
behavior:
- mrf_stream_renders_bare_madmin_documents: envelope keys leak, no
per-entry documents
- mrf_stream_renders_empty_body_for_no_entries: empty backlog still
renders the envelope (phantom row)
- mrf_aggregate_envelope_retains_counters: PerObjectEntriesAvailable
never advertises the enumerable stream
fix(site-replication): lift a rejoined site's restarted edit counter over stale fence marks
A site removed while unreachable (unilateral removal: the receiver never
dropped it from its peer map, so parse_site_replication_state's load-time
mark pruning never fired) that later rejoins recreates its state object
and restarts edit_generation at zero. The receiver's surviving high-water
mark then silently fences out every stamped delivery from that origin —
peer edits and the add finalize fan-out alike are acked without applying
— until the restarted counter catches up.
Allocate the generation as a hybrid logical clock instead:
max(wall clock in unix nanoseconds, previous + 1), still inside the state
transaction under the distributed state-object lock. Every value a
lifetime hands out is capped by the wall clock at its own allocation, so
a recreated lifetime's first allocation exceeds them all and clears the
stale mark, while a pre-removal delivery still in flight stays below the
new floor and remains correctly fenced. previous+1 keeps allocations
strictly increasing across same-tick allocations and mid-lifetime clock
regressions.
Nothing changes on the wire or in the persisted schema: editGeneration
stays the single fence param and edit_generation the single counter
field, so pre-hybrid receivers get the fix as soon as the sender
upgrades, old binaries preserve the field across rolling up/downgrades,
and marks recorded by plain-counter receivers (small values) are cleared
by any wall-clock allocation. A clock that regresses across a
delete/recreate degrades to a fence that self-heals once real time
passes the previous lifetime's last allocation, and introduces no
rollback window beyond what the plain counter already had.
An epoch-based design (editEpoch wire param + per-origin epoch marks)
was built first and rejected under adversarial review: old binaries
rewriting the state object drop the unknown epoch fields, which both
disarms the fix mid-rolling-upgrade and — because epoch adoption lowers
the generation mark — reopens the pre-restart rollback the fence exists
to prevent; a backwards clock also fences an origin permanently instead
of self-healing. The hybrid clock has none of these modes.
* fix(storage): restore multipart disk compression and make the legacy decompressor resumable
Multipart uploads have bypassed disk compression since #5169 removed the session marker as a stopgap for mid-stream GET failures. The actual root cause was never the multipart layout: the legacy DecompressReader reset its payload consumption state on every poll re-entry, so a Poll::Pending in the middle of a block payload (routine under the erasure duplex) desynchronized the block framing and surfaced as LZ4 frameType errors. This rewrites the decoder as a resumable state machine, restores the multipart session compression marker, reports logical part sizes in ListParts, and makes the rebalance migration read raw stored bytes so compressed and encrypted objects survive migration verbatim.
Fixes#5957. Internal tracking: backlog#1848, backlog#1850.
* feat(storage): stage multipart compression behind RUSTFS_COMPRESSION_MULTIPART_ENABLED
Review follow-up: a rolling-upgrade window must not create new compressed multipart objects while pre-fix nodes (whose decompressor is not resumable) may still serve reads. The session marker is now additionally gated on RUSTFS_COMPRESSION_MULTIPART_ENABLED, default off, so the restored capability stays dark until the operator confirms fleet convergence. The default flips per the multipart-compression-default-off-window entry in docs/architecture/compat-cleanup-register.md once the minimum supported direct-upgrade release ships the resumable decoder.
* chore(compat): satisfy the cleanup-register guard for the multipart compression switch
The architecture guard requires every backticked identifier in a register entry to carry a RUSTFS_COMPAT_TODO source marker: keep only the entry slug in backticks, and add the marker (with its literal Remove-after condition) at the switch definition.
* chore(rio): drop a dead store in the poison guard and note the end-block branch
Review follow-up: the poison gate re-assigned an already-true flag, and the COMPRESS_TYPE_END branch reads as dead without stating that the writer never emits an end block — that absence is exactly what lets concatenated per-part streams decode as one.
* fix(s3): report empty compressed multipart part size
* fix(s3): report empty encrypted multipart part size
* fix(site-replication): route every state RMW through the locked transaction
P1-15 PR2 (rustfs/backlog#1796, batch B2 of rustfs/backlog#1675), the
follow-up promised by rustfs/rustfs#5882.
PR1 left ~26 read-modify-write call sites on
config/site-replication/state.json in the pre-transaction shape: a
process-local mutex around load / mutate / save, each IO taking its own
object lock. Nothing held a distributed lock across the whole sequence, so
two nodes of one site still lost each other's updates, and the transitional
mutex kept the old shape available to copy.
Every remaining RMW now runs inside update_site_replication_state;
read-only sites use load_site_replication_state, whose object read comes
with the object-level read lock. SITE_REPLICATION_STATE_LOCK and its owner
helper are gone, together with their architecture-guard allowlist entry and
inventory row.
The multi-stage flows (add / edit / peer join / peer edit / remove / rotate)
keep their updated_at and pending-id CAS, but the CAS now runs inside the
transaction that writes, against the state that transaction loaded. Peer
probes, IAM work and fan-outs run between transactions and hold no lock at
all — the add no longer blocks every writer of the site across its peer join
round trips, and it re-checks the precondition right after the capability
probes so the common race is rejected before any IAM write or remote join.
When the add's commit CAS still fails, the error says the peers may already
be joined and that re-running the add reconverges. The add adopts only the
fields it computed (exhaustive destructure — adding a state field is a
compile error until classified); fields owned by writers that do not bump
updated_at keep their freshly loaded values.
Ordering of peer-edit deliveries now rests on the generation fence landed in
PR1 rather than on a guard that could never order two nodes: the add's
finalize fan-out carries the generation allocated in its commit. An accepted
peer join PRESERVES the applied-generation high-water marks — join fan-outs
are routine (adds and rotations both deliver SRPeerJoin to existing peers),
so wiping them would let stalled older edits land after any join; the
unilateral-removal rejoin misfence that a wipe would have patched is
pre-existing since the fence landed and needs an epoch in the fence instead.
The rotation handler now takes the lifecycle guard: the background
service-account reconciler runs its repair under a lifecycle try-acquire,
and its pending-rotation precheck is only sound if a rotation cannot start
mid-repair — an exclusion the removed process mutex used to provide as a
side effect.
update_site_replication_state_when_changed adds persist-or-skip so ack
markers and pending-clearing paths stop rewriting the object on a miss —
load-bearing, because the shared persist helper clears the whole object for
a ≤1-peer pending-free state — and save_site_replication_state is now
cfg(test): the pre-P1-15 shape can no longer be written in production code.
No on-disk format change.
Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (181 passed); site-replication
dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D
warnings; make pre-commit. Mutation checks: dropping the state-object lock
from the boundary reds the separate-node concurrency tests; flipping a
persist-or-skip miss to a persist reds
test_missed_pending_clear_must_not_rewrite_the_state_object. Reviewed by
three independent adversarial passes (correctness/concurrency,
security/compatibility, simplicity/test-coverage); their confirmed findings
are folded in.
* fix(site-replication): serialize peer-join admission around its IAM write
Review follow-up (overtrue): two joins accepted by the same node could
interleave as "A checks a stale snapshot and pauses reading its body, B
applies secret B and commits, A resumes, overwrites IAM with secret A, and
A's commit is refused as superseded" — the persisted state advertised B's
contract while IAM only accepted A's secret, failing every peer
control-plane call. The pre-P1-15 process mutex serialized same-node joins
end to end; removing it dropped that exclusion.
admit_peer_join now runs the staleness check, the IAM upsert and the state
commit under the lifecycle guard, with the authoritative pre-check taken
against a load under that guard BEFORE IAM changes anything. The closing
transaction still re-checks staleness: the guard is process-local (exactly
as far as the old mutex reached) and the state-object lock arbitrates joins
accepted by different nodes. The body is fully read before the guard so a
stalling sender cannot block add/remove/rotate/reconciler.
The IAM step is injected, and the gated-body regression test reproduces the
review's ordering: join A is held mid-IAM while a newer join B arrives; B
must wait at the guard, and both IAM order and the final persisted state end
on B. Mutation-verified: removing the lifecycle guard from admit_peer_join
turns the test red.
Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (182 passed);
site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs
--all-targets -D warnings; make pre-commit.
* fix(site-replication): fence peer-join admission across nodes
Review follow-up (overtrue, round 2): the lifecycle guard only serializes
joins within one process. Node A could pass the staleness check for an
older T1, node B write secret B to IAM and commit a newer T2, and node A
then overwrite IAM with secret A while its own state commit is refused as
superseded — state advertising T2's contract while IAM only accepts A's
secret.
The admission (staleness check -> IAM upsert -> state commit) now also runs
under a distributed join-admission lock, a namespace-lock key with no
backing object, following the repair execution lock's pattern — including
its nesting of config-object locks (admission -> state), and delegating
crash safety to the lock subsystem's lease expiry instead of a hand-rolled
TTL. The staleness check runs against a load taken inside the lock, before
IAM changes anything, so a superseded join exits without touching IAM. The
closing transaction keeps its re-check for defence in depth and for
old-version nodes that do not take the admission lock during a rolling
upgrade (that mixed-version window keeps today's behavior and closes when
the upgrade completes).
admit_peer_join_across_nodes is the admission minus the process-local
lifecycle guard — exactly what a second node runs — and the new
separate-nodes regression test drives it directly with join A gated
mid-IAM: join B must wait at the distributed lock, and both the IAM write
order and the final persisted state end on B. Mutation-verified: removing
the admission lock turns the test red while the same-node test (which
drives the full admit_peer_join) stays green.
Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (183 passed);
site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs
--all-targets -D warnings; make pre-commit.
Completes PR3 of the issue.
Store doubles: NoopTableCatalogStore (193 lines, a pure stub answering "nothing here") and TestTableCatalogStore (416 lines, a stateful fake with commit pauses and failure injection) move into test_support.rs, along with TestCatalogPublishPause which the latter needs. Per the issue's ruling both shapes are kept — they are different tools, not duplicates of each other. Being honest about the benefit: this does not reduce the number of TableCatalogStore implementations, it puts both in one file so a trait change is one file to edit instead of two.
row_level_conflict fold: rejects_stale_new_manifest_sequence, rejects_stale_added_entry_sequence, and rejects_historical_change_in_new_manifest were identical apart from four literals (manifest-list sequence, data-file name, manifest-entry snapshot id, failure message). They become one table-driven test with three rows, each row keeping its original values, and every assertion carries the case name.
Verification: cargo test -p rustfs --lib table_catalog 479 passed and --lib admin::handlers::table_catalog 165 passed (both down exactly 2 from the 3->1 fold; the store filter is a substring match that also covers the admin tests); clippy --lib --tests -D warnings clean; make pre-commit green.
Ref rustfs/backlog#1837 (PR3).
Stream materialized GET bodies by moving the buffered Bytes once instead of wrapping the stream in an extra bytes_stream layer.
Add an operations runbook for object I/O tuning A/B sweeps.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(table-catalog): move the store-side stateful object backend into test_support
First half of the issue's PR2: the store tests' TestCatalogObjectBackend cluster (state/record/locks/pause types, the seed/fail/pause instrumented inherent impl, the TableCatalogObjectBackend trait impl, and the BlockingObjectPublication/UnserializedTestPublication commit-publication fakes — 544 lines) moves verbatim from table_catalog/tests.rs into test_support.rs, with pub(crate) visibility on the items and fields the tests reach directly. Pure move, no behavior change; the admin handler tests' TestTableCatalogObjectBackend union (its put barrier / fail-path / lock-attempt instrumentation folding into this fake) is the second half.
Verification: cargo test -p rustfs --lib table_catalog 481 passed; clippy --lib --tests -D warnings clean; make pre-commit green.
Ref rustfs/backlog#1837 (PR2, part 1).
* test(table-catalog): fold the admin object backend into the shared fake
Second half of PR2: the admin handler tests' TestTableCatalogObjectBackend (struct, inherent impl, trait impl, lock alias — 201 lines) is deleted and its instrumentation folded into the shared TestCatalogObjectBackend, which the admin tests now take through a type alias so no call site is renamed.
Two behavioral differences between the two fakes were found by the test suites rather than assumed away, and both are preserved:
- Lock observability: the admin fake implemented only acquire_write_lock, so the trait's default acquire_read_lock -> acquire_write_lock delegation made read acquisitions visible in lock_attempts. The shared fake implements both independently, so five fence/lock tests timed out until the read path also records attempts.
- Etag generation: the admin fake used content-addressed sha256 etags (its tests observe an etag and expect rewriting identical bytes to reproduce it) while the store fake uses an incrementing counter. Instead of silently picking one, the union carries a content_addressed_etags flag; the 80 admin construction sites go through TestCatalogObjectBackend::content_addressed() and the store tests keep counter semantics.
The six one-shot path-keyed injection knobs (fail/corrupt put, missing/fail read, put barrier) run before the store fake's attempt-indexed injection maps, matching each fake's original ordering.
Verification: cargo test -p rustfs --lib table_catalog 481 passed; --lib admin::handlers::table_catalog 167 passed; clippy --lib --tests -D warnings clean; make pre-commit green.
Ref rustfs/backlog#1837 (PR2, part 2).
* refactor(sse): sink managed-SSE attribution into the shared encryption-keys module
Moves the managed-SSE classifier — stored_managed_encryption_key, contains_managed_encryption_metadata, normalize_managed_metadata — and the SSEType enum from rustfs/src/storage/sse.rs into crates/utils/src/http/object_encryption_keys.rs, the module that already owns every constant they read. This is PR-B0 of rustfs/backlog#1643: crates/scanner must never depend on the rustfs binary crate, so encryption attribution has to live in a shared lower layer before the scanner can report per-scheme coverage without growing a second classifier.
SSEType moves wholesale (option a): its only impl is the dependency-free audit_label(), so the enum relocates verbatim (audit_label becomes pub) and rustfs::storage::sse re-exports it, keeping every existing path compiling. The one piece that cannot move verbatim is normalize_managed_metadata's KMS-context branch, which needs base64 and serde_json — dependencies rustfs-utils does not have and does not gain here. The shared normalizer instead takes an injected Option<fn(&str) -> Option<String>> context recoder; sse.rs passes recode_minio_kms_context, the old inline chain verbatim including the silent skip on decode failure. stored_managed_encryption_key passes no recoder because the context mapping only ever inserts the context key, which the key-id lookup never reads, so its output is identical.
Every metadata lookup stays a case-sensitive exact match (lowercase x-amz-* stored forms, TitleCase MinIO-internal names) per the backlog#1775 trap; new shared-module tests pin that, and a source-scan test in sse.rs asserts the classifier has exactly one definition so a second copy cannot silently return.
* fix(utils): satisfy encryption key test clippy
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Keep S3 CopyObject's real outer owner task alive across caller cancellation so the source/destination bucket guards, same-key copy guard, storage commit, and post-commit publication hooks complete as one request-owned transaction boundary.
Co-authored-by: heihutu <heihutu@gmail.com>
The two table_catalog test files (27.5K lines combined) each maintained a parallel constructor stack for Iceberg metadata JSON and avro manifest-list/manifest bytes. Per the issue's adversarial ruling the parameterized admin variants are canonical (the store file hardcoded sequence 7 / snapshot 20); the two stacks were verified structurally identical first — schemas byte-equal, field lists and values aligned.
New #[cfg(test)] table_catalog/test_support.rs owns the seven constructors (metadata JSON, three manifest-list variants, two manifest variants, nullable_long). The admin tests import them under their old names; the store tests keep their historical signatures as thin delegates passing the fixed values explicitly — every produced byte is identical to the pre-extraction fixtures (the delegate's argument order was cross-checked against the canonical destructuring after an initial swap surfaced as five sequence-bound validation failures).
Ref rustfs/backlog#1837 (PR1).
Use BytesMut spare capacity for direct and pooled small PUT body reads while preserving exact-length validation.
Co-authored-by: heihutu <heihutu@gmail.com>
RemoteAddr, the DependencyReadiness family (DependencyReadiness, ReadinessDegradedReason, DependencyReadinessReport), and convert_ecstore_object_info (with its offset_date_time_to_timestamp helper) are consumed across app, infra, and interface layers but lived under server, so every lower-layer import was an upward app->interface or infra->interface edge the layer guard had to baseline.
They now live in a new layer-neutral rustfs/src/shared_types.rs (classified infra by the guard, making all consumer imports downward or lateral). server::readiness and server::event re-export for their own internals; the eight consumer sites (admin_usecase, bucket_usecase, object_usecase, cluster_snapshot, storage/access, storage/helper, plus the admin handler tests) import from the new home. Pure move: no type, impl, or behavior change.
The regenerated layer-dependency baseline shrinks by exactly eight lines with zero additions — the ratchet's intended direction. The two remaining readiness entries (collect/snapshot fn imports) need the collection machinery itself extracted from server and are left for the issue's PR5 scope.
Ref rustfs/backlog#1834 (PR4).
The 14 tests carried #[ignore = "requires isolated global object layer state"], and the only CI lane that runs ignored tests filters for lifecycle tests — so they executed nowhere. Under nextest, the authoritative runner, every test owns its process and the stale reason no longer applies; all 14 pass.
Ten of them assert the InternalError path taken while the global object layer is uninitialized, a premise a sibling test can destroy under the documented shared-process cargo test fallback. Those ten now start with an explicit premise guard: when a sibling already initialized the store the test skips with a message instead of asserting against a scenario it does not describe. Under nextest the guard never fires and the assertions always run.
Dual-runner evidence: nextest 79 passed; cargo test module-scoped 79 passed; the full storage-tree cargo test sweep returns to its pre-existing baseline (8 unrelated in-process failures, none introduced or worsened here). No test deleted.
Ref rustfs/backlog#1830 (PR1).
* perf(metrics): attribute PUT stage costs
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): move PUT metadata during shuffle
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(s3): reuse PUT object lock state
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): trim PUT metadata fanout clones
Build per-disk PUT metadata only for committed writer slots, move the response metadata out of the fanout vector, and preserve fresh FileInfo shuffle semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(metrics): make PUT stage attribution opt-in
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): commit inline PUT shards directly
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): streamline rename staging cleanup
Use the directory-specific removal operation for rename_data staging parents. This avoids a guaranteed failed file-removal probe on Unix-like hosts and lets Windows remove the empty directory directly while preserving best-effort non-empty handling.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover inline PUT rename failures
Cache the detailed stage metrics gate once per PUT and exercise exact-quorum and quorum-minus-one failures after inline shard encoding.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The peer-edit delivery fence from #5882 treated an equal applied generation as stale. One edit legitimately fans out one delivery per peer record under a single generation (the ILM-expiry edit sends every peer's record), so the receiver applied only the first body, raised its high-water mark, and silently acked-success while dropping the rest — enableILMExpiryReplication never converged on receiving sites and the three-node nightly e2e failed deterministically (issue #5767).
Only a strictly newer applied generation is stale now. Equal generation implies the same logical edit and re-applying a delivery is idempotent (update_peer overwrites the peer record; the mark is raised with max), while strictly older deliveries — the cross-node ordering case the fence exists for — stay rejected.
Adds a composed unit test driving three same-generation bodies through the receiver's fenced sequence, and widens the replication e2e's two site-replication wait helpers from a 10s polling ceiling to the 30s deadline the file's other waits use.
PutBucketVersioning with Status=Suspended on a bucket that carries a replication configuration now fails with InvalidBucketState, matching AWS S3 and MinIO. Suspension would start minting null versions that the versioned replication engine can never converge — the state is unreachable on AWS and MinIO, and the nightly acceptance-matrix e2e that tried to exercise it failed every night since it landed (issue #5767).
The acceptance-matrix test tail now pins the rejection contract (InvalidBucketState) and verifies a fresh matched PUT still replicates with a real version id after the rejected suspension.
The shared module rustfs_utils::http::object_encryption_keys is the single source of truth for encryption metadata key names, but three call sites still carried their own copies or bare literals: crates/kms/src/service.rs (two private constants plus four bare x-rustfs-encryption-* literals on both the write and read path), rustfs/src/app/select_object.rs (six SELECT_* copies), and rustfs/src/storage/options.rs (two private prefix copies now imported from header_compat). All values are unchanged, so the change is a compiler-verified rename.
The reader-only x-rustfs-internal-server-side-encryption- family gets a named constant with the verified judgment recorded on it: no writer emits these keys anywhere in the repo (the SSE writer persists the MinIO-branded keys verbatim for interop), the two comments claiming the dual-key invariant writes this twin were wrong and are corrected, and the defensive redaction/strip readers are kept because removing them is risk-asymmetric.
rustfs-kms's rustfs-utils dependency now declares the http feature it uses instead of relying on feature unification from sibling crates.
Refs rustfs/backlog#1775, rustfs/backlog#1562.
test_format_v1 (ecstore layout::format) only printed its results; the pinned v1 format.json literal never parsed at all because "this": null fails Uuid deserialization, and the Err was silently discarded. Fix the fixture to the real on-disk shape (MinIO and RustFS always write a concrete disk UUID there) and assert a serialize->parse roundtrip identity plus every pinned field of the literal.
test_console_cors_configuration discarded all four parse_cors_origins results; parse_cors_origins returns an opaque CorsLayer, so the test now drives real CORS preflight requests through an axum router and asserts the allow-origin outcomes: wildcard answers any origin with *, a configured list echoes listed origins and refuses unlisted ones, empty/unset configurations allow no cross-origin caller.
test_heal_channel_processor_new only constructed the processor; it now asserts the response channel accepts a send.
Ref rustfs/backlog#1836 (PR1).
select_object.rs re-declared six interop header names as SELECT_* locals (five X-Minio-Internal-Server-Side-Encryption-* markers plus x-rustfs-encryption-key-id). The canonical owners live in rustfs-utils' object_encryption_keys module, which the rustfs crate already depends on with the full feature set. Import them under their canonical names and drop the local copies; SELECT_KMS_ARN_PREFIX stays local because no canonical owner exists for the KMS ARN prefix.
Values are byte-identical, so no behavior change.
Ref rustfs/backlog#1833 (PR3).
唐小鸭