* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(kms): report a missing KMS key as 400 KMS.NotFoundException
A PutObject whose resolved SSE-KMS key (request header or bucket default
rule) does not exist in the KMS answered 500 InternalError with a generic
message: KmsError::KeyNotFound fell through to the default arm of the
StorageError-to-ApiError mapping. S3 reports this client mistake as 400
KMS.NotFoundException; the mapping now does the same and names the key.
s3s has no status for a custom code, so the ApiError-to-S3Error conversion
supplies it.
The legacy create-key aliases behind /minio/admin/v3/kms/key/create ignored
the key-id query parameter that mc sends, creating a key under a generated
id instead of the requested name. The alias now honors key-id (and its
keyId/key spellings) alongside the name tag, and refuses a request whose
two sources disagree.
Refs: rustfs/backlog#2330 (KMS-312, KMS-110)
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* feat(heal): expose admission observability
Track heal admission outcomes and bounded lock-phase latency through the existing operations snapshot so distributed E2E gates can assert duplicate, forceStart, and displacement behavior without relying on logs.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* feat(observability): expose pool write-block diagnostics
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* fix(s3): reject unsigned x-amz headers on presigned requests
A SigV4 presigned URL only binds the headers listed in X-Amz-SignedHeaders, but the handlers applied every x-amz-* request header regardless. The holder of a presigned PutObject URL signed with SignedHeaders=host could add x-amz-tagging, x-amz-storage-class, x-amz-website-redirect-location, ACL, metadata, Object Lock or SSE headers and have them applied (GHSA-g8w9-qw9q-fghr). Reject such requests at the S3 access boundary with 403 AccessDenied and the AWS message "There were headers present in the request which were not signed"; x-amz-cf-id stays tolerated for CloudFront. SigV2 and header-signed SigV4 requests are unchanged.
Regression tests are named after the advisory (unit tests in rustfs/src/auth.rs, e2e in crates/e2e_test/src/presigned_negative_test.rs with a signed-tagging positive control); the security smoke floor rises to 20 and the e2e selection digests are refreshed for the two new cases.
* fix(s3): apply presigned signed-header rule to custom routes and harden parsing
Move the GHSA-g8w9-qw9q-fghr check to the first statement of S3Access::check, apply it in S3Router::check_access so admin, console, STS and extension routes that never reach the access hook enforce the same rule, read X-Amz-SignedHeaders with the exact key the verifier uses and treat a duplicate as signing nothing, and log the rejection as a warn event with the repository field shape. Add presigned GET, unsigned x-amz-copy-source and unsigned Content-Type e2e cases plus a router unit test; raise the security smoke floor to 26 and refresh the selection digests.
* docs(testing): list the full GHSA-g8w9 regression set
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A multipart UploadPart queued for a foreground write permit is not read while it waits, so the client's socket write stalls for the whole wait and the client's own write timeout decides the outcome; botocore reports that as ConnectionClosedError. Lower the default queue wait from 30 s to 10 s so the part receives SlowDown before mainstream SDK timeouts, and stop forcing a 4 MiB SO_RCVBUF on the API listener (new RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES restores a fixed size) so a queued connection no longer lets up to 8 MiB of unread body accumulate in kernel memory.
Fixes#7385.
Co-authored-by: houseme <housemecn@gmail.com>
Add a Unix process-kill fixture for the authoritative successor fsync boundary where the scoped journal is durable while the legacy mirror still contains the startup epoch. Also extend heal-control transport replay coverage so a duplicate producer after a lost response receives the canonical merged receipt without creating another task.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(replication): resolve drifted replicas via a target version ledger
A replication target that mints its own version ids (Wasabi, AWS S3)
never answers to the source uuid, so every version-addressed mutation
after the initial PUT failed forever: permanent version deletes answered
NoSuchVersion every heal cycle, and tag / retention / legal-hold updates
re-PUT the object, minting one more target version per update
(rustfs/backlog#2340).
Record the id the target assigned as a per-target ledger on the source
version (replication-target-version-<arn>, written through the existing
status writeback) and resolve every later mutation through it: version
deletes DELETE the ledger id, metadata updates go through the
metadata-only Object Lock and tagging APIs. Replicas written before the
ledger existed are located by exact key and ETag, minus the candidates
other generations of the key already claim through their own ledgers; an
ambiguous remainder is refused with a backoff instead of guessed, since a
wrong pick would destroy a live generation. A fresh write never consults
content identity. NoSuchVersion on a version-addressed DELETE counts as
purged.
The fake target gains the Wasabi shape (404 NoSuchVersion on an unknown
id, per-version Object Lock APIs) and the matrix covers the three
mutation classes plus the same-bytes generation case.
* fix(scanner): drop the unused Digest import
Same one-line change as rustfs/rustfs#7366 (main is red with it under -D warnings); carried here so the stacked PRs' merge commits compile until that fix lands.
* fix(admin): probe replication-check mutations by the assigned version id (#7373)
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names (#7377)
* fix(admin): probe replication-check mutations by the assigned version id
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names
A permanent version delete whose replication keeps failing stays in
xl.meta as a PENDING purge, hidden from listings, until every target
confirms it. Once the operator removes the replication configuration or
the rule naming that target nothing ever confirms it: the heal path
derived its delete decision from the configuration (the decision string
is not persisted) and skipped the version forever, so DeleteBucket
answered BucketNotEmpty for a residue the client could neither list nor
remove (rustfs/backlog#2340).
Owe a version purge to the targets its purge state names, let the heal
path through without a configuration, and have the delete worker settle
a target the configuration no longer names as abandoned: the purge is
reported complete locally through the normal writeback, the replica on
the former target is left alone, and the event
replication_purge_abandoned plus a counter are the record.
* fix(admin): send replication-check marker creation without a version id
Running the DeleteMarker / VersionDelete phases on a target that mints
its own version ids exposed two probe-shape bugs on real Wasabi:
- the DeleteMarker phase put the assigned version id on its DELETE. A
RustFS peer reads the source-deletemarker header and creates a marker,
but a generic S3 target executes it as a permanent delete of the probe
version, so VersionDelete then answered NoSuchVersion. Use the same wire
shape as live delete replication: no versionId on a marker creation.
- cleanup treated NoSuchVersion on the version the VersionDelete phase had
already removed as a failure (RustFS/MinIO answer 204 there).
Also gate the no-configuration heal pass-through for pending purges on a
purge state that actually names targets, so a purge without a recorded
target keeps the ordinary skip (scanner unit test), and merge origin/main
(#7365 settles the pool-metadata probe test that failed in CI).
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(scanner): remove unused digest import
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): add raw page owner index (#7375)
* feat(scanner): add raw page owner index
Add a serializable raw enumeration page owner index for scanner resume work.
The index exposes unsupported, building, and ready states, validates committed page identity by recomputing digests, and uses generation checks for CAS-style page commits.
Focused tests cover small-budget restart progress, page digest/source drift rejection, corrupt deserialized state, CAS failure, precommit crash, empty sources, and invalid entry boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): persist raw page owner resume state (#7379)
Wire the scanner raw enumeration partial-cache writer to the raw page owner index so interrupted bucket walks can retain validated page-builder state across scanner restarts.
Keep complete owner sources terminal-only, add partial-source ingestion for in-progress raw directory reads, and validate the persisted page index through bucket checkpoint preparation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): fence segment producer observations (#7381)
Require the segment observation fixture to carry source, incarnation, key-format, baseline, process epoch, generation-window, gap, overflow, and producer-coverage proof before accepting a narrowed proposal. Keep the diagnostic path fixture-only and remove its ordinary stderr output.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate pool metadata read probes (#7367)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF crash successor matrix (#7369)
* test(heal): cover MRF crash successor matrix
Add process-boundary MRF replay coverage for the successor snapshot window after a retained startup journal is flushed but before cleanup deletes it. Extend the mixed authoritative/legacy reader fixture with a scoped v2 journal epoch to pin the no-merge contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover service-kill MRF replay (#7380)
Add a Unix process fixture that waits after publishing the pending MRF successor snapshot, then is terminated by the parent before restart replay.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover transport-lost start receipts (#7371)
Add gRPC transport fault fixtures for heal-control start admission. The tests distinguish pre-admission transport loss from post-admission response loss, then verify exact envelope retries reuse the canonical receipt while fresh forceStart requests create distinct tasks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add crash-restart heal evidence case (#7370)
* test(scanner): add crash-restart heal evidence case
Add a distinct W21 background target crash case to the scanner/heal evidence registry and oracle path.
Keep the existing restart lane on graceful process restart, keep the crash lane on hard kill, and make the wiring checker reject evidence/oracle mismatches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): support older Python wiring checks
Let the scanner/heal evidence wiring checker run under Python 3.9/3.10 by falling back to tomli and chunked SHA-256 hashing when the Python 3.11 standard APIs are unavailable.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject stale raw page source seeds (#7382)
Do not prefill a resumed raw page owner with previously indexed entries when starting a new raw directory observation pass. The next pass must observe the same prefix again before the page index can advance; otherwise the index is discarded fail-closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): defer raw page revalidation until observed (#7384)
A resumed raw page owner index must not prefill entries from older cache state, but it also must not discard a valid multi-entry index before the current raw directory pass has observed enough entries to prove identity. Track the persisted index floor and only run the strict owner identity check once the current pass reaches that floor.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): drop the unused Digest import
* fix(replication): carry the compression layout through SSE-C passthrough (#7372)
* fix(replication): queue an in-flight version only once (#7376)
Multipart parts shared the 250 ms direct-PutObject wait on the foreground
write permit pool, so SDK-default concurrency (many parts per upload in
flight at once) was rejected wholesale with SlowDown at stock settings.
Keep the pool that bounds in-flight bodies, but let parts wait in a
bounded queue with their own timeout before body ingest, report the
queue depth in the ForegroundWrite admission snapshot, and document the
foreground write admission environment variables.
Commit 086ee8e48 moved authorize_recovery_admin_request above authorize_transition_admin_request and left the transition gate's doc comment attached to the recovery gate, which returns a hashed actor rather than the masked access key the comment describes. Move the comment back onto the transition gate and give the recovery gate its own accurate description. No behavior change.
* fix(scanner): require root publication proof before dirty ack
Bind ACK expectations to validated scan candidates and confirm the actual primary-root revision and readback. Retain saved outcomes and dirty responsibility when stronger evidence is unavailable. Isolate CAS attempt confirmation and invalidate proof after scope mutations.
Revalidate observed candidate reuse before issuing a new publication proof, preserve the exact validated authoritative baseline work digest, and settle fixture commit tails before stable maintenance scans. Keep scoped ACK production disabled.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix: restore ILM transition and lifecycle validation (#7316)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Run one supervised cleanup attempt for an existing operator reset, with
strict phase and revision checks under the original leader lock. Keep v3
reset authorization and responses unchanged, report deferred status, and
bound probe and shutdown waits without aborting in-flight reset ownership.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
唐小鸭