mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-12 15:48:35 +00:00
498080dec4eea447259bb2b060f097ec6732be2d
22 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 唐小鸭
|
aeaba86d73 |
fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)
* fix(replication): total-order rule sort and honor V1 top-level Prefix Rule matching had two defects from the pre-GA replication audit (rustfs/backlog#2367 C-1 and C-2): - The actionable-rule sort compared same-destination rules by priority but answered Equal for any other pair, which is not a total order; the standard library sort panics on such comparators once a slice exceeds the insertion-sort threshold, so an object matching more than 20 enabled rules across two or more targets could panic the PUT or DELETE task. Rules now sort by priority descending with destination and id as tie-breakers, and filter_target_arns preserves that order instead of draining a HashSet. - A V1 rule written without a <Filter> carries its prefix at the top level; that field was never read, so <Prefix>logs/</Prefix> matched every object. ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping precedence. The existing prefix fixtures were built this way and had been asserting nothing. * fix(admin): advertise data-usage and listen capabilities to rc The rc client gated `rc du` and `rc watch` on a pinned contract that matched server versions by the string prefix `1.0.0-rc.`; a server that reports `1.0.0` no longer matches, and the dynamic `advertised` list did not carry either name, so `rc du` against a GA server fails with an unsupported-capability error (rustfs/backlog#2367 E-2). Advertise `admin.data-usage` from the admin route inventory like the IAM entries, and `listen_notification` for the bucket `?events=` extension route the admin router dispatches. The client merges advertised entries ahead of its pinned contract, so no version sniffing is needed. * fix(site-replication): stop notifying the local site on remove and rotate The pending-remove and pending-rotation notification loops skipped the local site by endpoint only, while finalization identifies it by deployment id or endpoint. The reconcile tick resolves the local peer from the node's own listen address (and a handler from the request Host), so `remove --all` dialed the site's registered endpoint, waited out the request timeout against the lifecycle lock it was holding, and answered `Partial: failed to notify 1 peer(s)` for a removal that had succeeded (rustfs/backlog#2367 A-4, backlog#2195 item 3). Both loops now iterate the peers still awaiting notification through one helper that applies the finalization identity. * fix(site-replication): promote and settle IAM retries without a tick of slack Two retry-queue behaviours kept an IAM change from converging for ten to twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3, backlog#2305): - The lightweight 30-second pass filtered its reachability probe to bucket ops, so a backed-off IAM or bucket-metadata snapshot waited for the 600-second tick to notice the peer. It now probes every backed-off class and still replays only bounded bucket ops; promotion is a state flip the heavyweight tick acts on. - Backoffs are multiples of the tick interval, so a failure stamped δ seconds after a tick was 600 − δ old at the next tick and slipped a whole extra interval. The heavyweight drain now evaluates backoff halfway to its next tick. - An IAM entry first created by a non-deletion failure (the add bootstrap's snapshot send, the drain's own replay, an import-iam schedule) was never stamped `deletions_recorded`, so a later recorded deletion could not settle it and it escalated to the marker only `replicate repair` clears. Entries created by this binary now start recorded; a row persisted by an older binary keeps the escalation semantics. * fix(site-replication): reload peer node caches after bucket wiring writes Every S3 bucket-config write ends by asking the other nodes of the cluster to reload the bucket's metadata; the site-replication writers never did. On a multi-node site the node that ran the pairing (or applied a peer's bucket-meta item) rewrote the bucket targets and the derived replication rules on disk, while every other node kept serving its cached copy for up to the 15-minute refresh. A `resync start` routed to such a node reported every freshly wired bucket as `Config not found` and a bucket whose operator target the pairing had replaced as `recorded remote target no longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional SITE-105). Add one best-effort reload helper in the site-replication hooks and call it after the bucket setup, versioning, peer bucket-meta apply, removed-peer cleanup, make-with-versioning, and endpoint-refresh writes; the ensure helpers now report whether they wrote so unchanged passes stay silent. The resync manifest and start now read the persisted wiring instead of the node-local cache, matching the target read the start path already did. The new four-node e2e pairs two clusters and starts a resync through a non-coordinator node right after pairing; it also covers an IAM user created on a non-coordinator node converging to the peer site. * test(e2e): cover delete-marker replication from a multi-node source The functional suite reported delete markers created on a 3-node source never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report was a probe defect, but the shape had no coverage: the existing delete-marker e2e runs a single-node source. Pin it against a four-node source replicating to a four-node peer and to a single-node target, with the write and the delete issued through different nodes. * ci(e2e): refresh the distributed selection for the new replication cases Four distributed cases were added (two site-replication, two delete-marker replication). The linux digest is derived from the last CI listing of the lane (34 cases, matching the previous pin) plus the four new names; the darwin digest is the local listing, which selects the same 38 cases. |
||
|
唐小鸭
|
46907c05cf |
fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. |
||
|
唐小鸭
|
899f81f3ad |
fix(replication): resolve drifted replicas via a target version ledger (#7368)
* fix(replication): resolve drifted replicas via a target version ledger A replication target that mints its own version ids (Wasabi, AWS S3) never answers to the source uuid, so every version-addressed mutation after the initial PUT failed forever: permanent version deletes answered NoSuchVersion every heal cycle, and tag / retention / legal-hold updates re-PUT the object, minting one more target version per update (rustfs/backlog#2340). Record the id the target assigned as a per-target ledger on the source version (replication-target-version-<arn>, written through the existing status writeback) and resolve every later mutation through it: version deletes DELETE the ledger id, metadata updates go through the metadata-only Object Lock and tagging APIs. Replicas written before the ledger existed are located by exact key and ETag, minus the candidates other generations of the key already claim through their own ledgers; an ambiguous remainder is refused with a backoff instead of guessed, since a wrong pick would destroy a live generation. A fresh write never consults content identity. NoSuchVersion on a version-addressed DELETE counts as purged. The fake target gains the Wasabi shape (404 NoSuchVersion on an unknown id, per-version Object Lock APIs) and the matrix covers the three mutation classes plus the same-bytes generation case. * fix(scanner): drop the unused Digest import Same one-line change as rustfs/rustfs#7366 (main is red with it under -D warnings); carried here so the stacked PRs' merge commits compile until that fix lands. * fix(admin): probe replication-check mutations by the assigned version id (#7373) On a target that mints its own version ids the DeleteMarker and VersionDelete phases of ?replication-check were skipped: they addressed the source id, which such a target never had. The replication worker now addresses the id the target assigned (the target-version ledger), and the probe already holds that id from its own PUT, so run both phases against it. VersionFidelity keeps failing with the mismatch code and the target stays FAILED; the phases report whether ledger-addressed purges work against this endpoint (rustfs/backlog#2340). * fix(replication): abandon purges to targets the bucket no longer names (#7377) * fix(admin): probe replication-check mutations by the assigned version id On a target that mints its own version ids the DeleteMarker and VersionDelete phases of ?replication-check were skipped: they addressed the source id, which such a target never had. The replication worker now addresses the id the target assigned (the target-version ledger), and the probe already holds that id from its own PUT, so run both phases against it. VersionFidelity keeps failing with the mismatch code and the target stays FAILED; the phases report whether ledger-addressed purges work against this endpoint (rustfs/backlog#2340). * fix(replication): abandon purges to targets the bucket no longer names A permanent version delete whose replication keeps failing stays in xl.meta as a PENDING purge, hidden from listings, until every target confirms it. Once the operator removes the replication configuration or the rule naming that target nothing ever confirms it: the heal path derived its delete decision from the configuration (the decision string is not persisted) and skipped the version forever, so DeleteBucket answered BucketNotEmpty for a residue the client could neither list nor remove (rustfs/backlog#2340). Owe a version purge to the targets its purge state names, let the heal path through without a configuration, and have the delete worker settle a target the configuration no longer names as abandoned: the purge is reported complete locally through the normal writeback, the replica on the former target is left alone, and the event replication_purge_abandoned plus a counter are the record. * fix(admin): send replication-check marker creation without a version id Running the DeleteMarker / VersionDelete phases on a target that mints its own version ids exposed two probe-shape bugs on real Wasabi: - the DeleteMarker phase put the assigned version id on its DELETE. A RustFS peer reads the source-deletemarker header and creates a marker, but a generic S3 target executes it as a permanent delete of the probe version, so VersionDelete then answered NoSuchVersion. Use the same wire shape as live delete replication: no versionId on a marker creation. - cleanup treated NoSuchVersion on the version the VersionDelete phase had already removed as a failure (RustFS/MinIO answer 204 there). Also gate the no-configuration heal pass-through for pending purges on a purge state that actually names targets, so a purge without a recorded target keeps the ordinary skip (scanner unit test), and merge origin/main (#7365 settles the pool-metadata probe test that failed in CI). --------- Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Zhengchao An
|
7df0920c80 | test(replication): bind writable paths to DTO fields (#6923) | ||
|
cxymds
|
0c18012442 | fix(admin): version remote target credential capabilities (#6876) | ||
|
cxymds
|
1e8c8d4cd5 | feat(replication): support temporary target credentials (#6860) | ||
|
cxymds
|
64cca79fbb | feat(admin): expose remote target credential capability state (#6857) | ||
|
唐小鸭
|
4ddc728c9d |
fix(replication): deny non-owner replication config edits under site replication (#6375)
* fix(replication): deny non-owner replication config edits under site replication Under site replication a user holding only bucket-scoped s3:PutReplicationConfiguration could rewrite or erase the operator-managed site-repl-* rules, with the change broadcast to every peer (backlog#1948, audit A1/P2-17). - Gate PutBucketReplication/DeleteBucketReplication in the S3 handlers: when site replication is enabled and the requester is not the owner, return MinIO-parity XMinioReplicationDenyEdit (HTTP 400). The gate runs after policy authorization and only on the external S3 path; the reconciler and peer bucket-meta ingestion are unaffected. - Defense in depth in the bucket usecase: PUT merges the incoming config with the stored site-repl-* rules (same merge as peer ingestion) instead of overwriting verbatim; DELETE keeps the site-repl-* rules and never garbage-collects a bucket target a surviving site-replication rule still references. - Move is_site_replication_rule / merge_incoming_replication_config / replication_target_arn_deployment_id from the admin site-replication handler down to rustfs-replication so the app layer can reuse them without new layering violations. * fix(replication): scope site-owned rule detection to reconciler-derived rules The `site-repl-*` prefix alone classified any rule as site-owned, so on a bucket outside site replication an owner's `site-repl-user` rule survived DeleteBucketReplication (rule and target kept, success returned). Rule ids do not reserve that namespace. A rule is reconciler-owned only when it matches what the reconciler derives: id `site-repl-<deployment id>` for a current remote site replication peer and a destination ARN naming that same deployment id. The S3 put/delete path reads the remote peer set (empty when site replication is disabled) and keeps exactly those rules; everything else is operator state the request replaces or deletes. An incoming rule that claims a current peer's id is dropped so the reconciler rule's id stays unique. The peer ingestion path and the reconciler keep their prefix predicate unchanged. * fix(replication): keep operator rule priorities across site rule merges Merging stored site-replication rules into a PutBucketReplication body renumbered every rule 1..n in list order, rewriting the submitted policy: overlapping same-target rules submitted as priority 5 then 1 became 1 then 2, so the delete-marker-disabled rule won the replication decision. The reconciler and the peer-removal prune renumbered the same way. Operator priorities now stay verbatim everywhere; only the reconciler's derived rules move, to the lowest priorities no operator rule uses, via one pure helper shared by the S3 edit merge, the peer ingestion merge, the reconciler pass and the prune. Being a pure function of the rule list it is idempotent, so the reconciler's no-op check still holds after a merged write, and an on-disk config in the historical layout (operator rules 1..k, site rules k+1..n) yields the same bytes, so nothing is rewritten on upgrade. * fix(replication): pass site peer ids into the bucket usecase from the interface layer The review fix made the bucket usecase read the site-replication peer set through the admin handlers, an app->interface import the layer guard rejects. The S3 handlers (interface) now read the peer set and pass it in, so the usecase stays a pure function of its inputs; a state-read failure still fails the edit closed, just one layer up. * fix(replication): classify peer-ingested rules by the derived id/ARN contract The peer ingestion merge still treated every incoming `site-repl-*` id as reconciler-owned, so an owner-authored `site-repl-user` rule that the S3 merge now keeps on the editing site was dropped on every peer and the sites persisted different operator configs. The ingestion merge now classifies by the same derived contract as the S3 merge: a rule is the reconciler's only when its `site-repl-<id>` names the deployment its destination ARN targets and that deployment is a site of the cluster (the receiver's own id included, since the sender's rule towards the receiver names it). The reconciler, the peer-removal prune and the target-online probe switch from the id prefix to the derived shape as well, so the rule survives their passes too; rules in the derived shape that name a removed peer or this site are still rebuilt away. Regression: a PutBucketReplication merged on site A and ingested on site B keeps `site-repl-user` on both and the operator rule sets agree. * fix(replication): keep an operator role target through site rule merges The S3 and peer-ingestion merges cleared `Role` whenever it parsed as a site-replication ARN, which an owner-submitted remote target with an empty region (`arn:minio:replication::<id>:<bucket>`) also does. The merged config then selected the rule destination ARNs instead of the validated role target. Only a role naming a current site of the cluster is the holder's identity (the reconciler's per-peer target lookup reads it); every other role passed target validation and stays. The reconciler's repair pass applies the same rule. Regression: an owner role target survives both merges and `filter_target_arns` / `replication_target_arns` select it; a role naming a current peer is still cleared. * fix(replication): gate operator priority preservation on a peer contract probe Keeping operator rule priorities verbatim is not rolling-upgrade safe: a peer still running the pre-contract code renumbers every rule 1..n in list order on ingest and on each reconciler pass, so an upgraded site broadcasting `5,1` leaves that peer on `1,2` — which can select the other overlapping rule — and the sites never reconverge. Operator rules now merge under an explicit contract: - `OperatorRuleContract::Derived`: site rules are the derived id/ARN shape, operator priorities stay verbatim (the behavior of the previous commits). - `OperatorRuleContract::Legacy`: byte-for-byte what a pre-contract peer does — `site-repl-*` ids are all site rules, a site-replication-shaped `Role` is dropped, every rule is renumbered 1..n in list order. The S3 merge additionally lists the operator rules in priority order first, so the renumbering keeps their relative order and the winning rule per target is the one the operator submitted. The S3 PutBucketReplication/DeleteBucketReplication path probes every remote peer through the existing `peer/edit-capabilities` endpoint (capability `derived-rule-contract`; pre-contract peers answer `success:false` or 404) and merges under Derived only when every peer supports it; any refusal or probe failure pins that edit to Legacy. Every bucket-meta item this site sends (S3 hooks, bootstrap plan, retry snapshots, tombstones) carries `derivedRuleContract: true`; a receiver merges a payload without the marker the Legacy way, so an item from a pre-contract sender is handled exactly as its own peers handle it. Rolling upgrade: while any site runs the older code every edit is canonicalized cluster-wide (numbers lost, order kept); once the last site is upgraded the next edit keeps its priorities. Configs canonicalized during the mixed period are not renumbered back — the derived priority assignment is a no-op on the canonical layout — so an operator who wants the original values re-submits the config after the upgrade completes. Adding a site that runs the older code after priorities were preserved is not gated and would desynchronize that bucket until the next edit. --------- Co-authored-by: houseme <housemecn@gmail.com> |
||
|
唐小鸭
|
a930152d5a |
fix(admin): expose per-target disableProxy through remote target admin API (#6376)
The read-proxy selector already honors a target's disable_proxy flag (PR #6172), but the admin API still rejected the field, so the only way to set it was importing a MinIO-written bucket-targets.json. - move disableProxy from REMOTE_TARGET_UNSUPPORTED_FIELDS to REMOTE_TARGET_WRITABLE_FIELDS (set-remote-target create accepts it) - add TargetUpdateOp::Proxy so set-remote-target?update=true&proxy=true overlays only the proxy group (MinIO TargetUpdateType parity) - bump REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION 1 -> 2 and update the runtime capability pin tests - keep edge/edgeSyncBeforeExpiry rejected (no implementation behind them) - pin that a published TargetClient carries disable_proxy, the field the proxy-target selector consults Refs rustfs/backlog#1950 |
||
|
Zhengchao An
|
a07ad4a9ff | test(replication): cover rule id byte limit (#5873) | ||
|
唐小鸭
|
3792fed827 |
fix(replication): madmin reset/diff wire compat and config validation (#5799)
* fix(admin): align replication-reset responses with madmin ResyncTargetsInfo shape
The replication-reset and replication-reset-status responses serialized
their shell as "Targets" and per-target fields in PascalCase, while
madmin-go ResyncTargetsInfo/ResyncTarget expect the "target" shell key
and lowercase field tags (arn/resetid/resyncStatus/replicationCount/
completedReplicationSize/failedReplicationCount/failedReplicationSize).
Go json decoding is case-insensitive per field, but Targets vs target,
Status vs resyncStatus and the size/count key names cannot match, so
mc replicate resync decoded empty results.
Rename the serde tags to the exact madmin wire shape, keep the
ResetBeforeDate/Error RustFS extension keys (unknown keys are ignored
by Go decoders), pin the shape with a snapshot unit test, and update
the e2e client DTO to decode the madmin shape.
* fix(admin): stream bare madmin DiffInfo documents from replication diff
POST /v3/replication/diff returned a single enveloped object
({Entries, IsTruncated, ScannedVersions}) while madmin-go
BucketReplicationDiff decodes the body with a json.Decoder loop over
bare DiffInfo documents. The envelope decoded as exactly one DiffInfo
with an empty object, so mc replicate diff printed a phantom empty row
instead of the real backlog.
Emit one DiffInfo JSON document per line by default, using the exact
madmin json tags (object/versionId/rStatus/deletemarker/lastModified;
Size stays as a RustFS extension key that Go decoders ignore). The
enveloped shape moves to the opt-in ?aggregate=true RustFS extension,
which remains the only carrier of scan-coverage metadata; a truncated
default-mode scan is surfaced via a warn tracing event instead of
in-stream. Pin both shapes with unit tests and tighten the e2e helper
to reject any envelope in the stream.
* feat(replication): validate replication config structure before persisting
PutBucketReplication accepted structurally invalid configurations that
MinIO's replication.Config.Validate rejects: empty or oversized rule
lists, duplicate or negative rule priorities, over-long rule IDs,
filters carrying more than one of Prefix/Tag/And, and delete marker
replication enabled on tag-filtered rules. Such configs persisted
silently and later produced undefined routing (e.g. ambiguous priority
ties) instead of failing the PUT.
Add validate_replication_config_structure as a pure function in
rustfs-replication (limits documented as constants), surface it through
the ecstore api facade, and run it first in the PUT capability gate so
defects are named before any metadata write. Missing Priority counts as
zero for the uniqueness check, matching Go's zero-value semantics. The
self-target rejection deliberately stays at set-remote-target, where the
endpoint is known; a config can never reference a self-pointing ARN.
Document the rule-level Destination.StorageClass contract (use the
remote target's storage_class instead) and renumber the acceptance
matrix e2e to unique priorities, which MinIO would also require.
* test(replication): pin duplicated wire types with boundary reconciliation tests
rustfs-filemeta (xl.meta disk format) and rustfs-replication (MRF/resync
persistence format) deliberately each own ReplicationStatusType,
VersionPurgeStatusType and ReplicationState; the boundary converts
between them via as_str(), whose From<&str> impls fall back to Empty on
unknown tokens — a variant added on one side silently degrades to Empty
on the other.
Add reconciliation tests in replication_filemeta_boundary: exhaustive
matches with no wildcard arm on both sides of both enums (a new variant
fails compilation until the mapping is reconsidered), string-token
round-trip asserts (a token the other side does not recognize fails
instead of quietly becoming Empty), and a full-field ReplicationState
round-trip. Cross-reference the tests from both type definitions.
Struct drift was already compile-guarded by the exhaustive struct
literals in the conversion functions.
* docs(replication): define split completion criteria and milestone sequence
The ecstore replication split plan had no completion measure — the
boundary scaffolding risked ossifying because nothing said when the
migration counts as done. Record the criteria in the module inventory:
done means the Required Contracts table's 'Current dependency to
remove' column is empty; the end state moves pool/resyncer/state into
crates/replication, with the boundary micro-files dissolving as code
crosses the crate line (batch-merging them beforehand is explicitly
rejected — the guard scripts anchor on their file names, so merging is
churn with zero functional gain; only datatypes.rs can retire early).
Sequence the remaining work as M2 (resyncer pure decision logic, after
the oversized function splits) → M3 (worker runtime, highest risk,
last) → M4 (retire boundaries and guard entries). Refresh the stale
first-step text — the event sink / runtime contracts already landed —
and update the split-plan status table accordingly.
* fix(replication): align structural validator with MinIO semantics after adversarial review
Three interop corrections found by adversarial review of the new
structural validator, plus review fallout fixes:
- Delete-marker replication is now rejected only for a direct Filter.Tag,
not for tags inside Filter.And — MinIO's validator only inspects the
direct tag, and mc replicate add --tags "k1=v1&k2=v2" (delete-marker
replication on by default) puts multiple tags into And.Tags, so the
stricter check rejected mc-generated configs MinIO accepts.
- Rule ID length is measured in bytes (Go len semantics), not chars —
a 255-char multibyte ID must not round-trip into a config MinIO
rejects.
- An empty <Tag/> element (no key) counts as absent, matching MinIO's
Tag.IsEmpty(); console form serializers emit empty tags, which would
otherwise trip the exactly-one-of and delete-marker checks.
Also: repair the store-uninitialized PUT test whose empty-rules fixture
now (correctly) fails structural validation before reaching the store
lookup; pin the previously untested startTime madmin key in the
reset-status shape test; and signal a truncated default-mode diff scan
via the x-rustfs-replication-diff-truncated response header — the bare
madmin stream has no envelope, so a truncated scan was otherwise
indistinguishable from a complete healthy one (madmin/mc ignore unknown
headers).
* test(e2e): activate SSE-S3 replication contract and pin resync fail-closed path
The SSE-S3 replication contract e2e was ignored under backlog#1291
(silent plaintext replication); the fail-closed gate in
replication_target_boundary.rs closed that hole, so the ignore reason
expired. Un-ignore the test — it now pins the current fail-closed
contract (FAILED status, failure event, readable encrypted source,
stable absence of all target versions), verified green.
Add test_bucket_replication_sse_s3_resync_stays_fail_closed: drives the
existing-object resync path (PUT ?replication-reset) over a FAILED
SSE-S3 object and asserts the resync generation reaches a terminal
state without ever materializing a target version, with the
stays-absent window also spanning fast-scanner heal cycles. The new
start_bucket_replication_reset helper doubles as the madmin
ResyncTargetsInfo shape assertion (target[0].arn/resetid) for the
reset-start response.
Refresh the stale nextest count commentary (the module is at 20 fast +
36 nightly = 56 tests by cargo nextest list; the SSE-S3-ignored note no
longer holds).
|
||
|
唐小鸭
|
733c7b0f67 |
fix(replication): accept remote target healthCheckDuration nanoseconds (#5754)
* test(replication): accept madmin nanosecond healthCheckDuration payloads Red-phase TDD tests for P0-7: mc 'replicate add' sends the madmin default healthCheckDuration=60s as a Go time.Duration nanosecond integer (60000000000), which RustFS currently rejects as an unsupported field and would misread as seconds. Also pins the defensive seconds-or-nanos read for persisted bucket-targets metadata and the capability contract listing healthCheckDuration as writable. Currently failing (red): - remote_target_request_accepts_go_duration_wire_values - remote_target_request_accepts_legacy_seconds_health_check - remote_target_health_check_duration_is_declared_writable - bucket_target_reads_go_nanosecond_durations_defensively - runtime_capabilities_response_reports_missing_topology_before_storage_init * fix(replication): accept remote target healthCheckDuration nanoseconds mc 'replicate add' always sends the madmin default healthcheck-seconds=60 serialized as a Go time.Duration nanosecond integer (60000000000), so the default mc link-creation path (and 'mc replicate update') failed with InvalidRequest. Move healthCheckDuration from the unsupported to the writable remote-target field list; the capability contract in the runtime capabilities response follows the constants automatically. Fix the unit mismatch in both directions: - Request parsing and persisted bucket-targets reads decode the value defensively: below 10^7 it is legacy RustFS seconds, otherwise Go time.Duration nanoseconds (also covers MinIO-written metadata). totalDowntime shares the same wire shape and gets the same handling. - The list-remote-targets admin response re-encodes only these two fields as nanoseconds via a dedicated serialization path, leaving the persisted seconds-based wire format untouched for existing readers. The per-target health-check interval is accepted for mc compatibility but not yet applied; the heartbeat keeps its global env-configured interval, and the explicit 'healthcheck' update op stays rejected. disableProxy, edge, and edgeSyncBeforeExpiry remain explicitly rejected. |
||
|
唐小鸭
|
15b9c1f4e3 |
fix(replication): make bucket replication rules editable from clients (#5715)
* fix(replication): accept explicit STANDARD destination storage class The replication engine never reads Rule.Destination.StorageClass (replica placement comes from the bucket-target config or the source object), yet the validator rejected any config carrying the field. The console's add-rule form always sends StorageClass=STANDARD, so every rule created through it failed with InvalidRequest. Tolerate exactly STANDARD as a no-op — semantically identical to omitting the field — and keep rejecting every other value, which would be silently ignored rather than honored. Document the deliberate omission from the replication capability contract. * feat(admin): support MinIO-style partial updates for set-remote-target set-remote-target?update=true previously replaced every stored field and required complete credentials in the body, so flipping a target's sync mode from the console forced operators to re-enter the secret key, and real mc replicate update bodies (madmin Clone() strips the secret) failed to deserialize at all. Adopt MinIO's TargetUpdateType contract: query params creds/sync/bandwidth/ path name the field groups to overlay onto the stored target, everything else keeps its persisted value, and unsupported groups (proxy, healthcheck, edge, edgeSyncBeforeExpiry) fail loudly. Credentials updates are skipped for site-replication peer targets — probed by both scheme derivations of the stored endpoint and the stored deployment id — because an operator never knows the site replicator's credentials, and a body-supplied deployment id is ignored on update since it anchors peer identity. madmin JSON aliases (bandwidthlimit, storageclass, resetID, deploymentID, sessionToken) let mc bodies parse under deny_unknown_fields. e2e: cover a credential-free sync-only update preserving the stored connection and the zero-ops no-op contract; align the missing-arn assertion with the earlier validation error. * chore(scripts): add two-site replication lab manager site_replication_smoke.py spawns and manages two local rustfs processes, pairs them via the site-replication admin API (idempotent), and verifies bidirectional object replication. Subcommands: up/down/restart/status/logs/ smoke/info/remove/clean. Stdlib-only; requests are SigV4-signed the same way as crates/e2e_test. * chore(scripts): rename direction-suffixed payload variables for typos check The typos linter reads the _ba suffix in payload_ba as a misspelling of "by"; use payload_a_to_b / payload_b_to_a instead. --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
cxymds
|
380ec74ece |
fix(replication): persist force-delete handoff state (#5641)
* fix(replication): persist force-delete handoff state * fix(arch): route force-delete config access through boundary * style: format force-delete imports --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
cxymds
|
1fdcbd9225 |
fix(replication): fail closed on destination encryption (#5633)
* fix(replication): fail closed on destination encryption * test(replication): avoid Debug bound in encryption assertion |
||
|
cxymds
|
114b2420a2 |
feat(admin): expose versioned replication capabilities (#5631)
* feat(admin): expose replication capabilities * fix(admin): route replication capabilities through facades |
||
|
cxymds
|
378c9ba67f | fix(replication): enforce bucket write contract (#5629) | ||
|
cxymds
|
c1955a8498 | fix(replication): harden live delete admission (#5599) | ||
|
唐小鸭
|
c8016cbcdb |
fix(replication): enforce bucket replication switches (#5449)
* fix(replication): enforce bucket replication switches * fix(replication): satisfy delete admission clippy lint * fix(replication): restore MinIO tag filter behavior --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
Zhengchao An
|
819e1422fb | fix(replication): resolve same-destination rules highest-priority first (#4452) | ||
|
Zhengchao An
|
b972e25c51 | refactor(replication): move app decisions into crate (#4191) | ||
|
Zhengchao An
|
b57820a486 | refactor(replication): move config contracts into crate (#4162) |