* fix(replication): deny non-owner replication config edits under site replication
Under site replication a user holding only bucket-scoped
s3:PutReplicationConfiguration could rewrite or erase the operator-managed
site-repl-* rules, with the change broadcast to every peer (backlog#1948,
audit A1/P2-17).
- Gate PutBucketReplication/DeleteBucketReplication in the S3 handlers:
when site replication is enabled and the requester is not the owner,
return MinIO-parity XMinioReplicationDenyEdit (HTTP 400). The gate runs
after policy authorization and only on the external S3 path; the
reconciler and peer bucket-meta ingestion are unaffected.
- Defense in depth in the bucket usecase: PUT merges the incoming config
with the stored site-repl-* rules (same merge as peer ingestion) instead
of overwriting verbatim; DELETE keeps the site-repl-* rules and never
garbage-collects a bucket target a surviving site-replication rule still
references.
- Move is_site_replication_rule / merge_incoming_replication_config /
replication_target_arn_deployment_id from the admin site-replication
handler down to rustfs-replication so the app layer can reuse them
without new layering violations.
* fix(replication): scope site-owned rule detection to reconciler-derived rules
The `site-repl-*` prefix alone classified any rule as site-owned, so on a
bucket outside site replication an owner's `site-repl-user` rule survived
DeleteBucketReplication (rule and target kept, success returned). Rule ids
do not reserve that namespace.
A rule is reconciler-owned only when it matches what the reconciler
derives: id `site-repl-<deployment id>` for a current remote site
replication peer and a destination ARN naming that same deployment id.
The S3 put/delete path reads the remote peer set (empty when site
replication is disabled) and keeps exactly those rules; everything else
is operator state the request replaces or deletes. An incoming rule that
claims a current peer's id is dropped so the reconciler rule's id stays
unique. The peer ingestion path and the reconciler keep their prefix
predicate unchanged.
* fix(replication): keep operator rule priorities across site rule merges
Merging stored site-replication rules into a PutBucketReplication body
renumbered every rule 1..n in list order, rewriting the submitted policy:
overlapping same-target rules submitted as priority 5 then 1 became 1
then 2, so the delete-marker-disabled rule won the replication decision.
The reconciler and the peer-removal prune renumbered the same way.
Operator priorities now stay verbatim everywhere; only the reconciler's
derived rules move, to the lowest priorities no operator rule uses, via
one pure helper shared by the S3 edit merge, the peer ingestion merge,
the reconciler pass and the prune. Being a pure function of the rule
list it is idempotent, so the reconciler's no-op check still holds after
a merged write, and an on-disk config in the historical layout (operator
rules 1..k, site rules k+1..n) yields the same bytes, so nothing is
rewritten on upgrade.
* fix(replication): pass site peer ids into the bucket usecase from the interface layer
The review fix made the bucket usecase read the site-replication peer set
through the admin handlers, an app->interface import the layer guard
rejects. The S3 handlers (interface) now read the peer set and pass it in,
so the usecase stays a pure function of its inputs; a state-read failure
still fails the edit closed, just one layer up.
* fix(replication): classify peer-ingested rules by the derived id/ARN contract
The peer ingestion merge still treated every incoming `site-repl-*` id as
reconciler-owned, so an owner-authored `site-repl-user` rule that the S3
merge now keeps on the editing site was dropped on every peer and the
sites persisted different operator configs.
The ingestion merge now classifies by the same derived contract as the
S3 merge: a rule is the reconciler's only when its `site-repl-<id>` names
the deployment its destination ARN targets and that deployment is a site
of the cluster (the receiver's own id included, since the sender's rule
towards the receiver names it). The reconciler, the peer-removal prune
and the target-online probe switch from the id prefix to the derived
shape as well, so the rule survives their passes too; rules in the
derived shape that name a removed peer or this site are still rebuilt
away.
Regression: a PutBucketReplication merged on site A and ingested on
site B keeps `site-repl-user` on both and the operator rule sets agree.
* fix(replication): keep an operator role target through site rule merges
The S3 and peer-ingestion merges cleared `Role` whenever it parsed as a
site-replication ARN, which an owner-submitted remote target with an
empty region (`arn:minio:replication::<id>:<bucket>`) also does. The
merged config then selected the rule destination ARNs instead of the
validated role target.
Only a role naming a current site of the cluster is the holder's
identity (the reconciler's per-peer target lookup reads it); every other
role passed target validation and stays. The reconciler's repair pass
applies the same rule.
Regression: an owner role target survives both merges and
`filter_target_arns` / `replication_target_arns` select it; a role naming
a current peer is still cleared.
* fix(replication): gate operator priority preservation on a peer contract probe
Keeping operator rule priorities verbatim is not rolling-upgrade safe: a
peer still running the pre-contract code renumbers every rule 1..n in
list order on ingest and on each reconciler pass, so an upgraded site
broadcasting `5,1` leaves that peer on `1,2` — which can select the
other overlapping rule — and the sites never reconverge.
Operator rules now merge under an explicit contract:
- `OperatorRuleContract::Derived`: site rules are the derived id/ARN
shape, operator priorities stay verbatim (the behavior of the previous
commits).
- `OperatorRuleContract::Legacy`: byte-for-byte what a pre-contract peer
does — `site-repl-*` ids are all site rules, a site-replication-shaped
`Role` is dropped, every rule is renumbered 1..n in list order. The S3
merge additionally lists the operator rules in priority order first,
so the renumbering keeps their relative order and the winning rule per
target is the one the operator submitted.
The S3 PutBucketReplication/DeleteBucketReplication path probes every
remote peer through the existing `peer/edit-capabilities` endpoint
(capability `derived-rule-contract`; pre-contract peers answer
`success:false` or 404) and merges under Derived only when every peer
supports it; any refusal or probe failure pins that edit to Legacy.
Every bucket-meta item this site sends (S3 hooks, bootstrap plan, retry
snapshots, tombstones) carries `derivedRuleContract: true`; a receiver
merges a payload without the marker the Legacy way, so an item from a
pre-contract sender is handled exactly as its own peers handle it.
Rolling upgrade: while any site runs the older code every edit is
canonicalized cluster-wide (numbers lost, order kept); once the last
site is upgraded the next edit keeps its priorities. Configs
canonicalized during the mixed period are not renumbered back — the
derived priority assignment is a no-op on the canonical layout — so an
operator who wants the original values re-submits the config after the
upgrade completes. Adding a site that runs the older code after
priorities were preserved is not gated and would desynchronize that
bucket until the next edit.
---------
Co-authored-by: houseme <housemecn@gmail.com>
* test(site-replication): pin MinIO IAMUserType wire semantics for policy mappings
Red tests for P0-4: MinIO peers send SRPolicyMapping.UserType using the
madmin IAMUserType table (unknown=-1, regUser=0, stsUser=1, svcUser=2),
while RustFS deserializes the field as u64 and decodes it with the
internal RPC table (None=0, Svc=1, Sts=2, Reg=3).
- userType -1 (MinIO group mappings) fails to deserialize, rejecting the
whole IAM item: group mappings never sync from MinIO.
- stsUser=1 decodes as Svc, landing federated STS mappings under the
wrong prefix and silently dropping their effect.
* fix(site-replication): translate policy mapping userType at MinIO wire boundary
SRPolicyMapping.userType travels on the wire using MinIO's IAMUserType
table (unknown=-1, regUser=0, stsUser=1, svcUser=2), but RustFS stored
the field as u64 and reused the internal RPC encoding
UserType::to_u64/from_u64 (None=0, Svc=1, Sts=2, Reg=3) at the site
replication boundary. Consequences: MinIO group mappings (userType -1)
failed to deserialize and the whole IAM item was rejected, and MinIO STS
mappings (1) were stored as service-account mappings, silently dropping
federated users' policies.
- Widen SRPolicyMapping.user_type and SRCredInfo.iam_user_type to i64 so
MinIO's -1 deserializes.
- Add sr_wire_user_type / user_type_from_sr_wire in rustfs-iam as the
dedicated SR wire codec: MinIO table on both directions, groups always
encoded as 0, and wire value 3 kept forever as an alias for Reg so
mappings from pre-fix RustFS peers still decode; unknown values fail
closed.
- Route the SR inbound (apply_iam_item) and outbound
(mapped_policy_to_sr_mapping, policy-mapping change hooks) paths
through the codec.
The internal UserType::to_u64/from_u64 encoding is untouched: it is the
intra-cluster node RPC contract and changing it would break rolling
restarts. Outbound compatibility with old RustFS peers is preserved
because UserType::None and Reg share the users prefix in
get_mapped_policy_path, so wire 0 lands in the same location Reg=3 did.
* test(site-replication): expect MinIO sts-account IAM item type
MinIO madmin-go replicates STS credentials with SRIAMItem type
"sts-account" (SRIAMItemSTSAcc), but RustFS emits and accepts only
"sts-credential", so cross-implementation STS replication fails in
both directions (MinIO returns errSRInvalidRequest, RustFS returns
NotImplemented).
Red-light tests:
- pin the outbound AssumeRole replication item type to "sts-account"
(construction extracted into assume_role_site_replication_item so it
is testable, behavior unchanged in this commit)
- update the federated identity replication item snapshot to
"sts-account"
- inbound apply_iam_item must dispatch both "sts-account" and the
legacy "sts-credential" alias to the STS arm instead of the
unknown-type NotImplemented fallback
* fix(site-replication): use MinIO-compatible sts-account IAM item type
MinIO madmin-go replicates STS credentials with SRIAMItem type
"sts-account" (SRIAMItemSTSAcc). RustFS emitted "sts-credential" and
accepted only that value inbound, so STS credential replication with
MinIO peers failed in both directions: MinIO rejected RustFS items as
errSRInvalidRequest and RustFS answered MinIO items with
NotImplemented.
- define SR_IAM_ITEM_STS_ACC ("sts-account") and
SR_IAM_ITEM_STS_ACC_LEGACY ("sts-credential") in rustfs-madmin
- emit "sts-account" from both outbound sites (AssumeRole hook and
federated identity OIDC hook)
- accept both types inbound; the legacy alias remains permanently for
mixed-version RustFS rolling upgrades
Token verification and the retry/event mechanism are unchanged.
* fix(site-replication): keep reverse direction after config broadcast
`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.
Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.
Four defects kept that invisible or unrecoverable:
- `update_all_targets` discarded target-client build errors silently, and
`replicate_object` logged the resulting missing-target drop at debug
while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
destination named a remote site, so two sites holding identical
configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
encrypted with a previous root secret decode as "no such account".
Startup now reconciles the account, reseeding from the secret every
site-replication bucket target already stores, and refuses the
delete-then-create sequence when the parent cannot back an account.
Bucket rules are reconciled too, so an already-broken site heals on
upgrade.
- A joined site never verified it could reach the initiator, whose
endpoint is derived from the Host header of the admin request that
created the topology. The join now probes each peer and reports through
`initial_sync_error_message`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): report unreachable targets and reconcile on a timer
Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.
Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.
The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(site-replication): invert reconcile dependency to satisfy layers
`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.
Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.
Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.
The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): harden reconcile per review feedback
Addresses the automated review on #5292.
Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.
Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.
Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.
Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.
Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
唐小鸭