The site replication add preflight probed peer sites serially while
holding the process-wide lifecycle lock, so k unreachable sites held the
lock for k peer-request timeouts, and every concurrent
add/remove/refresh waited on an unbounded lock acquire for the whole
time. Probe all sites concurrently (matching the file's other peer
fan-outs) so k unreachable sites cost roughly one timeout, and bound the
lifecycle lock acquire at 30s, returning a retryable 503 to waiters
instead of hanging indefinitely.
Regression tests pin the preflight fan-out concurrency, the bounded
acquire's 503, and the 10s/3s peer client timeout constants.
Refs rustfs/backlog#1952, rustfs/backlog#1946, rustfs/backlog#1889
Co-authored-by: houseme <housemecn@gmail.com>
* fix(replication): deny non-owner replication config edits under site replication
Under site replication a user holding only bucket-scoped
s3:PutReplicationConfiguration could rewrite or erase the operator-managed
site-repl-* rules, with the change broadcast to every peer (backlog#1948,
audit A1/P2-17).
- Gate PutBucketReplication/DeleteBucketReplication in the S3 handlers:
when site replication is enabled and the requester is not the owner,
return MinIO-parity XMinioReplicationDenyEdit (HTTP 400). The gate runs
after policy authorization and only on the external S3 path; the
reconciler and peer bucket-meta ingestion are unaffected.
- Defense in depth in the bucket usecase: PUT merges the incoming config
with the stored site-repl-* rules (same merge as peer ingestion) instead
of overwriting verbatim; DELETE keeps the site-repl-* rules and never
garbage-collects a bucket target a surviving site-replication rule still
references.
- Move is_site_replication_rule / merge_incoming_replication_config /
replication_target_arn_deployment_id from the admin site-replication
handler down to rustfs-replication so the app layer can reuse them
without new layering violations.
* fix(replication): scope site-owned rule detection to reconciler-derived rules
The `site-repl-*` prefix alone classified any rule as site-owned, so on a
bucket outside site replication an owner's `site-repl-user` rule survived
DeleteBucketReplication (rule and target kept, success returned). Rule ids
do not reserve that namespace.
A rule is reconciler-owned only when it matches what the reconciler
derives: id `site-repl-<deployment id>` for a current remote site
replication peer and a destination ARN naming that same deployment id.
The S3 put/delete path reads the remote peer set (empty when site
replication is disabled) and keeps exactly those rules; everything else
is operator state the request replaces or deletes. An incoming rule that
claims a current peer's id is dropped so the reconciler rule's id stays
unique. The peer ingestion path and the reconciler keep their prefix
predicate unchanged.
* fix(replication): keep operator rule priorities across site rule merges
Merging stored site-replication rules into a PutBucketReplication body
renumbered every rule 1..n in list order, rewriting the submitted policy:
overlapping same-target rules submitted as priority 5 then 1 became 1
then 2, so the delete-marker-disabled rule won the replication decision.
The reconciler and the peer-removal prune renumbered the same way.
Operator priorities now stay verbatim everywhere; only the reconciler's
derived rules move, to the lowest priorities no operator rule uses, via
one pure helper shared by the S3 edit merge, the peer ingestion merge,
the reconciler pass and the prune. Being a pure function of the rule
list it is idempotent, so the reconciler's no-op check still holds after
a merged write, and an on-disk config in the historical layout (operator
rules 1..k, site rules k+1..n) yields the same bytes, so nothing is
rewritten on upgrade.
* fix(replication): pass site peer ids into the bucket usecase from the interface layer
The review fix made the bucket usecase read the site-replication peer set
through the admin handlers, an app->interface import the layer guard
rejects. The S3 handlers (interface) now read the peer set and pass it in,
so the usecase stays a pure function of its inputs; a state-read failure
still fails the edit closed, just one layer up.
* fix(replication): classify peer-ingested rules by the derived id/ARN contract
The peer ingestion merge still treated every incoming `site-repl-*` id as
reconciler-owned, so an owner-authored `site-repl-user` rule that the S3
merge now keeps on the editing site was dropped on every peer and the
sites persisted different operator configs.
The ingestion merge now classifies by the same derived contract as the
S3 merge: a rule is the reconciler's only when its `site-repl-<id>` names
the deployment its destination ARN targets and that deployment is a site
of the cluster (the receiver's own id included, since the sender's rule
towards the receiver names it). The reconciler, the peer-removal prune
and the target-online probe switch from the id prefix to the derived
shape as well, so the rule survives their passes too; rules in the
derived shape that name a removed peer or this site are still rebuilt
away.
Regression: a PutBucketReplication merged on site A and ingested on
site B keeps `site-repl-user` on both and the operator rule sets agree.
* fix(replication): keep an operator role target through site rule merges
The S3 and peer-ingestion merges cleared `Role` whenever it parsed as a
site-replication ARN, which an owner-submitted remote target with an
empty region (`arn:minio:replication::<id>:<bucket>`) also does. The
merged config then selected the rule destination ARNs instead of the
validated role target.
Only a role naming a current site of the cluster is the holder's
identity (the reconciler's per-peer target lookup reads it); every other
role passed target validation and stays. The reconciler's repair pass
applies the same rule.
Regression: an owner role target survives both merges and
`filter_target_arns` / `replication_target_arns` select it; a role naming
a current peer is still cleared.
* fix(replication): gate operator priority preservation on a peer contract probe
Keeping operator rule priorities verbatim is not rolling-upgrade safe: a
peer still running the pre-contract code renumbers every rule 1..n in
list order on ingest and on each reconciler pass, so an upgraded site
broadcasting `5,1` leaves that peer on `1,2` — which can select the
other overlapping rule — and the sites never reconverge.
Operator rules now merge under an explicit contract:
- `OperatorRuleContract::Derived`: site rules are the derived id/ARN
shape, operator priorities stay verbatim (the behavior of the previous
commits).
- `OperatorRuleContract::Legacy`: byte-for-byte what a pre-contract peer
does — `site-repl-*` ids are all site rules, a site-replication-shaped
`Role` is dropped, every rule is renumbered 1..n in list order. The S3
merge additionally lists the operator rules in priority order first,
so the renumbering keeps their relative order and the winning rule per
target is the one the operator submitted.
The S3 PutBucketReplication/DeleteBucketReplication path probes every
remote peer through the existing `peer/edit-capabilities` endpoint
(capability `derived-rule-contract`; pre-contract peers answer
`success:false` or 404) and merges under Derived only when every peer
supports it; any refusal or probe failure pins that edit to Legacy.
Every bucket-meta item this site sends (S3 hooks, bootstrap plan, retry
snapshots, tombstones) carries `derivedRuleContract: true`; a receiver
merges a payload without the marker the Legacy way, so an item from a
pre-contract sender is handled exactly as its own peers handle it.
Rolling upgrade: while any site runs the older code every edit is
canonicalized cluster-wide (numbers lost, order kept); once the last
site is upgraded the next edit keeps its priorities. Configs
canonicalized during the mixed period are not renumbered back — the
derived priority assignment is a no-op on the canonical layout — so an
operator who wants the original values re-submits the config after the
upgrade completes. Adding a site that runs the older code after
priorities were preserved is not gated and would desynchronize that
bucket until the next edit.
---------
Co-authored-by: houseme <housemecn@gmail.com>
The peer reload_pool_meta handler blindly replaced in-memory pool
metadata with the persisted snapshot, so a delayed or out-of-order
reload could roll back newer local queued/canceled/failed/complete
decommission state, and a missing pool.bin wiped local state to an
empty default.
Route peer reload through the same monotonic merge used by the admin
status refresh (merge_pool_status_refresh): entries are replaced only
when strictly newer and no local worker is active; missing snapshots
fail closed. The helper now reports whether any entry was replaced or
appended, and rejected stale/missing reloads are logged. The RPC
handler spawns missing decommission workers only after a reload
actually merged newer state, so duplicate deliveries cannot start
workers for an older generation.
Fixesrustfs/backlog#1917
* feat(allocator): replace mimalloc/libmimalloc-sys with rustfs-mimalloc/rustfs-mimalloc-sys
Replace the upstream xonatius/mimalloc_rust.git fork (mimalloc + libmimalloc-sys)
with the published rustfs-mimalloc (v0.5.0) and rustfs-mimalloc-sys (v0.5.0) crates
from crates.io.
The new crates are based on mimalloc V3 (v3.5.0) and provide:
- MiMalloc global allocator with safe API (collect, stats_json, process_info)
- Heap management and arena operations (heap module)
- Full FFI bindings to mimalloc V3
Changes:
- Workspace deps: mimalloc + libmimalloc-sys (git) → rustfs-mimalloc + rustfs-mimalloc-sys (crates.io)
- allocator_reclaim.rs: libmimalloc_sys::mi_collect → rustfs_mimalloc::MiMalloc::collect
- memory_observability.rs: raw FFI mi_stats_get_json → MiMalloc::stats_json()
- main.rs: heap ownership tests use Heap::contains() (V3 API)
- deny.toml: remove xonatius/mimalloc_rust.git from allow-git
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): default rename fanout to parallel early-ack path
Switch the default rename_data commit fanout from serial join_all to the
parallel JoinSet early-ack path. The serial path (#5987) was the primary
cause of the 1MiB PUT regression (-71.7%) observed in rc.3 benchmarks.
A/B verification on testing 4-node cluster (c=64, 1MiB PUT, 2min):
- Serial (join_all): 96.99 MiB/s, P50=644ms
- Early ack (JoinSet): 177.46 MiB/s, P50=407ms (+83%)
Also:
- Update rename_data_reclaims_synthetic_inline_rollback_dir_after_commit
to use rename_data_owned and await tail_drain for proper cleanup.
- Update rename_data_waits_for_tail_disk_after_write_quorum to explicitly
test the serial path (now non-default) via env override.
- Add error source chain to HTTP Body stream transport error log
(backlog#2005) so the underlying cause is visible.
Ref: rustfs/backlog#2005
Ref: rustfs/backlog#1792#issuecomment-5384346238
Ref: rustfs/backlog#1792#issuecomment-5384370938
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Change DEFAULT_FSYNC_BLOCKING_THREADS from 0 to 64 to isolate
fsync/fdatasync operations into a dedicated blocking thread pool.
A/B validation on 4-node EC cluster (testing, 10.0.0.5/8/9/11:9000):
PUT 256KiB c16: p99 226ms → 135ms (−40%), p50 60ms → 19ms (−68%)
GET 256KiB c16: p99 3.97ms → 3.63ms (−9%), throughput +1.8%
GET 4KiB c64: neutral (pure read, no fsync involvement)
Without isolation, fsync operations contend with read I/O (pread/stat/open)
on the main blocking pool, causing device-bound fsync to starve read
operations under mixed PUT+GET workloads.
Co-authored-by: heihutu <heihutu@gmail.com>
SSE and compression wrap the payload so its length is unknown and
advertise HashReader::SIZE_PRESERVE_LAYER (-1). Every layer preserved
that sentinel except create_bitrot_writer, which clamped it to 0 before
calling DiskAPI::create_file. RemoteDisk forwards that size verbatim in
the put_file_stream query, so remote peers were told the body was empty.
Since the authenticated put-file trailer (#5868) the receiver used the
declared size to split body from trailer, turning the clamp into a fatal
"auth trailer has trailing data" failure for every SSE PUT on multi-node
deployments (rc.2). #6320 relaxed the receiver to only trust size > 0;
this change fixes the sender so the sentinel survives end to end and the
wire no longer conflates empty objects with unknown-length streams.
Refs #6331
Replace the upstream xonatius/mimalloc_rust.git fork (mimalloc + libmimalloc-sys)
with the published rustfs-mimalloc (v0.5.0) and rustfs-mimalloc-sys (v0.5.0) crates
from crates.io.
The new crates are based on mimalloc V3 (v3.5.0) and provide:
- MiMalloc global allocator with safe API (collect, stats_json, process_info)
- Heap management and arena operations (heap module)
- Full FFI bindings to mimalloc V3
Changes:
- Workspace deps: mimalloc + libmimalloc-sys (git) → rustfs-mimalloc + rustfs-mimalloc-sys (crates.io)
- allocator_reclaim.rs: libmimalloc_sys::mi_collect → rustfs_mimalloc::MiMalloc::collect
- memory_observability.rs: raw FFI mi_stats_get_json → MiMalloc::stats_json()
- main.rs: heap ownership tests use Heap::contains() (V3 API)
- deny.toml: remove xonatius/mimalloc_rust.git from allow-git
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(admin): send versioningEnabled on site replication make-bucket ops
The outbound make-with-versioning bucket-op query only carried
operation/createdAt/lockEnabled. MinIO's own create-bucket hook sends
versioningEnabled=true on this op, so align the outbound query with
MinIO's site-replication make-bucket wire contract. Route both outbound
builders (bootstrap plan and create-bucket hook) through one shared
builder that always appends versioningEnabled=true. RustFS's own inbound
handler force-enables versioning either way, so RustFS-to-RustFS
behavior is unchanged; the MinIO release verified against
(RELEASE.2025-09-07) also force-enables versioning regardless of the
flag, so this aligns the wire contract rather than changing observable
behavior there.
* fix(admin): propagate purge-deleted-bucket errors in site replication
The purge-deleted-bucket branch of the peer bucket-ops handler dropped
the delete_bucket error and answered 200, so a peer-driven purge that
failed (disk full, quorum loss) was reported as success while the
bucket survived on this site. Tolerate only bucket-not-found (the purge
raced an earlier replay or a local delete) and propagate every other
error through ApiError like the sibling delete branches do.
* fix(admin): derive fallback site deployment ID with UUIDv5
deployment_id_for_endpoint used DefaultHasher, whose algorithm is not
guaranteed stable across Rust releases. The fallback fires when a peer
response carries an empty deploymentID; the result is persisted in
site-replication state, used for collision disambiguation, and
broadcast to peers, so a toolchain bump could re-derive a different ID
for the same endpoint. Note that the add preflight currently rejects
that case upstream of this fallback. Derive UUIDv5 (NAMESPACE_URL) over
the canonical endpoint instead, and log a structured warn when a peer
metainfo response arrives without a deploymentID. Already persisted
fallback IDs are non-empty and therefore never re-derived, so existing
state is unaffected.
* fix(admin): stream site replication devnull body without 1MB cap
The site-replication devnull endpoint buffered the request body through
read_plain_admin_body, which enforces the 1MB admin body cap. MinIO
peers stream multi-megabyte probe bodies to this endpoint during site
netperf link checks and expect an unbounded discard, so any larger
probe got a 400 and was misreported as a broken link. Stream and
discard the body chunk by chunk with no size cap instead, mirroring
MinIO's io.Discard drain. The response stays 204 with an empty body.
* refactor(e2e/kms): replace fixed startup sleeps with KMS readiness probe
Replace 33 hard-coded sleep(3s) / sleep(2s) startup waits in KMS e2e tests
with an active readiness probe (wait_for_kms_ready) that polls the KMS
status endpoint with exponential backoff (200ms→1s, 5s budget).
This cuts per-test startup latency from a fixed 3s to ~200-500ms while
remaining robust against slow CI machines.
Non-startup sleeps (ILM polling loops, fault-recovery detection delays,
test-runner inter-test pauses) are left untouched.
* style: cargo fmt
* fix(kms): use .expect() instead of ? in test functions that return ()
7 call sites of wait_for_kms_ready() used ? in async test functions
that return () instead of Result. Changed to .expect("KMS ready").
* fix(kms): enforce readiness probe deadline
* fix(kms): validate readiness backend status
The inline_block threshold used floor division (DEFAULT_INLINE_OBJECT_BUDGET
/ data_shards) while shard_file_size uses ceiling division (div_ceil). For
EC 12:4 with 256KiB objects, this caused a 1-byte discrepancy:
- inline_block = 262144 / 12 = 21845 (floor)
- shard_file_size = 262144.div_ceil(12) = 21846 (ceil)
- should_inline(21846, 12, false) = false (wrong!)
Fix by using div_ceil for the inline_block calculation, so both sides
use the same rounding and the inline path is correctly triggered.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(data-usage): ReplicationStats -> ReplicationTargetUsage
Rename the data-usage crate's ReplicationStats to ReplicationTargetUsage.
Serde field names are byte-identical (only the Rust type name changed;
field identifiers that rmp encodes are untouched). An rmp round-trip test
guards against future drift.
Scanner test imports updated to match.
* style: cargo fmt
The read-proxy selector already honors a target's disable_proxy flag
(PR #6172), but the admin API still rejected the field, so the only way
to set it was importing a MinIO-written bucket-targets.json.
- move disableProxy from REMOTE_TARGET_UNSUPPORTED_FIELDS to
REMOTE_TARGET_WRITABLE_FIELDS (set-remote-target create accepts it)
- add TargetUpdateOp::Proxy so set-remote-target?update=true&proxy=true
overlays only the proxy group (MinIO TargetUpdateType parity)
- bump REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION 1 -> 2 and update the
runtime capability pin tests
- keep edge/edgeSyncBeforeExpiry rejected (no implementation behind them)
- pin that a published TargetClient carries disable_proxy, the field the
proxy-target selector consults
Refs rustfs/backlog#1950
cxymds