* fix(replication): deny non-owner replication config edits under site replication
Under site replication a user holding only bucket-scoped
s3:PutReplicationConfiguration could rewrite or erase the operator-managed
site-repl-* rules, with the change broadcast to every peer (backlog#1948,
audit A1/P2-17).
- Gate PutBucketReplication/DeleteBucketReplication in the S3 handlers:
when site replication is enabled and the requester is not the owner,
return MinIO-parity XMinioReplicationDenyEdit (HTTP 400). The gate runs
after policy authorization and only on the external S3 path; the
reconciler and peer bucket-meta ingestion are unaffected.
- Defense in depth in the bucket usecase: PUT merges the incoming config
with the stored site-repl-* rules (same merge as peer ingestion) instead
of overwriting verbatim; DELETE keeps the site-repl-* rules and never
garbage-collects a bucket target a surviving site-replication rule still
references.
- Move is_site_replication_rule / merge_incoming_replication_config /
replication_target_arn_deployment_id from the admin site-replication
handler down to rustfs-replication so the app layer can reuse them
without new layering violations.
* fix(replication): scope site-owned rule detection to reconciler-derived rules
The `site-repl-*` prefix alone classified any rule as site-owned, so on a
bucket outside site replication an owner's `site-repl-user` rule survived
DeleteBucketReplication (rule and target kept, success returned). Rule ids
do not reserve that namespace.
A rule is reconciler-owned only when it matches what the reconciler
derives: id `site-repl-<deployment id>` for a current remote site
replication peer and a destination ARN naming that same deployment id.
The S3 put/delete path reads the remote peer set (empty when site
replication is disabled) and keeps exactly those rules; everything else
is operator state the request replaces or deletes. An incoming rule that
claims a current peer's id is dropped so the reconciler rule's id stays
unique. The peer ingestion path and the reconciler keep their prefix
predicate unchanged.
* fix(replication): keep operator rule priorities across site rule merges
Merging stored site-replication rules into a PutBucketReplication body
renumbered every rule 1..n in list order, rewriting the submitted policy:
overlapping same-target rules submitted as priority 5 then 1 became 1
then 2, so the delete-marker-disabled rule won the replication decision.
The reconciler and the peer-removal prune renumbered the same way.
Operator priorities now stay verbatim everywhere; only the reconciler's
derived rules move, to the lowest priorities no operator rule uses, via
one pure helper shared by the S3 edit merge, the peer ingestion merge,
the reconciler pass and the prune. Being a pure function of the rule
list it is idempotent, so the reconciler's no-op check still holds after
a merged write, and an on-disk config in the historical layout (operator
rules 1..k, site rules k+1..n) yields the same bytes, so nothing is
rewritten on upgrade.
* fix(replication): pass site peer ids into the bucket usecase from the interface layer
The review fix made the bucket usecase read the site-replication peer set
through the admin handlers, an app->interface import the layer guard
rejects. The S3 handlers (interface) now read the peer set and pass it in,
so the usecase stays a pure function of its inputs; a state-read failure
still fails the edit closed, just one layer up.
* fix(replication): classify peer-ingested rules by the derived id/ARN contract
The peer ingestion merge still treated every incoming `site-repl-*` id as
reconciler-owned, so an owner-authored `site-repl-user` rule that the S3
merge now keeps on the editing site was dropped on every peer and the
sites persisted different operator configs.
The ingestion merge now classifies by the same derived contract as the
S3 merge: a rule is the reconciler's only when its `site-repl-<id>` names
the deployment its destination ARN targets and that deployment is a site
of the cluster (the receiver's own id included, since the sender's rule
towards the receiver names it). The reconciler, the peer-removal prune
and the target-online probe switch from the id prefix to the derived
shape as well, so the rule survives their passes too; rules in the
derived shape that name a removed peer or this site are still rebuilt
away.
Regression: a PutBucketReplication merged on site A and ingested on
site B keeps `site-repl-user` on both and the operator rule sets agree.
* fix(replication): keep an operator role target through site rule merges
The S3 and peer-ingestion merges cleared `Role` whenever it parsed as a
site-replication ARN, which an owner-submitted remote target with an
empty region (`arn:minio:replication::<id>:<bucket>`) also does. The
merged config then selected the rule destination ARNs instead of the
validated role target.
Only a role naming a current site of the cluster is the holder's
identity (the reconciler's per-peer target lookup reads it); every other
role passed target validation and stays. The reconciler's repair pass
applies the same rule.
Regression: an owner role target survives both merges and
`filter_target_arns` / `replication_target_arns` select it; a role naming
a current peer is still cleared.
* fix(replication): gate operator priority preservation on a peer contract probe
Keeping operator rule priorities verbatim is not rolling-upgrade safe: a
peer still running the pre-contract code renumbers every rule 1..n in
list order on ingest and on each reconciler pass, so an upgraded site
broadcasting `5,1` leaves that peer on `1,2` — which can select the
other overlapping rule — and the sites never reconverge.
Operator rules now merge under an explicit contract:
- `OperatorRuleContract::Derived`: site rules are the derived id/ARN
shape, operator priorities stay verbatim (the behavior of the previous
commits).
- `OperatorRuleContract::Legacy`: byte-for-byte what a pre-contract peer
does — `site-repl-*` ids are all site rules, a site-replication-shaped
`Role` is dropped, every rule is renumbered 1..n in list order. The S3
merge additionally lists the operator rules in priority order first,
so the renumbering keeps their relative order and the winning rule per
target is the one the operator submitted.
The S3 PutBucketReplication/DeleteBucketReplication path probes every
remote peer through the existing `peer/edit-capabilities` endpoint
(capability `derived-rule-contract`; pre-contract peers answer
`success:false` or 404) and merges under Derived only when every peer
supports it; any refusal or probe failure pins that edit to Legacy.
Every bucket-meta item this site sends (S3 hooks, bootstrap plan, retry
snapshots, tombstones) carries `derivedRuleContract: true`; a receiver
merges a payload without the marker the Legacy way, so an item from a
pre-contract sender is handled exactly as its own peers handle it.
Rolling upgrade: while any site runs the older code every edit is
canonicalized cluster-wide (numbers lost, order kept); once the last
site is upgraded the next edit keeps its priorities. Configs
canonicalized during the mixed period are not renumbered back — the
derived priority assignment is a no-op on the canonical layout — so an
operator who wants the original values re-submits the config after the
upgrade completes. Adding a site that runs the older code after
priorities were preserved is not gated and would desynchronize that
bucket until the next edit.
---------
Co-authored-by: houseme <housemecn@gmail.com>
The peer reload_pool_meta handler blindly replaced in-memory pool
metadata with the persisted snapshot, so a delayed or out-of-order
reload could roll back newer local queued/canceled/failed/complete
decommission state, and a missing pool.bin wiped local state to an
empty default.
Route peer reload through the same monotonic merge used by the admin
status refresh (merge_pool_status_refresh): entries are replaced only
when strictly newer and no local worker is active; missing snapshots
fail closed. The helper now reports whether any entry was replaced or
appended, and rejected stale/missing reloads are logged. The RPC
handler spawns missing decommission workers only after a reload
actually merged newer state, so duplicate deliveries cannot start
workers for an older generation.
Fixesrustfs/backlog#1917
* feat(allocator): replace mimalloc/libmimalloc-sys with rustfs-mimalloc/rustfs-mimalloc-sys
Replace the upstream xonatius/mimalloc_rust.git fork (mimalloc + libmimalloc-sys)
with the published rustfs-mimalloc (v0.5.0) and rustfs-mimalloc-sys (v0.5.0) crates
from crates.io.
The new crates are based on mimalloc V3 (v3.5.0) and provide:
- MiMalloc global allocator with safe API (collect, stats_json, process_info)
- Heap management and arena operations (heap module)
- Full FFI bindings to mimalloc V3
Changes:
- Workspace deps: mimalloc + libmimalloc-sys (git) → rustfs-mimalloc + rustfs-mimalloc-sys (crates.io)
- allocator_reclaim.rs: libmimalloc_sys::mi_collect → rustfs_mimalloc::MiMalloc::collect
- memory_observability.rs: raw FFI mi_stats_get_json → MiMalloc::stats_json()
- main.rs: heap ownership tests use Heap::contains() (V3 API)
- deny.toml: remove xonatius/mimalloc_rust.git from allow-git
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): default rename fanout to parallel early-ack path
Switch the default rename_data commit fanout from serial join_all to the
parallel JoinSet early-ack path. The serial path (#5987) was the primary
cause of the 1MiB PUT regression (-71.7%) observed in rc.3 benchmarks.
A/B verification on testing 4-node cluster (c=64, 1MiB PUT, 2min):
- Serial (join_all): 96.99 MiB/s, P50=644ms
- Early ack (JoinSet): 177.46 MiB/s, P50=407ms (+83%)
Also:
- Update rename_data_reclaims_synthetic_inline_rollback_dir_after_commit
to use rename_data_owned and await tail_drain for proper cleanup.
- Update rename_data_waits_for_tail_disk_after_write_quorum to explicitly
test the serial path (now non-default) via env override.
- Add error source chain to HTTP Body stream transport error log
(backlog#2005) so the underlying cause is visible.
Ref: rustfs/backlog#2005
Ref: rustfs/backlog#1792#issuecomment-5384346238
Ref: rustfs/backlog#1792#issuecomment-5384370938
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
SSE and compression wrap the payload so its length is unknown and
advertise HashReader::SIZE_PRESERVE_LAYER (-1). Every layer preserved
that sentinel except create_bitrot_writer, which clamped it to 0 before
calling DiskAPI::create_file. RemoteDisk forwards that size verbatim in
the put_file_stream query, so remote peers were told the body was empty.
Since the authenticated put-file trailer (#5868) the receiver used the
declared size to split body from trailer, turning the clamp into a fatal
"auth trailer has trailing data" failure for every SSE PUT on multi-node
deployments (rc.2). #6320 relaxed the receiver to only trust size > 0;
this change fixes the sender so the sentinel survives end to end and the
wire no longer conflates empty objects with unknown-length streams.
Refs #6331
The inline_block threshold used floor division (DEFAULT_INLINE_OBJECT_BUDGET
/ data_shards) while shard_file_size uses ceiling division (div_ceil). For
EC 12:4 with 256KiB objects, this caused a 1-byte discrepancy:
- inline_block = 262144 / 12 = 21845 (floor)
- shard_file_size = 262144.div_ceil(12) = 21846 (ceil)
- should_inline(21846, 12, false) = false (wrong!)
Fix by using div_ceil for the inline_block calculation, so both sides
use the same rounding and the inline path is correctly triggered.
Co-authored-by: heihutu <heihutu@gmail.com>
The read-proxy selector already honors a target's disable_proxy flag
(PR #6172), but the admin API still rejected the field, so the only way
to set it was importing a MinIO-written bucket-targets.json.
- move disableProxy from REMOTE_TARGET_UNSUPPORTED_FIELDS to
REMOTE_TARGET_WRITABLE_FIELDS (set-remote-target create accepts it)
- add TargetUpdateOp::Proxy so set-remote-target?update=true&proxy=true
overlays only the proxy group (MinIO TargetUpdateType parity)
- bump REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION 1 -> 2 and update the
runtime capability pin tests
- keep edge/edgeSyncBeforeExpiry rejected (no implementation behind them)
- pin that a published TargetClient carries disable_proxy, the field the
proxy-target selector consults
Refs rustfs/backlog#1950
* fix(scanner): defer usage publication during pool recovery
* fix(scanner): preserve metrics when publication is deferred
* fix(scanner): route test types through storage boundary
* fix(scanner): keep cache floor deferred during movement
Add deterministic rename_data coverage for tail-disk success/failure, cancellation serialization, and strict quorum rollback visibility after disk reopen. This establishes the safety boundary before experimenting with write-quorum early ACK and background tail completion for backlog #925.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(lifecycle): safely expire all object versions
* fix(lifecycle): preserve delete-all replication purges
* fix(lifecycle): remove dead replication journal
* fix(ci): avoid lifecycle transition test stack overflow
* fix(lifecycle): release recovery locks before tier IO
* test(lifecycle): align object-lock error assertions
* test(lifecycle): avoid scanner restore stack overflow
* test(scanner): avoid stack overflow in transition and restore flow test (#6300)
* refactor(scanner): split remote_scanner.rs into stream child module (#6289)
Split the 3080-line remote_scanner.rs (47% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:
- remote_scanner.rs (~320): protocol constants, process statics, and
the request decode/validate/admit/preflight/claim API plus root
re-exports
- remote_scanner/stream.rs (~1340): wire/frame types, replay cache,
FrameAuthenticator, serve path, local bucket scan + persist, client
scan, and the bounded stream plumbing
- remote_scanner/stream/tests.rs (~1470): the inline test module as a
child module of stream so it can reach both parents' private items
All crate paths are unchanged: lib.rs re-exports
(serve_remote_scanner_request, RemoteScannerRequest, ...) resolve
through root re-exports, and scanner_io's crate::remote_scanner::
{scan_remote_bucket, RemoteScannerScanSpec, RemoteScannerOutcome}
paths resolve through pub(crate) re-exports. Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain;
no item's effective visibility widens. Code is moved verbatim apart
from those markers, per-module import headers, and rustfmt line
re-wraps.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(heal): split resume.rs into focused child modules (#6290)
Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:
- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
ResumeManager core (constructors, load/discovery, progress mutators,
ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module
All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(scanner): split scanner_io.rs into child modules (#6294)
Split the 5369-line scanner_io.rs (39% inline tests) into a canonical
scanner_io.rs + scanner_io/ module tree with zero behavior change:
- scanner_io.rs (~660): constants, metadata-error constructors, the
bucket scan plan, cycle-status classification helpers, the ScannerIO /
ScannerIOCache / ScannerIODisk traits, and ScannerCycleResult
- scanner_io/dirty_usage.rs (~300): process-wide dirty-usage statics
and the acknowledgment protocol
- scanner_io/guards.rs (~270): concurrency gauges and RAII guards
- scanner_io/cache.rs (~410): scanner cache locks and the snapshot
persist/publish path
- scanner_io/io_cycle.rs (~390), io_cache.rs (~1160), io_disk.rs
(~230): the ECStore / SetDisks / Disk trait implementations
- scanner_io/publish_gate_tests.rs (~750) and tests.rs (~1340): the two
inline test modules as child modules
All crate paths are unchanged: the lib.rs scanner_io re-exports and
every crate::scanner_io:: consumer (scanner.rs, remote_scanner,
scanner_folder, and cross-crate rustfs users) resolve through root
re-exports with their original visibilities (pub stays pub, pub(crate)
stays pub(crate)). Cross-module items gain pub(super), whose scope
equals the old single-module privacy domain. Code is moved verbatim
apart from those markers, per-module import headers, and rustfmt
re-wraps.
The logging-guardrail nsscanner_disk skip-set_disks rule now points at
scanner_io/io_disk.rs where the function moved; the pattern and
thresholds are unchanged.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(scanner): split data_usage_define persistence and tests (#6292)
Split the 3655-line data_usage_define.rs (59% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:
- data_usage_define.rs (~950): cache constants and revision helpers,
the data-usage tree types, DataUsageCacheInfo with its hand-written
Serialize, the in-memory tree operations, dui, and marshal/unmarshal
- data_usage_define/persistence.rs (~580): the load/backup/restore
ladder (load, try_load_inner, revision_for_path) and the CAS save
path with its retry policy and save metrics
- data_usage_define/tests.rs (~2155): the inline test module as a child
module
All module paths are unchanged (the lib.rs data_usage_define::* glob
re-export and every crate::data_usage_define:: consumer resolve as
before). The hand-written map-encoded Serialize for
DataUsageCacheInfo is moved byte-for-byte per the AGENTS.md
cross-cutting invariant; on-disk names and the cache key format const
stay in the root. Four persistence helpers used by tests gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): bump datafusion to 55.0.0 (#6288)
* refactor(heal): split task.rs per heal kind (#6293)
* feat(ecstore): batch small file fdatasync commits (#6297)
* feat(ecstore): batch small file fdatasync commits
Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): wait for compression S3 readiness
Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover multi-committed mutation intents (#6296)
* fix(tier): recover multi-committed mutation intents
* fix(tier): recover committed mutations on standalone nodes
* test(scanner): avoid stack overflow in transition test
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Add PUT-stage diagnostics for file fdatasync group commit wait time, per-group outstanding depth, and rename disk completion position. These metrics keep the existing default-off PUT stage gate and do not change group commit scheduling or quorum behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): batch small file fdatasync commits
Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): wait for compression S3 readiness
Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
houseme