* perf(put): add eager path metrics and isolation tooling
* fix(decommission): persist progress adaptively (#3497)
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* refactor: move bucket operations contract (#3507)
* fix(s3): handle multipart flexible checksums (#3508)
* fix(io-core): avoid blocking on pooled buffer return
* perf(put): add slow inflight diagnostics
* perf(put): fix 16KiB regression with threshold and pool bypass
- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold
Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.
* perf(put): gate stage metrics behind observability flag
Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.
The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).
This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.
* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID
Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects
Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)
Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)
Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request
Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs
* chore: add pyroscope feature flag and clean up imports
- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency
* style: fix import ordering and code formatting
- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs
* fix(test): use tokio::test for request_logging_layer tests
The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.
* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms
When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.
Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.
Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).
Refs: PR #3514 review comments by chatgpt-codex-connector
* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics
- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
(rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
METRICS_FLAG_LOCK mutex and explicitly set desired state at test start
Refs: PR #3514 review comments by chatgpt-codex-connector
* style: apply cargo fmt to metacache_set.rs
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* perf(ecstore): reuse erasure-decode shard buffers across stripes
ParallelReader::read allocated and zero-filled a fresh
`vec![0u8; shard_size]` per shard on every erasure stripe. Erasure::decode
builds one ParallelReader and loops stripes, so those buffers can be
reused: BitrotReader::read overwrites buf[..n] and the caller truncates to
n, so leftover bytes from the prior stripe are never observed. The decode
loop now hands each stripe's shard buffers back to the reader, which
reuses them (resized to shard_size) on the next stripe. The heal path,
which consumes its buffers into Bytes, never hands them back and is
unchanged.
This path runs on every object GET. Streaming-decode benchmark
(Erasure::decode end-to-end, median):
4+2 16MiB 1MiB-block: -9.6% verify-on, -23.8% verify-off
6+3 24MiB 1MiB-block: -9.7% verify-on, -22.5% verify-off
4+2 4MiB 64KiB-block: -5.5% verify-on, -16.3% verify-off
Adds a streaming-decode benchmark covering the ParallelReader path and a
regression test that decodes a multi-stripe object with missing data
shards (so reconstructed buffers are recycled) with bitrot verification
both on and off, asserting byte-exact output.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* perf(ecstore): tighten decode buffer reuse and bench harness
Address PR review feedback on the erasure-decode buffer-reuse change:
- Only claim a recycled shard buffer inside the `Some(reader)` branch of
`ParallelReader::read`, so the take is co-located with the read that
uses it and missing shards no longer touch the recycle slot.
- Move per-shard `BitrotReader` construction into Criterion `iter_batched`
setup so reader/UUID construction stays out of the timed decode path,
and assert decode success plus full output length each iteration.
Re-measured with the cleaner harness (median, before -> after, all
p < 0.05): verify-on -5.3%..-11.7%, verify-off -15.8%..-25.3%.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* perf(ecstore): drop dead recycle slots and tidy decode bench
Address the second PR review pass:
- In `Erasure::decode`, clear recycled slots for missing-shard (None-reader)
indices before handing buffers back. `ParallelReader::read` only reuses
slots with an active reader, and `decode_data` always re-allocates the
reconstructed shard, so retaining those buffers only held memory that could
never be reused in degraded reads.
- Move the output sink into the benchmark's `iter_batched` setup and validate
decode success/output length once per config instead of inside the timed
loop, so neither sink construction nor assertions touch the measurement.
Re-measured with the final harness (median, before -> after, all p < 0.05):
verify-on -5.2%..-9.8%, verify-off -19.4%..-25.8%.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(bucket-repl): persist MRF retry queue to disk and reload on startup
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(bucket-repl): address three blocking MRF issues from review
1. MRF replay loses delete operations — add `MrfOpKind` discriminator to
`MrfReplicateEntry` (Object | Delete, default=Object for backward
compat). `DeletedObjectReplicationInfo::to_mrf_entry` now persists
`op=Delete`, `version_id`, `delete_marker_version_id`, and
`delete_marker`. `start_mrf_processor` branches on `op`: delete
entries skip `get_object_info` and replay via
`schedule_replication_delete` with `ReplicationType::Heal`; object
entries follow the existing heal path.
2. `flush_mrf_to_disk` cleared the in-memory batch even on encode/write
failure — changed return type to `bool` and callers now only
`pending.clear()` on `true`, so a transient storage error retries
on the next tick instead of silently dropping the batch.
3. Add focused tests: encode/decode roundtrips for object, delete-marker,
versioned-delete, and mixed-batch entries; a routing test confirming
op-kind propagates correctly and that the default is Object for
legacy files; a legacy-compat test verifying old entries round-trip
cleanly through the new format.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: fix clippy redundant-clone in MRF tests
Replace &[entry.clone()] with std::slice::from_ref(&entry) in two
encode_mrf_file call sites flagged by clippy's redundant_clone lint
under --all-targets --features rio-v2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(bucket-repl): strengthen legacy MRF compat test with hand-built msgpack
The previous mrf_legacy_file_without_op_field_decoded_as_object test
round-tripped through encode_mrf_file, so it exercised the new format
and never touched a truly-legacy payload.
Replace it with a hand-built msgpack payload that genuinely omits the
"op", "deleteMarker", and "deleteMarkerVersionID" keys — exactly what
the old binary would have written before MrfOpKind existed. The test
now fails if #[serde(default)] is removed from the op field, which
proves real backward compatibility rather than round-trip stability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
fix(scanner): skip startup delay when replication is active so failed objects heal promptly after restart
Co-authored-by: houseme <housemecn@gmail.com>
fix(bucket-repl): honor op_type in replicate_object so ExistingObject resync respects DISABLED targets
replicate_object was calling filter_target_arns with hard-coded
op_type: Object and existing_object: false regardless of what was
stored in roi.op_type. This meant that a resync worker setting
roi.op_type = ExistingObject (resync_bucket, line 889) had no effect
on target filtering: all configured targets were included, even ones
whose rule had ExistingObjectReplicationStatus::DISABLED.
Fix: pass op_type: roi.op_type and derive existing_object from it
(true only for ExistingObject, not Heal — Heal intentionally bypasses
the existing-object opt-out to repair past failures).
Also add warn! logs at all four MRF channel-overflow sites that were
previously silently returning Missed with no observability.
Verified with a live two-instance test: after resync, objects reached
the ENABLED target and were correctly blocked from the DISABLED target.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix(site-repl): clamp replication_cfg_mismatch to owning deployments and add resync status branch
Fix 3: replication_cfg_mismatch was set on ALL deployments for a bucket when
any replication-config discrepancy existed, even on deployments that simply have
no config. mc computes "in sync" as max_buckets minus per-deployment mismatch
entries; with N deployments all flagged for 1 bucket, the result underflows to
1-N = -1. Now only deployments that own a replication config are flagged.
Fix 4: SiteReplicationResyncOpHandler accepted "start" and "cancel" but returned
an empty body (and later a parse error on the client) for any other operation
string. Add a SITE_REPL_RESYNC_STATUS="status" arm that returns the stored
SRResyncOpStatus or an explicit {"status":"not-found"} object so mc never sees
an unexpected end of JSON input.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): derive sync_state from reachability and replication rule completeness
PeerInfo.sync_state was set to SyncStatus::Unknown at every construction site
and never updated from real signals. build_status_info now tracks which peers
were reachable during the metainfo fetch phase and, after all bucket stats are
merged, derives sync_state as:
- Enable : reachable AND no replication_cfg_mismatch for any bucket
- Disable : reachable BUT at least one bucket has incomplete/missing rules
- Unknown : unreachable (fetch failed)
The "Sync" column in mc admin replicate status will now reflect actual state
rather than always showing Unknown.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): add SRRotateServiceAccountHandler for split-brain svc-acct repair
When site-replicator-0 gets desynced (different secret on different peers), calls
via the old service account return 403, blocking remove and replicate operations
with no in-band recovery path.
SiteReplicationRemoveHandler already purges local state unconditionally before
sending peer notifications, so force-remove works locally even when peers reject
the 403. The new POST /v3/site-replication/rotate-svc-acct endpoint provides the
missing recovery path: it generates a fresh service-account secret, applies it
locally, and pushes a peer/join to every member. Each peer's SRPeerJoinHandler
accepts the join idempotently (update if exists, create otherwise), repairing the
desynced credential without a full teardown + restart.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): back-fill pre-existing buckets and objects on replicate add
SiteReplicationAddHandler and SRPeerJoinHandler both called persist_site_replication_state
and returned success without doing anything for buckets that existed before the
sites were linked. Only buckets created after the link (via site_replication_make_bucket_hook)
ever replicated.
Introduce backfill_existing_buckets_after_add which, after persisting the new
state, iterates every local bucket and:
1. Ensures versioning is enabled (required by replication).
2. Reconciles bucket targets so a target entry exists for each remote peer.
3. Reconciles the replication config so a rule pointing to each peer is present.
4. Broadcasts a make-bucket-hook to peers (idempotent) so they create the bucket.
5. Kicks start_site_bucket_resync toward every remote peer so pre-existing
objects travel across.
Errors per bucket are logged but never abort the overall add — manual resync
remains available as a fallback. Both the initiator and the receiving (join) side
run the backfill so convergence happens regardless of which side held data first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): reconcile replication config to restore bidirectional replication
Root cause: ensure_site_replication_bucket_replication_config bailed with Ok(())
the moment any replication config was found on the bucket. When bucket B was
propagated to site-2 via the make-with-versioning bucket-op, site-2's
configure-replication step loaded the freshly-written config and immediately
returned, never adding the reverse-direction rule pointing back to site-1. Result:
objects uploaded to site-2 failed with "replication head_object fallback failed
... service error" because no rule targeted the originating site.
Fix: drop the early-return. Instead load the existing rules, build the full
desired config via build_site_replication_config, and MERGE — adding only the
rules that are absent (identified by their "site-repl-<deployment_id>" id). Re-
number priorities after each merge to avoid conflicts. Existing non-site-repl
rules are preserved. The write is skipped entirely when all desired rules are
already present, so repeat calls remain cheap.
Together with Fix 1 (backfill), any file written to ANY member now replicates to
ALL members. The offline-and-recover case also benefits: when a peer returns,
the per-bucket rules are complete and the resyncer can catch up.
Also register the new rotate-svc-acct route in route_policy and
route_registration_test so the route inventory assertions stay green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(replication): remove broken resync worker-signal gate
The resync_bucket task opened a new broadcast receiver via
worker_rx.resubscribe(), which positions the receiver at the current
write-head of the ring buffer — past all 10 bootstrap signals written
in ReplicationResyncer::new(). Every spawned resync task therefore
blocked on recv() forever, making `mc admin replicate resync start`
report "started" while no objects ever moved.
Remove the dead wait entirely. Each resync_bucket call is already
spawned on-demand (tokio::spawn in start_bucket_resync / load_resync),
so no additional gate is needed. The per-object concurrency limit is
already enforced by the inner mpsc worker channels (line ~877). Also
remove the now-dead worker_tx/worker_rx fields, bootstrap loop, and
signal-send in resync_bucket_mark_status.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs(site-repl): document rotate-svc-acct idempotency and partial-failure behavior
* style: cargo fmt and remove redundant clones
* fix(site-repl): preserve object-lock state when back-filling existing buckets
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
houseme