Single-part encrypted objects in the legacy format could not serve range
reads without decrypting from byte 0: v1 frames are emitted per upstream
read, so no closed-form plaintext-to-ciphertext mapping exists. The v2
layout fixed the frame length (8218 ciphertext bytes per 8 KiB plaintext
frame), making the mapping closed-form.
Consume it:
- Single-part PUTs that encrypt locally under the v2 write switch stamp
the frame-layout marker, valued with the object's data_dir token -
ciphertext passthrough, data movement and copies mint a new data_dir
or strip the marker, so a re-homed marker disqualifies itself.
- The encrypted read plan seeks marked, uncompressed single-part objects
to frame_index * 8218 and decrypts from that frame: the frame index
rides the plan's sequence-number slot into DecryptReader::new_at_block,
whose nonce and AAD bind absolute indices. New metric path label
frame_seek.
- A lying marker fails closed: v2 authentication rejects bytes at a fake
frame boundary; plaintext is never served from the wrong offset.
Compressed objects and multipart sub-part seeks keep the conservative
paths (follow-up work); reading needs no switch - seekability follows
the marker.
The legacy rio v1 stream format authenticates only each frame's
ciphertext: the 8-byte header (length + plaintext CRC32) and the end
marker sit outside the AEAD, frames carry no position binding, and
nothing marks the last frame - header rewrites, frame reordering and
truncation of trailing frames are not cryptographically detected.
Add a v2 layout in the same format family, dispatched per frame by the
type byte:
- the header plus the frame's index are AEAD associated data (0x01), so
header tampering, reordering and cross-position splicing fail
authentication;
- the final frame carries its own authenticated type byte (0x02); a
clean EOF or an end marker before it is an error, every stream
(including the empty one) ends in an authenticated final frame, and a
v2 multipart stream fails if it ends before all listed part segments;
- the writer accumulates full 8 KiB blocks, so non-final frames are
fixed-length (8218 ciphertext bytes) and single-part objects gain a
closed-form offset mapping for the follow-up range seek.
Key hierarchy, nonce derivation, envelopes and metadata are unchanged;
v1 objects stay readable forever, while v2 frames reject the historical
nonce fallbacks and unknown frame types become a hard error.
Write side ships off by default (RUSTFS_ENCRYPTION_FRAME_V2): mixed
version clusters cannot read v2 frames, and encrypted ciphertext travels
verbatim through transition, decommission and SSE-C replication
passthrough. This release ships read support; the default flips in a
following release.
Range GETs on encrypted objects read the whole ciphertext from offset 0
and discarded the decrypted prefix, because the part-boundary seek
shipped behind RUSTFS_ENCRYPTED_RANGE_SEEK defaulted to false
(backlog#1316 Phase A).
Flip the default to true. Safety rests on the marker chain: MPUs created
without a candidate layout marker never become seek-eligible,
CompleteMultipartUpload promotes the candidate to the quorum marker only
after revalidating it against the object's data_dir under the uploadId
write lock, and reads seek only when the quorum marker matches the
current data_dir. Single-part, compressed and markerless objects keep
the full-read path; RUSTFS_ENCRYPTED_RANGE_SEEK=false remains the kill
switch.
The stale default-off regression test becomes
test_legacy_range_seek_defaults_enabled: the unset-env default must
match the explicit opt-in plan, seek past the leading parts, and not
span the whole ciphertext.
* feat(mimalloc): add arena diagnostics and configuration
Based on mimalloc maintainer feedback (microsoft/mimalloc#1372),
add diagnostics to check mimalloc arena configuration at runtime.
Changes:
- Add rustfs-mimalloc-sys to workspace dependencies
- Add log_mimalloc_diagnostics() function to check:
- arena_max_object_size
- pagemap_commit status
- mimalloc version
- Add memory_observability module with mimalloc diagnostics
This helps diagnose why allocations might be going outside arenas,
which is the suspected root cause of futex contention.
Ref: rustfs/backlog#2005
Ref: microsoft/mimalloc#1372
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): add Vec<u8> buffer pool for EC operations
Add a general-purpose buffer pool to reduce Vec<u8> allocations
in hot paths like EC encoding/decoding.
Changes:
- Add BufferPool struct in crates/ecstore/src/erasure/codec/buffer_pool.rs
- Thread-safe pool with capacity-based bucketing (power-of-two)
- Global EC_BUFFER_POOL instance with 16 buffers per bucket
- Add buffer_pool module to codec/mod.rs
Expected impact:
- Reduce heap allocations in EC encode/decode paths
- Avoid memzero overhead (proven 4.8% CPU saving in ShardBufferPool)
- Reduce mimalloc lock contention
Note: Main bottleneck remains mimalloc internal synchronization
(futex 98.64% time). Buffer pool provides modest improvement (+2-5%).
Ref: rustfs/backlog#2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and related files
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): add #[allow(dead_code)] to buffer pool
The BufferPool infrastructure is ready but not yet integrated
into the EC hot paths. Add #[allow(dead_code)] with clear
documentation about integration status.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): integrate BufferPool into bitrot verify path
Replace vec![0; shard_size] with get_ec_buffer() in the bitrot
verification hot path to reduce heap allocations and avoid memzero.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and bitrot changes
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(ecstore): clean up buffer pool code
- Remove unnecessary #[allow(dead_code)] attributes
- Update module documentation to reflect current integration status
- Simplify code structure
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(runtime): cap default worker threads at 16
Testing showed 16 worker threads outperforms 32+ for 1KiB PUT
workloads due to reduced mimalloc lock contention.
A/B test results (testing 4-node cluster, c=64):
- worker_threads=32: 740 obj/s (baseline)
- worker_threads=16: 785 obj/s (+6.1%)
The default was detect_cores() which returned 32 on our testing
nodes. Cap at 16 for optimal small-object performance.
Ref: rustfs/backlog#2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and runtime changes
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): remove unused BufferPool::new() function
The new() function was never used since EC_BUFFER_POOL
initializes directly with with_limits(16).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): update buffer_pool tests to use with_limits
Replace BufferPool::new() with BufferPool::with_limits(16) in tests
since new() was removed in favor of with_limits().
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): optimize opts.clone() and FileInfo clone patterns
## Changes
1. ObjectOptions helper methods:
- add as_commit_opts(): creates commit options with no_lock=true,
metadata_cache_safe=false, include_part_checksums=true
- add as_read_opts(): creates read options with
include_part_checksums=true
- add with_no_lock(): creates options with modified no_lock field
2. Replace opts.clone() in hot paths:
- commit_opts = opts.as_commit_opts() (was 4-line manual clone)
- read_opts = opts.as_read_opts() (was 2-line manual clone)
3. Optimize FileInfo clone in rename path:
- avoid double clone: clone once and modify erasure.index in place
- pass &file_info reference to rename_data_borrowed_with_fence
## A/B Results (4-node cluster, c=64)
| Size | main | optimized | Change |
|------|------|-----------|--------|
| 1KiB | 892 obj/s | 920-976 obj/s | +3%~+9% |
| 4KiB | 957 obj/s | 903 obj/s | -5.7% |
| 16KiB | 922 obj/s | 855 obj/s | -7.3% |
Note: 1KiB improvement is consistent. 4KiB/16KiB variance
likely due to test noise; needs more rounds to confirm.
Ref: rustfs/backlog#2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): add BytesMut buffer pool to EC encoding path
Pre-allocate a Vec<BytesMut> pool in the EC encoding loop to avoid
repeated heap allocations for ingest buffers.
Changes:
- Pre-allocate buffer pool with capacity 4
- Reuse buffers from pool after encoding
- Return buffers to pool when capacity is sufficient
Expected impact: +10-20% in EC encoding path by reducing
BytesMut allocation overhead.
Ref: rustfs/backlog#2005
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: hector <hetor@rustfs.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): handle metadata-less bucket residue
Diagnose metadata-less on-disk residue before non-force DeleteBucket reaches physical deletion, and keep scanner-discovered metadata-missing objects on a non-destructive heal path.
Add explicit heal --remove cleanup for unversioned metadata-less data directories, using the existing data-dir delete primitive and fail-closed shape checks so pre-commit or unknown residue is preserved.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(connect): adapt offline array validator
Wrap the filesystem summary validator in a closure so Option::is_some_and can pass the concrete array reference accepted by serde_json::Value::as_array.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(connect): remove redundant offline test clones
Move the temporary path into the swap closure after deriving the output path, keeping clippy's redundant-clone lint clean for offline bundle tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(mimalloc): add arena diagnostics and configuration
Based on mimalloc maintainer feedback (microsoft/mimalloc#1372),
add diagnostics to check mimalloc arena configuration at runtime.
Changes:
- Add rustfs-mimalloc-sys to workspace dependencies
- Add log_mimalloc_diagnostics() function to check:
- arena_max_object_size
- pagemap_commit status
- mimalloc version
- Add memory_observability module with mimalloc diagnostics
This helps diagnose why allocations might be going outside arenas,
which is the suspected root cause of futex contention.
Ref: rustfs/backlog#2005
Ref: microsoft/mimalloc#1372
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): add Vec<u8> buffer pool for EC operations
Add a general-purpose buffer pool to reduce Vec<u8> allocations
in hot paths like EC encoding/decoding.
Changes:
- Add BufferPool struct in crates/ecstore/src/erasure/codec/buffer_pool.rs
- Thread-safe pool with capacity-based bucketing (power-of-two)
- Global EC_BUFFER_POOL instance with 16 buffers per bucket
- Add buffer_pool module to codec/mod.rs
Expected impact:
- Reduce heap allocations in EC encode/decode paths
- Avoid memzero overhead (proven 4.8% CPU saving in ShardBufferPool)
- Reduce mimalloc lock contention
Note: Main bottleneck remains mimalloc internal synchronization
(futex 98.64% time). Buffer pool provides modest improvement (+2-5%).
Ref: rustfs/backlog#2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and related files
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): add #[allow(dead_code)] to buffer pool
The BufferPool infrastructure is ready but not yet integrated
into the EC hot paths. Add #[allow(dead_code)] with clear
documentation about integration status.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): integrate BufferPool into bitrot verify path
Replace vec![0; shard_size] with get_ec_buffer() in the bitrot
verification hot path to reduce heap allocations and avoid memzero.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and bitrot changes
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(ecstore): clean up buffer pool code
- Remove unnecessary #[allow(dead_code)] attributes
- Update module documentation to reflect current integration status
- Simplify code structure
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(runtime): cap default worker threads at 16
Testing showed 16 worker threads outperforms 32+ for 1KiB PUT
workloads due to reduced mimalloc lock contention.
A/B test results (testing 4-node cluster, c=64):
- worker_threads=32: 740 obj/s (baseline)
- worker_threads=16: 785 obj/s (+6.1%)
The default was detect_cores() which returned 32 on our testing
nodes. Cap at 16 for optimal small-object performance.
Ref: rustfs/backlog#2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: apply cargo fmt to buffer pool and runtime changes
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): remove unused BufferPool::new() function
The new() function was never used since EC_BUFFER_POOL
initializes directly with with_limits(16).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): update buffer_pool tests to use with_limits
Replace BufferPool::new() with BufferPool::with_limits(16) in tests
since new() was removed in favor of with_limits().
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: hector <hetor@rustfs.com>
Co-authored-by: heihutu <heihutu@gmail.com>
PR #6369 awaits record_durable_ilm_decommission_progress/terminal inline
from save/delete_transition_transaction_record. Their state machines are
large and sit on the already-deep transition worker poll chain
(worker -> transition -> transaction record -> delete_config -> full
store delete fanout), which overflowed the default 2 MiB tokio worker
stack in debug builds: app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions died
with SIGABRT in under a second (first-bad commit via git bisect
1.0.0-rc.3..1ec1a8d90: 34bbc1adb, #6369).
41546dee5 already unblocked the test by moving it onto a dedicated
32 MiB thread; this change removes the underlying stack growth so every
caller of the transaction-record helpers keeps its previous headroom.
With it, the test also passes on a plain 2 MiB tokio worker.
* fix(ecstore): fence pool metadata replica updates
* fix(ecstore): block decommission on unsafe pool metadata
* fix(ecstore): block writes after pool metadata save errors
* fix(ecstore): latch pool metadata writes before await
* fix(scanner): bootstrap pristine usage baseline
* fix(replication): tolerate orphaned resync intents at startup
Since #5215 (1.0.0-beta.12) startup reconciles every pending/started
resync intent in resync.bin against the bucket's configured targets and
aborts the whole server when an intent has no matching target ARN. A
resync whose remote target was later removed leaves exactly such an
orphan on disk, so every later start fails with "accepted replication
resync target ... is not configured" regardless of the binary version.
Skip orphaned intents with a warning instead of failing startup; the
resync routine already settles them to ResyncFailed. Cancel the intent
when its remote target is removed so the orphan is not created again.
Fixes#4784
* fix(replication): cancel removed-target resync under the admission lock
Canceling through this node's cached whole-bucket status map could
persist a map that predates another node's admission, erasing that
node's durable restart intent. Reload resync.bin under the bucket
admission lock, publish the fresh map, and only then mark the removed
target's intent canceled. Two-node regression covers the clobber.
* fix(replication): persist resync status via ETag CAS merge
mark_status, the periodic saver, admission, and removed-target
cancellation all persisted their node's cached whole-bucket map, so any
one node's stale cache could resurrect states another node had already
finalized (a canceled intent flipping back to Pending, an admission
vanishing). All resync.bin writers now go through update_resync_status_cas:
load the freshest document with its ETag, apply a per-target mutation
with staleness and canceled-is-terminal guards re-checked against the
persisted entry, and save conditionally, retrying on concurrent writes.
The periodic saver merges per target, letting terminal states and newer
admissions recorded elsewhere win. Cache convergence stays per-target so
locally running resyncs keep their authoritative progress counters.
Regressions: stale_peer_status_write_cannot_resurrect_canceled_intent
(node B's pre-cancel cache marking its own run Started must not revive
node A's canceled intent) plus unit coverage for the periodic-save merge.
* test(ecstore): rename resync test helper off the guarded contract name
fn resync_target is on the architecture guard's reserved list for
crates/replication operation contracts; the merge-test helper now reads
resync_target_state.
* fix(replication): serialize resync status updates
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): fence pool metadata replica updates
* fix(ecstore): block decommission on unsafe pool metadata
* fix(ecstore): block writes after pool metadata save errors
* fix(ecstore): latch pool metadata writes before await
* fix(ecstore): persist unresolved decommission entries
* fix(ecstore): type decommission completion result
* fix(ecstore): allow intentional decommission listing signatures under strict clippy
The sftp/swift feature-matrix clippy gates run with -D warnings and
flag the unresolved-entry resolver (large Err payload by design, 8
context parameters) and the decommission listing driver (9 args).
Document why and align with the existing decommission_entry precedent.
cargo fmt --check has been failing since #6410 landed: the merged
decommission import groups in core/pools.rs were not canonical
rustfmt output. Apply formatting verbatim, no code changes.
A single object's SourceChanged during decommission cleanup no longer
cancels the shared worker token and fails the whole pool operation.
Cleanup preflight and source-cleanup outcomes now retry per entry with
bounded attempts and cancellation-aware backoff, applied uniformly to
ordinary versions, delete markers, and tiered copies (removing the
try-once-only branches); every retry re-lists the entry and redoes
version multiset validation before touching the source. Only quorum
loss, unrecoverable system errors, or exceeding a pool-level
SourceChanged exhaustion threshold still fails the decommission, and
exhausted entries never delete their source versions.
Retry attempts, backoff, and deferred/exhausted reasons are logged per
entry for observability. Heavy regression tests spawn on dedicated
32MiB stacks following the existing store-test pattern.
Fixesrustfs/backlog#1913
Co-authored-by: houseme <housemecn@gmail.com>
Zhengchao An