* refactor(ecstore): extract read IO primitives to set_disk::core::io_primitives (backlog#820)
P5 step 1 of the SetDisks God-Object split (tracking backlog#815, issue
backlog#820). Relocate the module-level low-level read/erasure primitives out of
the 5,898-line set_disk/read.rs into a new core/io_primitives.rs module home:
- Metadata-fanout quorum machinery (MetadataQuorumAccumulator,
MetadataFanoutDiagnostics/Observation, MetadataEarlyStopDecision,
MetadataCacheLookup).
- Bitrot reader scheduling/creation (BitrotReaderSetup and the
create_bitrot_readers_* / schedule / fill helpers).
- Shard-cost classification, read-repair heal dedup, and codec-streaming
reader helpers.
Pure move + visibility adjustment, zero logic change. The moved block is
byte-identical to the pre-move read.rs source modulo the module header swap
(use super::* -> use super::super::*) and item visibility widening
(module-private / pub(super) -> pub(in crate::set_disk) so read.rs still
reaches them). read.rs's impl SetDisks body and imports are byte-identical to
before; mod.rs only gains 'mod core;' and repoints two
GetCodecStreamingReaderBuildOutcome references to the new path.
Verification:
- cargo check / clippy -D warnings -p rustfs-ecstore --all-targets: clean
- cargo test -p rustfs-ecstore --lib: 1840 passed, 0 failed
- normalized + token-stream diff of moved block vs original: identical modulo
visibility tokens and rustfmt signature re-wrapping
* fix(arch): allow io_primitives as READ_REPAIR_HEAL_CACHE owner (backlog#820)
The READ_REPAIR_HEAL_CACHE owner-path guard in
check_architecture_migration_rules.sh pinned the cache to
set_disk/read.rs. P5 step 1 relocated the cache (and its accessor
helpers) to set_disk/core/io_primitives.rs, so allow that path too.
Guard intent is unchanged: the cache stays behind the ECStore set-disk
read-owner helpers.
---------
Co-authored-by: houseme <housemecn@gmail.com>
fix(internode): relax aggressive HTTP/2 keepalive & RPC timeouts to stop large-object GET truncation
The internode gRPC channel used a 3s HTTP/2 keepalive timeout and a 10s
overall RPC timeout. Under high-concurrency large-object reads a saturated
peer's PING ACK is legitimately delayed past 3s, so the whole channel (and
every RPC/stream on it) is torn down as a 'dead peer'. In-flight peer shard
reads then fail mid-object and large GETs truncate after headers+Content-Length
are already sent, surfacing to clients as 'download error: unexpected EOF'.
Reproduced on a 4-node erasure cluster with pure GET-only warp (no concurrent
writes) at 64 concurrency: 10MiB GET ~31 unexpected-EOF / 3min. Raising the
internode keepalive timeout (3s->30s via env) alone cut that to ~7; also raising
the RPC timeout cut it to ~3.
- Raise DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS 3 -> 20
- Raise DEFAULT_INTERNODE_RPC_TIMEOUT_SECS 10 -> 30
- Wire the data-plane rio HttpReader keepalive timeout (was hardcoded 3s) to the
same env/default so control- and data-plane stay consistent.
Refs backlog#832.
P1 of the SetDisks split (tracking backlog#815, depends on P0 backlog#816).
Give the Heal operation family its own module home: relocate set_disk/heal.rs
to set_disk/ops/heal.rs and move the HealOperations storage-api contract impl
beside its inherent helpers. The contract stays implemented for SetDisks so its
associated-type bounds are unchanged (ecstore_contract_compat_test still
covers it).
Method bodies are moved unchanged. The four inherent helpers widen from
pub(super) to pub(in crate::set_disk) to preserve their exact prior visibility
from the deeper module. get_pool_and_set now reads topology through
SetDisksCtx to keep the Heal family aligned with the P0 borrow pattern; the
read is provably identical (ctx.format()/pool_index() alias the core fields).
Runtime behavior is unchanged.
P0 of the SetDisks split (tracking backlog#815). Introduce SetDisksCtx<'a>,
a Copy borrow handle over the shared SetDisks core state (topology/config,
disks, locker trio), and validate the pattern by routing the List family's
delete_all through a ListOperations<'a> service unit.
No trait impl is moved and runtime behavior is byte-for-byte unchanged; the
public SetDisks::delete_all entry point is preserved and delegates to the
borrow-based unit.
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking
DecompressReader::poll_read and DecryptReader::poll_read sliced the block
body with a fixed `[0..16]` index to read the length varint. The body length
comes from an untrusted 24-bit header field, so a corrupted/truncated block
shorter than 16 bytes made the slice panic and crash the request task — a
read-path DoS on GET of tiered/corrupted data.
Pass the whole (arbitrary-length-safe) slice to uvarint and reject a
non-positive or out-of-range length prefix with InvalidData. Adds a repro
test for each reader; all existing round-trip tests still pass.
Refs rustfs/backlog#812
* fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses
validate_outbound_ip branched on the IpAddr variant, and the V6 branch's
is_loopback/is_unicast_link_local/is_unique_local checks never inspect the
embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard
also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1,
::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard,
letting an attacker reach loopback/private/metadata endpoints.
Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which
matches only the true mapped form) before classification. Adds reject tests
for mapped loopback/private/metadata and an allow test for public IPv6.
Refs rustfs/backlog#813
* fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment
Four confirmed data-reliability defects:
- put_object_multipart_stream: the CompleteMultipartUpload part-collection loop
used exclusive `1..total_parts_count`, dropping the final part (and collecting
zero parts for a single-part object) — silently truncating the completed object.
Extracted collect_complete_parts (1..=total_parts_count) with unit tests.
- GCS warm backend get() ignored the requested byte range, returning the whole
object for a Range GET; now applies ReadRange::segment like the other backends.
- GCS warm backend remove() was an empty stub, so deleting a tiered object left
it on GCS forever; now deletes via StorageControl (added a control-plane client),
and in_use() actually lists (prefix-scoped) instead of always returning false.
- stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a
compressed, misaligned error vector; heal_object_dir then zipped it against the
full disks array and could make_volume on the WRONG disk. Now returns one
index-aligned entry per slot (None -> DiskNotFound), and heal no longer
pre-fills the drive report (which would double it). Added an alignment test.
Refs rustfs/backlog#807
* fix(kms): stop Vault backend from destroying/reviving keys on failure
Two confirmed key-safety defects in the Vault KV2 backend:
- get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a
fresh random master key and overwriting the stored value. That destroys the
original key material, making every DEK ever wrapped by it permanently
undecryptable. Decryption must never mutate the stored key: both branches now
return a cryptographic_error instead. (The empty-material bootstrap path, which
only fills a never-initialized key, is intentionally left intact.)
- cancel_key_deletion() reset key_state to Enabled only in the returned response
and never persisted it, so the key stayed PendingDeletion in storage and would
still be reaped. It now writes the state back via update_key_metadata_in_storage
and fails the request if the write fails.
Adds ignored (Vault-requiring) integration tests documenting both behaviours.
The third item (VaultTransit key state only in memory -> revived as Enabled after
restart) is deferred: a fail-closed guard would break restart availability for all
transit keys; the correct fix needs a persistent metadata store + Vault integration
testing. Tracked in rustfs/backlog#808.
Refs rustfs/backlog#808
* fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk
Two confirmed admin-API defects:
- Standard AssumeRole used the raw client-supplied DurationSeconds with no upper
bound, so a caller could mint near-permanent temporary credentials. Clamp it to
the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared
clamp_assume_role_duration helper, and build the exp claim with saturating_add.
This matches the existing AssumeRoleWithWebIdentity path.
- ImportBucketMetadata only mutated an in-memory map and returned 200, silently
dropping every imported config. It now persists each non-empty config via
metadata_sys::update (which merges onto existing on-disk metadata) and returns
InternalError if a write fails. Mapping extracted to imported_configs_to_persist
with unit tests.
Refs rustfs/backlog#809
* fix(heal): enqueue displacing request in release builds
push_displacing_lower_priority folded the real enqueue call into
debug_assert_eq!(self.push(request), Accepted). In release builds
(debug_assertions off) the whole macro — including its argument — is compiled
out, so after evicting a lower-priority queued item the new high-priority
request was silently dropped and never healed. Hoist self.push(request) out of
the assertion so the side effect runs in all builds. Adds a --release regression
test.
Refs rustfs/backlog#811
* fix(iam): propagate real delete_policy backend errors instead of swallowing them
delete_policy's is_from_notify path had its error handling inverted: a real
backend failure (disk IO / insufficient quorum) evicted the cache and returned
Ok(()), reporting a phantom success while policy.json survived on disk (to be
reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent
success — returned Err. Propagate real errors and let NoSuchPolicy fall through
to the idempotent cache-evict + Ok, matching delete_user / the notification
handler in the same file. Adds a backend-error-injection regression test.
Refs rustfs/backlog#810
* fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard
The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the
deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254)
still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::)
first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests
for compatible-form loopback/metadata and confirms ::1 / :: stay rejected.
Found by adversarial review of the initial fix. Refs rustfs/backlog#813
* fix(ecstore): fix the same last-part loss in the parallel streaming path
put_object_multipart_stream_parallel had the identical off-by-one
(1..total_parts_count) that truncated the last part / produced zero parts for a
single-part upload — reachable when concurrent stream parts are enabled. Reuse
collect_complete_parts, which now returns an error instead of panicking on a gap
in the parts map. Adds a missing-part error test.
Found by adversarial review of the initial fix. Refs rustfs/backlog#807
* fix(kms): local backend must preserve key material on status change
LocalKmsClient (the default KMS backend) regenerated the master key material on
enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status
change. A single disable+enable cycle therefore destroyed the original key,
making every DEK ever wrapped by it permanently undecryptable (silent data loss,
no network needed). Preserve the existing material via get_key_material and
re-save with only the status changed. Adds a hermetic regression test that wraps
a DEK, cycles all four status methods, and asserts the DEK still decrypts.
Found by adversarial review of the Vault fix. Refs rustfs/backlog#808
* test(rio): cover the length-prefix guard; correct its comment
Add a DecompressReader test that feeds an unterminated length varint so uvarint
returns 0 and the new guard (not the downstream codec) produces the InvalidData
error, and reword the guard comment which overclaimed that the > len bound
prevents a reachable panic (it is belt-and-suspenders). No behavior change.
Found by adversarial review. Refs rustfs/backlog#812
* test(rio): build test block headers via vec! to satisfy clippy
The new corrupted-block tests built the header with Vec::new() + repeated push,
tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed
header bytes with vec![] instead. No behavior change.
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(object-capacity): stop cancelled/remote-disk refreshes from corrupting capacity
Two independent capacity-refresh bugs (backlog rustfs/backlog#805):
- refresh_or_join set the singleflight `running` flag then awaited the
refresh future with no drop guard. When the admin request that became
leader was cancelled (client disconnect) mid-await, `running` stayed true
forever: joiners blocked indefinitely and the 120s scheduled refresh could
never start again. catch_unwind covered panics but not cancellation. A
RefreshLeaderGuard now resets the state and publishes an error on drop.
- capacity_disk_refs mapped the cluster-wide storage_info disk list without
filtering non-local disks, so admin-triggered refreshes ran a local WalkDir
over remote disks' drive_path. On multi-node clusters this double-counted
local bytes (shared mount layout) or hit NotFound (per-node layouts), and
poisoned the per-disk cache the scheduled local-only refresh depends on,
making the cached total oscillate. Filter to local disks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(rio): harden internode HTTP client build, cache, and PUT body integrity
Follow-ups from the internode HTTP review (backlog rustfs/backlog#805):
- handle_put_file accepted a truncated body as success: a HttpWriter dropped
mid-stream closes the chunked body cleanly, indistinguishable from EOF, and
the server never compared bytes copied against the declared size. Reject
size mismatches on the create path (append/unknown-size writes send size<=0
and are exempt).
- build_http_client used `.expect()` on ClientBuilder::build(), which runs
lazily on the first request and on every TLS generation bump (cert
rotation), so a build failure panicked a serving task. It now returns an
io::Error; get_http_client falls back to the previous TLS generation when a
rebuild fails instead of failing the request.
- CLIENT_CACHE was a tokio::Mutex taken on every stream open (data_shards
times per GET). Replaced with arc_swap::ArcSwapOption for lock-free reads on
the hot path; the generation-monotonic replacement guard is preserved via rcu.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
perf(get,heal): land verified fixes for backlog #800-#804
Five fixes from the GET-path performance audit and scanner/heal
completeness audit (rustfs/backlog#800..#804), each verified locally:
- backlog#800 (heal checkpoint O(N^2)): ResumeCheckpoint object sets are
now HashSet (Vec::contains was O(n) per healed object; 1.5ms at n=1M
vs 11ns measured), and per-object checkpoint/resume-state persistence
is batched (1000 mutations / 5s) instead of rewriting the whole file
per object. complete_page() prunes the sets at page boundaries so
memory stays bounded; positions still persist unconditionally and
legacy Vec-format checkpoints still deserialize.
- backlog#801 (DiskInfo.healing never set): erasure-set heal now writes
a healing marker (.rustfs.sys/healing.bin) on the disks it rebuilds
(endpoints plumbed via HealRequest/HealTask.heal_endpoints from the
auto disk scanner) and clears it on success. LocalDisk::disk_info
surfaces the marker, so scanner heal coordination, lock selection and
admin/metrics healing counts see the rebuild.
- backlog#802 (cache probe after data read): new GetObjectBodyCacheHook
in ecstore lets the app-layer object data cache serve the body inside
get_object_reader, after metadata quorum resolution (etag known) but
before the erasure shard read/decode. Previously a hit still paid the
full disk read. Hook is None/no-op when the cache is disabled.
- backlog#803 (GET hot-path redundant work): ObjectInfo is cloned for
event notification only when an event will actually be built (GET and
HEAD paths; events are currently suppressed so the clone was pure
waste); get_opts/put_opts/del_opts resolve bucket versioning with one
metadata-sys lookup instead of two; skip_verify_bitrot and
get_lock_acquire_timeout env reads are cached via OnceLock; the
io-priority metric is no longer double-counted; GetObject input fields
are cloned selectively instead of cloning the whole input.
- backlog#804 (disk permit starvation): the disk-read permit wait is now
bounded (RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT, default 5s, 0 =
previous unbounded behavior); on timeout the GET proceeds without a
permit and the bypass is counted. DiskReadPermitReader also releases
the permit at body EOF instead of holding it until the client drops
the stream.
Verification: make pre-commit; cargo clippy -D warnings on the four
changed crates; full rustfs lib suite (2096 tests) green; rustfs-heal
lib suite green with new unit tests for checkpoint pruning/legacy
format/throttle, permit EOF release, and the cache hook (hit + SSE
skip). The heal_integration_test and one set_disk listing test fail
identically on unmodified main (pre-existing global-state ordering
flakes, verified via git stash A/B).
Co-authored-by: houseme <housemecn@gmail.com>
* feat(cache): add object data cache engine
* feat(cache): wire app-layer object cache flow
* refactor(cache): streamline app-layer cache flow
* refactor(cache): tighten cache flow internals
* refactor: address final clippy cleanup
* chore(deps): update quick-xml to 0.41.0
* feat(cache): wire object data cache env config
* fix(cache): gate materialize fill by cache plan
* chore(cache): add object data cache benchmark gate
* fix(cache): guard object cache fill size mismatches
* refactor(cache): streamline object cache body planning
* fix(cache): align object cache rollout config
* test(cache): cover buffered object cache benchmark
* test(cache): isolate object cache benchmark metrics
* test(cache): mark materialize rollout experimental
* test(cache): tighten object cache benchmark gate
* fix(cache): address review findings for object data cache
- singleflight: clean up leader entry on cancellation (Drop impl) so a dropped GET future can no longer wedge all subsequent fills for the same key; switch the fill map to a std Mutex and add a regression test
- adapter: honor RUSTFS_OBJECT_DATA_CACHE_ENABLE=true by defaulting to hit_only when no explicit mode is set (explicit mode still wins)
- planner: treat nil version UUIDs as "no value" per repo convention so unversioned objects key under the canonical "null" instead of fragmenting the key space
- multipart: invalidate the object cache on the quota-exceeded rollback delete after complete-multipart, closing a stale-cache window
- layering: move the disabled-cache fallback into app::context and drop the new infra->app layer-dependency baseline entry
* fix(cache): close invalidation races and drop full-cache scan on writes
- index: make identity-index insert/remove/prune atomic via starshard compute_if_present/compute_if_absent so concurrent fills can no longer drop each other's keys (lost keys made entries unreachable to invalidation until TTL); add a concurrency regression test
- fill: register the key in the identity index before the entry becomes visible in the cache and re-check the index afterwards, undoing the fill when an invalidation raced in between (new skipped_invalidation_race fill result)
- invalidate: with the index now authoritative, remove the full-cache iter() fallback that made every PUT/DELETE of a never-cached object O(total cache entries) (two scans per PUT, 2N per batch delete)
- materialize-fill: fail the GET instead of falling back to the partially consumed stream after a mid-read error (the fallback would send a body missing its prefix under a full-length Content-Length), and log the same size-mismatch warning as the sibling buffering paths
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(storage): fix media-dependent buffer clamp expectation
test_concurrency_manager_multi_factor_strategy_buffer_clamp asserted media_cap.min(MI_B), but the implementation's final safety clamp is [32KiB, media_cap.max(MI_B)] — deliberately so a media cap above 1MiB (NVMe's 2MiB default) stays effective. The test only passed on machines detected as SSD/Unknown (cap == 1MiB) and failed on NVMe-backed CI runners with 2MiB != 1MiB. Assert the media cap itself, which is what the strategy actually guarantees on every environment.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(storage): format buffer clamp assertion
* chore(logging): update tier guardrail path
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Zhengchao An