perf(get,heal): land verified fixes for backlog #800-#804
Five fixes from the GET-path performance audit and scanner/heal
completeness audit (rustfs/backlog#800..#804), each verified locally:
- backlog#800 (heal checkpoint O(N^2)): ResumeCheckpoint object sets are
now HashSet (Vec::contains was O(n) per healed object; 1.5ms at n=1M
vs 11ns measured), and per-object checkpoint/resume-state persistence
is batched (1000 mutations / 5s) instead of rewriting the whole file
per object. complete_page() prunes the sets at page boundaries so
memory stays bounded; positions still persist unconditionally and
legacy Vec-format checkpoints still deserialize.
- backlog#801 (DiskInfo.healing never set): erasure-set heal now writes
a healing marker (.rustfs.sys/healing.bin) on the disks it rebuilds
(endpoints plumbed via HealRequest/HealTask.heal_endpoints from the
auto disk scanner) and clears it on success. LocalDisk::disk_info
surfaces the marker, so scanner heal coordination, lock selection and
admin/metrics healing counts see the rebuild.
- backlog#802 (cache probe after data read): new GetObjectBodyCacheHook
in ecstore lets the app-layer object data cache serve the body inside
get_object_reader, after metadata quorum resolution (etag known) but
before the erasure shard read/decode. Previously a hit still paid the
full disk read. Hook is None/no-op when the cache is disabled.
- backlog#803 (GET hot-path redundant work): ObjectInfo is cloned for
event notification only when an event will actually be built (GET and
HEAD paths; events are currently suppressed so the clone was pure
waste); get_opts/put_opts/del_opts resolve bucket versioning with one
metadata-sys lookup instead of two; skip_verify_bitrot and
get_lock_acquire_timeout env reads are cached via OnceLock; the
io-priority metric is no longer double-counted; GetObject input fields
are cloned selectively instead of cloning the whole input.
- backlog#804 (disk permit starvation): the disk-read permit wait is now
bounded (RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT, default 5s, 0 =
previous unbounded behavior); on timeout the GET proceeds without a
permit and the bypass is counted. DiskReadPermitReader also releases
the permit at body EOF instead of holding it until the client drops
the stream.
Verification: make pre-commit; cargo clippy -D warnings on the four
changed crates; full rustfs lib suite (2096 tests) green; rustfs-heal
lib suite green with new unit tests for checkpoint pruning/legacy
format/throttle, permit EOF release, and the cache hook (hit + SSE
skip). The heal_integration_test and one set_disk listing test fail
identically on unmodified main (pre-existing global-state ordering
flakes, verified via git stash A/B).
Co-authored-by: houseme <housemecn@gmail.com>
* feat(cache): add object data cache engine
* feat(cache): wire app-layer object cache flow
* refactor(cache): streamline app-layer cache flow
* refactor(cache): tighten cache flow internals
* refactor: address final clippy cleanup
* chore(deps): update quick-xml to 0.41.0
* feat(cache): wire object data cache env config
* fix(cache): gate materialize fill by cache plan
* chore(cache): add object data cache benchmark gate
* fix(cache): guard object cache fill size mismatches
* refactor(cache): streamline object cache body planning
* fix(cache): align object cache rollout config
* test(cache): cover buffered object cache benchmark
* test(cache): isolate object cache benchmark metrics
* test(cache): mark materialize rollout experimental
* test(cache): tighten object cache benchmark gate
* fix(cache): address review findings for object data cache
- singleflight: clean up leader entry on cancellation (Drop impl) so a dropped GET future can no longer wedge all subsequent fills for the same key; switch the fill map to a std Mutex and add a regression test
- adapter: honor RUSTFS_OBJECT_DATA_CACHE_ENABLE=true by defaulting to hit_only when no explicit mode is set (explicit mode still wins)
- planner: treat nil version UUIDs as "no value" per repo convention so unversioned objects key under the canonical "null" instead of fragmenting the key space
- multipart: invalidate the object cache on the quota-exceeded rollback delete after complete-multipart, closing a stale-cache window
- layering: move the disabled-cache fallback into app::context and drop the new infra->app layer-dependency baseline entry
* fix(cache): close invalidation races and drop full-cache scan on writes
- index: make identity-index insert/remove/prune atomic via starshard compute_if_present/compute_if_absent so concurrent fills can no longer drop each other's keys (lost keys made entries unreachable to invalidation until TTL); add a concurrency regression test
- fill: register the key in the identity index before the entry becomes visible in the cache and re-check the index afterwards, undoing the fill when an invalidation raced in between (new skipped_invalidation_race fill result)
- invalidate: with the index now authoritative, remove the full-cache iter() fallback that made every PUT/DELETE of a never-cached object O(total cache entries) (two scans per PUT, 2N per batch delete)
- materialize-fill: fail the GET instead of falling back to the partially consumed stream after a mid-read error (the fallback would send a body missing its prefix under a full-length Content-Length), and log the same size-mismatch warning as the sibling buffering paths
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(storage): fix media-dependent buffer clamp expectation
test_concurrency_manager_multi_factor_strategy_buffer_clamp asserted media_cap.min(MI_B), but the implementation's final safety clamp is [32KiB, media_cap.max(MI_B)] — deliberately so a media cap above 1MiB (NVMe's 2MiB default) stays effective. The test only passed on machines detected as SSD/Unknown (cap == 1MiB) and failed on NVMe-backed CI runners with 2MiB != 1MiB. Assert the media cap itself, which is what the strategy actually guarantees on every environment.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(storage): format buffer clamp assertion
* chore(logging): update tier guardrail path
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Add crates/e2e_test/src/chaos.rs with in-process fault-injection
primitives for a single-node multi-disk RustFS server: take a disk
offline (rename to <dir>.offline), bring it back online, replace a
disk with a fresh empty directory, corrupt an object's erasure shard
(part.* byte flips, xl.meta untouched), and SIGKILL/restart the server
with the same volumes and port.
Add reliability tests on a 4-disk (EC 2+2) topology, verified via
sha256 manifests recorded at write time:
- degraded read/write with one disk offline (incl. multipart object)
- bitrot read-through with two corrupted shards per object
- fresh-disk replacement healed via admin deep heal after a SIGKILL
restart
Also reuse the shared signed_admin_post helper from chaos.rs in the
heal regression suite instead of a duplicated local copy.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(core-storage): fix critical correctness defects from core-storage audit
Fixes verified defects found in a deep audit of the core storage path
(erasure coding, disk persistence, quorum, heal, replication resync):
- ecstore/disk: rewrite live xl.meta atomically (temp+rename) in
delete_versions_internal and write_metadata instead of in-place
truncate, which exposed torn metadata to concurrent readers and
crashes on the DeleteObjects hot path
- ecstore/erasure: allow heal to reconstruct from exactly data_shards
bitrot-verified sources; requiring data_shards+1 made objects
permanently unhealable after losing parity_shards disks
- ecstore/set_disk: direct-memory inline GET applied the erasure
distribution permutation twice (shuffled inputs re-indexed through
distribution), concatenating wrong shards into the response body in
degraded reads; collect from canonical disk-ordered inputs
- ecstore/set_disk: heal now preserves the committed inline layout
instead of recomputing it with a hardcoded unversioned threshold,
which split quorum identity of healed replicas and caused endless
re-heal churn
- ecstore/replication: resync results channel switched from
broadcast(1) to mpsc; a lagged broadcast receiver ended the stats
collector and every subsequent failure went uncounted, letting
failed resyncs be marked completed
- ecstore/replication: ignore an empty persisted resync checkpoint;
resuming with one skipped every object and marked the resync
completed without replicating anything
- ecstore/replication: fix inverted not-found error classification in
replicate_object/replicate_delete logging paths
- ecstore/erasure: guard decode paths against zero block_size or
data_shards from corrupt on-disk metadata (divide-by-zero panic)
- ecstore/disk: os::read_dir no longer consumes the entry limit on
entries it does not return (is_empty_dir misjudgment); create_file
opens with O_TRUNC to avoid stale trailing bytes
- filemeta: treat Some(nil) version id as a null version in
matches_not_strict; disk-loaded headers never store None, so the
mod_time quorum guard for unversioned overwrites never fired and an
interrupted overwrite could displace the committed version in merge
- filemeta: fix msgpack skip lengths for fixext (missed the ext type
byte) and ext16/32 (over-skipped) unknown fields
- filemeta: return FileCorrupt instead of usize underflow when
xl.meta is truncated inside the CRC trailer
- filemeta: surface delete-marker insertion failure in delete_version
instead of reporting success when the data dir is shared
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(replication): drop duplicate cfg(test) etag import from boundary module
The test module already imports content_matches_by_etag locally, so the
top-level cfg(test) import is unused under -D warnings and fails clippy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix(ecstore): purge orphan empty-directory trees on folder delete (#4189)
Empty "phantom" folders — on-disk directory trees with no xl.meta
anywhere — show up in listings as common prefixes but cannot be removed
by any S3 call: DeleteObject on the folder key encodes it to __XLDIR__,
finds no metadata, and returns NotFound, which the API layer masks as a
fake 204 while the tree survives on disk.
Make handle_delete_object attempt a guarded purge when a dir-object key
resolves to NotFound:
- sweep every erasure set of every pool (orphan fragments can sit on
any set, left behind by whichever sets stored the deleted children)
- per set, refuse if ANY disk holds a regular file under the prefix,
so degraded/healable objects are never destroyed
- remove empty directories children-first with non-recursive deletes;
a racing PutObject makes the rmdir fail with DirectoryNotEmpty and
the entry is skipped
This state is unrepresentable on AWS S3 (CommonPrefixes derive only
from real keys; DeleteObject is an idempotent 204), so the purge moves
visible behavior closer to S3. It deliberately goes beyond MinIO, whose
DeleteObject leaves these trees stranded and whose heal only removes
dangling (minority-presence) dirs — a long-standing complaint
(minio/minio#15257, #19516, #18444).
Also move the test-only content_matches_by_etag import in
replication_target_boundary.rs into the cfg(test) module; it failed
cargo clippy -D warnings and blocked the pre-PR gate.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
content_matches_by_etag is only used inside the #[cfg(test)] module, so
the lib-scope import from #4211 fails cargo clippy -D warnings on every
non-test build.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Remove or consolidate 57 test cases that cannot catch regressions
(literal-constant asserts, construct-then-assert, derived-serde
round-trips, near-duplicate env/getter matrices) in common, config,
iam, madmin, and object-capacity, keeping all wire-format and
error-path guards. Add 13 tests for previously uncovered high-risk
behavior: filemeta version-sort determinism and merge resilience to
garbage headers, zip extraction path-traversal rejection and exact
limit boundaries, JWT tampered-signature rejection, and the bytes
variant of dual-key (rustfs/minio) metadata fallback and precedence.
Test-only change; no production code touched.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): remove reachable panics in tiering, replication, and heal paths
- Parse x-amz-expiration leniently in tier PUT responses; any lifecycle
rule on the remote tier bucket returns an RFC1123 date that the previous
ISO8601 unwrap turned into a panic of the ILM transition worker
- Skip invalid user-metadata header values (with a warning) when building
tier and replication PUT headers instead of panicking on non-ASCII input
- Heal: tolerate absent data_dir for delete markers and remote objects
- transition_object: don't unwrap version_id on unversioned buckets when
recording partial writes for offline disks
- Admin server info: use port_or_known_default() so default-port (80/443)
endpoints don't panic is_server_resolvable
- Tier ListObjectsV2 client: decode response body with from_utf8_lossy
- walk_internal: log merge_entry_channels errors instead of dropping them
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): fail heal explicitly when data_dir is missing
Address review feedback: unwrap_or_default() silently substituted a nil
UUID when latest metadata lacked data_dir. Delete markers and remote
objects legitimately have no data_dir and skip the data-heal block, but
for a regular object a missing data_dir means corrupt metadata — return
FileCorrupt with a descriptive log instead of building part paths under
a nil UUID directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Agent-instruction and architecture docs had drifted from the code:
- CLAUDE.md: slim to commands + pointers; fix wrong claim that
`make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
hand-maintained (already stale) scoped-AGENTS index; link
architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
(rebalance-decommission phases, migration-progress ledger, PR
template) to docs/superpowers/plans with archive headers; fix stale
source paths in kept inventories (core/sets.rs, core/pools.rs,
store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
.gitignore now keeps .claude/skills and docs/operations committable
Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhengchao An