mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-27 16:48:58 +00:00
d73c8a783adee1dfdd02c8acc65fbb6ceb9999cd
988 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
ea2e24ac13 |
test/ci(ecstore): fix MinIO SSE interop size assertion + nightly dockerized interop check (#4809)
* test(ecstore): assert decrypted_size for MinIO SSE interop round-trip The ignored MinIO interop round-trip tests asserted `ObjectInfo.size` against the plaintext length. For SSE objects `size` is the on-disk DARE-encrypted size (plaintext + 32 bytes per 64 KiB block), so the assertion can never hold once real fixtures are present — the two `#[ignore]` tests failed the moment a real MinIO-written fixture was fed in, even though the decoded data was byte-identical. The client-visible object size comes from `decrypted_size()` / `get_actual_size()`, which correctly reads MinIO's `x-*-internal-actual-size` metadata (verified: both SSE-S3 and SSE-KMS 8 MiB multipart fixtures now report 8388608). Assert against that instead and keep the plaintext length and SHA-256 data checks. With real 4-drive MinIO fixtures (RELEASE.2025-09-07) all four tests pass, confirming RustFS reads MinIO erasure-coded SSE objects with byte-identical data and correct logical size. Co-Authored-By: heihutu <heihutu@gmail.com> * ci(ecstore): nightly MinIO interop check + dockerized fixture capture Wire the ignored MinIO on-disk interop reader tests into a nightly, non-required CI job, and make their fixtures reproducible without a host MinIO install. - Dockerfile + capture_via_docker.sh: build a throwaway image carrying the official MinIO server binary (pinned RELEASE.2025-09-07) plus the fixture lab on a small Python base, then run `lab.py capture-matrix` to write the SSE-S3 / SSE-KMS multipart fixtures the tests consume. lab.py drives MinIO's S3 API directly, so no `mc` is needed. - .github/workflows/minio-interop.yml: nightly + manual workflow on GitHub-hosted ubuntu-latest (reliable Docker + Python, unlike the self-hosted fleet — see e2e-s3tests.yml infra note). Regenerates the gitignored fixtures each run and executes the #[ignore] reader tests. Not a PR gate. - README: document the Docker capture path. Validated end to end: the script builds the image, captures the two multipart cases, and `cargo nextest run --run-ignored ignored-only` passes all four interop tests. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
c53e34f13b |
test(replication): lock outbound checksum consistency for XXHash/SHA-512/MD5 (#4808)
test(replication): lock outbound checksum consistency for new algorithms (T3) Adds a consistency test at the replication put-options boundary confirming that the AWS 2026-04 additional checksum algorithms (XXHash3/64/128, SHA-512, MD5) are forwarded into replication user_metadata identically to the classic five. The outbound replication path routes a stored object checksum through the algorithm-agnostic decrypt_checksums -> user_metadata flow, so the new algorithms (already covered by rustfs-rio read_checksums) need no new-algorithm-specific handling. This locks that behavior against regressions. Investigation summary (no code change needed on the outbound side): the per-algorithm ChecksumMode selection path is dormant (opts.checksum is never set to a specific algorithm; tiering uses Content-MD5; the add_crc bool is dead code), so extending ChecksumMode was unnecessary. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
750e5d15eb |
feat(checksums): add native S3 additional checksum support (#4805)
* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2) Add the AWS 2026-04 additional checksum algorithms as base types in rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len, hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the single source of truth, allocate the new base-type bits append-only above bit 9 to preserve the on-disk varint format, and add streaming hashers whose digest uses the S3 canonical big-endian encoding (seed 0). The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is rejected and they are never routed through add_part()/can_merge(). A round-trip guardrail test asserts every base type survives all dispatch sites, failing loudly if a future algorithm is added but a match arm or the mask is forgotten. Refs rustfs/backlog#1254 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3) Lock the byte order and seed of the new algorithms against the OFFICIAL upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128, SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte identical to what AWS SDKs (awscrt) compute — the interop correctness this feature hinges on. Add a non-empty regression lock (official "fox" vectors) that also asserts the encoded field is the standard-base64 of the raw digest. Refs rustfs/backlog#1255 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8) Cover the xl.meta varint (de)serialization for the new algorithms: to_bytes() -> read_checksums() must recover the value under the Display key for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node reading a future, unknown base-type bit degrades safely — skips the entry and returns without panicking or mis-decoding a length. Combined with the append-only bit allocation from S2, this protects mixed-version clusters. Refs rustfs/backlog#1260 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5) HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128 and SHA-512 checksums that S3 stored, closing the head_object gap in #4800. s3s HeadObjectOutput has no typed field for these, so they are emitted as raw response headers via response.headers (the same mechanism RustFS already uses for tagging-count), keyed by ChecksumType::key(). The existing five typed algorithms are unchanged. Also carries the Cargo.lock update for the xxhash-rust dependency introduced in S2. Refs rustfs/backlog#1257 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(checksums): fail-closed on unknown checksum algorithm (S7) A. Harden unknown/unsupported checksum algorithms to fail closed instead of panicking. ChecksumMode::base() in the outbound S3 client (crates/ecstore/src/client/checksum.rs) previously did `panic!("enum err.")` for any mode without a concrete base algorithm (e.g. a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added unit tests proving base() never panics and hasher() returns Err for unsupported modes. rustfs-checksums FromStr already returns Err on unknown names; added a regression test asserting garbage/unknown names fail closed. B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str, into_impl, name constants, the x-amz-checksum-* header constants and the HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128 big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each digest against a direct library computation. MD5 stays intentionally rejected (PR #4513) and is left untouched. C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8" with 7 variants already consuming 7 bits; adding the 4 new algorithms would overflow u8 and require a breaking repr change, so ChecksumMode is left unchanged. The new algorithms are available through the rustfs-checksums ChecksumAlgorithm path. Refs rustfs/backlog#1259 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4) Complete the additional-checksum round-trip so AWS SDKs can verify integrity on download and confirm it on upload: - GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and SHA-512 checksums (the download-side path SDKs auto-verify). The values flow from build_get_object_checksums through GetObjectOutputContext into finalize_get_object_response and are emitted after wrap_response_with_cors. - PutObject echoes the server-computed additional checksum on its response, captured at the want_checksum set points before opts is moved. Both reuse a single centralized helper, inject_additional_checksum_headers, which HeadObject now also uses. This is the ONLY place that emits these headers, so when s3s gains typed fields for these algorithms the migration is one spot (fill the typed field, drop the insert) with no risk of duplicate headers. The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value lands after the body) is left for e2e coverage in S10. Refs rustfs/backlog#1257 rustfs/backlog#1256 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9) Make multipart uploads work end-to-end for the composite-only algorithms (XXHash3/64/128, SHA-512): - complete_part_checksum previously returned the outer None for any algorithm outside the five typed ones, which failed CompleteMultipartUpload with InvalidPart. It now accepts any valid base type with no double-check value (Some(None)) — mirroring the missing-value path of the typed algorithms — since s3s CompletePart has no field to carry a client-supplied per-part value and the part was already verified server-side at UploadPart. Genuinely unset/invalid types are still rejected. - The existing COMPOSITE assembly (Checksum::new_from_data over the concatenated per-part raw digests; full_object_requested() is false so add_part() is correctly bypassed) already works for these algorithms via the S2 wiring. A rio test locks the assembly and that add_part refuses them. - UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on their responses via the shared inject_additional_checksum_headers helper (now pub(crate)), since s3s has no typed output field. Refs rustfs/backlog#1261 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6) Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest, COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector. Thanks to the single-source-of-truth wiring from S2, every dispatch site (GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all arm — no handler changes needed. Tests are extended to cover MD5 across them. Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client reject "md5" so it could never silently fall back to CRC32. This change is on the server-side rio path and never falls back — it implements MD5 correctly rather than substituting another algorithm — so the #4513 intent is preserved, and the outbound client keeps rejecting md5 (S7). Refs rustfs/backlog#1258 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11) from_string_with_obj_type ran alg.to_uppercase() on every checksummed request, allocating a String just to compare against a fixed set of algorithm names. Replace it with eq_ignore_ascii_case, which is allocation-free and, for the ASCII algorithm names involved, exactly equivalent. A test locks that case-insensitivity, the CRC64NVME full-object assumption, composite-only FULL_OBJECT rejection, and unknown/empty handling are all unchanged. The other S11 notes are intentionally not acted on: the Phase-0 header scan is N/A (we chose full support over rejection, so there is no reject guard), and parallelizing the serialized hash passes is deferred pending a measured need. Refs rustfs/backlog#1263 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(checksums): collapse 5 duplicated response-checksum loops into one Review of the accumulated commits found the same "iterate decrypted checksums, match five typed algorithms, drop the rest" loop copy-pasted across five response paths (GetObject, HeadObject, GetObjectAttributes object-level and part-level, CompleteMultipartUpload). That was patch-on-patch duplication. Collapse it into a single source of truth: - rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the five-typed vs additional-algorithm split. - object_usecase gains ResponseChecksums + classify_response_checksums(), which performs the typed/extra split once. All five call sites now destructure its result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a hand-rolled five-way comparison. Behaviour is unchanged (GetObjectAttributes still cannot surface the additional algorithms — an s3s XML-body limitation, now documented in one spot). One pass over the map; extra pairs pushed only when a new-algorithm checksum is present. Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(checksums): unit tests for classifier/echo helpers + fix unused import Add direct unit tests for the refactored single-source-of-truth helpers: - rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification. - object_usecase classify_response_checksums() — typed fields vs `extra` headers, the checksum-type marker, and empty input. - additional_checksum_echo_pairs() — echo pair only for additional algorithms, none for the five typed ones, none for None. - inject_additional_checksum_headers() — writes all pairs; empty is a no-op. Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left by the classifier refactor (would fail the -D warnings gate). Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length) The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword the S8 forward-compat comment; no code change. Refs rustfs/backlog#1260 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10) Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04 additional algorithms. aws_sdk_s3 has no typed builder for these, so the x-amz-checksum-<algo> header is injected via mutate_request (value from rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic checksum calculation disabled (request_checksum_calculation=WhenRequired) so the injected header is the only checksum on the wire. For each of XXHash3/64/128, SHA-512 and MD5: a correct value is accepted and the object stored intact; a mismatched value is rejected with BadDigest and nothing is stored. Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that additionally confirms the HEAD/GET header echo (14/14). Refs rustfs/backlog#1262 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * style(get): allow too_many_arguments on finalize_get_object_response The classifier refactor added an extra_checksum_headers parameter, pushing finalize_get_object_response to 8 args and tripping clippy::too_many_arguments under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already carry; no behavior change. Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
escapecode
|
a80699b6dd |
feat: add an opt-in NATS JetStream publish path for the notify and audit targets (#4634)
feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets The NATS notify and audit targets publish through NATS Core, which returns before the server has durably accepted the message. A broker restart or a connection drop between the publish and the flush loses the event, even though the send queue has already cleared it, and no acknowledgement gates that clear. An opt-in JetStream publish path clears a queued event only after the server returns a durable PublishAck, so delivery is at-least-once across a broker restart or a reconnect. It applies to both the notify and audit NATS targets, is off by default, and is byte-identical to the NATS Core path when disabled. The path includes durable store-and-forward, a stable dedup id sent as the Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate window, pre-flight stream validation, and a bounded failed-events store for terminally-failed and retry-exhausted events. Three configuration keys per target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_ prefixes. The on-disk batch filename separator changes from colon to underscore so batch names are valid on Windows filesystems, with transparent read-back of files written under the previous separator. The migration affects the shared queue store for every target type and lands with this feature because the store gains its first Windows-exercised paths here. Co-authored-by: houseme <housemecn@gmail.com> |
||
|
cxymds
|
25f81f812c |
feat(site-replication): support custom TLS peers (#4802)
* feat(madmin): add site replication TLS settings * feat(site-replication): support custom TLS peers * test(site-replication): remove redundant clones * test(site-replication): avoid needless resolver collection |
||
|
Zhengchao An
|
e9a0200a72 | fix(ecstore): hedge slow shard reads in lockstep GET to cut the large-object first-byte tail (#4799) | ||
|
cxymds
|
7ece747eab | fix(ecstore): suppress missing rollback rename warnings (#4792) | ||
|
Zhengchao An
|
6096bb189d | fix(ecstore): demote reliable_rename NotFound WARN to debug (#4789) | ||
|
houseme
|
1553dc3f62 |
Address P2 follow-ups from the 2026-07-10..12 merged-PR review (backlog#1210-1220) (#4783)
* fix(obs): open cleaner compression source with O_NOFOLLOW The compressor opened the source log via File::open, which follows a symlink at the final path component. Between the scanner selecting a regular file and this open, an attacker with write access to the log directory could swap the entry for a symlink (TOCTOU) pointing at, say, /etc/shadow, whose contents would then be copied into an archive. Open the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the temp/archive path already refused symlinks, this closes the source side. Refs rustfs/backlog#1210 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): recompress instead of trusting leftover cleaner archives archive_header_ok only checked the first 2-4 magic bytes before treating an existing .gz/.zst as a completed prior result and letting the caller delete the source log. A file with valid magic but a truncated or forged body passes that check, so an attacker with write access to the log directory (or a crashed prior run) could plant such a stub and make the cleaner delete the real log without ever producing a usable archive — silent audit-data loss. Chosen fix: stop trusting cross-process leftovers entirely and always recompress the source in this pass, rather than fully decoding every leftover to validate it. Full-decode validation would add real CPU cost and decode-bug surface for a rare crash-recovery case; the existing atomic create_new+rename already overwrites whatever sits at the archive path (a planted symlink is replaced, never followed) with a freshly written, fsync'd archive, so a partial/forged leftover can never gate source deletion. This is the lowest-regression option. Refs rustfs/backlog#1211 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): cap memory-gate reservation at cache growth headroom The memory gate subtracts `admitted_since_refresh` from the snapshot's available bytes so a burst arriving faster than the 5 s refresh cannot over-allocate. That counter is GROSS: it only rolls over on the refresh and never rolls back when a fill is later evicted, cancelled, or loses the invalidation race. Under sustained high-throughput churn (net footprint flat and far below `max_capacity`) the raw counter balloons past the memory the cache actually holds, so `effective_available` collapses and the gate reports false memory pressure — skipping the hottest fills with SkippedMemoryPressure until the next 5 s refresh. This only lowers hit rate; it never returns wrong data and self-heals each refresh. Fix direction 1 (minimal regression): cap the reservation deduction at the cache's own growth headroom (`max_capacity - weighted_size()`) instead of letting the unbounded gross counter shrink the system-available budget. The cache can never hold more than `max_capacity`, so a burst adds at most that headroom of real memory before moka evicts to stay bounded (net-zero churn beyond that point) — capping the deduction there keeps the reservation honest without treating gross churn as growth. Chosen over net-accounting (direction 2, releasing bytes on every failure/cancel/eviction path) because that only plugs the leak on failed fills and would not address the core defect: churn of *successful* insert/evict fills over the 5 s window. It also touches only the gate plus one call site rather than every failure path in moka_backend. The cap only ever raises `effective_available`, so real memory pressure (a low snapshot at refresh) still suppresses fills; when the cache is at capacity the headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn. `MokaBackend` now stores `max_capacity` and passes the live headroom into `allows_fill`. Adds targeted gate tests: gross churn far above headroom no longer falsely suppresses, yet the reservation still bounds a burst while the cache can genuinely grow. Refs rustfs/backlog#1212 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): assert native O_DIRECT path runs in uring read test uring_preserves_o_direct_for_eligible_reads only compared bytes through LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the read silently degrades to the buffered StdBackend fallback and the byte check still passes, so the test could go green without the native read_at_direct path ever executing -- a vacuous pass. Add a per-disk native_direct_reads counter on UringBackend, incremented only when pread_uring_direct completes, and rebuild the test to drive a real UringBackend's pread_bytes and assert the counter is non-zero (every eligible read went through the native tier). When io_uring or O_DIRECT is unavailable on the host filesystem (restricted CI runners, tmpfs), the test skips loudly via eprintln instead of asserting a tautology, while still checking byte-correctness on whatever tier served the read. The counter also gives a gray release a positive signal that the O_DIRECT tier is serving reads, not just a fallback count. Refs rustfs/backlog#1213 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads classify_direct_read_error is only reached from the read side: the O_DIRECT open in pread_uring_direct already succeeded (an open-time refusal is handled earlier as DirectOpenError::ODirectRefused). So an EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel accepted for O_DIRECT -- far more likely an alignment bug in the aligned read path than an unsupported filesystem. The old code latched the disk's native path off with only a once-per-disk debug trace, hiding a potential correctness regression behind a silent buffered-read downgrade. Diagnostics only: the fallback behaviour is unchanged (the native path is still latched off and the caller still reads via StdBackend). This adds a rustfs_io_uring_direct_read_einval_total counter and promotes the once-per-disk trace from debug to warn so an operator can see an alignment regression instead of an unexplained latency/CPU shift. Refs rustfs/backlog#1214 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document data-blocks-first default and its tail-latency cost DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay true: deferred-parity is the deliberate, already-rolled-out full-object GET default from backlog#1159/#923. Flipping it back to false in code would silently revert that rollout for every deployment that has not set the env var, so this commit only documents -- no behaviour change. The added notes explain what data-blocks-first does (schedule data shards up front, engage parity lazily on a missing/corrupt data shard), the known trade-off (parity is engaged late, so a slow-but-not-dead data drive raises GET p99 because the faster parity shards are not raced against it until a data shard is declared missing), and the operational rollback switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is intentionally an env override rather than a code default change. No metric was added: the low-risk observability hook for "slow data drive engaged deferred parity" would live at the deferred-stripe engage point, which is out of this file's scope; this change stays documentation-only to avoid touching the hot GET path. Refs rustfs/backlog#1215 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document wide-directory walk stall hazard and tuning list_dir enumerates a whole directory in one os::read_dir call (count = -1), and the walk caller bounds that entire enumeration with the per-read stall budget (default 5s) as if it were a single read. For a wide, flat prefix -- one directory holding millions of immediate children -- a single readdir can exceed the budget on a healthy disk, trip DiskError::Timeout, and surface as a ListObjects 500 quorum failure though the drive is fine (a #2999 sub-class). This is documented, not rewritten: turning the one-shot readdir into a streaming/batched enumeration that refreshes the stall deadline between chunks is an architecture-level change with high regression surface (ordering, the count contract, quorum merge) and belongs in a separate follow-up. The supported mitigation today is operational, so the comments point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS and the high-latency drive-timeout profile, which widen the budget with no code change. Notes were added at list_dir, the scan_dir call site, and get_drive_walkdir_stall_timeout. No behaviour change. Refs rustfs/backlog#1216 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document consumer-peek vs producer-stall coupling In list_path_raw the consumer's peek_timeout is drawn from the same source and same value (walkdir_stall_timeout, default 5s) as the producer-side walk stall budget, but the two measure different things: the producer stall bounds a single drive read, while the consumer peek bounds the gap between two ADJACENT entries arriving from a reader. Because they share a value, the consumer cannot wait meaningfully longer for the next entry than the producer is allowed to spend producing one. Walking a region dense with non-listable internal items can make a HEALTHY drive miss the budget between visible entries; the consumer then declares it stalled and detaches it, dropping a good drive from the merge and capping the "large prefix succeeds" guarantee. Documented, not decoupled: giving the consumer peek an independent, strictly-larger budget would cut these false detaches but equally delays detaching a genuinely dead drive and shifts listing tail-latency semantics, so it wants soak data before changing the default. The comment records the invariant any such follow-up must keep -- consumer peek >= producer stall, never stricter -- so it can never fail a drive before the producer would. No behaviour change. Refs rustfs/backlog#1217 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(io-metrics): add time-based trigger for low-IOPS latency percentiles Percentiles were recomputed only every 128 IOs and seeded to 0, so a low-traffic deployment exported p95/p99 = 0/stale for a long time after startup. Add a 10s wall-clock trigger alongside the count throttle so the first recompute can fire before 128 samples accrue. Hot-path per-op mean update is unchanged. Refs rustfs/backlog#1218 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): cover codec-streaming parity under fault injection and NoSuchKey The codec-streaming compat A/B previously ran only against a healthy 4-disk EC set with successful full GETs: the DiskFaultHarness was constructed but never faulted, the error path was untested, and the range assertion silently compared legacy-vs-legacy (ranges always fall back to the duplex path), overstating what it proved. Add two genuinely-failable scenarios reusing the existing harness and fixtures: - Parity reconstruction A/B: take one data disk offline and re-run the full object matrix on both phases while the EC 2+2 set rebuilds each large object from the surviving shards. Assert codec == legacy byte-for-byte (sha256) and header-for-header, and assert the codec phase served the reconstructed objects with zero duplex-pipe fallback (the reader gate is drive-health-independent, so the codec fast path is really exercised through reconstruction). - NoSuchKey negative path: compare the HTTP status + S3 error code of a missing-key GET across the legacy and codec phases and require them to be identical (404/NoSuchKey), guarding against the codec env perturbing the error path. Also clarify the range-phase comment so it is not misread as codec-range correctness coverage: both sides are served by the same legacy range path, so the assertion only proves ranges keep working and keep falling back to legacy with the gates open. Verified: cargo check/--no-run pass and the test passes locally (1 passed; dup_codec=0 confirms the codec path ran). Refs rustfs/backlog#1219 Co-Authored-By: heihutu <heihutu@gmail.com> * ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback The uring-integration leg ran on the runner's default TMPDIR, which may sit on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct path silently latches off to the aligned StdBackend fallback. Mount a dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep (bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are actually covered rather than validated only by signature diffing. Refs rustfs/backlog#1220 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
63b4568f85 |
fix(ecstore): reclaim orphan data dirs on the healthy heal path (#4781)
PR #4356 wired `reclaim_orphan_data_dirs` only into `heal_object`'s post-heal tail, which runs after the `disks_to_heal_count == 0` early return. That early return is exactly the state of the objects the sweep targets: a valid `xl.meta` with all shards present plus a leaked pre-#3510 data dir needs no shard healing, so a healthy heal returned before reclaim and swept nothing. On a healthy deployment (single node, no degraded disks) the reclaim was therefore dead code — an admin heal walked the objects, "healed" them, and reclaimed no leaked space. Run the best-effort reclaim on the `disks_to_heal_count == 0` path as well, gated on `!opts.dry_run`. The shared match+log block is factored into `reclaim_orphan_data_dirs_best_effort` so both exits behave identically. A reclaim failure still never fails the heal. Adds an end-to-end regression: put a healthy non-inline object, plant an unreferenced UUID data dir under it on every disk that holds the object, then drive `heal_object`. A dry-run heal must leave the stray in place; a real heal must reclaim it while preserving the live data dirs, `xl.meta`, and object contents. The test fails against the pre-fix control flow. Refs #3231, #3191, #4356. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
418f505a81 | fix(ecstore): allow concurrent directory scans (#4778) | ||
|
houseme
|
13bdca6762 |
build(toolchain): switch Rust channel to stable (#4775)
* Change Rust toolchain channel to stable Signed-off-by: houseme <housemecn@gmail.com> * style: apply clippy --fix and cargo fix lint suggestions Run `cargo clippy --fix --all-targets --all-features` and `cargo fix --lib --all-targets` across the workspace, then resolve the remaining warnings by hand: - collapse needless borrows in `format!` args, prefer `?` over explicit early returns, and use `.values()` / `.flatten()` iterator adapters - rewrite the `Md5` scan loop via `manual_flatten` and re-indent the `select!` macro body (rustfmt skips macro interiors) - annotate the intentional dead-code `Md5` inherent methods (constructed only by the test factory) with `#[allow(dead_code)]` Behavior is unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
d8e69a3adf |
fix(logging): enforce single-writer sinks and bound tracing (#4765)
* fix(obs): prevent rolling log stdout aliasing * fix(ecstore): bound hot-path tracing payloads * test(logging): guard service and disk log invariants * fix(obs): silence useless_conversion on st_dev for Linux clippy rustix's Stat.st_dev is u64 on Linux/glibc, making u64::try_from a no-op that trips clippy::useless_conversion under -D warnings. The conversion is still needed on macOS/BSD where st_dev is a signed dev_t, so suppress the lint on that line rather than dropping the portable fallible conversion. * fix(logging): keep stdout sink validation portable (#4769) * fix(obs): keep stdout device conversion portable * docs(logging): update single-writer plan status --------- Co-authored-by: overtrue <anzhengchao@gmail.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
2ddafb4ed9 |
test(ecstore): bound file sync probe waits (#4767)
Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
Zhengchao An
|
4c9431704c | fix(ecstore): cancel orphaned listing walks (#4773) | ||
|
Zhengchao An
|
b540c7e2d0 | test(ecstore): cover list marker key stripping (#4757) | ||
|
Zhengchao An
|
a5765274fc |
fix: auto-repair test_rename_data_shares_file_sync_limit hang on macOS (#4758)
fix(test): use canonicalized disk root for file_sync_probe in rename_data test On macOS, tempfile::tempdir returns /var/folders/... while LocalDisk resolves the root to /private/var/folders/... via dunce::canonicalize. The file_sync_probe::enter() path check uses starts_with(), so passing the non-canonical tempdir path caused the probe to never activate, making wait_for_active() hang indefinitely. Use disk.root (already canonicalized) for the probe instead. |
||
|
houseme
|
6886dca7d1 |
feat(ecstore): make GET codec-streaming a single rollout switch (backlog#1183) (#4752)
backlog#1183, staged rollout step. Simplify enabling the zero-duplex GET
codec-streaming fast path to a single switch — RUSTFS_GET_CODEC_STREAMING_ROLLOUT
(default "off") — now that body/header parity is proven (parity e2e net + bench
A/B on backlog#1183).
- Add a clean production rollout token "on" (aliases "full"/"production"); the
legacy "internal"/"benchmark" tokens remain accepted.
- Flip DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE and the two ..._COMPAT_CONFIRMED
defaults to true. They are retained as emergency kill-switches (set any to
false to force the fast path off) but no longer gate enablement — the rollout
switch does.
No production behavior change: with no env set the rollout switch defaults to
"off", so GET stays on the legacy duplex path exactly as before. Flipping the
hard default to on is deferred to a follow-up after a production soak.
Note (intentional semantics change): with the rollout switch opted in
("on"/"internal"/"benchmark"), codec streaming now activates without also
setting the two ..._COMPAT_CONFIRMED vars — compatibility is confirmed, so those
confirmations are baked in.
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
Ramakrishna Chilaka
|
028ba6a675 |
perf(ecstore): parallelize multipart shard syncing (#4734)
Bound large shard-directory syncs per disk and process while preserving small-directory and durability behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
houseme
|
89557a7ffe |
perf(ecstore): cache io_uring fallback root label (#4747)
`record_uring_fallback` is called at multiple sites in the io_uring read path whenever a read falls back to `StdBackend`. It formatted `self.root.display().to_string()` on every call, heap-allocating a `String` from a `Path` that never changes after construction — pure per-read waste when io_uring is degraded. Cache the label once in `UringBackend::try_new` as a `String` field (`root_label`) and clone it per emission. The metric name and the `"root"` label value are unchanged; only the redundant `Path` formatting is removed. The clone is a single alloc of an already-short string. Refs: rustfs/backlog#1185 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
763f246f8a |
test(ecstore): add shared MockWarmBackend test utility for lifecycle and tier tests (#4716)
* test(ecstore): extract shared MockWarmBackend into a test-util feature (backlog#1148 ilm-6) The tier/lifecycle integration tests carried two byte-for-byte copies of an in-memory WarmBackend mock — one in crates/scanner/tests and one in rustfs/src/app — plus duplicated register_mock_tier and polling helpers. Both implemented the same ecstore WarmBackend trait. Consolidate them into ecstore behind a new `test-util` feature, exposed via the `rustfs_ecstore::api::tier::test_util` facade: - MockWarmBackend: in-memory WarmBackend with an operation log (for ordering assertions such as "local delete precedes remote remove") and fault injection (FaultConfig): unreachable, HTTP 5xx, credential rejection, injected latency, plus external_remove to simulate an out-of-band remote deletion. - register_mock_tier / register_mock_tier_backend: register the mock into any TierConfigMgr handle (the global manager used by scanner tests or a per-instance one used by the app tests). - xl.meta transition assertion helpers: read_transition_meta, assert_transition_meta_consistent (cross-shard consistency of the status/tier/remote-key/remote-version-id tuple plus free-version count), and free_version_count. - polling helpers: wait_for_remote_absence, wait_for_object_count, wait_for_free_version_absence. Both existing copies now consume this single definition; `rg 'struct MockWarmBackend'` collapses to one. The feature is enabled only from [dev-dependencies], so it never links into the production binary (resolver 3). Designed for downstream ilm-8 (restore lifecycle) and ilm-11 (tier fault injection matrix). Coordinates with #4706 (ilm-2), which adds op-logging to the scanner mock — that op-logging is now part of this shared surface, so #4706 should rebase onto it. Refs rustfs/backlog#1148 (ilm-6), rustfs/backlog#1155. * test(ecstore): fix shared MockWarmBackend usage after main merge - Access stored objects via MockWarmBackend::contains() instead of the now private inner objects map (fixes E0609 after the shared test-util refactor). - Drop dead ReadCloser/ReaderImpl/DiskAPI imports and the unused transition_api test re-exports the mock extraction left behind. - Reword the scanner/rustfs test-util dependency comments so they no longer embed the literal rustfs_ecstore:: path that trips the ECStore architecture-migration guard. |
||
|
houseme
|
264b2dd480 |
perf(metrics): drop needless per-emission work on hot metric paths (#4743)
* perf(metrics): drop needless per-emission work on hot metric paths Audit of the metrics hot paths surfaced four low-risk wins where emission did work it did not need to: - `record_file_cache_reclaim_success/error` (disk/local.rs) called `.to_string()` on `kind` (already `&'static str`) and on the `"ok"`/`"err"` literals, heap- allocating up to four `String`s per page-cache reclaim window — which runs per read-stream reclaim. The `metrics` macros accept `&'static str` label values directly, so pass them as-is. - `record_read_repair_dedup` (set_disk/core/io_primitives.rs) likewise `.to_string()`-ed an already-`&'static str` `reason`. - `SetDisks::get_object_reader` (set_disk/ops/object.rs) captured `Instant::now()` and emitted the `rustfs.lock.acquire.*` counter and histogram unconditionally on every GET, right beside an already-gated stage timer. Gate them behind `get_stage_metrics_enabled()` too, so an inactive observability config pays no per-GET clock read or recorder lookups. - The per-response-body-chunk counter in server/http.rs re-ran the `counter!` registry lookup on every chunk (a streamed GET emits many). Resolve the label-less handle once into a `LazyLock<metrics::Counter>`; the global recorder is installed at startup before any response streams, so the cached handle binds to the final recorder. No metric names or label values change. The only behavior change is that the `rustfs.lock.acquire.*` GET-path metrics now follow the GET stage-metrics flag, consistent with the neighbouring stage timings. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(metrics): gate page-cache reclaim metrics behind metrics_enabled() `record_file_cache_reclaim_success/error` run per read-stream reclaim window on large-object reads and emitted unconditionally. When general metrics are disabled the `counter!`/`histogram!` macros still construct three metric keys per call for nothing. Skip the emission behind `rustfs_io_metrics::metrics_enabled()`, matching how the io-metrics free functions self-gate. The serial reclaim-metrics test now enables the flag (save/restore) alongside the existing stage gate. Left ungated deliberately: `record_read_repair_dedup` (rare read-repair path, and its non-serial test would need a global-flag toggle), and the HTTP body-chunk counter (its cached handle already makes the disabled case a no-op increment). Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
e742a540a4 |
test(cache): guard the body-cache eligibility gate against deny-list regressions (#1146) (#4742)
`full_object_plaintext_len` decides whether a body-cache hook hit may serve bytes in place of the erasure read. It is a fail-closed allow-list: it excludes every read whose `ReadPlan::build` applies some other transform (ranged/part, raw/data-movement, restore, encrypted, remote) with an early `return None`, then returns a `Some(..)` length only for the whole-plaintext cases. A newly added `ReadPlan` branch that nobody teaches this gate about falls through to `None` and safely bypasses the cache. Flip it to a deny-list and the same new branch silently serves bytes in the wrong representation — the backlog#1108 / #1109 / #1146 class of bug. The existing unit and e2e tests only cover the branches that exist today. This adds `scripts/check_body_cache_whitelist.sh`, a structural guard wired into pre-commit / pre-pr / dev-check and CI, that asserts every exclusion predicate and a `return None` still precede the first `Some(..)`. Reordering a predicate, dropping one, moving the positive return ahead of the gate, or renaming/removing the function all fail; wording, formatting, and adding a new exclusion in the same gate do not. Mutation-tested against all four regression shapes. This machine-enforces the structural invariant that backlog#1146 was kept open to guard by hand. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
1ab66e124c |
test(ecstore): pin the remaining io_uring fd-cache invariants (#1180) (#4739)
Close the three test-completeness items in rustfs/backlog#1180 that the earlier hardening (rustfs/rustfs#4726, #4729) left unpinned; the other two of the five (sharded cancel routing, bailout-handle error) already landed in rustfs/uring. - rename_data end-to-end invalidation: drive the real `LocalDisk::rename_data` commit (non-inline part, so `invalidate_part_paths` is non-empty) and assert the destination part descriptor is dropped — not merely `rename_file`/`delete`. Both production `rename_data` call sites share this invalidation, so a "fix one copy, miss the other" regression is now caught. The test is non-vacuous: it seeds the cache, removes the on-disk data dir out of band (the cached fd keeps the old inode alive and clears the path for the directory rename `rename_data` performs), and asserts a read still returns the OLD bytes before the commit — which fails outright if the cache is off, so it cannot pass without a live cache. - FD_CACHE_TTL backstop: an injected short TTL proves the cache self-evicts a descriptor with no explicit invalidation; a static check pins the 5s value. - zero-length read bounds parity on the cache-HIT path: a `length == 0` read past EOF must be rejected identically to the miss path and StdBackend, pinning the #1173 fix against regression. Refactors `FdCache::new` to delegate to a private `with_ttl(ttl)` helper so the TTL backstop can be exercised with a short TTL instead of a multi-second wait. Verified: `cargo test -p rustfs-ecstore` on Linux with real io_uring (seccomp=unconfined, RLIMIT_NOFILE raised, RUSTFS_URING_TESTS_MUST_RUN=1 so a degraded skip fails rather than passing vacuously). Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
793b2a06e2 | perf(ecstore): snapshot per-IO env config on local disk read/write paths (#4736) | ||
|
houseme
|
9d64d71bd7 | fix(ecstore): reduce GET reader setup shard fanout (#4735) | ||
|
houseme
|
c79366d42c |
refactor(ecstore): tidy io_uring read backend (docs + fd-cache handle) (#4732)
Two behavior-preserving cleanups to the io_uring local read backend that accumulated as later work layered onto it: - Reunite the `UringBackend` struct doc comment. The backlog#1145 fd-cache constants were inserted into the middle of the struct's doc block, so its opening sentences were orphaned onto `ENV_RUSTFS_IO_URING_FD_CACHE` and the struct itself was documented by a sentence fragment starting mid-clause with "through rustfs-uring's...". Move the opening back onto the struct and give the constant its own one-line doc. - Fold the fd-cache handle and its lookup key into a single `Option<(&FdCache, FdKey)>` in `pread_uring`, so the get and the insert_if_fresh sites stop re-deriving `self.fd_cache.as_ref()` and re-matching presence. Semantics are identical, including the backlog#1176 generation guard (`gen_at_open` snapshot before open, insert only when the generation is unchanged). No functional change. The borrow/move pattern was validated against a host-compilable reduction (the cfg(linux) path cannot be built from macOS); CI compiles the Linux backend. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
4f29dcdcec |
chore(deps): bump rustfs-uring to 0.2.1 (#4731)
chore(deps): bump rustfs-uring to 0.2.1 from crates.io rustfs-uring 0.2.1 routes the driver's runtime diagnostics through `tracing` with structured fields instead of `eprintln!` (rustfs/uring#13). No public API change — UringDriver::probe_and_start_sharded, read_at, and read_at_direct keep their signatures — and the read path / cancel-safety ownership model are untouched, so this is a drop-in patch bump of the version requirement plus the lockfile entry. 0.2.1 adds a `tracing` dependency; it is recorded in the rustfs-uring lock entry. tracing is already in the workspace graph, so nothing new is pulled in. The Cargo.lock change is scoped to the rustfs-uring package only; unrelated lockfile drift is left for its own change. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
ffca98cdbf |
fix(ecstore): harden io_uring integration (#4726)
* fix(ecstore): close the fd-cache open-then-insert race with a generation guard (rustfs/backlog#1176) pread_uring's miss path opened a descriptor on the blocking pool and only then inserted it into the moka cache. moka's invalidations cover only entries present at call time, so a heal/delete commit that invalidated between the open and the insert could not stop the just-opened stale inode from being cached afterwards — serving the pre-heal/pre-delete inode for up to the TTL and defeating the heal. Add an invalidation generation to FdCache, bumped by invalidate_exact and invalidate_under before they touch moka. The read path snapshots the generation before opening and inserts via insert_if_fresh, which refuses the insert if the generation moved during the open and, with a post-insert re-check, removes the entry if an invalidation raced the insert itself. Reads that never miss are unaffected. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): invalidate the fd cache on the primary object-delete paths (rustfs/backlog#1175) The fd-cache invalidation contract was only wired into DiskAPI::delete, rename_file and rename_data, but object deletion almost never goes through LocalDisk::delete — DeleteObject(s) reach delete_version, delete_versions -> delete_versions_internal, and delete_paths, all of which remove a version's data dir (move_to_trash / rename_all staging) with no invalidation. A cached io_uring descriptor kept the deleted part.N inode readable for up to the TTL, so a GET in that window could still return deleted data. Invalidate every cached fd under the removed data dir at each site: in delete_version and delete_versions_internal the data_dir uuid and object path are in hand (invalidate_cached_fds_under(volume, "{path}/{uuid}")); delete_paths invalidates under each removed path. A later rollback that restores a data dir just causes the next read to re-open it. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): close remaining fd-cache invalidation gaps (rustfs/backlog#1177) Three residual paths could keep serving a stale descriptor: - delete_volume removed the whole bucket tree (remove_dir_all/remove_dir) with no invalidation, and the cache-hit read path skips the volume-access check, so a cached fd kept a removed object readable. Add invalidate_cached_fds_for_volume (a per-volume moka predicate) and call it after the bucket is removed. - A retired LocalDisk instance (renew_disk on reconnect builds a fresh one) kept its populated cache alive while still referenced by in-flight ops, so invalidations through the new instance never reached it. close() now clears the backend's cache via clear_cached_fds. - rename_data's post-commit rollback (a commit-metadata fsync failure under strict durability) restored the old data dir without dropping fds cached during the committed window; the streaming branch now invalidates the dst part fds on those rollback paths. The inline branch's rollback runs inside spawn_blocking and is left to the TTL backstop. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): narrow the io_uring latch classes to match StdBackend (rustfs/backlog#1171) The runtime degradation classification reused the probe-time restriction errnos, which the driver's C7 contract explicitly warns against, so a single per-file error could latch a whole disk off io_uring: - is_io_uring_unsupported no longer includes EACCES: at read time on an already-open fd it is per-file (an LSM hooks security_file_permission on every read) and StdBackend hits the same denial, so falling back masks nothing and a full-disk latch would be wrong. ENOSYS and EPERM (seccomp/LSM applied after startup) remain. EOPNOTSUPP is now classified per-path by the caller. - pread_uring_direct's read-error arm now mirrors StdBackend: an O_DIRECT-shape error (EINVAL/EOPNOTSUPP) latches only direct_uring.supported so eligible reads take StdBackend's aligned path, instead of over-latching the whole io_uring backend or never latching a read-side EINVAL at all. - try_new only negative-caches genuine restriction-class probe failures in URING_UNSUPPORTED_DISKS; an unexpected (possibly transient) probe failure now falls back without latching, so the next reconnect re-probes. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ecstore): log when a disk latches io_uring off at runtime (rustfs/backlog#1172) A probe-gated gray release was flying blind: the permanent per-disk `active` latch flipped with no log and no metric, so the only message operators ever saw was the startup "io_uring read backend enabled" line — which stayed true on dashboards even after the very first read latched the disk back to StdBackend forever. Add latch_active_off, which flips the latch with `swap` and logs the true->false transition exactly once at warn with a dedicated event constant, disk root, and errno. Both the buffered and O_DIRECT read paths use it. A fallback/latch metric counter and periodic export of the driver StatsSnapshot (cq_overflow, cancel_already) remain as follow-ups that need rustfs_io_metrics plumbing. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(audit): correct the stale rustfs-uring license-allow rationale (rustfs/backlog#1181) The dependency-review allow said rustfs-uring is "pulled as a git dependency", but ecstore now pins it from crates.io. Update the rationale and scope the allow to the exact pinned version (pkg:cargo/rustfs-uring@0.1.0) so a future version bump forces a conscious re-review of the license/provenance claim instead of being waved through on an outdated justification. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): offload io_uring driver teardown off the tokio worker (rustfs/backlog#1170) UringBackend held Arc<UringDriver> and had no Drop, so when the last LocalDisk reference dropped in async context (disk reconnect via renew_disk, or shutdown), UringDriver's own Drop ran on that thread — sending Shutdown and joining each shard thread, which can block up to the bounded-drain timeout (5s) on a hung / D-state disk, stalling a tokio worker. Wrap the driver in ManuallyDrop (deref is transparent, so read call sites are unchanged) and add a Drop that takes the Arc and, when a runtime is present, drops it on a blocking thread so the potentially-blocking join never runs on a runtime worker. Off-runtime it drops inline. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): restore StdBackend read parity on the uring paths (rustfs/backlog#1173) Two byte-for-byte parity breaks against StdBackend on the io_uring read paths: - A zero-length read on an fd-cache hit returned Ok(empty) without any bounds check, while StdBackend and the uring miss path return FileCorrupt for an offset past EOF. Fstat the cached descriptor on the length==0 path and match. - reclaim_read_range fadvise(DONTNEED)'d the raw unaligned [offset, offset+len) range, but fadvise only drops fully-covered pages, so the head partial page stayed resident — whereas StdBackend's mmap path reclaims the page-aligned superset. Bitrot shards' 32-byte block headers keep offsets off page boundaries, so this diverged on the common case. Page-align the reclaim window to match the mmap path exactly. (The third parity item from the audit — a failed reclaim fadvise failing the read — is already parity: StdBackend's mmap path propagates the same fadvise error with `?`, so no change is needed.) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): bound worst-case in-flight memory by chunking huge uring reads (rustfs/backlog#1174) The driver's backpressure permits count operations, not bytes, and it zero-fills a full-size buffer per op, so a single unbounded read could pin ~length bytes per permit (128 permits x shards x up to ~2 GiB). ecstore passes a whole part's shard range as one pread_bytes with no upstream chunking. On the buffered path, split reads larger than URING_MAX_OP_LEN (128 MiB) into sequential chunks, awaited one at a time, so worst-case in-flight memory is bounded by permits x URING_MAX_OP_LEN per shard. The threshold is high enough that ordinary shard reads keep the single-op, zero-copy fast path unchanged. The O_DIRECT path (opt-in, alignment-constrained) is left for a follow-up. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): gate the io_uring fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178) The fd cache holds up to FD_CACHE_CAPACITY (512) descriptors per disk, but try_new cannot know the disk count and nothing checked the process fd budget. On a bare-metal / non-systemd run with the common 1024 soft RLIMIT_NOFILE, two disks would already exhaust fds with EMFILE surfacing on reads and probes. Check the soft limit at try_new: enable the cache only with ample headroom (>= 16384), otherwise log a warning once and fall back to open-per-read. The packaged systemd unit sets 1,048,576, so tuned deployments are unaffected. Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): make io_uring test skips visible and gate non-vacuity (rustfs/backlog#1179) The ecstore io_uring tests degrade to a silent pass when io_uring is unavailable (bare `return`s or plain eprintlns), so a CI leg on a restricted runner never exercises the real UringBackend/FdCache/latch paths yet still goes green — an integration regression could merge unseen. Add uring_test_skip: it emits a grep-able `SKIP <name>` line and, when RUSTFS_URING_TESTS_MUST_RUN is set (a CI leg that guarantees io_uring, e.g. a seccomp=unconfined container), panics instead of skipping. Route the silent-skip sites through it. Wiring a dedicated CI leg that sets that env on a capable runner is tracked in the issue; this provides the enforcement mechanism. Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): cover delete_paths fd-cache invalidation (rustfs/backlog#1180) Add an end-to-end test that seeds the descriptor cache with a read, removes the part via disk.delete_paths (one of the primary object-delete entry points that does not go through LocalDisk::delete), and asserts the next read no longer returns the removed inode — pinning the invalidation added in #1175. The sharded cancel-routing half of #1180 is covered in the rustfs-uring PR. Co-Authored-By: heihutu <heihutu@gmail.com> * io_uring audit follow-ups: O_DIRECT chunking, inline invalidation, metrics, CI leg (backlog#1160) (#4729) * fix(ecstore): chunk large O_DIRECT reads too, bounding in-flight memory (rustfs/backlog#1174) The buffered read path already splits reads above URING_MAX_OP_LEN into sequential chunks; do the same for the O_DIRECT path, which was left for a follow-up. read_at_direct aligns each chunk's sub-range internally, and chunk sizes are a multiple of URING_MAX_OP_LEN so a boundary re-read is at most one block. Extract classify_direct_read_error so the single-op and chunked paths share one copy of the EINVAL/EOPNOTSUPP-vs-subsystem latch classification rather than duplicating it. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): invalidate cached fds on the inline rename_data rollback (rustfs/backlog#1177) The streaming rename_data branch invalidates cached part fds on its post-commit rollback paths, but the inline branch runs its commit and rollback inside a single spawn_blocking closure where the async invalidate cannot be called, so it was left to the TTL backstop. Capture the closure's result instead of `??`-propagating it: on error (a commit-metadata fsync failure under strict durability rolls the committed rename back), invalidate the dst part paths at the async level before returning. Inline objects keep their data in xl.meta rather than separate part inodes, so this is largely defensive, but it removes the caveat and keeps the two branches consistent. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ecstore): export io_uring latch/fallback and driver stats metrics (rustfs/backlog#1172) Complete the gray-release observability. Beyond the warn log added earlier, emit metrics so a dashboard can answer "how much traffic is on io_uring vs falling back, and is any disk degrading": - rustfs_io_uring_latch_off_total — a disk latching io_uring off at runtime. - rustfs_io_uring_read_fallback_total — each io_uring -> StdBackend read fallback (latched-off short-circuit, O_DIRECT error, buffered error). - a low-frequency per-disk exporter of the driver StatsSnapshot as gauges (in_flight, cq_overflow, cancel_already), spawned in try_new. It holds only a Weak reference so it never keeps the driver alive, and drops any temporary strong reference on the blocking pool so a last-reference UringDriver::Drop join never runs on an async worker (rustfs/backlog#1170). submit_errors is deliberately not exported yet: it is a field added in the unreleased rustfs-uring 0.2.0, and ecstore still pins 0.1.0. It lands once the dependency is bumped (rustfs/backlog#1181). Co-Authored-By: heihutu <heihutu@gmail.com> * ci: add a real-io_uring integration leg on ubuntu-latest (rustfs/backlog#1179) The existing self-hosted sm-standard runners cannot guarantee io_uring is available (a container seccomp filter can block io_uring_setup), so the ecstore uring tests degrade to a silent skip and never exercise the real UringBackend/FdCache/latch paths in CI. Add a job on GitHub-hosted ubuntu-latest, which runs a recent kernel with no container seccomp filter, running the uring-named ecstore tests with RUSTFS_IO_URING_READ_ENABLE=true and RUSTFS_URING_TESTS_MUST_RUN=1 — the non-vacuity gate makes the leg fail rather than skip if io_uring is unavailable, so an integration regression can no longer merge green behind a vacuous pass. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
ab60244d7d |
chore(deps): bump rustfs-uring to 0.2.0 (#4730)
chore(deps): bump rustfs-uring to 0.2.0 from crates.io rustfs-uring 0.2.0 is published on crates.io. It carries the read-driver hardening from the backlog#1160 adversarial audit (cancel-safety and graceful-degradation fixes on the Linux io_uring read path). The public API ecstore consumes is unchanged — UringDriver::probe_and_start_sharded, read_at, read_at_direct all keep their 0.1.0 signatures — so this is a drop-in bump of the version requirement plus the lockfile entry. The Cargo.lock change is intentionally scoped to the rustfs-uring package only; unrelated lockfile drift is left for its own change. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
2ebe8e561b |
fix(replication): allow loopback replication targets under an explicit test opt-in (#4725)
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit
|
||
|
Zhengchao An
|
05fae6f939 |
test(ecstore): add TierConfigMgr state-machine unit coverage (#4713)
* test(ecstore): unit-test TierConfigMgr add/edit/remove/verify state machine (backlog#1148 ilm-4) Covers the tier config state machine and persistence paths that previously had only 4 codec tests and none for tier_config.rs: - add: non-uppercase name, duplicate name, unsupported type, missing backend payload, and a regression anchor documenting that AWS-reserved names (STANDARD) are not currently rejected. - edit: unknown tier, missing-credentials rejection for RustFS and MinIO. - remove: idempotent unknown-tier no-op, in-use rejection, empty-backend success, force skips the in_use probe, and probe-error surfacing. - verify: unknown tier, healthy backend, unhealthy backend. - pure query helpers (empty/is_tier_valid/tier_type/get/list_tiers). - persistence: JSON marshal/unmarshal roundtrip, external tier-config.bin roundtrip for Azure and GCS payload mapping, truncated/unknown-format/ unknown-version rejection, legacy v1 version-word acceptance, and encode failure on missing payload. Tests are hermetic: error paths return before backend construction, and a MockWarmBackend injected into driver_cache exercises remove/verify without any real remote. Refs backlog#1155. Co-authored-by: overtrue <anzhengchao@gmail.com> * fix(tier): reject reserved names STANDARD/RRS in TierConfigMgr::add (backlog#1148 ilm-4) (#4721) |
||
|
Zhengchao An
|
846aa95c32 |
test(security): GHSA-named regression tests for 3p3x and r5qv (backlog#1151 sec-6) (#4707)
test(security): add GHSA-named regression tests for 3p3x and r5qv (backlog#1151 sec-6) Anchor the two fixed advisories to discoverable, named regression tests so `rg -i "ghsa|3p3x|r5qv"` finds a guard for each, and future fixes are forced to update pinned behavior (red -> green). GHSA-3p3x-734c-h5vx (constant-time WebDAV/FTPS secret comparison, rustfs#4403): - ftps_core.rs: new `assert_ftps_ghsa_3p3x_wrong_credentials_rejected` drives the `ct_eq` reject branch in FtpsAuthenticator::authenticate; asserts wrong password and unknown user are both rejected (530) and indistinguishable. - webdav_core.rs: the auth-failure block now sends a valid access key with a wrong secret (exercising the `ct_eq` branch, not just the unknown-access-key path) plus an unknown user, asserting both 401 and indistinguishable. - Module doc comments map advisory -> tests -> fix PR on both files. GHSA-r5qv-rc46-hv8q (internode RPC fail-closed, rustfs#4402): - http_auth.rs: renamed the default-fallback rejection test to `ghsa_r5qv_resolve_shared_secret_rejects_default_fallback` (and broadened it to cover default env secret + blank secrets), and added `ghsa_r5qv_verify_rpc_signature_fails_closed_on_missing_or_invalid_auth` pinning the exact advisory scenario (missing/forged/cross-URL signature is rejected; a correctly signed request still passes). File-level doc maps the advisory. Docs: new docs/testing/security-regressions.md with the advisory -> test mapping table and where each layer runs; linked from docs/testing/README.md. sec-14 will formalize the written policy in AGENTS.md. The unit-level ghsa_r5qv_* tests run in the default CI pass. The WebDAV/FTPS e2e live in the protocols suite (fixed ports, --test-threads=1); they cannot join the e2e-smoke profile and are wired into CI by sec-5. Refs: rustfs/backlog#1151 (sec-6), rustfs/backlog#1155 |
||
|
Zhengchao An
|
ce53feee87 |
test(lifecycle): regression test for expire/GET race (#3491) (#4706)
Adds a serial-lane ILM integration regression test that pins the local-first ordering contract established by rustfs#3491, where `expire_transitioned_object` deletes local metadata BEFORE any remote tier cleanup so a concurrent GET can never observe live local metadata pointing at an already-removed remote tier version (user-visible `NoSuchVersion`). `serial_tests::test_expire_transitioned_object_never_races_concurrent_get` in `crates/scanner/tests/lifecycle_integration_test.rs`: - Transitions an object to a mock warm tier, then runs a tight concurrent GET loop while calling `expire_transitioned_object`; every GET must return a full, correct body or a clean object/version-not-found -- never a tier-fetch failure. - Deterministic ordering assertion (revert-proof): immediately after expiry the remote tier object is still present and the mock recorded zero remote `remove` calls, proving remote cleanup is deferred to free-version recovery. Reverting to remote-first ordering turns this assertion red. Supporting changes: - Re-export `expire_transitioned_object` from `api::bucket::lifecycle`. - Record remote-tier mutating ops in the scanner test `MockWarmBackend`. - Update tier-ilm-debugging.md to reference the new regression test. Runs in the CI ILM Integration (serial) lane (ci.yml test-ilm-integration-serial), picked up by the existing binary(lifecycle_integration_test) filter. Refs: rustfs/backlog#1148 (ilm-2), rustfs/backlog#1155 |
||
|
Henry Guo
|
3f25426534 |
fix(ecstore): reject incomplete listing usage refreshes (#4698)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> |
||
|
houseme
|
7437f99c45 |
fix(cache): key the object body cache on data_dir for write-uniqueness (#4703)
* refactor(object-data-cache): derive Default for ObjectDataCacheGetRequest The GET request literal is hand-listed field-by-field across ~13 test sites in two crates. Adding `mod_time_unix_nanos` in backlog#1111 had to touch every one and still missed a literal, producing a compile error caught only in a later CI lane. Derive `Default` and spread the engine-crate literals so the next field addition is absorbed rather than fanned out. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): key the object body cache on data_dir for write-uniqueness The cache key's correctness rested on being write-unique, but its only write-scoped component was `mod_time` — a wall-clock timestamp that is not monotonic and can be absent. Two writes that collide on MD5 (same etag+size) and land on an equal mod_time (clock skew, same-tick, or absent) derived the same key, so a node that never saw the overwrite could serve the previous body for up to the TTL, and the same collision turned the fill-after-invalidation race into a serving bug. This is the store's strong-read-after-write guarantee leaning on a probabilistic argument. Add `data_dir` — the xl.meta directory UUID ecstore regenerates on every body write — as the primary write-unique anchor: - surface `data_dir: Option<Uuid>` on ObjectInfo, copied from FileInfo in the single GET-path constructor `from_file_info`; - carry it into ObjectDataCacheKey as `data_dir_u128` (held as u128 to keep the engine crate free of a uuid dependency), derived in the one planner site both the ecstore hook and the usecase layer share, so both produce an identical key by construction; - keep `mod_time` as a second anchor and `etag+size` as belt-and-braces; an absent data_dir falls back to the prior behavior — strict improvement, no regression. Two writes distinct only by data_dir now derive different keys even under an MD5 collision with identical mod_time — the case mod_time alone cannot cover. Blast radius is compiler-guarded: ObjectInfo derives Default and every real construction site uses `..Default::default()`, so only from_file_info and one full-literal test needed the field. The three P0 body_cache_hook_e2e regressions, engine (80), and app (36) suites pass unchanged; a mutation that severs the planner wiring fails planner_key_changes_with_data_dir. Refs: backlog#1111, backlog#1118 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): thread data_dir field through the merged mutation-hook test Merging main (which landed the object-mutation-hook work, backlog#1131) brought in a GetRequest test literal that predates the data_dir field. Spread it via `..Default::default()` — the derive(Default) added here means this is the last such hand-listed literal to need touching. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
85fd824581 |
feat(object-data-cache): close write-side invalidation gaps and add an admin surface (#4694)
* feat(object-data-cache): close write/delete-side invalidation gaps The object data cache exposed only a single per-(bucket,object) invalidation primitive and no write-side ecstore hook, so several delete paths left dead bodies resident until TTL (hygiene/capacity, not stale-serving: lookups follow a fresh metadata quorum and cannot serve a gone object). This adds the missing primitives and wires them in. ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET body hook, registered next to it at startup, and call it from the ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`, `expire_transitioned_object` including the restored-copy branch, and `delete_object_versions`). The app impl is one `invalidate_object` call under a new `AfterLifecycleExpiry` reason. ODC-27 (backlog#1132): force prefix delete now invalidates the whole prefix, not just the prefix string. `store.delete_object(delete_prefix)` returns no deleted-name list, so this uses a new prefix primitive rather than the batch path. ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new bucket-scope primitive (covers force and non-force, which share the delete_bucket call). ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin handlers (GET stats, POST flush) routed through admin runtime_sources. The starshard identity index gains a single `remove_matching` full-scan API backing prefix/bucket/clear; it is documented as admin/delete-path only and never runs on the GET or fill hot path. New invalidation reasons and metric labels added; outcome (removed/noop) labelling kept correct for every new primitive. Also fixes a pre-existing broken intra-doc link in memory.rs. Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): extract the shared HookSlot behind both cache hooks This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs, which left two process-global registration slots whose register/get/clear bodies were line-for-line identical except the trait type and the WARN string: a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the poison-recovery closure, and the same read-lock-and-clone read. Two copies of the same swap-vs-warn logic can drift apart under maintenance. Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged public wrappers (register_/get_/clear_), so the crate's public surface and every call site are untouched — this is an internal consolidation, not a contract change. The load-bearing #1126 guarantee (newest registration wins, so a rebuilt AppContext is never stranded on a first-wins slot) previously had no direct test — the hook tests only covered register-then-notify. HookSlot now has its own unit tests including re_registration_swaps_to_the_latest_instance; mutation-testing confirms a first-wins regression fails exactly that test. No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e regressions, and the app-layer mutation-hook tests all pass unchanged. Refs: backlog#1126, backlog#1131 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(admin): register the object-data-cache routes in the policy inventory This PR added GET /object-data-cache/stats and POST /object-data-cache/flush but did not list them in the two registries that must account for every admin route: the route-policy inventory (route_policy.rs) and the route matrix (route_registration_test.rs). Their coverage tests — route_policy_inventory_covers_registered_routes and test_admin_route_matrix_matches_registered_routes — failed on CI because a registered route had no policy/matrix entry. These two tests are not part of `make pre-commit` (which runs fmt + arch + quick-check, not the full suite), so the gap passed local pre-commit and only surfaced in the CI Test-and-Lint lane. stats is a read (ServerInfoAdminAction, Sensitive); flush mutates (ConfigUpdateAdminAction, High) — matching the actions the handlers already enforce. The MinIO-alias matrix test is unaffected: these are native rustfs endpoints with no MinIO equivalent. Refs: backlog#1143 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
f9874b591a |
[DO NOT MERGE until published] chore(ecstore): switch rustfs-uring to crates.io 0.1.0 (#4701)
* chore(ecstore): switch rustfs-uring from the git rev to crates.io 0.1.0
Prepared ahead of the rustfs-uring 0.1.0 crates.io release (rustfs/uring):
flip ecstore's dependency from the pinned git revision to the published
`0.1.0`, and drop the now-unneeded git-source allowance in deny.toml.
DO NOT MERGE until rustfs-uring 0.1.0 is on crates.io. Until then cargo
cannot resolve `rustfs-uring = "0.1.0"`, so this branch will not build and
CI will be red — that is expected, not a defect in the change.
After the crate is published:
1. `cargo update -p rustfs-uring --precise 0.1.0` to regenerate Cargo.lock
(deliberately not touched here — the registry entry, with its checksum,
cannot be written before the crate exists);
2. push the Cargo.lock;
3. CI goes green; merge.
No code change. The guard `scripts/check_no_tokio_io_uring.sh` is unaffected
(it bans only tokio's io-uring runtime feature, not an explicit rustfs-uring
dependency). `audit.yml`'s `allow-dependencies-licenses` for rustfs-uring is
left in place — it is a first-party Apache-2.0 crate either way.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore: refresh rustfs-uring lockfile
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
houseme
|
6780140318 |
fix(object-data-cache): make the GET key write-unique and dedup the lookup (#4693)
fix(object-data-cache): make GET body cache key write-unique and dedup lookups Address four object-data-cache GET-path findings (backlog#1107 batch): ODC-06 (backlog#1111): the cache key was content-unique, not write-unique. Extend ObjectDataCacheKey with the resolved version's modification time (i128 unix nanoseconds, None -> 0), derived once in the shared planner so the ecstore hook and the usecase layer produce an identical key. An unversioned overwrite advances mod_time, so a stale node can no longer serve old bytes for up to the TTL under an MD5 collision; etag + size stay as belt-and-braces. ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes and lookups. GetObjectReader now carries a GetObjectBodySource marker (Unprobed / HookMissed / HookServed); the hook stamps it, and build_get_object_body_with_cache serves a hook-served body directly and skips its lookup whenever the hook already probed. One hook-served GET now records exactly one lookup. ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly, which never fills and keeps a permanent 0% hit rate. Default to FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is selected explicitly. ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was silently inert. Clamp the planner's size eligibility to min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible sizes plan SkipTooLarge instead of being reported eligible, and warn at startup when the excess is inert. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
d0ca14d8df |
test(ecstore): cover post-commit hard-crash in rename_data crash harness (#4689)
The rename_data crash-consistency harness (backlog#935/#878) only armed pre-commit crash points (AfterDataRename, AfterBackupBeforeMetaCommit), which assert the *old* version survives. The other half of the "partial commit -> old or new, never mixed" invariant — a hard power loss *after* the xl.meta commit rename must leave the *new* version — had no crash-point coverage. The existing after-metadata-commit failpoint only exercises the graceful in-process rollback (restores old), a different path from a hard crash (no rollback runs, commit stays on disk). Add a RenameDataCrashPoint::AfterMetaCommit injection right after the commit rename (cfg(test), compiles to a const-false no-op in production, like the existing points) and overwrite/fresh x strict/relaxed tests asserting the object reads back as the new version. The fresh case pins that the point is genuinely post-commit: a pre-commit misplacement would leave no readable object and fail the assertion. cargo test -p rustfs-ecstore crash_consistency: 9 passed. clippy -D warnings clean; fmt and arch guards clean. |
||
|
houseme
|
d715cb5c34 |
refactor(ecstore): single-source the bitrot read/verify path (backlog#1159) (#4697)
P-A (`read_appending`) and P-C (the in-memory fast path) each copied the hashed-read logic, so `BitrotReader` ended up with the hash verification, the short-read error, and the scratch-buffer fill written three times across `read` and `read_appending`'s two branches. That is patch-on-patch: a change to the bitrot contract would have to be made in three places and kept in sync by hand. Collapse the duplication onto three single-source pieces: - `split_and_verify` — a free function that splits `[hash][data]`, verifies (unless skip_verify), and returns the data slice plus the hash time. Free rather than a method so it can run while `self` is borrowed for the block. - `read_scratch_block` — the single-pass fill of `self.buf` with the short-read-to-UnexpectedEof contract. - `short_shard_read` / `begin_read` — the shared error and preamble. `read` and `read_appending` now differ only in what they must: how the block is acquired (caller's slice vs `try_take_block` vs scratch fill) and where the verified shard lands (`copy_from_slice` vs `extend_from_slice`). This also fixes a latent inconsistency the duplication hid: the old `read` did `copy_from_slice` *before* verifying, so on a hash mismatch it left the corrupt bytes in the caller's buffer before returning the error, while `read_appending` verified first. Both now verify before writing, so a shard that fails the hash never reaches the caller's buffer on either method — the stronger of the two behaviors. Net -19 lines; behavior otherwise unchanged. Verified: `erasure::` 215 passed, 0 failed (including the fast-path equivalence and corrupt-shard tests, and the existing `test_bitrot_read_hash_mismatch`); clippy --all-targets -D warnings clean. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
6f05a740b3 |
refactor(ecstore): extract the shared io_uring read preamble (backlog#1145) (#4696)
`pread_uring` and `pread_uring_direct` accumulated across seven rounds (#4632/#4635/#4645/#4649/#4653/#4658/#4662) and each carried its own copy of the same open preamble: resolve the bucket path, run the volume access check, resolve the object path, and check the path length. The only real difference is what happens after — one opens buffered and errors as `DiskError`, the other opens `O_DIRECT` and errors as `DirectOpenError` so it can latch an O_DIRECT refusal. Extract that shared sequence into `resolve_uring_object_path`, a blocking helper called inside both `spawn_blocking` closures. Each caller keeps exactly what differs — its open flags, its error type, its metadata/bounds/align handling — and no longer restates the resolution. No behavior change: the same checks run in the same order and produce the same errors (the O_DIRECT path still maps them through `DirectOpenError::Disk`). Net -12 lines. Verified on a real Linux host (16-core, real io_uring): clippy -p rustfs-ecstore --all-targets -D warnings clean; disk::local tests 144 passed, 0 failed — including the io_uring, O_DIRECT, fd-cache, and page-cache-reclaim cases that exercise both read paths. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
c2362bca14 |
perf(ecstore): slice in-memory shards instead of copying them twice (#4687)
* perf(ecstore): slice in-memory shards instead of copying them twice (backlog#1159)
The GET path reads a shard out of the page cache into a `Bytes`, then
`open_disk_reader` erased it behind `Box<dyn AsyncRead>` by wrapping it in
a `Cursor`. Downstream, `BitrotReader` could only get it back by copying:
once out of the `Cursor` into its scratch buffer, and once from there into
the caller's buffer. CPU profiling of a cached 1 MiB GET (device reads = 0)
attributed 8.23% of the whole server to `Cursor::poll_read` alone — a copy
of data that was already sitting in memory.
Keep the source concrete instead of erasing it. `ShardReader` is an enum of
`InMemory(Cursor<Bytes>)` and `Stream(Box<dyn AsyncRead ...>)`, and the new
`ShardSource::try_take_block(n)` lets an in-memory source hand over the
`[hash][data]` block as a slice. `read_appending` uses it to verify the hash
on the slice and `extend_from_slice` the shard straight into the caller's
buffer: one copy instead of two.
`try_take_block` defaults to `None`, so a streaming source keeps the old
path byte for byte, along with its short-read and EOF semantics. A source
that cannot serve `n` bytes declines rather than truncating, so a partial
block still becomes UnexpectedEof rather than a short shard. The hash is
still checked before anything is appended, so a corrupt shard never reaches
the caller's buffer on either path. The deferred parity reader opens its
source lazily and stays on the streaming path; parity is only read when a
data shard fails.
Tests gate equivalence and non-vacuity:
* `try_take_block` fires for `Cursor<Bytes>`, advances the position exactly
as a read of the same length would, declines when fewer than `n` bytes
remain, and returns `None` for a non-`Bytes` source — without this the
equivalence test below would silently compare one path against itself;
* both paths return identical bytes for the same shard;
* a corrupt shard fails on the fast path too, appending nothing.
Verified: clippy --tests -D warnings clean; `erasure::` 215 passed, 0 failed;
`set_disk::core::io_primitives` 49 and `io_support::` 22 pass.
Stacked on #4681 (`read_appending`), which this builds on.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): implement ShardSource for Cursor<&[u8]> used by the erasure bench
`crates/ecstore/benches/erasure_benchmark.rs` builds
`BitrotReader<Cursor<&[u8]>>`, which the new `ShardSource` bound on
`ParallelReader`/`decode` does not accept. `cargo clippy --tests` does not
compile bench targets, so this only surfaced in CI's `--all-targets` run.
A borrowed slice carries no `Bytes` to hand out, so it takes the default
`try_take_block` and keeps the old streaming copy path — no behavior change.
Verified with the same target set CI uses:
`cargo clippy -p rustfs-ecstore --all-targets -- -D warnings` clean.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
Zhengchao An
|
13f8768e7f |
feat(ecstore): make replication timing intervals env-overridable for tests (#4680)
Introduce RUSTFS_REPL_HEALTH_CHECK_INTERVAL_MS, RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS and RUSTFS_REPL_RESYNC_POLL_MAX_MS so tests and operators can shorten the replication background loops. Defaults are unchanged; invalid values fall back with a warn and values below 10ms are clamped to avoid busy-spin. Add replication_fast_env() e2e helper (backlog#1147 repl-4). |
||
|
houseme
|
5f1a475c56 |
perf(ecstore): stop zeroing pooled shard buffers on the GET path (backlog#1159) (#4681)
`ShardBufferPool::take` handed out a `resize(len, 0)`-ed buffer, and the
reader then overwrote every byte of it. CPU profiling of a cached 1 MiB
GET (device reads = 0, so all cost is CPU) attributed 4.81% of the whole
server to that memset — a buffer pool exists to reuse an allocation, and
memsetting it gives the saving straight back.
The zeroing was load-bearing only because `BitrotReader::read` takes
`&mut [u8]`, which must be initialized. But the reader never reads what
the caller put there, and never returns a partially filled buffer: both
the hashed and the no-hash path either fill the whole shard or fail with
UnexpectedEof, and a hash mismatch is an error rather than a short read.
So the initialization bought nothing observable.
Add `BitrotReader::read_appending(&mut Vec<u8>, want)`, which appends into
the buffer's spare capacity instead of demanding initialized bytes:
* hashed path — unchanged single copy, `extend_from_slice(data)` in place
of `copy_from_slice` into a pre-zeroed buffer, and only after the hash
verifies, so corrupt bytes never reach the caller's buffer;
* no-hash path — `read_buf` writes straight into the spare capacity and
advances the length only over bytes the reader actually wrote, so an
uninitialized tail can never be exposed.
`ShardBufferPool::take` now yields an empty buffer with capacity, and
`read_shard` no longer needs to `truncate`. `read` keeps its old signature
for the remaining callers.
Four tests gate the contract rather than the call:
* `read_appending` is byte-for-byte identical to `read` on both paths;
* a truncated shard is UnexpectedEof, never a partially filled buffer;
* bytes that fail the bitrot hash never reach the caller's buffer;
* `want > shard_size` is rejected;
plus the pool test now asserts the allocation is reused (same pointer) and
never zeroed.
Verified: `erasure::` 213 passed, 0 failed; on a real Linux host
`erasure::` 209 and `disk::local::` 143 pass serially, and the failures
seen in a parallel full-suite run reproduce identically on unmodified main
(they are ENOSPC from a full root filesystem plus pre-existing flakes).
Not claimed: an end-to-end throughput number. The A/B on the bench host was
too noisy to attribute (one rep pair was not fully cached, and its root
filesystem filled mid-run); what is measured is that the removed memset was
4.81% of GET CPU in the pre-change profile.
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
Zhengchao An
|
80ddd8fa7e |
fix(ecstore): roll back delete on disks that staged then errored (#4676)
fix(ecstore): roll back delete on disks that staged then errored (backlog#1158) #4300 rolls back a failed delete only on disks that returned Ok, skipping any disk that staged its rollback backup, applied the delete, and then errored -- leaving that disk deleted while its peers are restored. On rollback, fan the undo out to every online disk instead; the disk-side restore_delete_rollback is already idempotent (Ok no-op when nothing was staged), so unstaged disks are unaffected. The err.is_some() skip now applies only to the success/cleanup path. Covers both single-object and batch delete. Refs backlog#1158. |
||
|
houseme
|
8039a4ceae |
test(cache): end-to-end regressions for the body-cache hook P0s (#4675)
fix(ecstore): make body-cache hook re-registrable + add e2e regressions ODC-21 (backlog#1126): the GET body-cache hook lived in a first-wins OnceLock. When AppContext is rebuilt (config reload, test re-init) a fresh ObjectDataCacheAdapter is constructed and re-registered, but the OnceLock kept ecstore's GET probe pointed at adapter #1 while every usecase-layer fill and invalidation targeted adapter #2 — silently degrading the feature to a 0% hit rate with no error, log, or metric, and stranding entries in the unreachable cache until their TTL. Replace the slot with RwLock<Option<Arc<dyn GetObjectBodyCacheHook>>> so re-registration atomically swaps to the newest adapter, and log at WARN when a swap replaces a *different* instance (Arc::ptr_eq). RwLock over ArcSwapOption because arc-swap's RefCnt is impl<T> (Sized, thin *mut T) and cannot hold an Arc<dyn Trait> without a sized newtype wrapper; the probe reads the slot once per full-object GET but only clones an Arc, negligible next to the metadata quorum fan-out already done before the probe. Add a test-only clear_get_object_body_cache_hook so tests register/unregister deterministically. With the hook now re-registrable, add true end-to-end regressions that drive get_object_reader (not the full_object_plaintext_len predicate) against a real erasure-coded, genuinely-compressed object via the blackbox make_local_set_disks harness, with a stand-in hook playing the app-layer cache (the injection point production uses; the adapter itself lives above ecstore). These close the gap the predicate-only tests left — a caller that opens a new shortcut serving the cached body directly, the original form of both P0s: - backlog#1108: a raw_data_movement_read must yield the STORED (compressed) bytes, never the cached plaintext. - backlog#1109: a compressed cache hit must publish the DECOMPRESSED length as object_info.size (the UploadPartCopy invariant), with the streamed length matching. - backlog#1146: a restore read (restore_request.days) must serve STORED bytes, not the cache. Mutation-verified each e2e test bites: dropping the raw_data_movement_read gate serves plaintext (fails #1108); removing the hit-site size republication publishes 2972 vs 660000 (fails #1109); dropping the restore gate serves plaintext (fails #1146). Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
f83f9ada13 |
fix(ecstore): reclaim page cache after io_uring reads (#4662)
fix(ecstore): io_uring reads must reclaim the page cache like StdBackend (backlog#1145) `StdBackend::pread_bytes` calls `fadvise(DONTNEED)` over the range it just read whenever `should_reclaim_file_cache_after_read(length)` holds — `RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE` (on by default) above a 4 MiB threshold. Large object reads are usually cold, and leaving them resident evicts everything else, so this is a deliberate policy rather than a side effect of how StdBackend happens to read. `pread_uring` and `pread_uring_direct` never did it. Enabling io_uring on a disk therefore turned the policy off silently: shard reads at or above the threshold stayed resident and grew the page cache without bound. This is the same class of mistake as the O_DIRECT loss fixed earlier — copying the read itself while dropping the policy attached to it. It also badly distorts any benchmark. An end-to-end warp GET A/B on a 16-core host (4 MiB objects, conc 64) showed io_uring at 8782 MiB/s against StdBackend's 116 MiB/s. That is not a 76x speedup: with io_uring the run issued *zero* device reads and grew the page cache, while StdBackend read 2950 MiB from the device and shrank it. The two legs were not running the same policy. (Disabling the mmap read path changed nothing — 1.01x — so the mmap-vs-pread difference was not the cause either.) Both io_uring read paths now reclaim the range they read, with the same gate, the same range, and the same error handling as StdBackend. O_DIRECT should leave nothing resident anyway, but a filesystem that quietly buffered the read still has to honour the policy, so `pread_uring_direct` reclaims too. The test measures the policy, not the call: it reads an 8 MiB file through each backend and asks `mincore(2)` how much stayed resident. Crucially it also asserts the reclaim-disabled case leaves the range resident — without that gate, a backend that never reclaimed would pass silently. Verified: with the fix removed, the test fails with "uring: reclaim is on ... 2048/2048 pages remain"; with the fix it passes. Verified on a real Linux host (16-core, real io_uring): clippy --tests -D warnings clean; disk::local tests 143 passed, 0 failed. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Henry Guo
|
99eef032c6 |
fix(ecstore): restore Windows async read trait import (#4652)
* fix(ecstore): restore Windows async read trait import * fix(ecstore): gate async read import to non-Unix |
||
|
houseme
|
15b8b13698 |
feat(ecstore): cache part-file descriptors for io_uring reads (#4658)
* feat(ecstore): run one io_uring ring per shard on each disk (backlog#1145)
A buffered read that hits the page cache completes inline inside
`io_uring_enter`, so the thread driving a ring performs that read's
memcpy. One ring per disk therefore capped cache-hit reads at a single
core's memory bandwidth: measured on a 16-core host, one driver thread sat
pinned at 100% CPU while throughput stayed flat at ~5 GB/s regardless of
read size, against 50 GB/s for the blocking-pool baseline.
rustfs/uring#6 taught the driver to hold N independent rings, each with its
own thread, pending table, backpressure semaphore, and eventfd. Wire it up:
`UringBackend::try_new` now calls `probe_and_start_sharded`, and
`RUSTFS_IO_URING_SHARDS` selects the count per disk.
The default is a quarter of the available parallelism clamped to `1..=4`,
because the cost is `disks × shards` driver threads (each normally blocked
in `poll(2)`). Any override is clamped to `1..=16`, so a mistyped value can
neither disable the driver (0) nor spawn threads without bound; an
unparseable value falls back to the default.
Effect (warm page cache, 16-core, rustfs/uring's concurrent_pread_bench):
1 MiB, conc 8: 1 shard 4911 MB/s -> 8 shards 47361 MB/s (9.6x);
the blocking-pool baseline is 50662 MB/s
64 KiB, conc 32: StdBackend 153678 IOPS, p999 3030 us
8 shards 345402 IOPS, p999 897 us
64 KiB, conc 128: StdBackend 135155 IOPS, p999 10716 us
8 shards 389047 IOPS, p999 4092 us
Sharding removes the throughput deficit *and* keeps io_uring's tail-latency
advantage, rather than trading one for the other.
Unchanged: io_uring read stays gray-off by default
(`RUSTFS_IO_URING_READ_ENABLE`), reads are byte-for-byte identical to
StdBackend, the per-disk degradation latches and probe cache (backlog#1101)
and the O_DIRECT tiered fallback (backlog#1102) all still apply. Rings stay
per-disk, so a stalled disk cannot starve another disk's rings
(backlog#1055). Bumps the rustfs-uring pin to the merged #6 commit.
Verified on a real Linux host (16-core, real io_uring): cargo clippy
--tests -D warnings clean; disk::local tests 132 passed, 0 failed —
including the existing io_uring and O_DIRECT cases now running on the
sharded driver, plus a new test covering the shard-count default, override,
and clamping.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): cache part-file descriptors for io_uring reads (backlog#1145)
`pread_uring` opened the file on the blocking pool for every read, so each
read paid a `spawn_blocking` round trip — the very thread hop io_uring
exists to avoid. Sharding the driver (backlog#1145) removed the previous
ceiling and left this as the binding cost. Measured on a 16-core host with
a 4-shard driver, warm page cache:
64 KiB, conc 8: 143942 -> 263054 IOPS (+83%), p999 240 -> 65 us
64 KiB, conc 32: 150128 -> 204876 IOPS (+36%), p999 2508 -> 871 us
64 KiB, conc 128: 129172 -> 361287 IOPS (+180%), p999 15329 -> 3046 us
1 MiB, conc 32: 33875 -> 42301 IOPS (+25%)
At 64 KiB / conc 128 the open is what masked io_uring entirely: with it,
io_uring beat StdBackend by 3.5%; without it, by 189%.
Add a bounded per-disk descriptor cache used by the io_uring read path.
A hit takes no `open` and no `spawn_blocking`, so the read never leaves the
runtime worker.
Why caching a part-file descriptor is safe:
* only `<object>/<data_dir>/part.N` reaches this backend's `pread_bytes`;
`xl.meta` — the one path replaced in place — is read through `read_all`
/ `read_metadata` and never gets here;
* part files are never rewritten in place. A replacement is always
write-new-tmp then `rename`, which swaps the inode, so a cached
descriptor can never observe a torn shard.
Why invalidation is nevertheless REQUIRED: heal reuses the existing
version's `data_dir` and renames a rebuilt shard onto the SAME part path.
A cached descriptor would keep serving the pre-heal (corrupt) inode,
defeating the heal and eroding read quorum. `delete` likewise unlinks a
part that a cached descriptor would keep readable. So `rename_data`,
`rename_file`, and `delete` all call the new
`LocalIoBackend::invalidate_cached_fds` after they mutate, and a 5s TTL
bounds the blast radius should a future mutation path forget to.
Two preamble checks the miss path runs are not silently lost on a hit:
* bounds — the driver only short-reads at EOF (it resubmits otherwise),
so `bytes.len() != length` is exactly the old `meta.len() < end_offset`
check, and now yields the same `FileCorrupt`;
* volume access — skipped while an entry is live. An unreachable disk
keeps serving already-open descriptors for at most the TTL, after which
the re-open re-runs the check. Disk health is tracked independently of
this per-read probe.
Scope: buffered io_uring reads only. The O_DIRECT path keeps opening per
read (its reads are >= 4 MiB, so the open is a small fraction), and
StdBackend is untouched — it must take the blocking hop for the pread
regardless, so caching would buy it only 2-6% while carrying the same
invalidation risk. `RUSTFS_IO_URING_FD_CACHE=false` restores open-per-read.
Verified on a real Linux host (16-core, real io_uring): clippy --tests
-D warnings clean; disk::local tests 135 passed, 0 failed. The new
heal-staleness test first asserts a read still returns the PRE-heal bytes —
proving the cache is live and the hazard real — then that invalidation makes
the healed shard visible. A second test drives `rename_file` and `delete`
through `LocalDisk` to prove those paths actually invalidate, and a unit
test pins prefix invalidation to component boundaries (`a/b` must not drop
`a/bc`).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|