* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2)
Add the AWS 2026-04 additional checksum algorithms as base types in
rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len,
hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path
silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the
single source of truth, allocate the new base-type bits append-only above
bit 9 to preserve the on-disk varint format, and add streaming hashers whose
digest uses the S3 canonical big-endian encoding (seed 0).
The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is
rejected and they are never routed through add_part()/can_merge(). A
round-trip guardrail test asserts every base type survives all dispatch
sites, failing loudly if a future algorithm is added but a match arm or the
mask is forgotten.
Refs rustfs/backlog#1254 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3)
Lock the byte order and seed of the new algorithms against the OFFICIAL
upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128,
SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte
identical to what AWS SDKs (awscrt) compute — the interop correctness this
feature hinges on. Add a non-empty regression lock (official "fox" vectors)
that also asserts the encoded field is the standard-base64 of the raw digest.
Refs rustfs/backlog#1255rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8)
Cover the xl.meta varint (de)serialization for the new algorithms:
to_bytes() -> read_checksums() must recover the value under the Display key
for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node
reading a future, unknown base-type bit degrades safely — skips the entry and
returns without panicking or mis-decoding a length. Combined with the
append-only bit allocation from S2, this protects mixed-version clusters.
Refs rustfs/backlog#1260rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5)
HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128
and SHA-512 checksums that S3 stored, closing the head_object gap in #4800.
s3s HeadObjectOutput has no typed field for these, so they are emitted as raw
response headers via response.headers (the same mechanism RustFS already uses
for tagging-count), keyed by ChecksumType::key(). The existing five typed
algorithms are unchanged. Also carries the Cargo.lock update for the
xxhash-rust dependency introduced in S2.
Refs rustfs/backlog#1257rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(checksums): fail-closed on unknown checksum algorithm (S7)
A. Harden unknown/unsupported checksum algorithms to fail closed instead of
panicking. ChecksumMode::base() in the outbound S3 client
(crates/ecstore/src/client/checksum.rs) previously did
`panic!("enum err.")` for any mode without a concrete base algorithm (e.g.
a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added
unit tests proving base() never panics and hasher() returns Err for
unsupported modes. rustfs-checksums FromStr already returns Err on unknown
names; added a regression test asserting garbage/unknown names fail closed.
B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional
algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str,
into_impl, name constants, the x-amz-checksum-* header constants and the
HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio
spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128
big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each
digest against a direct library computation. MD5 stays intentionally
rejected (PR #4513) and is left untouched.
C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8"
with 7 variants already consuming 7 bits; adding the 4 new algorithms would
overflow u8 and require a breaking repr change, so ChecksumMode is left
unchanged. The new algorithms are available through the rustfs-checksums
ChecksumAlgorithm path.
Refs rustfs/backlog#1259rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4)
Complete the additional-checksum round-trip so AWS SDKs can verify integrity on
download and confirm it on upload:
- GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and
SHA-512 checksums (the download-side path SDKs auto-verify). The values flow
from build_get_object_checksums through GetObjectOutputContext into
finalize_get_object_response and are emitted after wrap_response_with_cors.
- PutObject echoes the server-computed additional checksum on its response,
captured at the want_checksum set points before opts is moved.
Both reuse a single centralized helper, inject_additional_checksum_headers,
which HeadObject now also uses. This is the ONLY place that emits these headers,
so when s3s gains typed fields for these algorithms the migration is one spot
(fill the typed field, drop the insert) with no risk of duplicate headers.
The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value
lands after the body) is left for e2e coverage in S10.
Refs rustfs/backlog#1257rustfs/backlog#1256rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9)
Make multipart uploads work end-to-end for the composite-only algorithms
(XXHash3/64/128, SHA-512):
- complete_part_checksum previously returned the outer None for any algorithm
outside the five typed ones, which failed CompleteMultipartUpload with
InvalidPart. It now accepts any valid base type with no double-check value
(Some(None)) — mirroring the missing-value path of the typed algorithms —
since s3s CompletePart has no field to carry a client-supplied per-part
value and the part was already verified server-side at UploadPart. Genuinely
unset/invalid types are still rejected.
- The existing COMPOSITE assembly (Checksum::new_from_data over the
concatenated per-part raw digests; full_object_requested() is false so
add_part() is correctly bypassed) already works for these algorithms via the
S2 wiring. A rio test locks the assembly and that add_part refuses them.
- UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on
their responses via the shared inject_additional_checksum_headers helper
(now pub(crate)), since s3s has no typed output field.
Refs rustfs/backlog#1261rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6)
Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from
the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest,
COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector.
Thanks to the single-source-of-truth wiring from S2, every dispatch site
(GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the
COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all
arm — no handler changes needed. Tests are extended to cover MD5 across them.
Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client
reject "md5" so it could never silently fall back to CRC32. This change is on
the server-side rio path and never falls back — it implements MD5 correctly
rather than substituting another algorithm — so the #4513 intent is preserved,
and the outbound client keeps rejecting md5 (S7).
Refs rustfs/backlog#1258rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11)
from_string_with_obj_type ran alg.to_uppercase() on every checksummed request,
allocating a String just to compare against a fixed set of algorithm names.
Replace it with eq_ignore_ascii_case, which is allocation-free and, for the
ASCII algorithm names involved, exactly equivalent. A test locks that
case-insensitivity, the CRC64NVME full-object assumption, composite-only
FULL_OBJECT rejection, and unknown/empty handling are all unchanged.
The other S11 notes are intentionally not acted on: the Phase-0 header scan is
N/A (we chose full support over rejection, so there is no reject guard), and
parallelizing the serialized hash passes is deferred pending a measured need.
Refs rustfs/backlog#1263rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(checksums): collapse 5 duplicated response-checksum loops into one
Review of the accumulated commits found the same "iterate decrypted checksums,
match five typed algorithms, drop the rest" loop copy-pasted across five
response paths (GetObject, HeadObject, GetObjectAttributes object-level and
part-level, CompleteMultipartUpload). That was patch-on-patch duplication.
Collapse it into a single source of truth:
- rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the
five-typed vs additional-algorithm split.
- object_usecase gains ResponseChecksums + classify_response_checksums(), which
performs the typed/extra split once. All five call sites now destructure its
result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a
hand-rolled five-way comparison.
Behaviour is unchanged (GetObjectAttributes still cannot surface the additional
algorithms — an s3s XML-body limitation, now documented in one spot). One pass
over the map; extra pairs pushed only when a new-algorithm checksum is present.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(checksums): unit tests for classifier/echo helpers + fix unused import
Add direct unit tests for the refactored single-source-of-truth helpers:
- rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and
that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification.
- object_usecase classify_response_checksums() — typed fields vs `extra` headers,
the checksum-type marker, and empty input.
- additional_checksum_echo_pairs() — echo pair only for additional algorithms,
none for the five typed ones, none for None.
- inject_additional_checksum_headers() — writes all pairs; empty is a no-op.
Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left
by the classifier refactor (would fail the -D warnings gate).
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length)
The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword
the S8 forward-compat comment; no code change.
Refs rustfs/backlog#1260
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10)
Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04
additional algorithms. aws_sdk_s3 has no typed builder for these, so the
x-amz-checksum-<algo> header is injected via mutate_request (value from
rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic
checksum calculation disabled (request_checksum_calculation=WhenRequired) so the
injected header is the only checksum on the wire. For each of XXHash3/64/128,
SHA-512 and MD5: a correct value is accepted and the object stored intact; a
mismatched value is rejected with BadDigest and nothing is stored.
Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that
additionally confirms the HEAD/GET header echo (14/14).
Refs rustfs/backlog#1262rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(get): allow too_many_arguments on finalize_get_object_response
The classifier refactor added an extra_checksum_headers parameter, pushing
finalize_get_object_response to 8 args and tripping clippy::too_many_arguments
under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already
carry; no behavior change.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): open cleaner compression source with O_NOFOLLOW
The compressor opened the source log via File::open, which follows a
symlink at the final path component. Between the scanner selecting a
regular file and this open, an attacker with write access to the log
directory could swap the entry for a symlink (TOCTOU) pointing at, say,
/etc/shadow, whose contents would then be copied into an archive. Open
the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the
temp/archive path already refused symlinks, this closes the source side.
Refs rustfs/backlog#1210
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): recompress instead of trusting leftover cleaner archives
archive_header_ok only checked the first 2-4 magic bytes before treating
an existing .gz/.zst as a completed prior result and letting the caller
delete the source log. A file with valid magic but a truncated or forged
body passes that check, so an attacker with write access to the log
directory (or a crashed prior run) could plant such a stub and make the
cleaner delete the real log without ever producing a usable archive —
silent audit-data loss.
Chosen fix: stop trusting cross-process leftovers entirely and always
recompress the source in this pass, rather than fully decoding every
leftover to validate it. Full-decode validation would add real CPU cost
and decode-bug surface for a rare crash-recovery case; the existing
atomic create_new+rename already overwrites whatever sits at the archive
path (a planted symlink is replaced, never followed) with a freshly
written, fsync'd archive, so a partial/forged leftover can never gate
source deletion. This is the lowest-regression option.
Refs rustfs/backlog#1211
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): cap memory-gate reservation at cache growth headroom
The memory gate subtracts `admitted_since_refresh` from the snapshot's
available bytes so a burst arriving faster than the 5 s refresh cannot
over-allocate. That counter is GROSS: it only rolls over on the refresh and
never rolls back when a fill is later evicted, cancelled, or loses the
invalidation race. Under sustained high-throughput churn (net footprint flat
and far below `max_capacity`) the raw counter balloons past the memory the
cache actually holds, so `effective_available` collapses and the gate reports
false memory pressure — skipping the hottest fills with SkippedMemoryPressure
until the next 5 s refresh. This only lowers hit rate; it never returns wrong
data and self-heals each refresh.
Fix direction 1 (minimal regression): cap the reservation deduction at the
cache's own growth headroom (`max_capacity - weighted_size()`) instead of
letting the unbounded gross counter shrink the system-available budget. The
cache can never hold more than `max_capacity`, so a burst adds at most that
headroom of real memory before moka evicts to stay bounded (net-zero churn
beyond that point) — capping the deduction there keeps the reservation honest
without treating gross churn as growth. Chosen over net-accounting (direction
2, releasing bytes on every failure/cancel/eviction path) because that only
plugs the leak on failed fills and would not address the core defect: churn of
*successful* insert/evict fills over the 5 s window. It also touches only the
gate plus one call site rather than every failure path in moka_backend.
The cap only ever raises `effective_available`, so real memory pressure (a low
snapshot at refresh) still suppresses fills; when the cache is at capacity the
headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn.
`MokaBackend` now stores `max_capacity` and passes the live headroom into
`allows_fill`. Adds targeted gate tests: gross churn far above headroom no
longer falsely suppresses, yet the reservation still bounds a burst while the
cache can genuinely grow.
Refs rustfs/backlog#1212
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): assert native O_DIRECT path runs in uring read test
uring_preserves_o_direct_for_eligible_reads only compared bytes through
LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the
read silently degrades to the buffered StdBackend fallback and the byte
check still passes, so the test could go green without the native
read_at_direct path ever executing -- a vacuous pass.
Add a per-disk native_direct_reads counter on UringBackend, incremented
only when pread_uring_direct completes, and rebuild the test to drive a
real UringBackend's pread_bytes and assert the counter is non-zero (every
eligible read went through the native tier). When io_uring or O_DIRECT is
unavailable on the host filesystem (restricted CI runners, tmpfs), the
test skips loudly via eprintln instead of asserting a tautology, while
still checking byte-correctness on whatever tier served the read.
The counter also gives a gray release a positive signal that the O_DIRECT
tier is serving reads, not just a fallback count.
Refs rustfs/backlog#1213
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads
classify_direct_read_error is only reached from the read side: the
O_DIRECT open in pread_uring_direct already succeeded (an open-time
refusal is handled earlier as DirectOpenError::ODirectRefused). So an
EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel
accepted for O_DIRECT -- far more likely an alignment bug in the aligned
read path than an unsupported filesystem. The old code latched the disk's
native path off with only a once-per-disk debug trace, hiding a potential
correctness regression behind a silent buffered-read downgrade.
Diagnostics only: the fallback behaviour is unchanged (the native path is
still latched off and the caller still reads via StdBackend). This adds a
rustfs_io_uring_direct_read_einval_total counter and promotes the
once-per-disk trace from debug to warn so an operator can see an alignment
regression instead of an unexplained latency/CPU shift.
Refs rustfs/backlog#1214
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document data-blocks-first default and its tail-latency cost
DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay
true: deferred-parity is the deliberate, already-rolled-out full-object
GET default from backlog#1159/#923. Flipping it back to false in code
would silently revert that rollout for every deployment that has not set
the env var, so this commit only documents -- no behaviour change.
The added notes explain what data-blocks-first does (schedule data shards
up front, engage parity lazily on a missing/corrupt data shard), the known
trade-off (parity is engaged late, so a slow-but-not-dead data drive
raises GET p99 because the faster parity shards are not raced against it
until a data shard is declared missing), and the operational rollback
switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is
intentionally an env override rather than a code default change.
No metric was added: the low-risk observability hook for "slow data drive
engaged deferred parity" would live at the deferred-stripe engage point,
which is out of this file's scope; this change stays documentation-only to
avoid touching the hot GET path.
Refs rustfs/backlog#1215
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document wide-directory walk stall hazard and tuning
list_dir enumerates a whole directory in one os::read_dir call (count =
-1), and the walk caller bounds that entire enumeration with the per-read
stall budget (default 5s) as if it were a single read. For a wide, flat
prefix -- one directory holding millions of immediate children -- a single
readdir can exceed the budget on a healthy disk, trip DiskError::Timeout,
and surface as a ListObjects 500 quorum failure though the drive is fine
(a #2999 sub-class).
This is documented, not rewritten: turning the one-shot readdir into a
streaming/batched enumeration that refreshes the stall deadline between
chunks is an architecture-level change with high regression surface
(ordering, the count contract, quorum merge) and belongs in a separate
follow-up. The supported mitigation today is operational, so the comments
point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS
and the high-latency drive-timeout profile, which widen the budget with no
code change. Notes were added at list_dir, the scan_dir call site, and
get_drive_walkdir_stall_timeout. No behaviour change.
Refs rustfs/backlog#1216
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document consumer-peek vs producer-stall coupling
In list_path_raw the consumer's peek_timeout is drawn from the same source
and same value (walkdir_stall_timeout, default 5s) as the producer-side
walk stall budget, but the two measure different things: the producer
stall bounds a single drive read, while the consumer peek bounds the gap
between two ADJACENT entries arriving from a reader. Because they share a
value, the consumer cannot wait meaningfully longer for the next entry
than the producer is allowed to spend producing one. Walking a region
dense with non-listable internal items can make a HEALTHY drive miss the
budget between visible entries; the consumer then declares it stalled and
detaches it, dropping a good drive from the merge and capping the "large
prefix succeeds" guarantee.
Documented, not decoupled: giving the consumer peek an independent,
strictly-larger budget would cut these false detaches but equally delays
detaching a genuinely dead drive and shifts listing tail-latency
semantics, so it wants soak data before changing the default. The comment
records the invariant any such follow-up must keep -- consumer peek >=
producer stall, never stricter -- so it can never fail a drive before the
producer would. No behaviour change.
Refs rustfs/backlog#1217
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(io-metrics): add time-based trigger for low-IOPS latency percentiles
Percentiles were recomputed only every 128 IOs and seeded to 0, so a
low-traffic deployment exported p95/p99 = 0/stale for a long time after
startup. Add a 10s wall-clock trigger alongside the count throttle so the
first recompute can fire before 128 samples accrue. Hot-path per-op mean
update is unchanged.
Refs rustfs/backlog#1218
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): cover codec-streaming parity under fault injection and NoSuchKey
The codec-streaming compat A/B previously ran only against a healthy
4-disk EC set with successful full GETs: the DiskFaultHarness was
constructed but never faulted, the error path was untested, and the
range assertion silently compared legacy-vs-legacy (ranges always fall
back to the duplex path), overstating what it proved.
Add two genuinely-failable scenarios reusing the existing harness and
fixtures:
- Parity reconstruction A/B: take one data disk offline and re-run the
full object matrix on both phases while the EC 2+2 set rebuilds each
large object from the surviving shards. Assert codec == legacy
byte-for-byte (sha256) and header-for-header, and assert the codec
phase served the reconstructed objects with zero duplex-pipe fallback
(the reader gate is drive-health-independent, so the codec fast path
is really exercised through reconstruction).
- NoSuchKey negative path: compare the HTTP status + S3 error code of a
missing-key GET across the legacy and codec phases and require them to
be identical (404/NoSuchKey), guarding against the codec env
perturbing the error path.
Also clarify the range-phase comment so it is not misread as
codec-range correctness coverage: both sides are served by the same
legacy range path, so the assertion only proves ranges keep working and
keep falling back to legacy with the gates open.
Verified: cargo check/--no-run pass and the test passes locally
(1 passed; dup_codec=0 confirms the codec path ran).
Refs rustfs/backlog#1219
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback
The uring-integration leg ran on the runner's default TMPDIR, which may sit
on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct
path silently latches off to the aligned StdBackend fallback. Mount a
dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep
(bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are
actually covered rather than validated only by signature diffing.
Refs rustfs/backlog#1220
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1183 tracks flipping the default GET data path from the legacy
tokio::io::duplex double-copy to the zero-duplex codec-streaming fast path.
That flip is gated behind RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED /
..._HEADER_COMPAT_CONFIRMED, which need empirical evidence the two paths are
byte- and header-identical.
Add an e2e regression net that runs the same object matrix twice against the
same on-disk EC shards, changing only the codec-streaming env gates: phase A
(default) takes the legacy duplex path, phase B (gates opened) takes codec
streaming. It asserts byte-for-byte (sha256) and header-for-header equality
across inline / small / multi-block (1.5M/3M/5M+) / multipart objects, plus a
ranged GET (which falls back to legacy by gate design). Path confirmation is
not assumed: the legacy path logs "Created duplex pipe ..." per full GET, so
the test counts that marker per phase and asserts the codec phase created zero
duplex pipes, proving the fast path actually ran rather than silently falling
back to the path it is compared against.
To capture the child server's logs for that assertion, add an optional
RustFSTestEnvironment.capture_log_path (default None = inherit stdio,
backward compatible) that redirects the spawned server's stdout+stderr to a file.
Co-authored-by: heihutu <heihutu@gmail.com>
Convert the two reliant lifecycle expiry tests from #[ignore] + a hardcoded
localhost:9000 server to self-managed RustFSTestEnvironment tests (random port,
isolated temp dir) and wire them into the PR e2e-smoke subset.
Time control is chosen per test and documented in-module:
- test_lifecycle_expiry_backdated_mtime: mod_time back-dating via the internal
source-replication backdoor (new put_object_with_backdated_mtime helper) so a
Days=1 rule is already due, with RUSTFS_ILM_PROCESS_TIME=1 shrinking the
rounding boundary. Asserts the backdated matching-prefix object expires and a
non-matching-prefix object with the same backdating survives (isolates the
prefix filter). Proves the backdoor does not trip the replication gate.
- test_lifecycle_versioned_current_version_expiry_creates_delete_marker:
RUSTFS_ILM_DEBUG_DAY_SECS=2 (ilm-5) accelerates the day length; asserts a
latest delete marker is created, the original data version is retained and
still readable by version id.
- test_lifecycle_zero_day_expiry: Days=0 immediate expiry; matching object
deleted, non-matching survives.
All three use RUSTFS_SCANNER_CYCLE=1 so the scanner runs every second; tests poll
for the terminal state instead of sleeping fixed wall-clock. Ignore count for
reliant/lifecycle.rs goes 2 -> 0.
CI wiring: added a distinct 'test(/^reliant::lifecycle::/)' clause to the
existing profile.e2e-smoke default-filter in .config/nextest.toml (the single
sanctioned e2e wiring mechanism per crates/e2e_test/README.md; no new e2e job).
Refs rustfs/backlog#1148 (ilm-3), rustfs/backlog#1155
test(e2e): negative presigned-URL SigV4 suite (backlog#1151 sec-2)
Add crates/e2e_test/src/presigned_negative_test.rs, the query-string
SigV4 sibling of the header-SigV4 suite (sec-1, #4708). Presigned URLs
were only exercised on the happy path, so nothing pinned our end-to-end
enforcement of expiry or query-signature verification.
Against a live single-node RustFSTestEnvironment (random port, isolated
temp dir), the suite asserts HTTP status + S3 error <Code> for:
- expired presigned GET -> 403 AccessDenied (Request has expired)
- tampered X-Amz-Signature -> 403 SignatureDoesNotMatch
- wrong secret key -> 403 SignatureDoesNotMatch
- tampered target key (swapped after signing) -> 403 SignatureDoesNotMatch
- tampered presigned PUT -> 403 SignatureDoesNotMatch + object not stored
- positive controls: valid presigned GET (body match) and PUT (HEAD verify)
Expiry is controlled without real waiting via the AWS SDK presigner's
start_time (sign as of one hour ago with a 60s window). s3s checks
expiry before the signature, so the expired case surfaces AccessDenied.
Wired into the e2e-smoke nextest profile (fast, single-node, no external
deps) so it runs on every PR per the crates/e2e_test/README.md admission
criteria.
Refs rustfs/backlog#1151 (sec-2), rustfs/backlog#1155.
test(admin-auth): unit + e2e coverage for the admin authorization gate
Adds the first tests for the central admin authorization gate
`rustfs/src/admin/auth.rs`, which previously had zero coverage
(backlog#1151 sec-4, master plan #1155).
Unit tests (rustfs/src/admin/auth.rs):
- Refactor the two `validate_admin_request*` entry points to share a
single generic decision core `evaluate_admin_actions<S: Store>` /
`check_admin_request_auth<S: Store>` (removes the duplicated
action-loop and makes the gate testable without a running cluster).
- Cover: owner/root credential allowed; authenticated non-admin
credential denied with AccessDenied; missing credential denied; and
the multi-action loop grants on any permitted action, denies when
none pass. Backed by an in-memory empty IamSys so the owner path
short-circuits to allow and every other principal resolves to deny.
E2E (crates/e2e_test/src/admin_auth_test.rs, raw SigV4 over HTTP, no
awscurl dependency):
- non_admin_credential_denied_on_admin_api: a limited IAM user gets
403 AccessDenied on GET /rustfs/admin/v3/info while root succeeds.
- root_credential_rotation_takes_effect: restart with rotated
--access-key/--secret-key; new credential works and the stale one is
rejected, on both the admin plane and the S3 data plane.
- default_credentials_emit_startup_warning: booting with the default
rustfsadmin credentials emits the default-credentials warning.
Refs rustfs/backlog#1151 (sec-4), #1155.
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit 5c7c757a3 (#4712) activated the previously-dormant replication e2e
suite (they had never run anywhere). All 9 fast tests then failed on main
because the SSRF egress guard rejects the 127.0.0.1 targets the e2e harness
configures: `target endpoint is not allowed: outbound URL host '127.0.0.1'
is not allowed: loopback address`. The whole harness runs on loopback, so
every replication test hit this before reaching its actual assertion.
Loopback is a genuine SSRF vector and must stay rejected in production, so
this does not relax the guard. Instead `validate_replication_target_endpoint`
gains an off-by-default opt-in (`RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET`)
that re-enables loopback targets (127.0.0.1 / ::1 / localhost) for single-host
multi-instance dev and the e2e harness. Private addresses stay unconditionally
allowed as before; the opt-in does not widen into link-local or the cloud
metadata endpoint. The e2e harness sets the env for every server it spawns
(single-node and cluster paths), overridable via extra_env.
Verified end-to-end: all 9 previously-failing replication_extension_test
smoke tests pass against a locally built binary. New unit tests in
bucket_target_sys pin the matrix — public/private always allowed, loopback
gated on the opt-in in both IP and hostname forms, and metadata/link-local
still rejected even with the opt-in on.
Refs: backlog#1147
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(replication): rename optin -> opt_in to satisfy typos check
Pure rename of three unit-test function names; no behaviour change.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
test(security): add GHSA-named regression tests for 3p3x and r5qv (backlog#1151 sec-6)
Anchor the two fixed advisories to discoverable, named regression tests so
`rg -i "ghsa|3p3x|r5qv"` finds a guard for each, and future fixes are forced
to update pinned behavior (red -> green).
GHSA-3p3x-734c-h5vx (constant-time WebDAV/FTPS secret comparison, rustfs#4403):
- ftps_core.rs: new `assert_ftps_ghsa_3p3x_wrong_credentials_rejected` drives
the `ct_eq` reject branch in FtpsAuthenticator::authenticate; asserts wrong
password and unknown user are both rejected (530) and indistinguishable.
- webdav_core.rs: the auth-failure block now sends a valid access key with a
wrong secret (exercising the `ct_eq` branch, not just the unknown-access-key
path) plus an unknown user, asserting both 401 and indistinguishable.
- Module doc comments map advisory -> tests -> fix PR on both files.
GHSA-r5qv-rc46-hv8q (internode RPC fail-closed, rustfs#4402):
- http_auth.rs: renamed the default-fallback rejection test to
`ghsa_r5qv_resolve_shared_secret_rejects_default_fallback` (and broadened it
to cover default env secret + blank secrets), and added
`ghsa_r5qv_verify_rpc_signature_fails_closed_on_missing_or_invalid_auth`
pinning the exact advisory scenario (missing/forged/cross-URL signature is
rejected; a correctly signed request still passes). File-level doc maps the
advisory.
Docs: new docs/testing/security-regressions.md with the advisory -> test
mapping table and where each layer runs; linked from docs/testing/README.md.
sec-14 will formalize the written policy in AGENTS.md.
The unit-level ghsa_r5qv_* tests run in the default CI pass. The WebDAV/FTPS
e2e live in the protocols suite (fixed ports, --test-threads=1); they cannot
join the e2e-smoke profile and are wired into CI by sec-5.
Refs: rustfs/backlog#1151 (sec-6), rustfs/backlog#1155
* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)
The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.
These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.
First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
- test_lifecycle_expiration (prefix Days=1/Days=5, scanner delete)
- test_lifecyclev2_expiration (same via ListObjectsV2)
- test_lifecycle_noncur_expiration (NoncurrentVersionExpiration)
- test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
- test_lifecycle_multipart_expiration (AbortIncompleteMultipartUpload)
Dropped from the batch, kept excluded with reason:
- test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
while AWS/this test expect InvalidArgument. Real validation gap.
Changes:
- scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
- scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
can point at an alternate whitelist.
- .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
no start delay) and a 2s stale-multipart cleanup interval, running the new
list serially.
- scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
scope=all sweep (plain server) does not misclassify expected failures.
- scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
- docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
excluded 274, behavior 5) and document the lane.
Refs backlog#1148 (ilm-10), master plan backlog#1155.
* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)
test(e2e): add negative header-SigV4 rejection suite (backlog#1151 sec-1)
RustFS delegates SigV4 verification to the s3s dependency, so nothing in
this repo pins OUR end-to-end wiring of it. Add negative_sigv4_test.rs
sending REJECTED header-SigV4 requests against a live RustFSTestEnvironment
and asserting the HTTP status plus the S3 error code XML:
- valid_header_sigv4_request_succeeds (positive control so the negatives
cannot pass for the wrong reason)
- tampered_signature_returns_signature_does_not_match -> 403 SignatureDoesNotMatch
- wrong_secret_key_returns_signature_does_not_match -> 403 SignatureDoesNotMatch
- tampered_payload_is_rejected (body != signed x-amz-content-sha256) -> not 200
- skewed_date_returns_request_time_too_skewed (>15min) -> 403 RequestTimeTooSkewed
- malformed_authorization_header_returns_clean_4xx (present-not-missing) -> 4xx, never 5xx
Signatures are hand-built via rustfs_signer::request_signature_v4 primitives
(get_signing_key/get_signature/get_scope) so the test controls the timestamp,
secret, signed payload hash, and final signature bytes. Missing-credential
negatives are intentionally not duplicated (covered by multipart_auth_test /
anonymous_access_test).
Refs backlog#1151 (sec-1), master plan backlog#1155.
Adds an S3-API-level e2e regression net proving that a large-object GET on a
degraded EC set never returns a silently truncated body under a full
Content-Length -- the historical "unexpected EOF" bug fixed on main by
rustfs#4594 (short-body GetObject stream -> UnexpectedEof), rustfs#4560
(in-place per-part legacy degradation for the lazy multipart reader), and
rustfs#4585 (DARE package-boundary truncation detection). Those fixes each
ship a *unit* regression; this covers the layer they do not -- the full HTTP
GET path streaming a real body reconstructed from real on-disk EC shards.
New file crates/e2e_test/src/degraded_read_eof_regression_test.rs, single-node
4-disk (EC 2+2) DiskFaultHarness, three scenarios:
(a) one disk offline: a 6 MiB single object (>=2 EC stripes) and a 3x5 MiB
multipart object GET back byte-identical with the correct Content-Length.
(b) mid-stream bitrot within quorum: 2 of 4 shards corrupted mid-file on a
large multipart object still reconstructs the full, hash-matching body.
(c) beyond read quorum (the heart of the net): 3 of 4 shards corrupted
mid-file -- the read MUST fail cleanly (non-2xx or a mid-stream body
error), NEVER close with a truncated body under the full Content-Length.
The shared get_checked() helper panics on the forbidden outcome (a clean 2xx
whose collected body is shorter than the advertised Content-Length), so the
truncation bug can never be silently tolerated.
CI placement: these spawn a 4-disk server per test and are resource-heavy, so
they stay OUT of the fast PR e2e-smoke filter. A new e2e-reliability nextest
test-group (max-threads=1) serializes them (and the existing
reliability_disk_fault_test) across nextest's process boundary; ci-7's nightly
full-e2e run picks them up. Visible via `cargo nextest list -p e2e_test`.
Refs rustfs/backlog#1150 (dist-13), rustfs/backlog#1155, rustfs#4594,
rustfs#4560, rustfs#4585, rustfs#2955.
Activate the 36 dormant replication e2e tests in
crates/e2e_test/src/replication_extension_test.rs (zero ran anywhere before).
Split via the ci-4 nextest profile mechanism, no hand-rolled cargo-test lane:
- PR smoke (profile.e2e-smoke, existing e2e-tests job): the 20 fast
bucket-replication tests (target-registration / replication-check / list /
remove / delete admin paths) that validate config synchronously and never
wait for async convergence. Each spawns its own single-node rustfs server(s)
on random ports with isolated temp dirs, so parallel-safe by construction
(serial_test's #[serial] is a no-op under nextest's process-per-test model;
no test-group needed).
- Nightly (profile.e2e-repl-nightly + .github/workflows/e2e-replication-nightly.yml):
the remaining 16 = 6 slow data-plane tests + 9 _real_dual_node + 1
_real_single_node. Defined as 'replication module MINUS the PR allowlist' so
new replication tests default to nightly and are never silently unrun.
The nightly workflow builds the binary once, installs awscurl so the STS
dual-node test runs (skips gracefully with a visible log line otherwise), and
routes scheduled failures through .github/actions/schedule-failure-issue
(ci-8). Explicit division of labor with ci-5 e2e-full: these run only here.
Counts (cargo nextest list): e2e-smoke 83 (63 + 20), e2e-repl-nightly 16.
Docs updated: e2e-suite-inventory.md, e2e_test/README.md.
Refs backlog#1147 repl-1, backlog#1155.
docs(e2e): write contributor guide for e2e_test crate (backlog#1153 infra-10)
Turn the e2e_test README skeleton into a dense, link-heavy contributor guide:
crate overview + grouped module map, how to run (whole crate / one module /
e2e-smoke profile / ILM serial lane / protocols suite), #[ignore] semantics as
a pointer to live sources, how to add a test (RustFSTestEnvironment +
RustFSTestClusterEnvironment, common.rs/chaos.rs helper inventory, isolation
rules, #[serial]-vs-nextest reality), a CI subset map table, and a
troubleshooting section (stale binary / RUSTFS_TEST_PORT / orphan processes).
Keeps the ci-4 "CI smoke subset" section intact and restructures around it.
Adds a reciprocal link from crates/e2e_test/AGENTS.md.
Refs rustfs/backlog#1153 (infra-10), #1155.
ci: run e2e smoke subset via nextest e2e-smoke profile (backlog#1149 ci-4)
The e2e-tests job previously ran a single test (delete_marker_migration
_semantics) out of ~400 in the e2e_test crate. Define a nextest
profile.e2e-smoke whose default-filter selects 17 fast single-node
dependency-free modules (63 tests) and run it in the job, reusing the
downloaded debug binary. The profile is the single wiring mechanism for
e2e tests in CI; admission criteria live in crates/e2e_test/README.md,
and docs/testing/e2e-suite-inventory.md records authoritative per-module
counts from cargo nextest list.
Introduce RUSTFS_REPL_HEALTH_CHECK_INTERVAL_MS, RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS
and RUSTFS_REPL_RESYNC_POLL_MAX_MS so tests and operators can shorten the
replication background loops. Defaults are unchanged; invalid values fall
back with a warn and values below 10ms are clamped to avoid busy-spin.
Add replication_fast_env() e2e helper (backlog#1147 repl-4).
test(e2e): align object-lock overwrite tests with AWS new-version semantics
PR #3179 made put-like writes (PUT, CopyObject, multipart create/complete)
on versioned object-lock buckets create a new version instead of being
blocked by the current version's legal hold or COMPLIANCE retention, which
is the correct AWS behavior. Six e2e assertions in object_lock_test.rs
still encoded the old blocking semantics and, because e2e_test is excluded
from the CI test run, stayed red unnoticed since 2026-06-03.
Verified against a real server built from current main: all six failed at
their "should be blocked" assertions. Rewritten to assert the AWS
contract: the write succeeds with a distinct new version id, the new
content is current, the protected version keeps its content and lock
metadata, and version-targeted deletion of the protected version stays
blocked (for COMPLIANCE even with bypass_governance). Full object_lock
module is green again (33/33).
fix(ecstore): emit CommonPrefix for object with same-named prefix in non-recursive listings (backlog#1042)
On single-disk / consistent deployments a non-recursive scan_dir that finds a/xl.meta classifies `a` as an object and does not descend, so the source never emits the prefix `a/`, dropping the CommonPrefix from delimiter listings even when `a/b` exists. This is the single-disk follow-up to backlog#880 (the multi-disk merge path was fixed in #4563).
Fix: in the scan_dir Ok branch, for a non-recursive listing of a plain object, probe whether `a/` holds a listing entry other than the object itself (new object_prefix_has_sibling_listing_entry, reusing directory_has_visible_listing_entry and skipping the object's own xl.meta and data dirs); if so, additionally schedule the prefix dir `a/`. The upstream fold in from_meta_cache_entries_sorted_infos already emits object `a` as Contents and dir `a/` as a CommonPrefix, so nothing changes there. Leaf objects (the common case) pay a single list_dir and return false immediately.
Verification: new scan_dir unit test (positive: `a` + `a/b` coexist; negative: leaf `c` produces no spurious `c/`); new end-to-end e2e (object `a` + `a/b` with delimiter "/" -> Contents `a` + CommonPrefix `a/`); existing list_objects_duplicates e2e (#1797 / #2439 regressions) stays green; set_disk::read 114 passed.
Note: lib-test compilation depends on #4573 (now merged), which added the allow_inplace_legacy_fallback argument to the codec streaming arity tests after #4560 / backlog#879.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899)
A held distributed write lock had a fixed 30s TTL and was never renewed, so
any operation exceeding it got its per-node lease reclaimed and stolen by a
contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the
#899 design (Phase 2 abort-on-loss is deferred and tracked in code comments).
Phase 0 (server refresh wiring, P0):
- node_service handle_refresh was a no-op stub that parsed args then always
returned success=true. Extract a testable `refresh_lock` free function that
actually delegates to the node's lock backend. Not-found maps to
success=false with no error_info, so RemoteClient::refresh keeps its
Ok(resp.success) semantics and yields a real not_found signal to the
coordinator heartbeat. Without this, client heartbeats were silently no-ops.
Phase 1 (client heartbeat + safe interval + observability):
- DistributedLockGuard spawns a heartbeat that refreshes every per-client lease
on a derived interval; interval is derived without Duration::clamp
(entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic.
- Add LockLostSignal: declare the lock lost when not_found exceeds
entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the
ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers.
- disarm(), release(), and Drop now abort the heartbeat before releasing so no
refresh races the unlock (refresh only extends, never creates, a lease).
- Reclaim path stays behaviorally unchanged but now warns with owner/resource/
lease age and records a metric (#698 scavenger preserved).
- Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default
for RPC back-compat) + builder, and lock lifecycle metrics.
Open questions adopt the design's documented defaults (marked TODO in code):
refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len();
DEFAULT_LOCK_REFRESH_INTERVAL=10s.
Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate),
server refresh delegation (rustfs), and end-to-end survives-past-ttl plus
crashed-owner-reclaim regressions (namespace).
test(e2e): add SHA256 verify-on-write & HEAD-checksum regression for #4341
Issue #4341 reported (on beta.8) that RustFS accepted mismatched S3
SHA256 checksums on PutObject (HTTP 200 instead of 400 BadDigest) and
did not return ChecksumSHA256 on HeadObject with ChecksumMode=ENABLED.
Both behaviors are already correct on current main, but the existing
e2e coverage only asserted the happy path (upload succeeds + GetObject
content matches). It did not encode the issue's two acceptance criteria,
so a future regression would go undetected.
Add two focused e2e tests that assert exactly the issue's contract:
- test_put_object_rejects_mismatched_sha256: a body that does not match
the declared x-amz-checksum-sha256 must be rejected with BadDigest and
must not be stored.
- test_head_object_returns_stored_sha256: after a correct upload,
HeadObject with ChecksumMode=ENABLED must return the stored base64
SHA256 digest.
Both tests pass against a real rustfs server (mismatch -> HTTP 400
BadDigest verified).
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Add crates/e2e_test/src/chaos.rs with in-process fault-injection
primitives for a single-node multi-disk RustFS server: take a disk
offline (rename to <dir>.offline), bring it back online, replace a
disk with a fresh empty directory, corrupt an object's erasure shard
(part.* byte flips, xl.meta untouched), and SIGKILL/restart the server
with the same volumes and port.
Add reliability tests on a 4-disk (EC 2+2) topology, verified via
sha256 manifests recorded at write time:
- degraded read/write with one disk offline (incl. multipart object)
- bitrot read-through with two corrupted shards per object
- fresh-disk replacement healed via admin deep heal after a SIGKILL
restart
Also reuse the shared signed_admin_post helper from chaos.rs in the
heal regression suite instead of a duplicated local copy.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(replication): support insecure https bucket targets
* feat(replication): support custom ca bucket targets
* test(replication): satisfy e2e clippy for TLS helpers
* fix(replication): avoid native root panic for custom trust stores
* test(replication): decouple private IP target test from TLS roots
* test(replication): use target TLS client in private IP unit test
houseme