mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-24 13:16:28 +00:00
Address P2 follow-ups from the 2026-07-10..12 merged-PR review (backlog#1210-1220) (#4783)
* fix(obs): open cleaner compression source with O_NOFOLLOW The compressor opened the source log via File::open, which follows a symlink at the final path component. Between the scanner selecting a regular file and this open, an attacker with write access to the log directory could swap the entry for a symlink (TOCTOU) pointing at, say, /etc/shadow, whose contents would then be copied into an archive. Open the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the temp/archive path already refused symlinks, this closes the source side. Refs rustfs/backlog#1210 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): recompress instead of trusting leftover cleaner archives archive_header_ok only checked the first 2-4 magic bytes before treating an existing .gz/.zst as a completed prior result and letting the caller delete the source log. A file with valid magic but a truncated or forged body passes that check, so an attacker with write access to the log directory (or a crashed prior run) could plant such a stub and make the cleaner delete the real log without ever producing a usable archive — silent audit-data loss. Chosen fix: stop trusting cross-process leftovers entirely and always recompress the source in this pass, rather than fully decoding every leftover to validate it. Full-decode validation would add real CPU cost and decode-bug surface for a rare crash-recovery case; the existing atomic create_new+rename already overwrites whatever sits at the archive path (a planted symlink is replaced, never followed) with a freshly written, fsync'd archive, so a partial/forged leftover can never gate source deletion. This is the lowest-regression option. Refs rustfs/backlog#1211 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): cap memory-gate reservation at cache growth headroom The memory gate subtracts `admitted_since_refresh` from the snapshot's available bytes so a burst arriving faster than the 5 s refresh cannot over-allocate. That counter is GROSS: it only rolls over on the refresh and never rolls back when a fill is later evicted, cancelled, or loses the invalidation race. Under sustained high-throughput churn (net footprint flat and far below `max_capacity`) the raw counter balloons past the memory the cache actually holds, so `effective_available` collapses and the gate reports false memory pressure — skipping the hottest fills with SkippedMemoryPressure until the next 5 s refresh. This only lowers hit rate; it never returns wrong data and self-heals each refresh. Fix direction 1 (minimal regression): cap the reservation deduction at the cache's own growth headroom (`max_capacity - weighted_size()`) instead of letting the unbounded gross counter shrink the system-available budget. The cache can never hold more than `max_capacity`, so a burst adds at most that headroom of real memory before moka evicts to stay bounded (net-zero churn beyond that point) — capping the deduction there keeps the reservation honest without treating gross churn as growth. Chosen over net-accounting (direction 2, releasing bytes on every failure/cancel/eviction path) because that only plugs the leak on failed fills and would not address the core defect: churn of *successful* insert/evict fills over the 5 s window. It also touches only the gate plus one call site rather than every failure path in moka_backend. The cap only ever raises `effective_available`, so real memory pressure (a low snapshot at refresh) still suppresses fills; when the cache is at capacity the headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn. `MokaBackend` now stores `max_capacity` and passes the live headroom into `allows_fill`. Adds targeted gate tests: gross churn far above headroom no longer falsely suppresses, yet the reservation still bounds a burst while the cache can genuinely grow. Refs rustfs/backlog#1212 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): assert native O_DIRECT path runs in uring read test uring_preserves_o_direct_for_eligible_reads only compared bytes through LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the read silently degrades to the buffered StdBackend fallback and the byte check still passes, so the test could go green without the native read_at_direct path ever executing -- a vacuous pass. Add a per-disk native_direct_reads counter on UringBackend, incremented only when pread_uring_direct completes, and rebuild the test to drive a real UringBackend's pread_bytes and assert the counter is non-zero (every eligible read went through the native tier). When io_uring or O_DIRECT is unavailable on the host filesystem (restricted CI runners, tmpfs), the test skips loudly via eprintln instead of asserting a tautology, while still checking byte-correctness on whatever tier served the read. The counter also gives a gray release a positive signal that the O_DIRECT tier is serving reads, not just a fallback count. Refs rustfs/backlog#1213 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads classify_direct_read_error is only reached from the read side: the O_DIRECT open in pread_uring_direct already succeeded (an open-time refusal is handled earlier as DirectOpenError::ODirectRefused). So an EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel accepted for O_DIRECT -- far more likely an alignment bug in the aligned read path than an unsupported filesystem. The old code latched the disk's native path off with only a once-per-disk debug trace, hiding a potential correctness regression behind a silent buffered-read downgrade. Diagnostics only: the fallback behaviour is unchanged (the native path is still latched off and the caller still reads via StdBackend). This adds a rustfs_io_uring_direct_read_einval_total counter and promotes the once-per-disk trace from debug to warn so an operator can see an alignment regression instead of an unexplained latency/CPU shift. Refs rustfs/backlog#1214 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document data-blocks-first default and its tail-latency cost DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay true: deferred-parity is the deliberate, already-rolled-out full-object GET default from backlog#1159/#923. Flipping it back to false in code would silently revert that rollout for every deployment that has not set the env var, so this commit only documents -- no behaviour change. The added notes explain what data-blocks-first does (schedule data shards up front, engage parity lazily on a missing/corrupt data shard), the known trade-off (parity is engaged late, so a slow-but-not-dead data drive raises GET p99 because the faster parity shards are not raced against it until a data shard is declared missing), and the operational rollback switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is intentionally an env override rather than a code default change. No metric was added: the low-risk observability hook for "slow data drive engaged deferred parity" would live at the deferred-stripe engage point, which is out of this file's scope; this change stays documentation-only to avoid touching the hot GET path. Refs rustfs/backlog#1215 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document wide-directory walk stall hazard and tuning list_dir enumerates a whole directory in one os::read_dir call (count = -1), and the walk caller bounds that entire enumeration with the per-read stall budget (default 5s) as if it were a single read. For a wide, flat prefix -- one directory holding millions of immediate children -- a single readdir can exceed the budget on a healthy disk, trip DiskError::Timeout, and surface as a ListObjects 500 quorum failure though the drive is fine (a #2999 sub-class). This is documented, not rewritten: turning the one-shot readdir into a streaming/batched enumeration that refreshes the stall deadline between chunks is an architecture-level change with high regression surface (ordering, the count contract, quorum merge) and belongs in a separate follow-up. The supported mitigation today is operational, so the comments point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS and the high-latency drive-timeout profile, which widen the budget with no code change. Notes were added at list_dir, the scan_dir call site, and get_drive_walkdir_stall_timeout. No behaviour change. Refs rustfs/backlog#1216 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document consumer-peek vs producer-stall coupling In list_path_raw the consumer's peek_timeout is drawn from the same source and same value (walkdir_stall_timeout, default 5s) as the producer-side walk stall budget, but the two measure different things: the producer stall bounds a single drive read, while the consumer peek bounds the gap between two ADJACENT entries arriving from a reader. Because they share a value, the consumer cannot wait meaningfully longer for the next entry than the producer is allowed to spend producing one. Walking a region dense with non-listable internal items can make a HEALTHY drive miss the budget between visible entries; the consumer then declares it stalled and detaches it, dropping a good drive from the merge and capping the "large prefix succeeds" guarantee. Documented, not decoupled: giving the consumer peek an independent, strictly-larger budget would cut these false detaches but equally delays detaching a genuinely dead drive and shifts listing tail-latency semantics, so it wants soak data before changing the default. The comment records the invariant any such follow-up must keep -- consumer peek >= producer stall, never stricter -- so it can never fail a drive before the producer would. No behaviour change. Refs rustfs/backlog#1217 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(io-metrics): add time-based trigger for low-IOPS latency percentiles Percentiles were recomputed only every 128 IOs and seeded to 0, so a low-traffic deployment exported p95/p99 = 0/stale for a long time after startup. Add a 10s wall-clock trigger alongside the count throttle so the first recompute can fire before 128 samples accrue. Hot-path per-op mean update is unchanged. Refs rustfs/backlog#1218 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): cover codec-streaming parity under fault injection and NoSuchKey The codec-streaming compat A/B previously ran only against a healthy 4-disk EC set with successful full GETs: the DiskFaultHarness was constructed but never faulted, the error path was untested, and the range assertion silently compared legacy-vs-legacy (ranges always fall back to the duplex path), overstating what it proved. Add two genuinely-failable scenarios reusing the existing harness and fixtures: - Parity reconstruction A/B: take one data disk offline and re-run the full object matrix on both phases while the EC 2+2 set rebuilds each large object from the surviving shards. Assert codec == legacy byte-for-byte (sha256) and header-for-header, and assert the codec phase served the reconstructed objects with zero duplex-pipe fallback (the reader gate is drive-health-independent, so the codec fast path is really exercised through reconstruction). - NoSuchKey negative path: compare the HTTP status + S3 error code of a missing-key GET across the legacy and codec phases and require them to be identical (404/NoSuchKey), guarding against the codec env perturbing the error path. Also clarify the range-phase comment so it is not misread as codec-range correctness coverage: both sides are served by the same legacy range path, so the assertion only proves ranges keep working and keep falling back to legacy with the gates open. Verified: cargo check/--no-run pass and the test passes locally (1 passed; dup_codec=0 confirms the codec path ran). Refs rustfs/backlog#1219 Co-Authored-By: heihutu <heihutu@gmail.com> * ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback The uring-integration leg ran on the runner's default TMPDIR, which may sit on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct path silently latches off to the aligned StdBackend fallback. Mount a dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep (bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are actually covered rather than validated only by signature diffing. Refs rustfs/backlog#1220 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
@@ -39,6 +39,20 @@
|
||||
//! **zero** — proving the codec path actually ran rather than silently falling
|
||||
//! back to the very path it is being compared against.
|
||||
//!
|
||||
//! Beyond the all-healthy happy path, the suite also drives the codec/legacy
|
||||
//! A/B under two conditions the `DiskFaultHarness` makes reachable:
|
||||
//!
|
||||
//! * Parity reconstruction: one data disk is taken offline
|
||||
//! (`take_disk_offline`) and the SAME object matrix is GET both ways while
|
||||
//! the EC 2+2 set rebuilds each large object from the surviving shards. The
|
||||
//! codec-streaming reader gate never inspects drive health, so the codec
|
||||
//! fast path is exercised end-to-end through reconstruction; the test
|
||||
//! asserts byte- and header-equality vs the legacy path AND that the codec
|
||||
//! phase never fell back to a duplex pipe while reconstructing.
|
||||
//! * Missing object: a GET for an absent key is compared across both phases
|
||||
//! to prove the error semantics (HTTP status + S3 error code) are identical
|
||||
//! — the codec env must not perturb the NoSuchKey negative path.
|
||||
//!
|
||||
//! Topology: single-node 4-disk EC set (default 2 data + 2 parity) via the
|
||||
//! in-process `DiskFaultHarness` (see `chaos.rs`). Small objects are inlined
|
||||
//! into `xl.meta` (served identically on both paths); large objects span one or
|
||||
@@ -49,6 +63,7 @@ mod tests {
|
||||
use crate::chaos::DiskFaultHarness;
|
||||
use crate::common::init_logging;
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
|
||||
use serial_test::serial;
|
||||
@@ -63,6 +78,9 @@ mod tests {
|
||||
const MIB: usize = 1024 * 1024;
|
||||
const BUCKET: &str = "codec-streaming-compat";
|
||||
const CONTENT_TYPE: &str = "application/x-rustfs-compat";
|
||||
/// A key that is never uploaded — used to compare the NoSuchKey negative
|
||||
/// path across the legacy and codec phases.
|
||||
const MISSING_KEY: &str = "does-not-exist/ghost.bin";
|
||||
|
||||
/// Marker the legacy duplex GET path logs once per full-object read
|
||||
/// (`crates/ecstore/src/set_disk/ops/object.rs`). Its presence/absence in a
|
||||
@@ -144,6 +162,23 @@ mod tests {
|
||||
})
|
||||
}
|
||||
|
||||
/// GET a key expected to be absent, projecting the wire-visible error
|
||||
/// semantics — HTTP status code plus the S3 error code — so the legacy and
|
||||
/// codec phases can be asserted to reject a missing object identically. A
|
||||
/// missing object fails during metadata resolution, before the reader-path
|
||||
/// gate is consulted, so both phases MUST agree; a divergence here would
|
||||
/// mean the codec env perturbed the negative path.
|
||||
async fn missing_key_semantics(client: &Client, key: &str) -> Result<(u16, String), Box<dyn Error + Send + Sync>> {
|
||||
match client.get_object().bucket(BUCKET).key(key).send().await {
|
||||
Ok(_) => Err(format!("GET {key} unexpectedly succeeded; expected a NoSuchKey error").into()),
|
||||
Err(err) => {
|
||||
let status = err.raw_response().map(|r| r.status().as_u16()).unwrap_or(0);
|
||||
let code = err.as_service_error().and_then(|e| e.code()).unwrap_or("<none>").to_string();
|
||||
Ok((status, code))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Env that opens every codec-streaming gate to 100% for the codec phase.
|
||||
/// Mirrors the exact knobs `get_codec_streaming_reader_gate` inspects
|
||||
/// (`crates/ecstore/src/set_disk/mod.rs`).
|
||||
@@ -335,6 +370,37 @@ mod tests {
|
||||
"baseline phase should have used the legacy duplex path for the {num_large} large objects, but only saw {dup_base} duplex markers in {base_log}"
|
||||
);
|
||||
|
||||
// Legacy negative path: a GET for an absent key must fail with a
|
||||
// well-formed NoSuchKey (404). Captured now so Phase B can prove the
|
||||
// codec env returns the identical error semantics.
|
||||
let legacy_missing = missing_key_semantics(&client, MISSING_KEY).await?;
|
||||
assert_eq!(
|
||||
legacy_missing,
|
||||
(404, "NoSuchKey".to_string()),
|
||||
"legacy GET of a missing key should be 404/NoSuchKey, got {legacy_missing:?}"
|
||||
);
|
||||
|
||||
// ---- Phase A degraded: pull one data disk, force parity reconstruction ----
|
||||
// With disk0 offline the EC 2+2 set must rebuild every large object's
|
||||
// data from the surviving data+parity shards. Record the legacy-duplex
|
||||
// bytes and headers produced under reconstruction so Phase B can prove
|
||||
// the codec path reconstructs the same bytes and headers. The duplex
|
||||
// marker snapshot (`dup_base`) is already taken, so these extra reads do
|
||||
// not affect the path-confirmation assertion above.
|
||||
harness.take_disk_offline(0)?;
|
||||
let mut baseline_degraded: BTreeMap<String, GetView> = BTreeMap::new();
|
||||
for (shape, data) in plain {
|
||||
let view = get_full(&client, shape.key).await?;
|
||||
assert_eq!(view.sha256, sha256_hex(data), "degraded baseline body mismatch for {}", shape.key);
|
||||
assert_eq!(view.len, data.len(), "degraded baseline length mismatch for {}", shape.key);
|
||||
baseline_degraded.insert(shape.key.to_string(), view);
|
||||
}
|
||||
let mp_view = get_full(&client, multipart_key).await?;
|
||||
assert_eq!(mp_view.sha256, sha256_hex(&multipart_body), "degraded baseline multipart body mismatch");
|
||||
baseline_degraded.insert(multipart_key.to_string(), mp_view);
|
||||
// Restore the disk so Phase B restarts from a clean, complete disk set.
|
||||
harness.bring_disk_online(0)?;
|
||||
|
||||
// ---- Phase B: codec streaming (gates opened) ----
|
||||
harness.kill_server();
|
||||
for (k, v) in codec_env() {
|
||||
@@ -355,6 +421,14 @@ mod tests {
|
||||
assert_eq!(mp_view.sha256, sha256_hex(&multipart_body), "codec multipart body mismatch");
|
||||
codec.insert(multipart_key.to_string(), mp_view);
|
||||
|
||||
// Negative-path equivalence: the codec env must return the exact same
|
||||
// status + error code as the legacy phase for a missing key.
|
||||
let codec_missing = missing_key_semantics(&client, MISSING_KEY).await?;
|
||||
assert_eq!(
|
||||
codec_missing, legacy_missing,
|
||||
"NoSuchKey error semantics diverged between codec and legacy phases: codec={codec_missing:?} legacy={legacy_missing:?}"
|
||||
);
|
||||
|
||||
// Snapshot the codec-phase duplex count BEFORE issuing the ranged GET
|
||||
// (range falls back to the duplex path by design and would pollute it).
|
||||
sleep(Duration::from_millis(300)).await;
|
||||
@@ -382,18 +456,75 @@ mod tests {
|
||||
"codec phase created {dup_codec} duplex pipe(s) for full GETs; the codec-streaming fast path was not exercised (see {codec_log})"
|
||||
);
|
||||
|
||||
// Range GET still served correctly while codec streaming is enabled
|
||||
// (ranges fall back to the legacy path by gate design).
|
||||
// Range GET while codec streaming is enabled. NOTE ON COVERAGE: the
|
||||
// reader gate unconditionally routes every ranged request back to the
|
||||
// legacy duplex path (`GetCodecStreamingFallbackReason::Range`), so both
|
||||
// `baseline_range` and `codec_range` are produced by the SAME legacy
|
||||
// path. This assertion therefore only verifies that ranged GETs keep
|
||||
// working (and keep falling back to legacy) with the codec gates open —
|
||||
// it does NOT exercise or validate a codec-streaming range reader, which
|
||||
// does not exist. It must not be read as codec range-correctness
|
||||
// coverage.
|
||||
let codec_range = get_range(&client, "large-3mib", range_spec).await?;
|
||||
assert_eq!(
|
||||
codec_range.sha256, baseline_range.sha256,
|
||||
"ranged GET body diverged under codec streaming"
|
||||
"ranged GET body diverged with codec streaming enabled (both served by the legacy range path)"
|
||||
);
|
||||
assert_eq!(
|
||||
codec_range.len, baseline_range.len,
|
||||
"ranged GET length diverged with codec streaming enabled"
|
||||
);
|
||||
|
||||
// ---- Phase B degraded: the same reconstruction, now on the codec path ----
|
||||
// Re-run the reconstruction A/B with the codec-streaming gates still
|
||||
// open. The reader gate decision is independent of drive health (it
|
||||
// never inspects disk state), so the codec fast path is exercised
|
||||
// end-to-end while the EC set rebuilds each large object from the
|
||||
// surviving shards — this is a real codec-vs-legacy reconstruction test,
|
||||
// not legacy-vs-legacy. Snapshot the duplex count first (the range GET
|
||||
// above already used the duplex path) so we can measure only the markers
|
||||
// these degraded codec GETs add.
|
||||
let dup_codec_before_degraded = count_marker(&codec_log, DUPLEX_MARKER);
|
||||
harness.take_disk_offline(0)?;
|
||||
let mut codec_degraded: BTreeMap<String, GetView> = BTreeMap::new();
|
||||
for (shape, data) in plain {
|
||||
let view = get_full(&client, shape.key).await?;
|
||||
assert_eq!(view.sha256, sha256_hex(data), "degraded codec body mismatch for {}", shape.key);
|
||||
codec_degraded.insert(shape.key.to_string(), view);
|
||||
}
|
||||
let mp_view = get_full(&client, multipart_key).await?;
|
||||
assert_eq!(mp_view.sha256, sha256_hex(&multipart_body), "degraded codec multipart body mismatch");
|
||||
codec_degraded.insert(multipart_key.to_string(), mp_view);
|
||||
harness.bring_disk_online(0)?;
|
||||
|
||||
// A/B under parity reconstruction: codec == legacy, byte-for-byte and
|
||||
// header-for-header, for every object in the matrix.
|
||||
for key in baseline_degraded.keys() {
|
||||
let b = &baseline_degraded[key];
|
||||
let c = &codec_degraded[key];
|
||||
assert_eq!(c.sha256, b.sha256, "degraded body hash diverged for {key} (parity reconstruction)");
|
||||
assert_eq!(c.len, b.len, "degraded body length diverged for {key} (parity reconstruction)");
|
||||
assert_eq!(
|
||||
c.headers, b.headers,
|
||||
"degraded response headers diverged for {key} (parity reconstruction)\nbaseline={:#?}\ncodec={:#?}",
|
||||
b.headers, c.headers
|
||||
);
|
||||
}
|
||||
|
||||
// Path confirmation under reconstruction: the codec fast path must have
|
||||
// served the reconstructed large objects without ever falling back to
|
||||
// the legacy duplex pipe. Without this, the equivalence above could be
|
||||
// legacy-vs-legacy and prove nothing about codec reconstruction.
|
||||
sleep(Duration::from_millis(300)).await;
|
||||
let dup_codec_degraded = count_marker(&codec_log, DUPLEX_MARKER).saturating_sub(dup_codec_before_degraded);
|
||||
assert_eq!(
|
||||
dup_codec_degraded, 0,
|
||||
"codec phase created {dup_codec_degraded} duplex pipe(s) while reconstructing large objects with disk0 offline; the codec fast path was not exercised under degraded reads (see {codec_log})"
|
||||
);
|
||||
assert_eq!(codec_range.len, baseline_range.len, "ranged GET length diverged under codec streaming");
|
||||
|
||||
info!(
|
||||
objects = baseline.len(),
|
||||
"codec streaming produced byte- and header-identical GET responses vs legacy duplex"
|
||||
"codec streaming produced byte- and header-identical GET responses vs legacy duplex (healthy + parity-reconstructed + NoSuchKey)"
|
||||
);
|
||||
|
||||
// Best-effort cleanup of the capture logs.
|
||||
|
||||
Reference in New Issue
Block a user