chore(ci): guard against committed planning docs; remove re-added ones
PR #4771 removed the docs/superpowers planning-doc archive and added a rule,
but #4765 force-added two more (git add -f bypasses .gitignore):
docs/superpowers/plans/2026-07-12-observability-single-writer.md and
docs/superpowers/specs/2026-07-12-observability-single-writer-design.md — an
agentic implementation plan and its design spec. Delete both.
Close the enforcement gap so this cannot recur:
- New scripts/check_no_planning_docs.sh fails if anything is tracked under
docs/superpowers/, regardless of how it was added.
- Wire it into make pre-commit/pre-pr/dev-check.
- Run it in CI: ci.yml for code/mixed PRs, and ci-docs-only.yml (which was a
green stub) so docs-only PRs can no longer slip a planning doc past the
required 'Test and Lint' check.
- Document the guard in AGENTS.md and the arch-checks skill.
* fix(obs/cleaner): fsync archive and log dir before deleting source logs
OLC-01: the compression path flushed the BufWriter but never synced the
archive data or the parent directory before renaming, and the source was
then unlinked with no durability barrier. A crash after rename but before
the page cache reached disk could leave a truncated/zero-length archive
while the source was already gone — permanent log/audit data loss. Because
the archive is always renamed to a brand-new name (guaranteed by the
existing exists() guard), ext4 auto_da_alloc does not mask this.
Hand the underlying File back from the writer closure, sync_all() it before
rename, fsync the parent directory, and fsync the log directory after the
unlinks so a delete cannot be reordered ahead of the archive it justified.
Guard the temp file with an RAII cleanup so an early return or panic cannot
leak a *.tmp orphan.
Ref: rustfs/backlog#1194 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): create compression temp file with O_EXCL and O_NOFOLLOW
OLC-03: the temp archive was opened with File::create at a predictable
`<source>.gz.tmp` path with no O_EXCL/O_NOFOLLOW, so an actor with write
access to the log directory could pre-plant that path as a symlink and have
the compressor follow it — truncating and overwriting an arbitrary external
file, then chmod-ing it to the source log's mode. This mirrors the symlink
refusal already enforced on the deletion path (secure_delete).
Route temp creation through create_tmp_archive(), which uses create_new
(O_CREAT|O_EXCL) to refuse a pre-existing entry and, on Unix, O_NOFOLLOW to
refuse a symlink at the final path component.
Ref: rustfs/backlog#1196 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): create compression temp file with restrictive mode
OLC-06: File::create left the temp archive world-readable (0644 & ~umask)
for the entire duration of compressing a large log, exposing the full
plaintext of a possibly-0600 audit log on shared hosts until the mode was
copied only after the write completed. Pass the source mode into
create_tmp_archive and open the temp file with it (default 0600) so it is
restrictive from creation; the post-write chmod still tightens/matches the
source mode exactly.
Ref: rustfs/backlog#1199 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): validate existing archive before skipping recompression
OLC-02: the idempotency guard used Path::exists() (which follows symlinks)
and trusted whatever it found, then the caller deleted the source. A planted
`<archive>.gz` symlink, or a zero-length/truncated archive left by a crashed
run (OLC-01), would green-light deleting the source with no valid backup —
data loss / log destruction.
Replace exists() with symlink_metadata (no follow) and only treat the entry
as a completed prior result when it is a regular, non-empty file whose header
matches the codec magic (gzip 1f 8b / zstd 28 b5 2f fd). Anything else falls
through to recompression, whose atomic create_new+rename replaces the bad
entry (a symlink is replaced, never followed or deleted through).
Ref: rustfs/backlog#1195 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): stop dry-run overstating reclaimed bytes for compression
OLC-08: in dry-run, compress_with_writer returned output_bytes = 0, so
projected_freed_bytes = input and delete_files reported the full input as
freed. A real run keeps the archive on disk (freed = input - archive), so
dry-run overstated reclaim by the whole archive footprint. Estimate the
archive with a deliberately conservative ratio so the projection never
exceeds what a real run reclaims.
Ref: rustfs/backlog#1201 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): make freed-byte accounting resilient; document steal metric
OLC-12: input/output byte sizes were read via metadata().unwrap_or(0), which
silently reports 0 on failure and skews freed-byte metrics (input - 0 = full
input, overstating reclaim). Use the copy() byte count as the authoritative
input size and, when the archive metadata read fails, conservatively assume
no savings instead of 0. Also document that the steal_success_rate counts
only victim steals (batch = one success), so it reads as a relative
rebalancing signal, not absolute task acquisition.
Ref: rustfs/backlog#1205 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): preserve level-0 semantics, allow zstd 22, log effective levels
OLC-09: build() and the codec calls clamped gzip/zstd levels to [1,9]/[1,21],
silently rewriting gzip level 0 (store) and zstd level 0 (codec default) to 1
and blocking the legal zstd maximum of 22. Clamp to [0,9]/[0,22] so those
meanings survive, and echo the effective (post-clamp) levels in the startup
log via new effective_gzip_level()/effective_zstd_level() getters so the log
matches what actually runs.
Ref: rustfs/backlog#1202 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): bound compressed archives by byte cap; warn on retention=0
OLC-04: archive expiry was gated on compressed_file_retention_days > 0, so
retention=0 disabled it entirely while compression kept producing archives,
and max_total_size_bytes only ever bounded uncompressed logs — unbounded disk
growth. Replace select_expired_compressed with select_archives_to_delete,
which applies age expiry (when retention is on) and, regardless of retention,
trims the oldest archives until the set fits under max_total_size_bytes. Also
warn at startup when compression is on with retention=0 so the "keep forever"
semantics are not mistaken for "delete immediately".
Ref: rustfs/backlog#1197 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn on invalid exclude glob instead of dropping silently
OLC-05: build() dropped unparseable exclude globs via filter_map(...ok()), so
a typo (or a literal comma splitting a char-class in the config string) turned
"protect this file" into "delete this file" with no signal. Log a warning per
rejected pattern with the raw string and parse error so the misconfiguration
is visible.
Ref: rustfs/backlog#1198 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(obs/cleaner): backoff idle workers, cap worker count, lower small-host floor
OLC-11: the work-stealing loop re-spun on Steal::Retry with no yield and used
yield_now on the empty path, burning CPU during redistribution windows;
worker_count had no upper bound so a mis-set parallel_workers over a directory
of thousands of logs could spawn thousands of threads; and
default_parallel_workers forced >=4 workers even on 1-2 vCPU hosts. Use
crossbeam_utils::Backoff (spin->yield, reset on work) on the idle paths, clamp
worker_count to MAX_PARALLEL_COMPRESS_WORKERS, and lower the default floor to 1.
Ref: rustfs/backlog#1204 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn when active-file guard is disabled by empty filename
OLC-13 (defense-in-depth): the scanner protects the live log purely by exact
filename equality against active_filename. An empty active_filename silently
disables that protection, so a non-empty file_pattern could make the live log
a deletion candidate via the public builder. Warn in build() when that unsafe
combination is configured. The audit's "never delete the newest match"
structural guard is intentionally not implemented: it would conflict with the
legitimate keep_files=0 semantics (purge all rotated logs). The naming
contract is instead locked by regression tests (OLC-14).
Ref: rustfs/backlog#1206 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn on unknown algorithm/match_mode, echo match_mode
OLC-07: from_config_str silently fell back to defaults for unrecognized
compression algorithm and match mode (any non-"prefix" value became Suffix),
hiding operator typos like "prefixx" that could make the cleaner match no
rotated logs. Warn on a non-empty unrecognized value in both parsers, and
echo the resolved match_mode in the startup log alongside the algorithm.
Ref: rustfs/backlog#1200 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): derive orphan .tmp suffixes and exempt them from min age
OLC-10: orphan `*.gz.tmp`/`*.zst.tmp` cleanup was gated by min_file_age_seconds
(default 3600), so crash-left orphans lingered up to an hour, and the tmp
suffix list was hardcoded rather than derived from compressed_suffixes() — a
new codec would leave `*.<ext>.tmp` orphans the scanner never recognizes.
Derive the temp suffix from CompressionAlgorithm::compressed_suffixes(), and
gate orphan removal on a small fixed grace window instead of min_file_age
(orphans are never live-written after the rename that would promote them).
Ref: rustfs/backlog#1203 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs/cleaner): cover symlink, archive expiry, idempotency, and edge cases
OLC-14: add regression tests for the previously-untested safety/correctness
branches — symlink rejection (external target never deleted), archive age
expiry vs fresh retention, archive byte-cap trim with retention disabled,
gz/zst classification, max_single_file_size selection, min_age protecting a
fresh non-empty log, active-file exclusion when the active name also matches
the pattern, invalid exclude glob not aborting build, dry-run + compression
creating no archive, gzip round-trip validity, and the idempotent-archive
branch trusting a valid prior archive.
Ref: rustfs/backlog#1207 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(obs/cleaner): apply rustfmt and collapse nested if (clippy)
Formatting-only cleanup over the audit fix series: rustfmt normalization of the
multi-line expressions introduced in compress.rs/core.rs, plus collapsing the
delete_files directory-fsync into a single let-chain to satisfy
clippy::collapsible_if. No behavior change.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(obs/cleaner): collapse redundant source stats in compression path
The per-issue fixes to compress_with_writer accumulated three metadata()
syscalls on the source in the real compression path: an input_bytes read that
was immediately shadowed by the copied byte count (a dead read), a source_mode
read (OLC-06), and the pre-OLC-06 post-write chmod re-reading the same mode.
Collapse to a single fd-based read — move the dry-run input_bytes read into
the dry-run branch, read source_mode from the already-open fd (no path stat,
no TOCTOU), and reuse it for the post-write chmod. Behavior is unchanged
(same inode's mode, written for input size); 3 source stats -> 1.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(obs/cleaner): fix typo flagged by CI (mis-set -> misconfigured)
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* Change Rust toolchain channel to stable
Signed-off-by: houseme <housemecn@gmail.com>
* style: apply clippy --fix and cargo fix lint suggestions
Run `cargo clippy --fix --all-targets --all-features` and
`cargo fix --lib --all-targets` across the workspace, then resolve the
remaining warnings by hand:
- collapse needless borrows in `format!` args, prefer `?` over explicit
early returns, and use `.values()` / `.flatten()` iterator adapters
- rewrite the `Md5` scan loop via `manual_flatten` and re-indent the
`select!` macro body (rustfmt skips macro interiors)
- annotate the intentional dead-code `Md5` inherent methods (constructed
only by the test factory) with `#[allow(dead_code)]`
Behavior is unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): prevent rolling log stdout aliasing
* fix(ecstore): bound hot-path tracing payloads
* test(logging): guard service and disk log invariants
* fix(obs): silence useless_conversion on st_dev for Linux clippy
rustix's Stat.st_dev is u64 on Linux/glibc, making u64::try_from a no-op
that trips clippy::useless_conversion under -D warnings. The conversion is
still needed on macOS/BSD where st_dev is a signed dev_t, so suppress the
lint on that line rather than dropping the portable fallible conversion.
* fix(logging): keep stdout sink validation portable (#4769)
* fix(obs): keep stdout device conversion portable
* docs(logging): update single-writer plan status
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
The s3-tests lifecycle expiration cases (test_lifecycle_expiration,
test_lifecyclev2_expiration, test_lifecycle_deletemarker_expiration)
flaked with counts stalling one plateau behind (e.g. `assert 6 == 4`).
Root cause: a compacted directory is only re-descended -- and its
objects re-evaluated against ILM rules -- once every
DATA_USAGE_UPDATE_DIR_CYCLES scanner cycles (default 16). At the lane's
accelerated RUSTFS_SCANNER_CYCLE=2 that is ~32s between evaluations, so
a Days=1 object due at debug_day (10s) is not actually expired until the
next ~32s boundary (~42s) -- just past the test's 4*lc_interval (40s)
poll window, so the list still returns the pre-expiry count.
Set RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 for this debug-only lane so
compacted directories are re-descended every cycle and ILM fires within
~2s of the due time, comfortably inside the poll window. This does not
touch the 4*lc_interval < 5*debug_day plateau invariant.
docs: remove agent-generated planning docs, forbid committing them
Delete one-shot planning/progress artifacts that were checked into the tree:
the 14 superpowers plan/tracker docs under docs/superpowers/plans/, plus
issue-scoped implementation plans, optimization conclusions, and dated
benchmark-result snapshots under docs/ (issue-4003 ListObjectsV2 plans,
get-small-file conclusion, issue824/issue829 benchmark results, issue-713
>1GiB GET baseline summary and ops guide).
Codify the rule so they do not come back:
- .gitignore drops the docs/superpowers whitelist, so anything new under
docs/ stays ignored unless force-added.
- AGENTS.md gains an explicit 'do not commit planning-type documents' rule
scoping version control to the durable architecture/operations/testing sets.
- docs/architecture/README.md, overview.md, arch-checks SKILL.md, and
check_doc_paths.sh drop their references to the removed archive.
The scheduled e2e-s3tests sweep failed at "Wait for RustFS ready" in both
topologies because the server never started (issue #4762).
Two independent startup faults, both surfaced now that the local
physical-disk-independence guard is enforced:
- single: RUSTFS_VOLUMES=/data/rustfs{0...3} all live on one physical
device on the runner, so the guard aborts startup. Set the
CI-sanctioned RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true (what the guard's own
error message and the e2e tests already use).
- multi: the entrypoint's process_data_volumes skipped every non-absolute
entry, so the distributed URL form
"http://rustfs{1...4}:9000/data/rustfs{0...3}" never created
/data/rustfs0..3. LocalDisk init then aborts with VolumeNotFound because
resolve_local_disk_root no longer auto-creates the disk root. This also
broke the shipped .docker/compose/docker-compose.cluster.yaml for real
distributed docker deployments.
entrypoint.sh now:
1. Expands multiple {N...M} ranges per token (the URL form carries two).
The previous single-pass expander collapsed it to "http://rustfs1"
and dropped the disk path; it now re-scans until no ranges remain,
operating on only the first brace to keep multi-range tokens intact.
2. Creates the local filesystem path for URL-form endpoints (stripping
scheme://host:port) without appending them as CLI args — rustfs reads
the distributed form from RUSTFS_VOLUMES directly.
Absolute-path and default (/data) inputs expand byte-identically to before.
The multi compose also gets the CI disk-check bypass, since the four disks
share one device inside each node's container.
fix(cache): drain pending removals until entry_count reaches zero in clear()
The previous clear() implementation used a single run_pending_tasks()
call after invalidate_all(), which was insufficient under concurrent
fill pressure — moka processes invalidations lazily in batches, so
entries can linger after a single maintenance pass.
Replace the fixed single call with a drain loop (up to 256 rounds) that
calls run_pending_tasks() and yields between iterations until
entry_count() reaches zero. This ensures the concurrency storm test
(moka_backend_concurrency_storm_leaves_no_leaked_state) passes reliably.
The earlier fix (#4759) added a pre-invalidate_all drain and an 8-pass
loop but reordered operations in a way that introduced a new race. This
commit keeps the original invalidate_all-first ordering and only adds
the drain loop after the initial run_pending_tasks() call.
The GET body cache exports 11 `rustfs_object_data_cache_*` metrics but no
bundled dashboard visualized them. This adds one so operators can see the
cache's behaviour without hand-writing PromQL.
New `grafana-object-data-cache.json` (auto-provisioned from the dashboards
directory, `${DS_PROMETHEUS}`, schema 38) with 19 panels covering every metric:
hit ratio and lookup outcomes, plan decisions and cacheable ratio, fill
outcomes and fill-duration quantiles, hit-vs-fill byte throughput, entries and
weighted bytes vs capacity, in-flight fills, memory-pressure skips,
invalidations by reason/outcome, and size-class breakdowns. PromQL matches each
metric's type — rate() for counters, histogram_quantile() for the fill-duration
histogram, direct reads for gauges.
The observability README (EN + ZH) dashboard table gains a matching row.
Refs: backlog#1107
Co-authored-by: heihutu <heihutu@gmail.com>
Six doc-comments on `OtelConfig` fields stated defaults that disagreed
with the constants the runtime actually applies in
`extract_otel_config_from_env`. Correct them to match:
- profiling_export_enabled: false -> true (DEFAULT_OBS_PROFILING_EXPORT_ENABLED)
- sample_ratio: 0.1 -> 1.0 (SAMPLE_RATIO)
- meter_interval: 15 -> 30 (METER_INTERVAL)
- logger_level: info -> error (DEFAULT_LOG_LEVEL)
- log_rotation_time: daily -> hourly (DEFAULT_LOG_ROTATION_TIME), and
document the full minutely/hourly/daily set plus the daily fallback
- log_cleanup_interval_seconds: 21600 -> 1800 (DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS)
Doc-comment only; no behavior change.
fix(test): use canonicalized disk root for file_sync_probe in rename_data test
On macOS, tempfile::tempdir returns /var/folders/... while LocalDisk
resolves the root to /private/var/folders/... via dunce::canonicalize.
The file_sync_probe::enter() path check uses starts_with(), so passing
the non-canonical tempdir path caused the probe to never activate,
making wait_for_active() hang indefinitely.
Use disk.root (already canonicalized) for the probe instead.
`report_metrics` runs on every metrics collection cycle and did three things it
did not need to, for every metric, every cycle:
- interned `metric.name`/`metric.help` through a `Mutex<HashMap>` even when the
`Cow` was already `Borrowed(&'static str)` (the common case for statically
named metrics);
- re-ran `describe_*!` (which re-locks the recorder's metadata map) although the
metadata never changes;
- allocated a fresh `Vec<(String, String)>`, cloning every label key and value,
even though `metric.labels` is already `[(&'static str, Cow<'static, str>)]`.
Now: names/help resolve to `&'static str` without touching the intern cache when
already borrowed; each metric is described once (tracked in a `HashSet`); and the
recorder is fed `&metric.labels` directly, removing the per-cycle label clone.
No metric names, help, label keys/values, or emitted values change. Verified by
building rustfs-obs and running the report unit tests (the `metrics` macro
accepts the borrowed label slice directly).
Addresses rustfs/backlog#1185 (P3, report path).
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1183, staged rollout step. Simplify enabling the zero-duplex GET
codec-streaming fast path to a single switch — RUSTFS_GET_CODEC_STREAMING_ROLLOUT
(default "off") — now that body/header parity is proven (parity e2e net + bench
A/B on backlog#1183).
- Add a clean production rollout token "on" (aliases "full"/"production"); the
legacy "internal"/"benchmark" tokens remain accepted.
- Flip DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE and the two ..._COMPAT_CONFIRMED
defaults to true. They are retained as emergency kill-switches (set any to
false to force the fast path off) but no longer gate enablement — the rollout
switch does.
No production behavior change: with no env set the rollout switch defaults to
"off", so GET stays on the legacy duplex path exactly as before. Flipping the
hard default to on is deferred to a follow-up after a production soak.
Note (intentional semantics change): with the rollout switch opted in
("on"/"internal"/"benchmark"), codec streaming now activates without also
setting the two ..._COMPAT_CONFIRMED vars — compatibility is confirmed, so those
confirmations are baked in.
Co-authored-by: heihutu <heihutu@gmail.com>
Bound large shard-directory syncs per disk and process while preserving small-directory and durability behavior.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The `metrics` macros re-run the recorder's `register_*` on every emission — a
`RwLock` read, a name-key hash, and an `Arc` clone — even for a metric that
never varies its key. For the hot, label-less recorders on the per-IO path
(`record_data_transfer`, `record_io_latency`, `record_io_latency_p95/p99`,
`record_io_queue_congestion`) that lookup is pure overhead once observability is
on.
Add `counter_increment_cached!` / `gauge_set_cached!` / `histogram_record_cached!`
that resolve the handle once via `LazyLock` in production and reuse it. Under
`cfg(test)` they re-resolve on every call, because the `metrics` crate resolves
against a thread-local recorder that `with_local_recorder` swaps per test — a
process-global cached handle would bind to whichever recorder was active first
and break test capture. The macros only wrap FIXED (label-less) keys, and the
`metrics_enabled()` gate still short-circuits before any emission when disabled.
Verified: the only callers of these functions are the collector (io-metrics'
own cfg(test) tests, which re-resolve) and production code; no cross-crate test
captures them. rustfs-io-metrics builds on both cfg paths, 147 unit + 4 doctests
pass, clippy clean.
Addresses rustfs/backlog#1185 (P3, per-emission handle caching).
Co-authored-by: heihutu <heihutu@gmail.com>
`record_uring_fallback` is called at multiple sites in the io_uring read
path whenever a read falls back to `StdBackend`. It formatted
`self.root.display().to_string()` on every call, heap-allocating a
`String` from a `Path` that never changes after construction — pure
per-read waste when io_uring is degraded.
Cache the label once in `UringBackend::try_new` as a `String` field
(`root_label`) and clone it per emission. The metric name and the
`"root"` label value are unchanged; only the redundant `Path` formatting
is removed. The clone is a single alloc of an already-short string.
Refs: rustfs/backlog#1185
Co-authored-by: heihutu <heihutu@gmail.com>
`cluster_peer_should_bypass` is called before every internode RPC when the
offline-bypass feature is enabled (remote disk `get_client`, remote locker), and
it took a single process-global `Mutex<HashMap>` even for the overwhelmingly
common case of an online or unknown peer, which is read-only. That serialized
all concurrent internode client acquisitions on one lock.
Switch `CLUSTER_PEER_HEALTH` to a `RwLock`:
- the hot check takes a shared read lock and returns immediately for unknown or
online peers, so concurrent RPCs no longer serialize;
- only an offline peer (rare) drops to the write lock to record a re-probe, with
a re-fetch/re-check because the state can flip back online between releasing the
read lock and taking the write lock;
- the write paths (dial reachable/unreachable) and the read-only
`cluster_peer_is_offline` move to `write()`/`read()` accordingly.
Behavior is unchanged on every path (online/unknown -> not bypassed; offline ->
bypass with one re-probe per interval); `Instant::now()` now runs only on the
offline path. Poison recovery is preserved. All internode unit tests pass.
Addresses rustfs/backlog#1185 (P2).
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1183 tracks flipping the default GET data path from the legacy
tokio::io::duplex double-copy to the zero-duplex codec-streaming fast path.
That flip is gated behind RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED /
..._HEADER_COMPAT_CONFIRMED, which need empirical evidence the two paths are
byte- and header-identical.
Add an e2e regression net that runs the same object matrix twice against the
same on-disk EC shards, changing only the codec-streaming env gates: phase A
(default) takes the legacy duplex path, phase B (gates opened) takes codec
streaming. It asserts byte-for-byte (sha256) and header-for-header equality
across inline / small / multi-block (1.5M/3M/5M+) / multipart objects, plus a
ranged GET (which falls back to legacy by gate design). Path confirmation is
not assumed: the legacy path logs "Created duplex pipe ..." per full GET, so
the test counts that marker per phase and asserts the codec phase created zero
duplex pipes, proving the fast path actually ran rather than silently falling
back to the path it is compared against.
To capture the child server's logs for that assertion, add an optional
RustFSTestEnvironment.capture_log_path (default None = inherit stdio,
backward compatible) that redirects the spawned server's stdout+stderr to a file.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(ecstore): extract shared MockWarmBackend into a test-util feature (backlog#1148 ilm-6)
The tier/lifecycle integration tests carried two byte-for-byte copies of an
in-memory WarmBackend mock — one in crates/scanner/tests and one in
rustfs/src/app — plus duplicated register_mock_tier and polling helpers. Both
implemented the same ecstore WarmBackend trait.
Consolidate them into ecstore behind a new `test-util` feature, exposed via the
`rustfs_ecstore::api::tier::test_util` facade:
- MockWarmBackend: in-memory WarmBackend with an operation log (for ordering
assertions such as "local delete precedes remote remove") and fault injection
(FaultConfig): unreachable, HTTP 5xx, credential rejection, injected latency,
plus external_remove to simulate an out-of-band remote deletion.
- register_mock_tier / register_mock_tier_backend: register the mock into any
TierConfigMgr handle (the global manager used by scanner tests or a
per-instance one used by the app tests).
- xl.meta transition assertion helpers: read_transition_meta,
assert_transition_meta_consistent (cross-shard consistency of the
status/tier/remote-key/remote-version-id tuple plus free-version count), and
free_version_count.
- polling helpers: wait_for_remote_absence, wait_for_object_count,
wait_for_free_version_absence.
Both existing copies now consume this single definition; `rg 'struct
MockWarmBackend'` collapses to one. The feature is enabled only from
[dev-dependencies], so it never links into the production binary (resolver 3).
Designed for downstream ilm-8 (restore lifecycle) and ilm-11 (tier fault
injection matrix). Coordinates with #4706 (ilm-2), which adds op-logging to the
scanner mock — that op-logging is now part of this shared surface, so #4706
should rebase onto it.
Refs rustfs/backlog#1148 (ilm-6), rustfs/backlog#1155.
* test(ecstore): fix shared MockWarmBackend usage after main merge
- Access stored objects via MockWarmBackend::contains() instead of the now
private inner objects map (fixes E0609 after the shared test-util refactor).
- Drop dead ReadCloser/ReaderImpl/DiskAPI imports and the unused
transition_api test re-exports the mock extraction left behind.
- Reword the scanner/rustfs test-util dependency comments so they no longer
embed the literal rustfs_ecstore:: path that trips the ECStore
architecture-migration guard.
`MetricsCollector::record_io_operation` recomputed P50/P95/P99 on every disk IO
by taking a read lock, collecting up to `max_latency_samples` (1000) into a
`Vec<u128>` and fully sorting it — an O(n log n) alloc+sort per operation on the
GET read path (gated by stage metrics). Both consumers sample only periodically:
the autotuner reads `avg_io_latency_us` on its tuning tick, and the P95/P99
values are never read internally — they only feed OTEL export. Nothing needs
per-IO freshness.
Split the two costs:
- The window mean (stored in `avg_io_latency_us`, read by the autotuner) is now
maintained in O(1) via a running sum kept in step with the window's push/pop,
and refreshed on every op — no sort, no warm-up regression.
- The P95/P99 sort is throttled to once per `PERCENTILE_RECOMPUTE_INTERVAL`
(128) operations. A bounded lag is invisible to the periodic autotuner tick and
OTEL export while the per-op sort cost is amortised ~128x away.
Semantics are otherwise unchanged: same sliding window, same percentile indices,
same mean value. Tests updated — the percentile test forces a recompute to
exercise the math directly, and a new test asserts the mean tracks every op while
the percentiles only recompute at the interval boundary.
Addresses rustfs/backlog#1185 (P1).
Co-authored-by: heihutu <heihutu@gmail.com>
Convert the two reliant lifecycle expiry tests from #[ignore] + a hardcoded
localhost:9000 server to self-managed RustFSTestEnvironment tests (random port,
isolated temp dir) and wire them into the PR e2e-smoke subset.
Time control is chosen per test and documented in-module:
- test_lifecycle_expiry_backdated_mtime: mod_time back-dating via the internal
source-replication backdoor (new put_object_with_backdated_mtime helper) so a
Days=1 rule is already due, with RUSTFS_ILM_PROCESS_TIME=1 shrinking the
rounding boundary. Asserts the backdated matching-prefix object expires and a
non-matching-prefix object with the same backdating survives (isolates the
prefix filter). Proves the backdoor does not trip the replication gate.
- test_lifecycle_versioned_current_version_expiry_creates_delete_marker:
RUSTFS_ILM_DEBUG_DAY_SECS=2 (ilm-5) accelerates the day length; asserts a
latest delete marker is created, the original data version is retained and
still readable by version id.
- test_lifecycle_zero_day_expiry: Days=0 immediate expiry; matching object
deleted, non-matching survives.
All three use RUSTFS_SCANNER_CYCLE=1 so the scanner runs every second; tests poll
for the terminal state instead of sleeping fixed wall-clock. Ignore count for
reliant/lifecycle.rs goes 2 -> 0.
CI wiring: added a distinct 'test(/^reliant::lifecycle::/)' clause to the
existing profile.e2e-smoke default-filter in .config/nextest.toml (the single
sanctioned e2e wiring mechanism per crates/e2e_test/README.md; no new e2e job).
Refs rustfs/backlog#1148 (ilm-3), rustfs/backlog#1155
* perf(metrics): drop needless per-emission work on hot metric paths
Audit of the metrics hot paths surfaced four low-risk wins where emission did
work it did not need to:
- `record_file_cache_reclaim_success/error` (disk/local.rs) called `.to_string()`
on `kind` (already `&'static str`) and on the `"ok"`/`"err"` literals, heap-
allocating up to four `String`s per page-cache reclaim window — which runs per
read-stream reclaim. The `metrics` macros accept `&'static str` label values
directly, so pass them as-is.
- `record_read_repair_dedup` (set_disk/core/io_primitives.rs) likewise
`.to_string()`-ed an already-`&'static str` `reason`.
- `SetDisks::get_object_reader` (set_disk/ops/object.rs) captured `Instant::now()`
and emitted the `rustfs.lock.acquire.*` counter and histogram unconditionally
on every GET, right beside an already-gated stage timer. Gate them behind
`get_stage_metrics_enabled()` too, so an inactive observability config pays no
per-GET clock read or recorder lookups.
- The per-response-body-chunk counter in server/http.rs re-ran the `counter!`
registry lookup on every chunk (a streamed GET emits many). Resolve the
label-less handle once into a `LazyLock<metrics::Counter>`; the global recorder
is installed at startup before any response streams, so the cached handle binds
to the final recorder.
No metric names or label values change. The only behavior change is that the
`rustfs.lock.acquire.*` GET-path metrics now follow the GET stage-metrics flag,
consistent with the neighbouring stage timings.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(metrics): gate page-cache reclaim metrics behind metrics_enabled()
`record_file_cache_reclaim_success/error` run per read-stream reclaim window on
large-object reads and emitted unconditionally. When general metrics are
disabled the `counter!`/`histogram!` macros still construct three metric keys
per call for nothing. Skip the emission behind `rustfs_io_metrics::metrics_enabled()`,
matching how the io-metrics free functions self-gate. The serial reclaim-metrics
test now enables the flag (save/restore) alongside the existing stage gate.
Left ungated deliberately: `record_read_repair_dedup` (rare read-repair path,
and its non-serial test would need a global-flag toggle), and the HTTP body-chunk
counter (its cached handle already makes the disabled case a no-op increment).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
`full_object_plaintext_len` decides whether a body-cache hook hit may serve
bytes in place of the erasure read. It is a fail-closed allow-list: it excludes
every read whose `ReadPlan::build` applies some other transform (ranged/part,
raw/data-movement, restore, encrypted, remote) with an early `return None`,
then returns a `Some(..)` length only for the whole-plaintext cases. A newly
added `ReadPlan` branch that nobody teaches this gate about falls through to
`None` and safely bypasses the cache. Flip it to a deny-list and the same new
branch silently serves bytes in the wrong representation — the backlog#1108 /
#1109 / #1146 class of bug.
The existing unit and e2e tests only cover the branches that exist today. This
adds `scripts/check_body_cache_whitelist.sh`, a structural guard wired into
pre-commit / pre-pr / dev-check and CI, that asserts every exclusion predicate
and a `return None` still precede the first `Some(..)`. Reordering a predicate,
dropping one, moving the positive return ahead of the gate, or renaming/removing
the function all fail; wording, formatting, and adding a new exclusion in the
same gate do not. Mutation-tested against all four regression shapes.
This machine-enforces the structural invariant that backlog#1146 was kept open
to guard by hand.
Co-authored-by: heihutu <heihutu@gmail.com>
Close the three test-completeness items in rustfs/backlog#1180 that the earlier
hardening (rustfs/rustfs#4726, #4729) left unpinned; the other two of the five
(sharded cancel routing, bailout-handle error) already landed in rustfs/uring.
- rename_data end-to-end invalidation: drive the real `LocalDisk::rename_data`
commit (non-inline part, so `invalidate_part_paths` is non-empty) and assert
the destination part descriptor is dropped — not merely `rename_file`/`delete`.
Both production `rename_data` call sites share this invalidation, so a
"fix one copy, miss the other" regression is now caught. The test is
non-vacuous: it seeds the cache, removes the on-disk data dir out of band (the
cached fd keeps the old inode alive and clears the path for the directory
rename `rename_data` performs), and asserts a read still returns the OLD bytes
before the commit — which fails outright if the cache is off, so it cannot pass
without a live cache.
- FD_CACHE_TTL backstop: an injected short TTL proves the cache self-evicts a
descriptor with no explicit invalidation; a static check pins the 5s value.
- zero-length read bounds parity on the cache-HIT path: a `length == 0` read
past EOF must be rejected identically to the miss path and StdBackend, pinning
the #1173 fix against regression.
Refactors `FdCache::new` to delegate to a private `with_ttl(ttl)` helper so the
TTL backstop can be exercised with a short TTL instead of a multi-second wait.
Verified: `cargo test -p rustfs-ecstore` on Linux with real io_uring
(seccomp=unconfined, RLIMIT_NOFILE raised, RUSTFS_URING_TESTS_MUST_RUN=1 so a
degraded skip fails rather than passing vacuously).
Co-authored-by: heihutu <heihutu@gmail.com>
The io-metrics crate already had per-stage GET/PUT switches, but the request
headers/summaries and ~40 general recorders (I/O scheduler, bytes-pool,
zero-copy, bandwidth, system-resource, error/timeout/retry) emitted
unconditionally, paying their label allocations and arithmetic even when no
metric recorder is installed.
Add `METRICS_ENABLED: AtomicBool` (default false) with `set_metrics_enabled()`
/ `metrics_enabled()`, isomorphic to the existing stage switches, and gate:
- GET headers/summaries via `get_stage_metrics_enabled()`:
record_get_object_request_start / _request_started / _request_result /
_timeout / _completion / _total_duration_with_path, plus legacy
record_get_object.
- PUT headers via `put_stage_metrics_enabled()`:
record_put_object_request_start / _request_result, plus legacy
record_put_object.
- 41 general recorders via `metrics_enabled()`.
Startup enables all three switches together under
`observability_metric_enabled()` (startup_observability.rs), with a passthrough
in startup_runtime_sources.rs. The global recorder is itself only installed
when observability metric export is on, so gating off is pure cost savings when
export is disabled and a no-op when it is enabled.
Deliberately NOT gated (would break function, not just observation): the EC
encode in-flight and GET buffered-bytes accounting guards
(add/remove_ec_encode_inflight_bytes, track_get_object_buffered_bytes) whose
values are read back by current_*; and the stateful sibling modules
(lock/deadlock/backpressure/autotuner/adaptive_ttl/collector/internode). The
console realtime metrics endpoint (/admin/v3/metrics) samples system state and
internode metrics directly, so it does not depend on the gated recorders.
Adds a toggle test exercising both the enabled and disabled paths, serialized
on the existing METRICS_FLAG_LOCK to stay robust under nextest's shared-process
model.
Co-authored-by: heihutu <heihutu@gmail.com>
Two behavior-preserving cleanups to the io_uring local read backend that
accumulated as later work layered onto it:
- Reunite the `UringBackend` struct doc comment. The backlog#1145 fd-cache
constants were inserted into the middle of the struct's doc block, so its
opening sentences were orphaned onto `ENV_RUSTFS_IO_URING_FD_CACHE` and the
struct itself was documented by a sentence fragment starting mid-clause with
"through rustfs-uring's...". Move the opening back onto the struct and give
the constant its own one-line doc.
- Fold the fd-cache handle and its lookup key into a single
`Option<(&FdCache, FdKey)>` in `pread_uring`, so the get and the
insert_if_fresh sites stop re-deriving `self.fd_cache.as_ref()` and
re-matching presence. Semantics are identical, including the backlog#1176
generation guard (`gen_at_open` snapshot before open, insert only when the
generation is unchanged).
No functional change. The borrow/move pattern was validated against a
host-compilable reduction (the cfg(linux) path cannot be built from macOS);
CI compiles the Linux backend.
Co-authored-by: heihutu <heihutu@gmail.com>
chore(deps): bump rustfs-uring to 0.2.1 from crates.io
rustfs-uring 0.2.1 routes the driver's runtime diagnostics through `tracing`
with structured fields instead of `eprintln!` (rustfs/uring#13). No public API
change — UringDriver::probe_and_start_sharded, read_at, and read_at_direct keep
their signatures — and the read path / cancel-safety ownership model are
untouched, so this is a drop-in patch bump of the version requirement plus the
lockfile entry.
0.2.1 adds a `tracing` dependency; it is recorded in the rustfs-uring lock
entry. tracing is already in the workspace graph, so nothing new is pulled in.
The Cargo.lock change is scoped to the rustfs-uring package only; unrelated
lockfile drift is left for its own change.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): close the fd-cache open-then-insert race with a generation guard (rustfs/backlog#1176)
pread_uring's miss path opened a descriptor on the blocking pool and only then
inserted it into the moka cache. moka's invalidations cover only entries present
at call time, so a heal/delete commit that invalidated between the open and the
insert could not stop the just-opened stale inode from being cached afterwards —
serving the pre-heal/pre-delete inode for up to the TTL and defeating the heal.
Add an invalidation generation to FdCache, bumped by invalidate_exact and
invalidate_under before they touch moka. The read path snapshots the generation
before opening and inserts via insert_if_fresh, which refuses the insert if the
generation moved during the open and, with a post-insert re-check, removes the
entry if an invalidation raced the insert itself. Reads that never miss are
unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate the fd cache on the primary object-delete paths (rustfs/backlog#1175)
The fd-cache invalidation contract was only wired into DiskAPI::delete,
rename_file and rename_data, but object deletion almost never goes through
LocalDisk::delete — DeleteObject(s) reach delete_version, delete_versions ->
delete_versions_internal, and delete_paths, all of which remove a version's data
dir (move_to_trash / rename_all staging) with no invalidation. A cached io_uring
descriptor kept the deleted part.N inode readable for up to the TTL, so a GET in
that window could still return deleted data.
Invalidate every cached fd under the removed data dir at each site: in
delete_version and delete_versions_internal the data_dir uuid and object path are
in hand (invalidate_cached_fds_under(volume, "{path}/{uuid}")); delete_paths
invalidates under each removed path. A later rollback that restores a data dir
just causes the next read to re-open it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): close remaining fd-cache invalidation gaps (rustfs/backlog#1177)
Three residual paths could keep serving a stale descriptor:
- delete_volume removed the whole bucket tree (remove_dir_all/remove_dir) with no
invalidation, and the cache-hit read path skips the volume-access check, so a
cached fd kept a removed object readable. Add invalidate_cached_fds_for_volume
(a per-volume moka predicate) and call it after the bucket is removed.
- A retired LocalDisk instance (renew_disk on reconnect builds a fresh one) kept
its populated cache alive while still referenced by in-flight ops, so
invalidations through the new instance never reached it. close() now clears the
backend's cache via clear_cached_fds.
- rename_data's post-commit rollback (a commit-metadata fsync failure under
strict durability) restored the old data dir without dropping fds cached during
the committed window; the streaming branch now invalidates the dst part fds on
those rollback paths. The inline branch's rollback runs inside spawn_blocking
and is left to the TTL backstop.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): narrow the io_uring latch classes to match StdBackend (rustfs/backlog#1171)
The runtime degradation classification reused the probe-time restriction errnos,
which the driver's C7 contract explicitly warns against, so a single per-file
error could latch a whole disk off io_uring:
- is_io_uring_unsupported no longer includes EACCES: at read time on an
already-open fd it is per-file (an LSM hooks security_file_permission on every
read) and StdBackend hits the same denial, so falling back masks nothing and a
full-disk latch would be wrong. ENOSYS and EPERM (seccomp/LSM applied after
startup) remain. EOPNOTSUPP is now classified per-path by the caller.
- pread_uring_direct's read-error arm now mirrors StdBackend: an O_DIRECT-shape
error (EINVAL/EOPNOTSUPP) latches only direct_uring.supported so eligible reads
take StdBackend's aligned path, instead of over-latching the whole io_uring
backend or never latching a read-side EINVAL at all.
- try_new only negative-caches genuine restriction-class probe failures in
URING_UNSUPPORTED_DISKS; an unexpected (possibly transient) probe failure now
falls back without latching, so the next reconnect re-probes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): log when a disk latches io_uring off at runtime (rustfs/backlog#1172)
A probe-gated gray release was flying blind: the permanent per-disk `active`
latch flipped with no log and no metric, so the only message operators ever saw
was the startup "io_uring read backend enabled" line — which stayed true on
dashboards even after the very first read latched the disk back to StdBackend
forever.
Add latch_active_off, which flips the latch with `swap` and logs the true->false
transition exactly once at warn with a dedicated event constant, disk root, and
errno. Both the buffered and O_DIRECT read paths use it. A fallback/latch metric
counter and periodic export of the driver StatsSnapshot (cq_overflow,
cancel_already) remain as follow-ups that need rustfs_io_metrics plumbing.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(audit): correct the stale rustfs-uring license-allow rationale (rustfs/backlog#1181)
The dependency-review allow said rustfs-uring is "pulled as a git dependency",
but ecstore now pins it from crates.io. Update the rationale and scope the allow
to the exact pinned version (pkg:cargo/rustfs-uring@0.1.0) so a future version
bump forces a conscious re-review of the license/provenance claim instead of
being waved through on an outdated justification.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): offload io_uring driver teardown off the tokio worker (rustfs/backlog#1170)
UringBackend held Arc<UringDriver> and had no Drop, so when the last LocalDisk
reference dropped in async context (disk reconnect via renew_disk, or shutdown),
UringDriver's own Drop ran on that thread — sending Shutdown and joining each
shard thread, which can block up to the bounded-drain timeout (5s) on a hung /
D-state disk, stalling a tokio worker.
Wrap the driver in ManuallyDrop (deref is transparent, so read call sites are
unchanged) and add a Drop that takes the Arc and, when a runtime is present,
drops it on a blocking thread so the potentially-blocking join never runs on a
runtime worker. Off-runtime it drops inline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): restore StdBackend read parity on the uring paths (rustfs/backlog#1173)
Two byte-for-byte parity breaks against StdBackend on the io_uring read paths:
- A zero-length read on an fd-cache hit returned Ok(empty) without any bounds
check, while StdBackend and the uring miss path return FileCorrupt for an
offset past EOF. Fstat the cached descriptor on the length==0 path and match.
- reclaim_read_range fadvise(DONTNEED)'d the raw unaligned [offset, offset+len)
range, but fadvise only drops fully-covered pages, so the head partial page
stayed resident — whereas StdBackend's mmap path reclaims the page-aligned
superset. Bitrot shards' 32-byte block headers keep offsets off page
boundaries, so this diverged on the common case. Page-align the reclaim window
to match the mmap path exactly.
(The third parity item from the audit — a failed reclaim fadvise failing the
read — is already parity: StdBackend's mmap path propagates the same fadvise
error with `?`, so no change is needed.)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): bound worst-case in-flight memory by chunking huge uring reads (rustfs/backlog#1174)
The driver's backpressure permits count operations, not bytes, and it zero-fills
a full-size buffer per op, so a single unbounded read could pin ~length bytes per
permit (128 permits x shards x up to ~2 GiB). ecstore passes a whole part's shard
range as one pread_bytes with no upstream chunking.
On the buffered path, split reads larger than URING_MAX_OP_LEN (128 MiB) into
sequential chunks, awaited one at a time, so worst-case in-flight memory is
bounded by permits x URING_MAX_OP_LEN per shard. The threshold is high enough
that ordinary shard reads keep the single-op, zero-copy fast path unchanged. The
O_DIRECT path (opt-in, alignment-constrained) is left for a follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): gate the io_uring fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178)
The fd cache holds up to FD_CACHE_CAPACITY (512) descriptors per disk, but
try_new cannot know the disk count and nothing checked the process fd budget. On
a bare-metal / non-systemd run with the common 1024 soft RLIMIT_NOFILE, two disks
would already exhaust fds with EMFILE surfacing on reads and probes.
Check the soft limit at try_new: enable the cache only with ample headroom
(>= 16384), otherwise log a warning once and fall back to open-per-read. The
packaged systemd unit sets 1,048,576, so tuned deployments are unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): make io_uring test skips visible and gate non-vacuity (rustfs/backlog#1179)
The ecstore io_uring tests degrade to a silent pass when io_uring is unavailable
(bare `return`s or plain eprintlns), so a CI leg on a restricted runner never
exercises the real UringBackend/FdCache/latch paths yet still goes green — an
integration regression could merge unseen.
Add uring_test_skip: it emits a grep-able `SKIP <name>` line and, when
RUSTFS_URING_TESTS_MUST_RUN is set (a CI leg that guarantees io_uring, e.g. a
seccomp=unconfined container), panics instead of skipping. Route the silent-skip
sites through it. Wiring a dedicated CI leg that sets that env on a capable
runner is tracked in the issue; this provides the enforcement mechanism.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover delete_paths fd-cache invalidation (rustfs/backlog#1180)
Add an end-to-end test that seeds the descriptor cache with a read, removes the
part via disk.delete_paths (one of the primary object-delete entry points that
does not go through LocalDisk::delete), and asserts the next read no longer
returns the removed inode — pinning the invalidation added in #1175. The sharded
cancel-routing half of #1180 is covered in the rustfs-uring PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* io_uring audit follow-ups: O_DIRECT chunking, inline invalidation, metrics, CI leg (backlog#1160) (#4729)
* fix(ecstore): chunk large O_DIRECT reads too, bounding in-flight memory (rustfs/backlog#1174)
The buffered read path already splits reads above URING_MAX_OP_LEN into
sequential chunks; do the same for the O_DIRECT path, which was left for a
follow-up. read_at_direct aligns each chunk's sub-range internally, and chunk
sizes are a multiple of URING_MAX_OP_LEN so a boundary re-read is at most one
block. Extract classify_direct_read_error so the single-op and chunked paths
share one copy of the EINVAL/EOPNOTSUPP-vs-subsystem latch classification rather
than duplicating it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate cached fds on the inline rename_data rollback (rustfs/backlog#1177)
The streaming rename_data branch invalidates cached part fds on its post-commit
rollback paths, but the inline branch runs its commit and rollback inside a
single spawn_blocking closure where the async invalidate cannot be called, so it
was left to the TTL backstop.
Capture the closure's result instead of `??`-propagating it: on error (a
commit-metadata fsync failure under strict durability rolls the committed rename
back), invalidate the dst part paths at the async level before returning. Inline
objects keep their data in xl.meta rather than separate part inodes, so this is
largely defensive, but it removes the caveat and keeps the two branches
consistent.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): export io_uring latch/fallback and driver stats metrics (rustfs/backlog#1172)
Complete the gray-release observability. Beyond the warn log added earlier, emit
metrics so a dashboard can answer "how much traffic is on io_uring vs falling
back, and is any disk degrading":
- rustfs_io_uring_latch_off_total — a disk latching io_uring off at runtime.
- rustfs_io_uring_read_fallback_total — each io_uring -> StdBackend read
fallback (latched-off short-circuit, O_DIRECT error, buffered error).
- a low-frequency per-disk exporter of the driver StatsSnapshot as gauges
(in_flight, cq_overflow, cancel_already), spawned in try_new. It holds only a
Weak reference so it never keeps the driver alive, and drops any temporary
strong reference on the blocking pool so a last-reference UringDriver::Drop
join never runs on an async worker (rustfs/backlog#1170).
submit_errors is deliberately not exported yet: it is a field added in the
unreleased rustfs-uring 0.2.0, and ecstore still pins 0.1.0. It lands once the
dependency is bumped (rustfs/backlog#1181).
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci: add a real-io_uring integration leg on ubuntu-latest (rustfs/backlog#1179)
The existing self-hosted sm-standard runners cannot guarantee io_uring is
available (a container seccomp filter can block io_uring_setup), so the ecstore
uring tests degrade to a silent skip and never exercise the real
UringBackend/FdCache/latch paths in CI.
Add a job on GitHub-hosted ubuntu-latest, which runs a recent kernel with no
container seccomp filter, running the uring-named ecstore tests with
RUSTFS_IO_URING_READ_ENABLE=true and RUSTFS_URING_TESTS_MUST_RUN=1 — the
non-vacuity gate makes the leg fail rather than skip if io_uring is unavailable,
so an integration regression can no longer merge green behind a vacuous pass.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(docker): warn instead of rejecting default or missing credentials
The entrypoint hard-reject from #4278 broke the container-first UX and
the repo's own scheduled e2e lanes (e2e-s3tests, mint boot rustfs-ci
images with default credentials and died at startup). Maintainer
decision: ship no baked-in credentials, warn instead of block.
- missing credentials: warn and start; wording accounts for the
RUSTFS_ROOT_*/MINIO_* alias sources the entrypoint does not inspect
- default rustfsadmin via env/CLI/file: warn and start; the warning
notes all-default pairs cannot derive an internode RPC secret
- malformed config stays fatal: source conflicts, unreadable files,
empty or whitespace-only values, flags missing their argument
- present-but-empty env vars now hit the empty-value hard failure
instead of running the binary with an empty root credential
- empty/default checks trim CR and blanks like the binary; files
without a trailing newline are no longer falsely rejected as empty
- the no-baked-credentials guard covers all four Dockerfiles, and the
test harness refuses hosts where /usr/bin/rustfs exists
- e2e-s3tests/mint move to non-default credentials (rustfsadmin-ci),
which also restores RPC-secret derivation for the multi-node lane
GHSA-68cw fail-closed RPC derivation (#4402) is untouched; helm stays
fail-closed by design.
* chore(docker): reword entrypoint comment flagged by typos check
chore(deps): bump rustfs-uring to 0.2.0 from crates.io
rustfs-uring 0.2.0 is published on crates.io. It carries the read-driver
hardening from the backlog#1160 adversarial audit (cancel-safety and
graceful-degradation fixes on the Linux io_uring read path). The public
API ecstore consumes is unchanged — UringDriver::probe_and_start_sharded,
read_at, read_at_direct all keep their 0.1.0 signatures — so this is a
drop-in bump of the version requirement plus the lockfile entry.
The Cargo.lock change is intentionally scoped to the rustfs-uring package
only; unrelated lockfile drift is left for its own change.
Co-authored-by: heihutu <heihutu@gmail.com>
test(e2e): negative presigned-URL SigV4 suite (backlog#1151 sec-2)
Add crates/e2e_test/src/presigned_negative_test.rs, the query-string
SigV4 sibling of the header-SigV4 suite (sec-1, #4708). Presigned URLs
were only exercised on the happy path, so nothing pinned our end-to-end
enforcement of expiry or query-signature verification.
Against a live single-node RustFSTestEnvironment (random port, isolated
temp dir), the suite asserts HTTP status + S3 error <Code> for:
- expired presigned GET -> 403 AccessDenied (Request has expired)
- tampered X-Amz-Signature -> 403 SignatureDoesNotMatch
- wrong secret key -> 403 SignatureDoesNotMatch
- tampered target key (swapped after signing) -> 403 SignatureDoesNotMatch
- tampered presigned PUT -> 403 SignatureDoesNotMatch + object not stored
- positive controls: valid presigned GET (body match) and PUT (HEAD verify)
Expiry is controlled without real waiting via the AWS SDK presigner's
start_time (sign as of one hour ago with a 60s window). s3s checks
expiry before the signature, so the expired case surfaces AccessDenied.
Wired into the e2e-smoke nextest profile (fast, single-node, no external
deps) so it runs on every PR per the crates/e2e_test/README.md admission
criteria.
Refs rustfs/backlog#1151 (sec-2), rustfs/backlog#1155.
test(admin-auth): unit + e2e coverage for the admin authorization gate
Adds the first tests for the central admin authorization gate
`rustfs/src/admin/auth.rs`, which previously had zero coverage
(backlog#1151 sec-4, master plan #1155).
Unit tests (rustfs/src/admin/auth.rs):
- Refactor the two `validate_admin_request*` entry points to share a
single generic decision core `evaluate_admin_actions<S: Store>` /
`check_admin_request_auth<S: Store>` (removes the duplicated
action-loop and makes the gate testable without a running cluster).
- Cover: owner/root credential allowed; authenticated non-admin
credential denied with AccessDenied; missing credential denied; and
the multi-action loop grants on any permitted action, denies when
none pass. Backed by an in-memory empty IamSys so the owner path
short-circuits to allow and every other principal resolves to deny.
E2E (crates/e2e_test/src/admin_auth_test.rs, raw SigV4 over HTTP, no
awscurl dependency):
- non_admin_credential_denied_on_admin_api: a limited IAM user gets
403 AccessDenied on GET /rustfs/admin/v3/info while root succeeds.
- root_credential_rotation_takes_effect: restart with rotated
--access-key/--secret-key; new credential works and the stale one is
rejected, on both the admin plane and the S3 data plane.
- default_credentials_emit_startup_warning: booting with the default
rustfsadmin credentials emits the default-credentials warning.
Refs rustfs/backlog#1151 (sec-4), #1155.
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit 5c7c757a3 (#4712) activated the previously-dormant replication e2e
suite (they had never run anywhere). All 9 fast tests then failed on main
because the SSRF egress guard rejects the 127.0.0.1 targets the e2e harness
configures: `target endpoint is not allowed: outbound URL host '127.0.0.1'
is not allowed: loopback address`. The whole harness runs on loopback, so
every replication test hit this before reaching its actual assertion.
Loopback is a genuine SSRF vector and must stay rejected in production, so
this does not relax the guard. Instead `validate_replication_target_endpoint`
gains an off-by-default opt-in (`RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET`)
that re-enables loopback targets (127.0.0.1 / ::1 / localhost) for single-host
multi-instance dev and the e2e harness. Private addresses stay unconditionally
allowed as before; the opt-in does not widen into link-local or the cloud
metadata endpoint. The e2e harness sets the env for every server it spawns
(single-node and cluster paths), overridable via extra_env.
Verified end-to-end: all 9 previously-failing replication_extension_test
smoke tests pass against a locally built binary. New unit tests in
bucket_target_sys pin the matrix — public/private always allowed, loopback
gated on the opt-in in both IP and hostname forms, and metadata/link-local
still rejected even with the opt-in on.
Refs: backlog#1147
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(replication): rename optin -> opt_in to satisfy typos check
Pure rename of three unit-test function names; no behaviour change.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all
closed). Review of every migration-era test/gate measure found three things
actually retirable or broken — everything else is a live anti-regression
guard and stays.
Remove:
- scripts/check_metrics_migration_refs.sh — guards a migration that
finished: rustfs_metrics:: has zero hits, the metrics crate no longer
exists, and the script was never wired into CI or make (only reference
was one line in config-model-boundary-adr.md, also removed).
- crates/obs init_metrics_collectors — the "backward-compatible alias kept
during migration" the removed script was guarding. Zero callers; pure
delegate to init_metrics_runtime.
Archive (docs/superpowers/plans/, continuing the 2026-07 convention,
with the standard archived banner):
- startup-timeline.md, scheduler-baseline.md,
profiling-numa-capability-inventory.md,
kms-development-defaults-inventory.md — one-shot snapshots whose only
consumer is the already-archived migration-progress ledger (their
same-dir links there start resolving again after the move); zero script
pins; fed the closed backlog#660/#665 architecture-review ledger.
Fixed the one outbound link (startup-timeline -> readiness-matrix) that
the move would have broken — check_doc_paths.sh deliberately does not
scan plans/, so nothing else would have caught it.
Wire (found orphaned by the same review):
- scripts/check_extension_schema_boundaries.sh guards a live contract
crate but was never invoked anywhere. Add lint-fmt.mak target, include
in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job
already installs ripgrep), sync the CONTRIBUTING.md enumerated list,
and harden the script against a silently-passing rg probe when src/
is missing.
Keep (verified live, documented so the next cleanup pass does not repeat
this analysis):
- scripts/check_architecture_migration_rules.sh — added a header stating
it is a permanent boundary guard, not retirable migration scaffolding;
'migration' in the name is historical.
- check_migration_gate_count.sh + floor, delete-marker e2e proof, all
pinned docs, compat-cleanup-register sync, remaining inventories
(referenced by live docs).
Verification: all 7 guard scripts pass, actionlint clean,
cargo check --workspace (excl e2e) clean, cargo fmt --check clean.
Adversarially reviewed by two independent skeptic passes; their 7
findings (alias left behind, broken outbound link, missing banners,
wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing
header rationale) are all folded in.
Three health handler tests assert on payload fields (degradedReasons,
details) that are absent when RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE is
true. The minimal-mode sibling test sets that env var via
temp_env::with_var, which leaks across parallel test threads.
Wrap the three affected tests with their own with_var guard pinning the
env var to false so they are deterministic regardless of thread order.
Zhengchao An