`full_object_plaintext_len` decides whether a body-cache hook hit may serve
bytes in place of the erasure read. It is a fail-closed allow-list: it excludes
every read whose `ReadPlan::build` applies some other transform (ranged/part,
raw/data-movement, restore, encrypted, remote) with an early `return None`,
then returns a `Some(..)` length only for the whole-plaintext cases. A newly
added `ReadPlan` branch that nobody teaches this gate about falls through to
`None` and safely bypasses the cache. Flip it to a deny-list and the same new
branch silently serves bytes in the wrong representation — the backlog#1108 /
#1109 / #1146 class of bug.
The existing unit and e2e tests only cover the branches that exist today. This
adds `scripts/check_body_cache_whitelist.sh`, a structural guard wired into
pre-commit / pre-pr / dev-check and CI, that asserts every exclusion predicate
and a `return None` still precede the first `Some(..)`. Reordering a predicate,
dropping one, moving the positive return ahead of the gate, or renaming/removing
the function all fail; wording, formatting, and adding a new exclusion in the
same gate do not. Mutation-tested against all four regression shapes.
This machine-enforces the structural invariant that backlog#1146 was kept open
to guard by hand.
Co-authored-by: heihutu <heihutu@gmail.com>
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all
closed). Review of every migration-era test/gate measure found three things
actually retirable or broken — everything else is a live anti-regression
guard and stays.
Remove:
- scripts/check_metrics_migration_refs.sh — guards a migration that
finished: rustfs_metrics:: has zero hits, the metrics crate no longer
exists, and the script was never wired into CI or make (only reference
was one line in config-model-boundary-adr.md, also removed).
- crates/obs init_metrics_collectors — the "backward-compatible alias kept
during migration" the removed script was guarding. Zero callers; pure
delegate to init_metrics_runtime.
Archive (docs/superpowers/plans/, continuing the 2026-07 convention,
with the standard archived banner):
- startup-timeline.md, scheduler-baseline.md,
profiling-numa-capability-inventory.md,
kms-development-defaults-inventory.md — one-shot snapshots whose only
consumer is the already-archived migration-progress ledger (their
same-dir links there start resolving again after the move); zero script
pins; fed the closed backlog#660/#665 architecture-review ledger.
Fixed the one outbound link (startup-timeline -> readiness-matrix) that
the move would have broken — check_doc_paths.sh deliberately does not
scan plans/, so nothing else would have caught it.
Wire (found orphaned by the same review):
- scripts/check_extension_schema_boundaries.sh guards a live contract
crate but was never invoked anywhere. Add lint-fmt.mak target, include
in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job
already installs ripgrep), sync the CONTRIBUTING.md enumerated list,
and harden the script against a silently-passing rg probe when src/
is missing.
Keep (verified live, documented so the next cleanup pass does not repeat
this analysis):
- scripts/check_architecture_migration_rules.sh — added a header stating
it is a permanent boundary guard, not retirable migration scaffolding;
'migration' in the name is historical.
- check_migration_gate_count.sh + floor, delete-marker e2e proof, all
pinned docs, compat-cleanup-register sync, remaining inventories
(referenced by live docs).
Verification: all 7 guard scripts pass, actionlint clean,
cargo check --workspace (excl e2e) clean, cargo fmt --check clean.
Adversarially reviewed by two independent skeptic passes; their 7
findings (alias left behind, broken outbound link, missing banners,
wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing
header rationale) are all folded in.
* fix(obs): remove the dial9 task-dump switch that could never do anything
Measured on a bench host (Linux x86_64) against the code merged in #4663: with
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and
`--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events.
dial9 captures a task dump only for futures it wrapped itself — those spawned
through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets
applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn`
throughout. Same workload, same binary, only the spawner changed:
tokio::spawn -> 0 dumps
dial9::spawn -> 14709 dumps, all with callchains
Upstream documents this (README line 151) and tracks the doc gap at
dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663,
and so shipped exactly the kind of lying configuration knob that PR set out to
delete. Remove it: the two environment variables, the config fields, the
`with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was
to constrain the build to Linux while recording nothing.
Re-adding it only makes sense together with migrating the paths under
investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157.
Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured:
dumps are captured with and without it (14709 vs 14674, within noise), and
upstream never asked for it. That requirement was mine, invented and untested.
Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): replace guessed dial9 retention numbers with measured ones
Three corrections, all to claims I wrote in #4663 without measuring them.
"Under a high poll rate that budget can wrap in minutes" was a guess. Measured on
a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent):
13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108
minutes. Even at ten times the throughput that is ~11 minutes. State the measured
rate and how to scale it instead.
dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O
on the blocking pool and through io_uring, never on an async worker, so a slow
drive never lengthens a poll. Injecting 200 ms of latency on one of four drives
cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms:
49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers
does not help: sched events are per-worker only, and the CPU profiler samples
on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at
the `rustfs_io_*` metrics instead.
What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no
fault injected at all — real worker stalls nothing else in the obs stack surfaces.
Lead with that.
Also link the two upstream issues filed for the gaps we documented:
dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs).
Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
CI runs `cargo clippy --all-targets -- -D warnings` without
`--all-features`, but the Makefile's clippy-check used `--all-features`.
After PR #4663 made the dial9 telemetry feature opt-in and removed
`--cfg tokio_unstable` from .cargo/config.toml, `--all-features`
activated dial9 (which requires tokio_unstable) and dial9-taskdump
(which requires tokio/taskdump, Linux-only), breaking local
`make pre-pr` on macOS.
Align the Makefile with CI by dropping `--all-features` from clippy-check.
build(make): require cargo-nextest for make test with explicit escape hatch (backlog#1153 infra-14)
cargo-nextest changes test semantics — it runs each test in its own process
(so serial_test's #[serial] mutex does not serialize across tests) and is the
only runner that honours .config/nextest.toml [test-groups] (e.g. the
ecstore-serial-flaky serialization guard). CI installs and runs nextest, but
`make test` only warned when it was missing and then silently fell back to
`cargo test`, which runs with different serialization behaviour than CI and can
mask or invent flakes.
Make cargo-nextest a hard dependency of `make test`:
- Missing nextest now fails `make test` with a clear message and install
instructions (`cargo install cargo-nextest --locked` or a prebuilt binary
via https://nexte.st/docs/installation/).
- Set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to opt into the plain `cargo test`
fallback anyway; it prints a loud warning that serialization semantics differ
from CI and .config/nextest.toml test-groups will NOT apply.
- The check stays cheap (command -v). The warn-only test-deps prerequisite is
dropped from check.mak in favour of recipe-level gating.
Install docs live in the make-layer error message plus a header comment in
tests.mak, with a single-line note in CONTRIBUTING.md (kept minimal to avoid
conflicting with #4667 / infra-9, which rewrites the verification sections).
No CI change: .github/workflows/ci.yml already runs cargo nextest and
.github/actions/setup/action.yml already installs it.
Refs rustfs/backlog#1153 (infra-14), master plan #1155.
Signed-off-by: Zhengchao An <anzhengchao@gmail.com>
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
Agent-instruction and architecture docs had drifted from the code:
- CLAUDE.md: slim to commands + pointers; fix wrong claim that
`make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
hand-maintained (already stale) scoped-AGENTS index; link
architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
(rebalance-decommission phases, migration-progress ledger, PR
template) to docs/superpowers/plans with archive headers; fix stale
source paths in kept inventories (core/sets.rs, core/pools.rs,
store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
.gitignore now keeps .claude/skills and docs/operations committable
Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(tier): stop sending nil/garbage versionId to warm backend S3
Three bugs caused NoSuchVersion errors when reading tiered objects:
1. warm_backend_s3sdk: GET and DELETE ignored rv/range opts entirely —
fixed to forward version_id and byte-range to the SDK request.
2. version.rs (MetaObject + MetaDeleteMarker): transition_version_id was
parsed with unwrap_or_default(), turning invalid/wrong-length bytes
into Uuid::nil(). The nil UUID was then serialized and sent as
?versionId=00000000-... to the tier backend -> NoSuchVersion.
Fixed: .and_then(.ok()).filter(!is_nil()) so only valid non-nil UUIDs
are forwarded as versionId.
3. bucket_lifecycle_ops: add debug/error logs in
get_transitioned_object_reader to record tier, tier_object, and
tier_version_id before and on failure of the tier GET.
Also adds tier transition fields to dump_fileinfo example for offline
xl.meta inspection, and fixes Docker build (cargo path + entrypoint).
Adds CLAUDE.md with tier architecture and debugging notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* more fixes for versionId
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>
* remove branch
* Add tests and fix cargo path, add load to build-docker
* update documentation (CLAUDE.md)
* more fixes for recover
* More fixes to ILM recover
* final fix
* chore: add missing-shard first-scene diagnostics (#3213)
chore(ecstore): add missing-shard first-scene diagnostics
Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.
Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.
* fix test anmd fmt
* fix cargo path
fix test
* fix(tier): format copy_object self-copy guard
---------
Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
houseme