First systematic HTTP e2e batch for the admin management surface
(backlog#1154 peri-2): full user -> canned-policy -> attach ->
service-account lifecycle with data-plane effect assertions (the
credential really gains and loses S3 access), plus a non-admin 403
probe per management endpoint targeting a different principal so
deny_only self-access semantics do not mask the gate. Wired into the
e2e-smoke nextest profile (ci-4 mechanism).
Pin the console listener's wire contract with a real binary on a random
port (backlog#1154 peri-4): unauthenticated version/license endpoints
answer complete JSON without credential material, the SPA prefix always
dispatches to the console static handler (never the S3 API), the admin
API and S3 root on the console listener stay 403 for anonymous callers,
and a console-disabled listener serves no console surface. Wired into
the e2e-smoke nextest profile (ci-4 mechanism).
Prove the swap-certificates-without-restart promise over real TLS
handshakes (backlog#1154 peri-5): new connections pick up the rotated
certificate within the reload interval, a session opened under the old
certificate survives the swap, and garbage material is fail-safe (the
old certificate keeps serving, the failure leaves a tls_reload_failed
log event, the process stays up). Wired into the e2e-smoke nextest
profile (ci-4 mechanism).
* ci: add e2e-full merge-gate job (single-node full e2e via nextest)
Adds the [profile.e2e-full] nextest profile and a matching e2e-full job in
ci.yml (push main + merge_group + workflow_dispatch) that runs the
never-automated user-visible e2e suites (KMS, object_lock, multipart_auth,
quota, checksum, encryption, security-boundary, ...) the fast PR e2e-smoke
subset skips.
The filter is the whole e2e_test crate minus the sets other lanes own:
protocols:: (ci-6/ci-7), the 6 RustFSTestClusterEnvironment cluster suites
(ci-7 nightly), replication_extension_test (repl-1's smoke + repl-nightly
lanes), and #[ignore]d tests (ci-13). The 4-disk reliability tests are
serialized, mirroring the ci profile. Reuses the downloaded debug binary and
uploads junit. backlog#1149 ci-5.
* ci: exclude characterized known-failures from e2e-full with tracked issues
First-ever automated run of the full suite (dispatch 29381309848: 341 ran /
32 failed / 460s) surfaced five real product-bug families, each now tracked:
rustfs#4842 (extract 500s, mtime=0 OffsetDateTime), rustfs#4843 (path-limit
vs ignore-errors), rustfs#4844 (anonymous POST SSE-S3 500), rustfs#4845
(object-lock POST + list metadata=true 403), rustfs#4846 (lock quorum
misclassified as timeout under load). Deterministic failures cannot be
retried away, so each family is excluded with its OPEN issue cited and the
fixing PR contract to delete the exclusion — passing negative-path siblings
stay in as regression guards.
ci(coverage): weekly cargo-llvm-cov workspace baseline (non-blocking)
Add the weekly line-coverage report (backlog#1153 infra-5):
- .github/workflows/coverage.yml — Sunday 07:00 UTC + workflow_dispatch;
runs `cargo llvm-cov nextest --workspace --exclude e2e_test` under
NEXTEST_PROFILE=ci (same scope and profile as the ci.yml test gate),
writes a per-crate line-coverage table to the job summary, uploads
lcov + JSON as a 90-day artifact, and routes scheduled failures
through the shared schedule-failure-issue action (ci-8). Runs only on
schedule/dispatch, so it can never become a required PR check.
- scripts/coverage_per_crate.py — stdlib-only aggregation of the
llvm-cov JSON export into the per-crate markdown table (worst-first,
TOTAL row), shared by the workflow and the local target.
- make coverage (.config/make/coverage.mak) — local equivalent with the
same command sequence; fails with install hints when cargo-llvm-cov
or cargo-nextest are missing. Listed in make help.
- docs/testing/README.md — Coverage section: cadence, where the table
and artifacts live, the trend-comparison method, and what is not
measured (doctests, e2e_test).
The header-SigV4 (sec-1), presigned-URL (sec-2), and admin-gate (sec-4) negative
auth-rejection e2e suites merged earlier but only presigned_negative was actually
selected by any CI profile; negative_sigv4_test and admin_auth_test compiled and
sat unrun. Add both to the e2e-smoke default-filter so all three attacker-facing
S3 auth-rejection suites execute on every PR. They already meet the smoke
admission criteria (RustFSTestEnvironment, random ports, parallel-safe, no
#[ignore], no feature gates), so this is a pure filterset change — the single
e2e-in-CI wiring mechanism (backlog#1149 ci-4), not a new job.
Because the filter selects by module name, a rename or deletion could silently
drop a suite out of the security gate with no CI signal. Add
scripts/check_security_smoke_count.sh (infra-12 count-floor mechanism): it lists
what the e2e-smoke profile selects and fails if the count of security
auth-rejection tests drops below the committed floor in
.config/security-smoke-floor.txt (16). Invoked from the e2e-tests job, before the
smoke run, so the nextest list compiles the binaries the run reuses.
The GHSA-3p3x FTPS/WebDAV constant-time e2e (protocols::test_protocol_core_suite)
stays out by topology: it binds fixed ports, needs the ftps,webdav features, and
is #[serial], so it cannot join the random-port default-feature smoke profile as
a filterset change (global ruling G5). Its GHSA-r5qv sibling is a unit test that
already runs in the default CI pass. docs/testing/security-regressions.md now
carries the full CI-execution map and flags the GHSA-3p3x e2e CI-lane gap as a
ci-domain follow-up.
Refs: backlog#1151 (sec-5)
chore(ci): guard against committed planning docs; remove re-added ones
PR #4771 removed the docs/superpowers planning-doc archive and added a rule,
but #4765 force-added two more (git add -f bypasses .gitignore):
docs/superpowers/plans/2026-07-12-observability-single-writer.md and
docs/superpowers/specs/2026-07-12-observability-single-writer-design.md — an
agentic implementation plan and its design spec. Delete both.
Close the enforcement gap so this cannot recur:
- New scripts/check_no_planning_docs.sh fails if anything is tracked under
docs/superpowers/, regardless of how it was added.
- Wire it into make pre-commit/pre-pr/dev-check.
- Run it in CI: ci.yml for code/mixed PRs, and ci-docs-only.yml (which was a
green stub) so docs-only PRs can no longer slip a planning doc past the
required 'Test and Lint' check.
- Document the guard in AGENTS.md and the arch-checks skill.
Convert the two reliant lifecycle expiry tests from #[ignore] + a hardcoded
localhost:9000 server to self-managed RustFSTestEnvironment tests (random port,
isolated temp dir) and wire them into the PR e2e-smoke subset.
Time control is chosen per test and documented in-module:
- test_lifecycle_expiry_backdated_mtime: mod_time back-dating via the internal
source-replication backdoor (new put_object_with_backdated_mtime helper) so a
Days=1 rule is already due, with RUSTFS_ILM_PROCESS_TIME=1 shrinking the
rounding boundary. Asserts the backdated matching-prefix object expires and a
non-matching-prefix object with the same backdating survives (isolates the
prefix filter). Proves the backdoor does not trip the replication gate.
- test_lifecycle_versioned_current_version_expiry_creates_delete_marker:
RUSTFS_ILM_DEBUG_DAY_SECS=2 (ilm-5) accelerates the day length; asserts a
latest delete marker is created, the original data version is retained and
still readable by version id.
- test_lifecycle_zero_day_expiry: Days=0 immediate expiry; matching object
deleted, non-matching survives.
All three use RUSTFS_SCANNER_CYCLE=1 so the scanner runs every second; tests poll
for the terminal state instead of sleeping fixed wall-clock. Ignore count for
reliant/lifecycle.rs goes 2 -> 0.
CI wiring: added a distinct 'test(/^reliant::lifecycle::/)' clause to the
existing profile.e2e-smoke default-filter in .config/nextest.toml (the single
sanctioned e2e wiring mechanism per crates/e2e_test/README.md; no new e2e job).
Refs rustfs/backlog#1148 (ilm-3), rustfs/backlog#1155
`full_object_plaintext_len` decides whether a body-cache hook hit may serve
bytes in place of the erasure read. It is a fail-closed allow-list: it excludes
every read whose `ReadPlan::build` applies some other transform (ranged/part,
raw/data-movement, restore, encrypted, remote) with an early `return None`,
then returns a `Some(..)` length only for the whole-plaintext cases. A newly
added `ReadPlan` branch that nobody teaches this gate about falls through to
`None` and safely bypasses the cache. Flip it to a deny-list and the same new
branch silently serves bytes in the wrong representation — the backlog#1108 /
#1109 / #1146 class of bug.
The existing unit and e2e tests only cover the branches that exist today. This
adds `scripts/check_body_cache_whitelist.sh`, a structural guard wired into
pre-commit / pre-pr / dev-check and CI, that asserts every exclusion predicate
and a `return None` still precede the first `Some(..)`. Reordering a predicate,
dropping one, moving the positive return ahead of the gate, or renaming/removing
the function all fail; wording, formatting, and adding a new exclusion in the
same gate do not. Mutation-tested against all four regression shapes.
This machine-enforces the structural invariant that backlog#1146 was kept open
to guard by hand.
Co-authored-by: heihutu <heihutu@gmail.com>
test(e2e): negative presigned-URL SigV4 suite (backlog#1151 sec-2)
Add crates/e2e_test/src/presigned_negative_test.rs, the query-string
SigV4 sibling of the header-SigV4 suite (sec-1, #4708). Presigned URLs
were only exercised on the happy path, so nothing pinned our end-to-end
enforcement of expiry or query-signature verification.
Against a live single-node RustFSTestEnvironment (random port, isolated
temp dir), the suite asserts HTTP status + S3 error <Code> for:
- expired presigned GET -> 403 AccessDenied (Request has expired)
- tampered X-Amz-Signature -> 403 SignatureDoesNotMatch
- wrong secret key -> 403 SignatureDoesNotMatch
- tampered target key (swapped after signing) -> 403 SignatureDoesNotMatch
- tampered presigned PUT -> 403 SignatureDoesNotMatch + object not stored
- positive controls: valid presigned GET (body match) and PUT (HEAD verify)
Expiry is controlled without real waiting via the AWS SDK presigner's
start_time (sign as of one hour ago with a 60s window). s3s checks
expiry before the signature, so the expired case surfaces AccessDenied.
Wired into the e2e-smoke nextest profile (fast, single-node, no external
deps) so it runs on every PR per the crates/e2e_test/README.md admission
criteria.
Refs rustfs/backlog#1151 (sec-2), rustfs/backlog#1155.
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit 5c7c757a3 (#4712) activated the previously-dormant replication e2e
suite (they had never run anywhere). All 9 fast tests then failed on main
because the SSRF egress guard rejects the 127.0.0.1 targets the e2e harness
configures: `target endpoint is not allowed: outbound URL host '127.0.0.1'
is not allowed: loopback address`. The whole harness runs on loopback, so
every replication test hit this before reaching its actual assertion.
Loopback is a genuine SSRF vector and must stay rejected in production, so
this does not relax the guard. Instead `validate_replication_target_endpoint`
gains an off-by-default opt-in (`RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET`)
that re-enables loopback targets (127.0.0.1 / ::1 / localhost) for single-host
multi-instance dev and the e2e harness. Private addresses stay unconditionally
allowed as before; the opt-in does not widen into link-local or the cloud
metadata endpoint. The e2e harness sets the env for every server it spawns
(single-node and cluster paths), overridable via extra_env.
Verified end-to-end: all 9 previously-failing replication_extension_test
smoke tests pass against a locally built binary. New unit tests in
bucket_target_sys pin the matrix — public/private always allowed, loopback
gated on the opt-in in both IP and hostname forms, and metadata/link-local
still rejected even with the opt-in on.
Refs: backlog#1147
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(replication): rename optin -> opt_in to satisfy typos check
Pure rename of three unit-test function names; no behaviour change.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all
closed). Review of every migration-era test/gate measure found three things
actually retirable or broken — everything else is a live anti-regression
guard and stays.
Remove:
- scripts/check_metrics_migration_refs.sh — guards a migration that
finished: rustfs_metrics:: has zero hits, the metrics crate no longer
exists, and the script was never wired into CI or make (only reference
was one line in config-model-boundary-adr.md, also removed).
- crates/obs init_metrics_collectors — the "backward-compatible alias kept
during migration" the removed script was guarding. Zero callers; pure
delegate to init_metrics_runtime.
Archive (docs/superpowers/plans/, continuing the 2026-07 convention,
with the standard archived banner):
- startup-timeline.md, scheduler-baseline.md,
profiling-numa-capability-inventory.md,
kms-development-defaults-inventory.md — one-shot snapshots whose only
consumer is the already-archived migration-progress ledger (their
same-dir links there start resolving again after the move); zero script
pins; fed the closed backlog#660/#665 architecture-review ledger.
Fixed the one outbound link (startup-timeline -> readiness-matrix) that
the move would have broken — check_doc_paths.sh deliberately does not
scan plans/, so nothing else would have caught it.
Wire (found orphaned by the same review):
- scripts/check_extension_schema_boundaries.sh guards a live contract
crate but was never invoked anywhere. Add lint-fmt.mak target, include
in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job
already installs ripgrep), sync the CONTRIBUTING.md enumerated list,
and harden the script against a silently-passing rg probe when src/
is missing.
Keep (verified live, documented so the next cleanup pass does not repeat
this analysis):
- scripts/check_architecture_migration_rules.sh — added a header stating
it is a permanent boundary guard, not retirable migration scaffolding;
'migration' in the name is historical.
- check_migration_gate_count.sh + floor, delete-marker e2e proof, all
pinned docs, compat-cleanup-register sync, remaining inventories
(referenced by live docs).
Verification: all 7 guard scripts pass, actionlint clean,
cargo check --workspace (excl e2e) clean, cargo fmt --check clean.
Adversarially reviewed by two independent skeptic passes; their 7
findings (alias left behind, broken outbound link, missing banners,
wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing
header rationale) are all folded in.
ci: pull replication tests out of e2e-smoke — loopback targets fail every PR
repl-1 (#4712) added 20 bucket-replication admin-path tests to the
e2e-smoke PR lane. They set a remote replication target at a loopback
endpoint (127.0.0.1, a second local server), which RustFS's target SSRF
guard rejects by default ('outbound URL host 127.0.0.1 is not allowed:
loopback address'). repl-1's local verification was inconclusive under
machine load, so this shipped broken and failed End-to-End Tests on
every PR based on post-repl-1 main (#4707 etc).
Move all replication e2e tests to the e2e-repl-nightly lane (now the full
replication_extension_test set, no allowlist split) to un-block PRs. The
nightly job needs loopback-target-allow server config for these to pass;
tracked as a repl-1 follow-up (backlog#1147). e2e-smoke returns to the
17 functional modules from ci-4 (63 tests).
Adds an S3-API-level e2e regression net proving that a large-object GET on a
degraded EC set never returns a silently truncated body under a full
Content-Length -- the historical "unexpected EOF" bug fixed on main by
rustfs#4594 (short-body GetObject stream -> UnexpectedEof), rustfs#4560
(in-place per-part legacy degradation for the lazy multipart reader), and
rustfs#4585 (DARE package-boundary truncation detection). Those fixes each
ship a *unit* regression; this covers the layer they do not -- the full HTTP
GET path streaming a real body reconstructed from real on-disk EC shards.
New file crates/e2e_test/src/degraded_read_eof_regression_test.rs, single-node
4-disk (EC 2+2) DiskFaultHarness, three scenarios:
(a) one disk offline: a 6 MiB single object (>=2 EC stripes) and a 3x5 MiB
multipart object GET back byte-identical with the correct Content-Length.
(b) mid-stream bitrot within quorum: 2 of 4 shards corrupted mid-file on a
large multipart object still reconstructs the full, hash-matching body.
(c) beyond read quorum (the heart of the net): 3 of 4 shards corrupted
mid-file -- the read MUST fail cleanly (non-2xx or a mid-stream body
error), NEVER close with a truncated body under the full Content-Length.
The shared get_checked() helper panics on the forbidden outcome (a clean 2xx
whose collected body is shorter than the advertised Content-Length), so the
truncation bug can never be silently tolerated.
CI placement: these spawn a 4-disk server per test and are resource-heavy, so
they stay OUT of the fast PR e2e-smoke filter. A new e2e-reliability nextest
test-group (max-threads=1) serializes them (and the existing
reliability_disk_fault_test) across nextest's process boundary; ci-7's nightly
full-e2e run picks them up. Visible via `cargo nextest list -p e2e_test`.
Refs rustfs/backlog#1150 (dist-13), rustfs/backlog#1155, rustfs#4594,
rustfs#4560, rustfs#4585, rustfs#2955.
Activate the 36 dormant replication e2e tests in
crates/e2e_test/src/replication_extension_test.rs (zero ran anywhere before).
Split via the ci-4 nextest profile mechanism, no hand-rolled cargo-test lane:
- PR smoke (profile.e2e-smoke, existing e2e-tests job): the 20 fast
bucket-replication tests (target-registration / replication-check / list /
remove / delete admin paths) that validate config synchronously and never
wait for async convergence. Each spawns its own single-node rustfs server(s)
on random ports with isolated temp dirs, so parallel-safe by construction
(serial_test's #[serial] is a no-op under nextest's process-per-test model;
no test-group needed).
- Nightly (profile.e2e-repl-nightly + .github/workflows/e2e-replication-nightly.yml):
the remaining 16 = 6 slow data-plane tests + 9 _real_dual_node + 1
_real_single_node. Defined as 'replication module MINUS the PR allowlist' so
new replication tests default to nightly and are never silently unrun.
The nightly workflow builds the binary once, installs awscurl so the STS
dual-node test runs (skips gracefully with a visible log line otherwise), and
routes scheduled failures through .github/actions/schedule-failure-issue
(ci-8). Explicit division of labor with ci-5 e2e-full: these run only here.
Counts (cargo nextest list): e2e-smoke 83 (63 + 20), e2e-repl-nightly 16.
Docs updated: e2e-suite-inventory.md, e2e_test/README.md.
Refs backlog#1147 repl-1, backlog#1155.
ci: quarantine walk_dir stall-budget flake under the ci profile (rustfs#4690)
First application of the flake policy from docs/testing/README.md: the
test failed on a zero-Rust-diff PR (#4674, run 29099382470) — timing
windows in the stall-budget accounting stretch under CI load. retries=2
under profile.ci only; local default profile still never retries.
Tracked by rustfs#4690 (30-day fix-or-delete deadline).
* fix(obs): remove the dial9 task-dump switch that could never do anything
Measured on a bench host (Linux x86_64) against the code merged in #4663: with
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and
`--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events.
dial9 captures a task dump only for futures it wrapped itself — those spawned
through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets
applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn`
throughout. Same workload, same binary, only the spawner changed:
tokio::spawn -> 0 dumps
dial9::spawn -> 14709 dumps, all with callchains
Upstream documents this (README line 151) and tracks the doc gap at
dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663,
and so shipped exactly the kind of lying configuration knob that PR set out to
delete. Remove it: the two environment variables, the config fields, the
`with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was
to constrain the build to Linux while recording nothing.
Re-adding it only makes sense together with migrating the paths under
investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157.
Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured:
dumps are captured with and without it (14709 vs 14674, within noise), and
upstream never asked for it. That requirement was mine, invented and untested.
Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): replace guessed dial9 retention numbers with measured ones
Three corrections, all to claims I wrote in #4663 without measuring them.
"Under a high poll rate that budget can wrap in minutes" was a guess. Measured on
a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent):
13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108
minutes. Even at ten times the throughput that is ~11 minutes. State the measured
rate and how to scale it instead.
dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O
on the blocking pool and through io_uring, never on an async worker, so a slow
drive never lengthens a poll. Injecting 200 ms of latency on one of four drives
cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms:
49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers
does not help: sched events are per-worker only, and the CPU profiler samples
on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at
the `rustfs_io_*` metrics instead.
What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no
fault injected at all — real worker stalls nothing else in the obs stack surfaces.
Lead with that.
Also link the two upstream issues filed for the gaps we documented:
dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs).
Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
ci: run e2e smoke subset via nextest e2e-smoke profile (backlog#1149 ci-4)
The e2e-tests job previously ran a single test (delete_marker_migration
_semantics) out of ~400 in the e2e_test crate. Define a nextest
profile.e2e-smoke whose default-filter selects 17 fast single-node
dependency-free modules (63 tests) and run it in the job, reusing the
downloaded debug binary. The profile is the single wiring mechanism for
e2e tests in CI; admission criteria live in crates/e2e_test/README.md,
and docs/testing/e2e-suite-inventory.md records authoritative per-module
counts from cargo nextest list.
CI runs `cargo clippy --all-targets -- -D warnings` without
`--all-features`, but the Makefile's clippy-check used `--all-features`.
After PR #4663 made the dial9 telemetry feature opt-in and removed
`--cfg tokio_unstable` from .cargo/config.toml, `--all-features`
activated dial9 (which requires tokio_unstable) and dial9-taskdump
(which requires tokio/taskdump, Linux-only), breaking local
`make pre-pr` on macOS.
Align the Makefile with CI by dropping `--all-features` from clippy-check.
The migration-proof step in ci.yml selects tests by name substring
(data_movement / rebalance / decommission / source_cleanup /
delete_marker), so a rename can silently thin the gate to zero with no
CI signal. Move the filter expression into
scripts/check_migration_gate_count.sh as its single source of truth:
the script counts the selected tests with cargo nextest list, fails if
the count drops below the committed floor in
.config/migration-gate-floor.txt, and then runs the gate with the same
filter so the check and the run cannot drift.
The floor equals the exact selected-test count at landing (571), so any
reduction fails CI and intentional shrinks must edit the floor file in
the same PR, keeping gate changes visible in diffs.
Refs rustfs/backlog#1153 (infra-12), rustfs/backlog#1155.
Signed-off-by: Zhengchao An <anzhengchao@gmail.com>
build(make): require cargo-nextest for make test with explicit escape hatch (backlog#1153 infra-14)
cargo-nextest changes test semantics — it runs each test in its own process
(so serial_test's #[serial] mutex does not serialize across tests) and is the
only runner that honours .config/nextest.toml [test-groups] (e.g. the
ecstore-serial-flaky serialization guard). CI installs and runs nextest, but
`make test` only warned when it was missing and then silently fell back to
`cargo test`, which runs with different serialization behaviour than CI and can
mask or invent flakes.
Make cargo-nextest a hard dependency of `make test`:
- Missing nextest now fails `make test` with a clear message and install
instructions (`cargo install cargo-nextest --locked` or a prebuilt binary
via https://nexte.st/docs/installation/).
- Set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to opt into the plain `cargo test`
fallback anyway; it prints a loud warning that serialization semantics differ
from CI and .config/nextest.toml test-groups will NOT apply.
- The check stays cheap (command -v). The warn-only test-deps prerequisite is
dropped from check.mak in favour of recipe-level gating.
Install docs live in the make-layer error message plus a header comment in
tests.mak, with a single-line note in CONTRIBUTING.md (kept minimal to avoid
conflicting with #4667 / infra-9, which rewrites the verification sections).
No CI change: .github/workflows/ci.yml already runs cargo nextest and
.github/actions/setup/action.yml already installs it.
Refs rustfs/backlog#1153 (infra-14), master plan #1155.
Signed-off-by: Zhengchao An <anzhengchao@gmail.com>
test(ci): add strict nextest ci profile with quarantine + flake policy
Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate
(ci-10, absorbs infra-15; backlog#1149).
- .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a
new race's first occurrence), fail-fast=false, and JUnit output at
target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get
retries=2 under the ci profile only; each entry links one OPEN issue. First
members are the two backlog#937 ecstore groups
(concurrent_resend_same_part_commits_one_generation and
store::bucket::tests::bucket_delete_*), which keep their existing
ecstore-serial-flaky test-group serialization. Local default profile still
never retries.
- .github/workflows/ci.yml: run the main test step with --profile ci and upload
the JUnit report (if: always(), 3-day retention, run-number in name). The
migration-proof step stays on the default profile to avoid clobbering the ci
JUnit artifact (its tests are not quarantined).
- docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding
the flake policy: discover -> open issue within 24h -> quarantine with issue
link -> fix or delete within 30 days. AGENTS.md points to it.
Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Two ecstore test groups pass in isolation but flake under the full parallel
nextest suite, producing CI noise on every in-flight PR (backlog #937):
- store::bucket::tests::bucket_delete_* race make_bucket into
InsufficientWriteQuorum because they share global disk-registry/lock-client
state with other concurrently running tests.
- bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
exceeds its already-maxed 60s lock-acquire deadline only when the suite
saturates disk I/O.
serial_test's #[serial] does not serialize these across nextest's per-test
process boundary. Add a nextest test-group (max-threads = 1) that matches both
groups so they run serialized, removing the load-induced flake. No test or
production code changes; long-term global-singleton isolation is tracked
separately.
Co-authored-by: heihutu <heihutu@gmail.com>
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
Agent-instruction and architecture docs had drifted from the code:
- CLAUDE.md: slim to commands + pointers; fix wrong claim that
`make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
hand-maintained (already stale) scoped-AGENTS index; link
architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
(rebalance-decommission phases, migration-progress ledger, PR
template) to docs/superpowers/plans with archive headers; fix stale
source paths in kept inventories (core/sets.rs, core/pools.rs,
store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
.gitignore now keeps .claude/skills and docs/operations committable
Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(tier): stop sending nil/garbage versionId to warm backend S3
Three bugs caused NoSuchVersion errors when reading tiered objects:
1. warm_backend_s3sdk: GET and DELETE ignored rv/range opts entirely —
fixed to forward version_id and byte-range to the SDK request.
2. version.rs (MetaObject + MetaDeleteMarker): transition_version_id was
parsed with unwrap_or_default(), turning invalid/wrong-length bytes
into Uuid::nil(). The nil UUID was then serialized and sent as
?versionId=00000000-... to the tier backend -> NoSuchVersion.
Fixed: .and_then(.ok()).filter(!is_nil()) so only valid non-nil UUIDs
are forwarded as versionId.
3. bucket_lifecycle_ops: add debug/error logs in
get_transitioned_object_reader to record tier, tier_object, and
tier_version_id before and on failure of the tier GET.
Also adds tier transition fields to dump_fileinfo example for offline
xl.meta inspection, and fixes Docker build (cargo path + entrypoint).
Adds CLAUDE.md with tier architecture and debugging notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* more fixes for versionId
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>
* remove branch
* Add tests and fix cargo path, add load to build-docker
* update documentation (CLAUDE.md)
* more fixes for recover
* More fixes to ILM recover
* final fix
* chore: add missing-shard first-scene diagnostics (#3213)
chore(ecstore): add missing-shard first-scene diagnostics
Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.
Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.
* fix test anmd fmt
* fix cargo path
fix test
* fix(tier): format copy_object self-copy guard
---------
Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Zhengchao An