mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-27 06:18:43 +00:00
Hauser
fea5a584c3
* perf(ilm): back off idle tier-delete journal recovery Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reconcile interrupted pool metadata publication Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback. Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): reuse proven pool metadata durability Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root. Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(multipart): exercise selective RPC delay and peer loss Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets. Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs: define multipart delay gates and receipt invariants Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(readiness): expose missing runtime drive slots Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retry committed promotion convergence safely Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement. Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): guard native metadata mutation aliases Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains. Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): register Darwin multipart RPC delay gates Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(multipart): fence native publication and recovery Publish multipart data through private hardlinks and atomically consume the prepared source head. Bind exact compensation capabilities and native recovery to the physical cohort, source generation and durable publication state. Require fresh terminal confirmation for native replay and preserve indeterminate data when no safe recovery witness exists. Release logical guards only after validated irreversible quorum success while retaining physical mutation leases, accounting and resource ownership through actual callbacks. Preserve local instance proof authority and historical Copy metadata without granting healed donors fresh publication authority. Keep general movement identity strict while preserving directory capacity compatibility in a separate checked comparison. Cover native fault and cancellation boundaries, zero-byte objects, real Copy, heal and migration, exact physical readback and distributed RPC delay. Retain all quorum and durability requirements and document rollout limitations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): fix ratchet scan and Windows fault hooks Move native pool metadata inventory tests after production items so the error-format ratchet sees the existing call site. Preserve the baseline instead of accepting a false count decrease. Make the directory-barrier failure hook available on every test platform and exercise the parent-barrier failure/retry cases in Windows CI. Keep production synchronization behavior and all existing assertions intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(protos): remove redundant recovery request clone Move the baseline into its final struct update while retaining the earlier clone needed to construct the first request variant. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align RPC delay gates with native abort * fix(storage): validate native source recovery and CI gates Preserve wire layouts while bounding native response sizes and fix the last-use recovery request clone. Avoid destination replay reads only for fully observed initial native sources; keep upgraded and recovered sources conservative. Cover real legacy publication after a cold restart. Align upgrade acceptance with the native mixed-version mutation pause and use exact upload-scoped test completion evidence before advancing time. Keep all original quorum, ownership, byte and deadline assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): verify native abort under partial admission Delay native cleanup at the exact upload root after all-participant closure. Verify quorum-only admissions retain every source byte on refused Abort and complete the same upload without creating a missing original participant. Keep sub-quorum orphans and their discoverable identities isolated while a fresh all-participant upload proves service recovery. Route shared constants through the existing E2E storage facade. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): avoid formatting a fixed invalid upload path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align native MPU fault recovery assertions Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): keep MPU tail deadlines on the wall clock Global Tokio time advancement can expire drive-health probes while real filesystem callbacks are pending. Use a short real drive deadline with the existing physical-entry and cleanup barriers, retaining the byte, lock, resource-permit and final-shard assertions. Fixes #8937 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): align native fixtures with instance ownership Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native multipart future construction Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): assert coordinator journal scope after rejoin Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): bound object repair future construction Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): qualify fixture disk identity checks Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native inspection from mutation state Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): pause abort time after native cleanup Wait for durable source deletion on the wall clock before advancing the capacity lease clock at the post-delete barrier. Paused time can otherwise expire the barrier while native filesystem cleanup is still pending. Retain lease-loss injection, durable inflight-byte accounting and restart reconciliation assertions. Document the test-util feature required to select the native decommission fixtures in package-scoped regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim data from owned multipart undo Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve unformatted bootstrap routes Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): bound CopyObject tracing poll frames Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): observe recovery after direct legacy bootstrap Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): prepare the homogeneous previous-release fleet Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets. Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): isolate release compatibility from fault profiles Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): use normal shutdown for planned release replacements Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * revert: retain the verified release compatibility fixture Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(copy): bound inherent handler polling stack Build the inherent CopyObject future through a synchronous noinline factory at the existing trait await. This keeps its large state out of the trait wrapper's active polling frame during historical self-copy. Preserve cancellation, tracing and post-write notifications, with one additional allocation restricted to COPY. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): validate authoritative quota and native pool heals Require actual authoritative quota usage in the existing readiness loop instead of treating HTTP success or unknown usage as a numeric baseline. Keep the original quota assertions and convergence budgets. Allow the owning pool metadata CAS to republish into a fresh valid data directory while retaining exact version, layout, completeness and shard fingerprints. Require every node's real metadata/read/write readiness before accepting recovery. Keep ordinary object manifests and the original 180-second recovery deadline unchanged, with negative oracle regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): distinguish certified native Abort recovery Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com>
RustFS Testing
Use this when: you need to pick a test layer for a change, name a test so a gate keeps selecting it, understand why #[serial] does nothing under nextest, or handle a flaky test.
Source of truth: .config/nextest.toml (profiles, test-groups, quarantine), .config/make/tests.mak (make test), .github/workflows/*.yml (what runs when; matrix in ci-gates.md).
Test taxonomy
Pick the lowest layer that can prove the change; add a higher-layer test only when the behaviour is not observable below it.
| Layer | What it covers | Entry command | When it runs (details: ci-gates.md) |
|---|---|---|---|
| Unit & crate integration | Per-crate logic and in-process integration tests | cargo nextest run --all --exclude e2e_test (or -p <crate>); make test wraps it |
Every PR, required (Test and Lint, ci profile) |
| ecstore black-box | Erasure-coded read/write/recovery validation; profiles quick / full / destructive / fuzz |
scripts/run_ecstore_validation_suite.sh --profile quick |
Local and release validation only; not wired into any workflow. Contract: ecstore-validation-suite-design.md |
e2e (e2e_test crate) |
A real rustfs binary per test, driven over the S3, admin, and protocol APIs |
python3 scripts/e2e_binary.py build --features e2e-test-hooks, then python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test |
PR: e2e-smoke (report-only); merge queue / main push: e2e-full; nightly: e2e-repl-nightly, e2e-nightly, e2e-protocols, e2e-distributed. Guide: crates/e2e_test/README.md; 4-node 4-disk map: distributed-e2e.md |
| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart, checksummed, SSE-S3) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | python3 scripts/e2e_binary.py build, then python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)' |
With e2e-repl-nightly; required locally for any change to outbound client defaults (SOP: docs/postmortems/2026-09-03-replication-checksum-default-regression.md) |
| s3s-e2e conformance | External S3 conformance tool against a live server | ./scripts/e2e-run.sh ./target/debug/rustfs <data-dir> |
PR, report-only (second half of the End-to-End Tests job) |
| S3 compatibility | ceph/s3-tests (boto3; allow-list scripts/s3-tests/implemented_tests.txt) and MinIO mint |
scripts/s3-tests/run.sh; mint via .github/workflows/mint.yml |
s3-tests: PR report-only plus a weekly full sweep; mint: weekly, report-only |
| Chaos / fault-injection | Single-node disk fault injection (crates/e2e_test/src/chaos.rs, crates/e2e_test/src/fault_proxy.rs) plus the 4-node kill/fresh-drive/blackhole cases in crates/e2e_test/src/distributed/chaos_test.rs |
Part of the e2e crate (e2e-reliability and e2e-distributed) |
Reliability cases with e2e-full; 4-node chaos on storage-sensitive PRs and nightly via e2e-distributed |
| Fuzz | cargo-fuzz targets over untrusted parsing surfaces; isolated sub-workspace under fuzz/ |
./scripts/fuzz/run.sh (see fuzz/README.md) |
PR smoke on the paths listed in .github/workflows/fuzz.yml, plus nightly corpus |
| Benchmarks | Criterion benches under each crate's benches/ |
cargo bench -p <crate> |
On demand; never a gate |
| Conditional reads | Paired ordinary/hit/miss GET latency, throughput and read-lock measurements | scripts/run_conditional_read_abba.py |
Local conditional-read acceptance; guide and scope |
Every script named above is indexed with status and wiring in scripts/README.md. Fixed GHSA advisories map to named regression tests in security-regressions.md.
The ECStore core invariant gates map required native commit, rollback, lock, read, multipart and metadata regressions to the existing CI run and distinguish them from process and network fault evidence.
The scanner checkpoint fixture diagnoses retained subtree coverage across budget interruption, persistence, reload, and plan invalidation.
The scanner cache cost profile separates clone, subtree copy, encoding, and counted save costs without changing production cache behavior.
The Pool layout compatibility reference defines the topology and EC regression matrix for single-drive, single-node multi-drive, and multi-node expansion pools.
The Scanner/Heal topology protection gates map G-EXP, G-REB and G-DEC and their fault windows to the scanner_heal_topology_ cases, their recorded inventory, and the basis of their timeouts.
The distributed multipart RPC delay gates define authenticated method-entry barriers, latency and peer-loss oracles, and the boundary between hook evidence and production incident attribution.
Naming conventions
Reserved test-name substrings (migration gate)
scripts/check_migration_gate_count.sh (runs in Test and Lint) selects migration-critical tests by name substring and fails when the count drops below .config/migration-gate-floor.txt. A rename that drops a substring silently thins the gate, so these substrings are reserved:
| Substring | Guards |
|---|---|
data_movement |
Cross-pool / cross-set object data-movement proofs |
rebalance |
Pool rebalance correctness |
decommission |
Pool decommission correctness |
source_cleanup |
Post-migration source cleanup |
delete_marker |
Delete-marker handling across migration |
A deliberate reduction lowers the floor in the same PR. The list above mirrors the script; change both together.
General naming
- Name a regression test after what it pins (issue or advisory number, or the invariant) so
rgfinds the guard for a past bug. - e2e lane membership is selected by test-name patterns in
.config/nextest.toml(for example_real_dual_node/_real_single_noderoute replication tests to the nightly lane). Follow the existing marker of the suite you extend. - Symbol naming follows the Rust API Guidelines (see
AGENTS.md).
nextest and #[serial]
cargo-nextest is the runner: make test requires it and CI installs it. nextest runs every test in its own process, so serial_test's in-process #[serial] mutex does not serialize tests against each other; it only affects the plain cargo test fallback. Cross-test serialization under nextest comes from a [test-groups] entry with max-threads = 1 in .config/nextest.toml (for example ecstore-serial-flaky, e2e-reliability) or from a -j 1 lane. Prefer making tests self-isolating (per-test instance context, random port, own temp dir) over adding serialization. RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test runs plain cargo test; its results are not authoritative because [test-groups] do not apply.
Package-scoped ECStore runs do not inherit workspace feature unification. The native decommission lock, capacity-lease and restart fixtures in core::pools_test::decommission_lock_order_tests require test-util. Select that module separately and fail an empty run, so unrelated tests matching another branch of a combined filter cannot hide its absence:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features test-util \
--no-tests fail -E 'test(/^core::pools_test::decommission_lock_order_tests::/)'
Time-driven tests use paused tokio time (start_paused plus tokio::time::advance) or explicit synchronization instead of fixed sleep windows.
Profiles
All profiles are defined in .config/nextest.toml; its block comments hold the filters and rationale.
| Profile | Role |
|---|---|
default |
Local runs; never retries |
ci |
PR gate for everything except e2e_test; global retries = 0 plus the quarantine list |
e2e-smoke |
PR subset of e2e_test |
e2e-full |
Merge-queue / main-push single-node e2e lane |
e2e-repl-nightly |
Nightly slow / cross-process replication lane |
e2e-nightly |
Nightly serial multi-process cluster fault lane |
e2e-distributed |
Storage-sensitive PR and nightly 4-node 4-disk S3 / lock / versioning / replication / quota / expand / decommission / rebalance / site-replication / chaos / upgrade (history + IAM AK/SK) lane |
e2e-protocols |
Nightly fixed-port FTPS/SFTP/WebDAV lane, run with -j 1 |
Membership of each e2e profile is pinned by a digest in .config/e2e-<profile>-selection.txt and checked by scripts/check_test_wiring.py --check-profile <profile> before the lane runs. To list what a profile selects on your platform (the result is platform-dependent because some modules are linux-only):
cargo nextest list -p e2e_test --profile e2e-smoke --message-format json \
| jq -r '.["rust-suites"][].testcases | to_entries[] | select(.value["filter-match"].status == "matches") | .key | split("::")[0]' \
| sort | uniq -c
Flake policy
A flaky test fails non-deterministically without a corresponding code change. Retry semantics live in .config/nextest.toml: default never retries, ci has global retries = 0, and only quarantined tests get retries = 2 under ci. A quarantined test that passes on retry is marked flaky in target/nextest/ci/junit.xml (uploaded as a CI artifact); that marker, not a green check, is how a live flake stays visible.
- Discover — a non-deterministic failure (CI or local) or a
flakyJUnit marker. - Open an issue within 24h describing symptom, suspected cause, and affected suite. No silent re-runs.
- Quarantine — add a
[[profile.ci.overrides]]entry withretries = 2and a comment linking exactly one OPEN issue. The current quarantine list is the[[profile.ci.overrides]]block in.config/nextest.toml. - Fix or delete within 30 days — make the test robust and remove the entry, or delete the test. An entry without a live OPEN issue link is a policy violation.
Coverage
.github/workflows/coverage.ymlmeasures workspace line coverage on its schedule and on manual dispatch:cargo llvm-cov nextest --workspace --exclude e2e_testunder theciprofile, the same scope as the PR gate. The per-crate table lands in the job summary; lcov and JSON exports are uploaded as an artifact (retention set in the workflow).- PRs touching the paths listed in
coverage.ymlalso run a report-only comparison against.config/coverage-baselines.tomlviascripts/check_security_coverage.py: a regression is recorded in the summary without failing the job; missing or malformed coverage evidence fails closed. make coverage(.config/make/coverage.mak) is the local equivalent; it writestarget/llvm-cov/lcov.infoandcoverage.jsonand prints the same table viascripts/coverage_per_crate.py.- Not measured: doctests (
ci.ymlruns them uninstrumented) and thee2e_testcrate. - A baseline change needs a linked coverage run and a reviewed explanation in the PR.
- For requested coverage assessments, reuse a report only when its revision, scope, and format match. If measurement is necessary, check disk/tool availability and use package/test-scoped
cargo llvm-covwith the repository nextest configuration; whole-workspace measurement requires that requested scope. - Missing branch/changed-line metrics are unknown, not zero. If measurement cannot run, mark coverage unverified and propose tests from inspected code without invented percentages.
- Rank evidenced gaps by changed behavior, data loss/security/compatibility risk, and uncovered branches. Pair each gap with a focused test, failure criterion, and scoped verification; do not introduce a threshold unless required.