Files
rustfs/docs/testing/ci-gates.md
T
houseme 5db0e14d04 test(scanner): preflight G09 evidence disk space (#7527)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* fix(targets): reject trailing batch items (#7508)

* test(scanner): preflight G09 evidence disk space

Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-08 23:58:08 +08:00

30 KiB

CI gate matrix

Use this when: a check is red and you need to know whether it blocks the merge, which workflow and job produced it, and how to reproduce it locally. Source of truth: the live main ruleset (command below) for required status; .github/workflows/<file>.yml for triggers, paths, timeout-minutes, and cron; .config/nextest.toml for e2e profile filters; .github/scheduled-validations.json for the freshness-watchdog list.

A job blocks a merge only when its exact check name is in the live main ruleset. A workflow name, a merge_group trigger, or a red PR check does not make a job required by itself.

Required merge checks

The main ruleset (6436880) requires exactly these contexts, with strict_required_status_checks_policy=false:

Required context Producer Validation
CLA Check cla.yml Contributor agreement
Quick Checks ci.yml job quick-checks Formatting and repository guard scripts
Test and Lint ci.yml job test-and-lint Clippy, workspace nextest (ci profile, excluding e2e_test), doctests, migration-gate count (scripts/check_migration_gate_count.sh)

For PRs limited to the paths-ignore list in ci.yml, ci-docs-only.yml reports Quick Checks and Test and Lint under the same names; it runs the quick checks and scripts/check_no_planning_docs.sh, not a Rust build or tests. scripts/check_ci_paths_sync.sh keeps the two path lists aligned.

Verify the live rule before changing merge policy:

gh api repos/rustfs/rustfs/rulesets/6436880 \
  --jq '.rules[] | select(.type == "required_status_checks") | .parameters'

Promotion rule: never promote a report-only lane to required from one green run. Require at least 14 days and 30 representative PRs with at least 99% complete execution, then update the ruleset and this file together.

Pull request and merge matrix

"Report-only" means visible and actionable but not in the required list. Budgets are each job's timeout-minutes in the named workflow and are not copied here.

Event Check name Workflow / job Merge status Reproduce
PR, non-doc change Quick Checks ci.yml quick-checks Required make pre-commit
PR, non-doc change Test and Lint ci.yml test-and-lint Required cargo clippy --all-targets -- -D warnings; cargo nextest run --profile ci --all --exclude e2e_test; cargo test --all --doc; scripts/check_migration_gate_count.sh
PR, non-doc change Typos ci.yml typos Report-only typos
PR, non-doc change ILM Integration (serial) ci.yml test-ilm-integration-serial Report-only exact command in the job
PR, non-doc change Test and Lint (rio-v2), Test and Lint (swift), Test and Lint (sftp) ci.yml test-and-lint-rio-v2, test-and-lint-protocols Report-only cargo nextest run with the job's --features
PR, non-doc change Connect Short Credential Boundary ci.yml connect-short-credential-boundary Report-only cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials; cargo check -p rustfs --release --features connect-e2e-short-credentials must fail
PR, non-doc change Build RustFS Debug Binary ci.yml build-rustfs-debug-binary Report-only; prerequisite for the black-box jobs cargo build -p rustfs --bins
PR, non-doc change io_uring Integration (real) ci.yml uring-integration Report-only cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture
PR, non-doc change End-to-End Tests ci.yml e2e-tests Report-only cargo nextest run --profile e2e-smoke -p e2e_test, then ./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>; membership guards scripts/check_test_wiring.py --check-profile e2e-smoke <listing.json> and scripts/check_security_smoke_count.sh check <listing.json>
PR, non-doc change S3 Implemented Tests ci.yml s3-implemented-tests Report-only build rustfs, then scripts/s3-tests/run.sh with the job's DEPLOY_MODE / TEST_MODE / MAXFAIL env
PR, non-doc change S3 Lifecycle Behavior Tests ci.yml s3-lifecycle-behavior-tests Report-only scripts/s3-tests/run.sh with the job's accelerated-scanner env
PR touching paths in audit.yml Cargo Deny, Workflow Pin Report, Dependency Review audit.yml cargo-deny, workflow-pin-report, dependency-review Report-only cargo deny check; scripts/security/check_workflow_pins.sh
PR touching paths in architecture-migration-rules.yml Architecture Migration Rules architecture-migration-rules.yml architecture-migration-rules Report-only scripts/check_architecture_migration_rules.sh
PR touching paths in nix.yml Nix Build & Check nix.yml nix-validation Report-only nix flake check
PR touching paths in fuzz.yml Build Fuzz Harness, Smoke / <target> fuzz.yml fuzz-build, pr-fuzz-smoke Report-only MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh
PR touching paths in windows-filesystem.yml Rename Safety windows-filesystem.yml rename-safety Report-only the cargo test -p rustfs-ecstore --lib <filter> commands in the job, on Windows
PR touching paths in coverage.yml Workspace line coverage coverage.yml coverage Report-only make coverage; python3 scripts/check_security_coverage.py target/llvm-cov/coverage.json
PR touching paths in e2e-upgrade.yml Direct upgrade from the previous release, Mixed-version rolling upgrade from the previous release, Bucket configuration survives the upgrade, Rollback reads current bucket metadata e2e-upgrade.yml upgrade matrix Report-only the cargo test --locked -p e2e_test command in the job with RUSTFS_UPGRADE_SOURCE_BINARY pointing at the pinned previous release (UPGRADE_SOURCE_VERSION)
PR touching paths in oidc-keycloak.yml OIDC Keycloak live gate oidc-keycloak.yml oidc-keycloak-live Report-only cargo build --locked -p rustfs --bin rustfs, then bash scripts/test/oidc_keycloak_live.sh ./target/debug/rustfs
PR touching paths in targets-integration.yml PostgreSQL, MySQL, AMQP, and NATS targets-integration.yml targets-live Report-only start the containers as in the job, export the RUSTFS_TEST_* DSNs, then the job's cargo test --locked -p rustfs-targets --test <name> -- --ignored --test-threads=1 commands
PR limited to main-CI-excluded paths Quick Checks, Test and Lint ci-docs-only.yml quick-checks, test-and-lint Required git diff --check; make doc-paths-check; scripts/check_no_planning_docs.sh
merge_group; push to main End-to-End Tests (full merge gate) ci.yml e2e-full Report-only cargo nextest run --profile e2e-full -p e2e_test

e2e filters live in .config/nextest.toml; extend a profile instead of adding a second selector. Before a profile runs, scripts/check_test_wiring.py compares its listing to the committed digest in .config/e2e-<profile>-selection.txt, so a silent test drop fails closed.

Scanner usage and heal rebuild coverage are intentionally split by risk and cost. data_usage_test runs in the PR e2e-smoke lane so changes that affect authoritative scanner usage publication, quota-visible usage, or admin usage snapshots get an end-to-end signal before merge review. heal_erasure_disk_rebuild_test runs in e2e-full so core erasure heal rebuild regressions are caught no later than the merge queue or main push lane; it also remains in e2e-nightly with the serialized cluster fault-domain suites for scheduled soak signal.

Scheduled validation

Scheduled lanes never block a PR. Their workflow-local gate fails the run, scheduled failures route to the shared failure-issue action, and scheduled-validation-freshness.yml fails when a workflow listed in .github/scheduled-validations.json has not run within its max_age_hours (a never_ran_grace_until entry covers the window before a newly enabled cron's first slot). Cadence is qualitative here; the cron lives in each workflow's on.schedule.

Workflow (cadence) Jobs Verdict and artifacts In freshness list Reproduce
ci.yml (weekly) full matrix, including the schedule/dispatch-only rio-v2 jobs build-rustfs-debug-binary-rio-v2 and e2e-tests-rio-v2 per-job yes dispatch ci.yml
build.yml (weekly) build-rustfs over the six-target platform matrix in prepare-platform-matrix (four Linux, macOS aarch64, Windows x86_64) build/package integrity yes dispatch build.yml with an exact platform set
e2e-replication-nightly.yml (nightly) repl-nightly, cluster-nightly, protocols-nightly three independent gates; JUnit, membership listing, server logs yes cargo nextest run --profile e2e-repl-nightly -p e2e_test; --profile e2e-nightly; -j 1 --profile e2e-protocols
e2e-distributed.yml (storage-sensitive PRs + nightly) distributed fail-closed 4-node 4-disk S3, durability, replication, movement, fault, and direct/rolling upgrade gate; JUnit, membership listing, per-node server logs yes, with never_ran_grace_until download the pinned previous release as in the workflow, export RUSTFS_UPGRADE_SOURCE_BINARY, then cargo nextest run --profile e2e-distributed -p e2e_test
e2e-s3tests.yml (weekly) s3tests (single and distributed, four shards each), upstream-head-canary compatibility gate; report, JUnit, node IDs, server logs yes scripts/s3-tests/run.sh against an existing single or distributed target
fuzz.yml (nightly) nightly-fuzz-corpus per target gate; corpus and crash artifacts yes MAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh
minio-interop.yml (nightly) minio-interop EC + SSE read-parity gate yes, with never_ran_grace_until pinned Docker fixture steps in the workflow
on-demand-migration-interop.yml (nightly) minio-source, cloud-source (aws, r2, gcs) report-only provider interop; one JSON report per provider naming cases, timings and source request counts, plus JUnit and MinIO logs. A cloud provider whose ODM_INTEROP_* secrets are absent is skipped with a summary note, not failed no start the pinned MinIO container as in the job, export the RUSTFS_ODM_INTEROP_* variables, then cargo nextest run --profile e2e-odm-interop -p e2e_test
performance-ab.yml (nightly) warp-ab regression-budget gate; A/B summaries and server logs yes bash scripts/run_hotpath_warp_abba.sh --help
nightly-gnu.yml (nightly) build, kms-vault-lane, kms-vault-ha-failover build, live Vault, and HA failover gates yes commands and pinned Vault images in the workflow
audit.yml (nightly) cargo-deny, workflow-pin-report dependency and workflow-pin gates yes cargo deny check; scripts/security/check_workflow_pins.sh
mint.yml (weekly) mint report-only by design; per-suite PASS/FAIL/NA and raw log.json yes pinned Docker sequence in the workflow
coverage.yml (weekly) coverage report-only trend; lcov and JSON artifact yes make coverage
runner-hygiene.yml (monthly) check-ephemerality runner ephemerality yes dispatch
e2e-upgrade.yml (weekly) upgrade (4-case matrix) upgrade and rollback gate; server logs no see the PR row
oidc-keycloak.yml (weekly) oidc-keycloak-live live OIDC gate no see the PR row
targets-integration.yml (nightly) targets-live live target gate; container logs no see the PR row
scheduled-validation-freshness.yml (nightly) check-freshness fails on a never-created or stale schedule n/a dispatch

Manual workflow_dispatch runs are debugging evidence and do not open scheduled-failure issues. A manual performance run may explicitly allow a known regression; that override is not a passing baseline.

Packaged functional acceptance

rustfs-functional-chain.yml dispatches the packaged-build suites in rustfs-*-test.yml on the shared lab runners. A failing suite step or job must fail its workflow. Report collection, cleanup, and dispatch of the next suite can still run with always(); continuing diagnostics does not make the failed suite successful.

Workflow status preserves errors that the test scripts report. It does not establish complete execution or a common package identity across the chain: inspect the current run's case results, package identity, and test-script revision as well. A script that returns zero after a failed tool invocation needs its own result check.

Release validation

Post-merge and tag-driven; not a substitute for a PR gate.

Trigger Workflow / job Result
Push to main, weekly schedule, dispatch build.yml build-rustfs (a development build on a main push restricts the matrix to the Linux targets) build artifacts; no release publication
Valid release or preview tag build.yml build-rustfs, create-release, upload-release-assets, publish-release draft release, checksummed assets, publish
Successful non-preview release-tag build (workflow_run) docker.yml build-docker, scan-docker-image multi-architecture images and vulnerability report
Successful release-tag build (workflow_run) package.yml package DEB/RPM packages and checksums uploaded to the release
Successful non-preview release-tag build (workflow_run) helm-package.yml build-helm-package, publish-helm-package versioned chart and repository index
Final tag's release published build.yml cleanup-preview-releases deletes every <target>-preview.<N> Release for that target; the tags are kept

Use an exact preview tag for an end-to-end release rehearsal. Manual dispatches are backfill/debug paths and do not prove the automatic workflow_run chain.

Change checklist

Update this file in the same PR when a job or check name changes, a workflow gains or loses a pull_request or schedule trigger, required contexts or strict/merge-queue policy change, report-only vs gating semantics change, or .github/scheduled-validations.json membership changes. Do not copy timeouts, crons, or test counts here.

ECStore invariant selection

The existing ci.yml test-and-lint job runs the ordinary ECStore and filemeta tests. After that run, scripts/check_test_wiring.py --check-core checks the same nextest profile and package selection against .config/ecstore-required-tests.json. Every named test must exist, match the filter, and be non-ignored; the job also requires a nonempty JUnit report. This checks membership without running the tests twice. core-test-listing.json, JUnit, and the run log are retained in the existing test-and-lint artifact.

The manifest records a minimum set of invariants: write quorum, metadata rollback, stale-writer lock loss, plaintext Range content, multipart cancellation, hiding uncommitted LIST versions, real MinIO metadata, corrupt part arrays, and the shared on-demand-migration source-backend contract for each provider dialect (S3, Azure, native GCS). The three contract entries live in the rustfs suite and reach the lane through that package's default features, so dropping gcs from rustfs's defaults fails this check instead of silently deselecting the GCS contract (rustfs/backlog#2323). Renaming or moving a required test must update the manifest in the same change after checking the compiled listing. Extend this list as new deterministic regressions land; it is not a claim that all storage invariants are covered.

The checked-in MinIO corpus is pinned by file SHA256 and its documented source release. The static wiring guard and the CI selection check both reject missing or changed fixtures. These are metadata fixtures, not a legacy shard-body corpus or proof of crash durability. Optional legacy_bitrot_read_test runs may still skip when their external corpus is absent; they do not satisfy a required compatibility lane. Real encrypted fixture reads remain in minio-interop.yml, and multi-node fault schedules remain in the existing nightly cluster lane. In-process reopen tests do not establish power-loss durability.

Run python3 scripts/check_test_wiring.py --self-test to exercise the negative cases: removed/ignored/filtered tests, malformed listing, absent fixtures, and wrong fixture hashes. Do not update hashes merely to silence the guard; a fixture change needs source/provenance and compatibility review.

Scanner/Heal Evidence Receipts

The existing scripts/check_test_wiring.py also validates Scanner/Heal case evidence registered in .config/scanner-heal-required-tests.json. It records already-built binaries and checks existing nextest output; it does not build, run tests, deploy servers, inject faults, or start another CI lane.

The initial case is background-target-restart, emitted by heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_remote_shards_after_background_target_restart. That test already runs in e2e-nightly. When RUSTFS_SCANNER_HEAL_RUN_DIR is set, it checks the actual server and test-executable hashes against run.json, pins the same server binary for all node starts, and writes its oracle only after the real assertions pass. The artifact contains the actual pre/post target PIDs, per-node S3 listings, expected and downloaded complete-body hashes/lengths, and target-disk VersionShardCensus fingerprints. Existing baseline objects must match their pre-fault physical manifests; the object created during the outage has no pre-fault target shard and is checked for complete physical parts and exact S3 content.

This case is a four-node, one-drive-per-node process-restart test. It is not power-loss validation, a 3x4 EC8+4 experiment, an all-version inventory, or proof of scanner enumeration, exact MRF disposition, legacy migration, or rollback. The schema 2 registry separates the implemented single-set restart lane from structured release lanes for authority coverage, checkpoint/crash, status and outcome, MRF responsibility, mixed-version rollback, scheduler pressure, maintenance producers, and EC8+4 multi-set coverage. All G01-G14/P1-P4 and R-E/R-D/R-L release requirements stay pending until their actual feature-specific oracles, measurements and required topologies exist. Missing cases cannot be supplied by synthetic W20 results. W20's bounded JSON and file-hash helpers are reused; its ABBA performance contracts remain in docs/operations/scanner-benchmark-runbook.md. Measured ABBA manifests must also carry the runbook's release_evidence contract. The runner rejects reports that cannot bind the exact 3x4 EC8+4 topology, multi-pool/multi-set shape, distributed same-window metrics endpoints, restart/crash modes, mixed-version reader/writer/rollback participation, and allocation/flamegraph/RSS/save-frequency profile artifact plan. Synthetic runs and manifests missing that contract remain harness-only evidence.

Recording One Case

Use a committed source tree, independently built current binaries, sufficient free disk space, and a task-owned artifact directory that does not yet exist. Set SERVER_BINARY and TEST_BINARY to those exact executable paths. The begin command requires the server's embedded --version commit to match the clean checkout and its embedded Git status to be clean. The E2E crate's build script embeds its build-time Git revision/dirty state, lockfile Git blob, enabled crate features, target, profile and encoded Rust flags. It tracks the crate/dependency trees, Cargo inputs and Git HEAD/ref/index, including common.rs restart logic. The producer checks this compiled identity against the receipt; it does not copy a current source revision into an older test binary's identity. The E2E uses its existing temporary cluster directories and cleanup. CARGO_TARGET_DIR controls compilation output; nextest's default report store remains the workspace's target/nextest. Execute the existing selected case as follows:

CASE=background-target-restart
FILTER='test(test_cluster_root_heal_recovers_remote_shards_after_background_target_restart)'
RUN_DIR="$PWD/artifacts/scanner-heal-run"
export RUSTFS_E2E_EXPECTED_FEATURES=default
scripts/python_bin.sh scripts/check_test_wiring.py \
  --begin-scanner-heal "$RUN_DIR" "$SERVER_BINARY" "$TEST_BINARY"
export RUSTFS_SCANNER_HEAL_RUN_DIR="$RUN_DIR"
export CARGO_BIN_EXE_rustfs="$SERVER_BINARY"
cargo nextest list --profile e2e-nightly -p e2e_test -E "$FILTER" \
  --message-format json > "$RUN_DIR/listing.json"
rm -f target/nextest/e2e-nightly/junit.xml
set +e
cargo nextest run --profile e2e-nightly -p e2e_test -E "$FILTER"
test_exit=$?
set -e
cp target/nextest/e2e-nightly/junit.xml "$RUN_DIR/junit.xml"
scripts/python_bin.sh scripts/check_test_wiring.py --finish-scanner-heal "$RUN_DIR" "$test_exit"
scripts/python_bin.sh scripts/check_test_wiring.py --check-scanner-heal "$RUN_DIR" "$CASE"

Set RUSTFS_E2E_EXPECTED_FEATURES to the actual intended e2e crate feature set, including default for a default-feature build, comma-separated for extra features, or empty for --no-default-features. It is mandatory when beginning a run. Crate features are distinct from the spawned server's build features.

Do not replace a nonzero command exit with zero. Missing JUnit or an oracle emission failure also fails acceptance. Each retry needs a new run directory; the producer refuses to overwrite an existing oracle. Keep failed-run logs and artifacts. The receipt pins source revision, actual binary hashes, run identity, start/finish times, and the artifact hashes. listing.json, junit.xml, and each oracle are limited to 1 MiB; object evidence has the fixture's 9..65 object bound. Credentials are not included in the receipt.

The checker binds nextest's flattened suite binary-id/binary-path to the actual test executable and requires the JUnit testcase's embedded execution timestamp to fall inside the receipt window (with millisecond precision). Copying an old JUnit file and refreshing its mtime does not make it new evidence. Schema versions, topology counts, PIDs, EC geometry and shard indices require actual integers: booleans and fractional values are rejected, and an index must fit the physical data-plus-parity geometry.

The checker rejects unselected/ignored tests, zero/duplicate JUnit cases, failures, skipped tests, retry/flaky records, stale or changed artifacts, different builds or run IDs, unchanged process IDs, wrong topology, missing shard parts, and mismatched S3 content/listings. The raw oracle JSON is emitted by the real E2E producer, not accepted from an adapter copying expectations.

--check-scanner-heal "$RUN_DIR" release checks available case evidence and returns nonzero for every pending release requirement. A focused case pass does not approve release. In particular, R-E requires fixed-budget real restarts without an unbudgeted final sweep, R-D requires the full manager/event/ledger disposition chain, and R-L requires source-conflict and crash/retirement evidence. Reader-only or unit fixtures cannot substitute for these. The external rustfs/auto-testing functional workflows propagate suite failures. Their workflow status does not establish this registry's required case coverage, build provenance, or object-level oracles.

For automation, --check-scanner-heal-release "$RUN_DIR" emits one compact JSON decision and exits nonzero while blocked. verified_cases contains only cases that pass the complete receipt, build provenance, nextest/JUnit and real oracle checks; rejected_cases names registered cases that do not, and pending_gates names the unimplemented release requirements and pending_lanes names the structured release lanes that still need real evidence. Schema 1 is deliberately marked release_schema_capable: false because it models only the single-version, unversioned-object restart/crash cases. Schema 2 can describe the wider release matrix, but approval still requires every registered case to verify and every required gate to leave pending only after a future checker can bind it to real feature-specific evidence. The current checker hard-rejects missing structured requirements and pending gates mapped to an implemented lane, so clearing pending text cannot become approval. A focused run, synthetic harness, compile-only result, skipped/retried test, ordinary CI success, or unregistered mixed-version, rollback, EC8+4 or performance claim therefore cannot become a release approval. For high-risk rollback gates, evidence_fields records the specific proof fields that a future real-evidence checker must bind before a pending gate can move out of the blocked set. G03 keeps scoped ACK tied to durable root publication, ACK request identity, participating peer capability snapshots, and mixed-peer fallback oracles; G09 keeps mixed-version reader, writer, and rollback payload evidence explicit. These fields are part of the release contract, not evidence by themselves.

The upgrade compatibility E2E can emit raw G09 JSON artifacts when RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR points at a fresh, task-owned directory. The rolling mixed-version test writes G09-mixed_version_reader_evidence.json and G09-mixed_version_writer_evidence.json after the old/new reader and writer assertions pass. The bucket-metadata rollback test writes G09-rollback_payload_evidence.json after the current -> previous -> current round trip has read back the known bucket configuration and objects. These artifacts are measured inputs for a later release bundle; the bundle must still record their relative paths, hashes, command provenance, timestamps, roles, participating revisions, and case lists before --check-scanner-heal-release-bundle can validate them.

For a release-candidate or PR-head Linux x86_64 host, run the full raw G09 artifact pass with:

scripts/run_scanner_heal_g09_upgrade_evidence.sh

The script mirrors the pinned previous-release asset used by the upgrade workflow, builds the current checkout, runs the mixed-version and rollback E2E lanes, and fails unless all three raw G09 artifacts are measured, revision-bound, and role-bound. Use --source-binary for a custom previous-release binary on another platform, or --test mixed-version|rollback while narrowing a failure. It performs a free-space preflight before building so a saturated validation host fails before producing partial evidence.

When the real release lanes have produced their dedicated artifacts, validate the complete hard-gate bundle with:

scripts/python_bin.sh scripts/check_test_wiring.py \
  --check-scanner-heal-release-bundle /path/to/release-evidence.json

The bundle checker is intentionally stricter than the case checker. It requires schema 2 registry metadata, evidence: measured, the current checkout revision, all G01-G14/P1-P4/R-E/R-D/R-L gates, per-gate status: pass, lane identity, relative artifact paths, matching SHA256 hashes, and non-empty summaries. It also binds each evidence field to its own run provenance: source_revision, run_id, measurement_window_id, timezone-qualified started_at and finished_at, command arguments, and artifact format. The field source_revision must match the bundle revision, and measured performance duration cannot exceed the recorded run window. When an evidence or profile artifact declares a JSON format, the checker also opens that artifact and requires its payload to repeat the same measured source_revision, run_id, measurement_window_id, gate and field identity; profile sub-artifacts must additionally name their artifact kind. Updating only the outer bundle hash cannot turn a stale JSON summary into current release evidence.

The hard evidence shape remains claim-specific: mixed-version gates must name at least two participating versions, crash/durable replay gates must include crash-boundary evidence, G14 must record EC8+4 with at least three nodes and four drives per node plus multi-set and multi-pool evidence, performance gates need measured durations, P3's pressure run needs at least two hours, and P1 needs a symbolized profile summary with resolved samples. Every G14 field and every performance gate's fields must also share one measurement_window_id, so EC8+4, multi-set/multi-pool, ABBA, throughput, and profiling artifacts cannot be stitched together from unrelated runs. P1 profile_evidence must bind every required profile artifact kind (allocation-profile, flamegraph, rss-samples, and save-frequency) with a relative path, artifact format, non-empty file, matching SHA256, and the same measurement window when a per-artifact window is declared. Missing, synthetic, stale, tampered, undersized, or topology-mismatched evidence returns a compact blocked or invalid JSON result and a nonzero exit.

The scheduler-pressure lane must also carry the numbers needed to close W09, W10, and W11: bounded deferred item/byte/age limits, zero duplicate tasks, pressure pacing engagement, recovery and lock-hold timings, fixed offered load, foreground p95/p99 latency, throughput, error count, attempt-cost samples, and completed heal object counts.

This command validates the evidence package; it does not create evidence. A handwritten JSON file, a synthetic harness pass, a single focused case, or a local unit fixture still cannot satisfy the distributed, mixed-version, crash-restart, durable MRF replay, EC8+4, ABBA, or profiling gates.

Run parser/receipt regressions with scripts/python_bin.sh scripts/check_test_wiring.py --self-test. Those fixtures validate the checker only and produce no runtime or performance evidence.

For local bundle-shape dry runs, generate a task-owned fixture directory with:

scripts/python_bin.sh scripts/check_test_wiring.py \
  --write-scanner-heal-release-bundle-fixture /path/to/fixture-dir

The generated file is marked fixture_only and is intentionally rejected by the release bundle checker. Use it to rehearse field names, artifact paths, hashes, profile artifact membership, mixed-version roles, and same-window provenance before copying the shape into a real measured bundle. It is not ABBA, profile, mixed-version, crash-restart, or release approval evidence.