* fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * fix(targets): reject trailing batch items (#7508) * test(scanner): preflight G09 evidence disk space Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: cui fliter <imcusg@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>
30 KiB
CI gate matrix
Use this when: a check is red and you need to know whether it blocks the merge, which workflow and job produced it, and how to reproduce it locally.
Source of truth: the live main ruleset (command below) for required status; .github/workflows/<file>.yml for triggers, paths, timeout-minutes, and cron; .config/nextest.toml for e2e profile filters; .github/scheduled-validations.json for the freshness-watchdog list.
A job blocks a merge only when its exact check name is in the live main ruleset. A workflow name, a merge_group trigger, or a red PR check does not make a job required by itself.
Required merge checks
The main ruleset (6436880) requires exactly these contexts, with strict_required_status_checks_policy=false:
| Required context | Producer | Validation |
|---|---|---|
CLA Check |
cla.yml |
Contributor agreement |
Quick Checks |
ci.yml job quick-checks |
Formatting and repository guard scripts |
Test and Lint |
ci.yml job test-and-lint |
Clippy, workspace nextest (ci profile, excluding e2e_test), doctests, migration-gate count (scripts/check_migration_gate_count.sh) |
For PRs limited to the paths-ignore list in ci.yml, ci-docs-only.yml reports Quick Checks and Test and Lint under the same names; it runs the quick checks and scripts/check_no_planning_docs.sh, not a Rust build or tests. scripts/check_ci_paths_sync.sh keeps the two path lists aligned.
Verify the live rule before changing merge policy:
gh api repos/rustfs/rustfs/rulesets/6436880 \
--jq '.rules[] | select(.type == "required_status_checks") | .parameters'
Promotion rule: never promote a report-only lane to required from one green run. Require at least 14 days and 30 representative PRs with at least 99% complete execution, then update the ruleset and this file together.
Pull request and merge matrix
"Report-only" means visible and actionable but not in the required list. Budgets are each job's timeout-minutes in the named workflow and are not copied here.
| Event | Check name | Workflow / job | Merge status | Reproduce |
|---|---|---|---|---|
| PR, non-doc change | Quick Checks |
ci.yml quick-checks |
Required | make pre-commit |
| PR, non-doc change | Test and Lint |
ci.yml test-and-lint |
Required | cargo clippy --all-targets -- -D warnings; cargo nextest run --profile ci --all --exclude e2e_test; cargo test --all --doc; scripts/check_migration_gate_count.sh |
| PR, non-doc change | Typos |
ci.yml typos |
Report-only | typos |
| PR, non-doc change | ILM Integration (serial) |
ci.yml test-ilm-integration-serial |
Report-only | exact command in the job |
| PR, non-doc change | Test and Lint (rio-v2), Test and Lint (swift), Test and Lint (sftp) |
ci.yml test-and-lint-rio-v2, test-and-lint-protocols |
Report-only | cargo nextest run with the job's --features |
| PR, non-doc change | Connect Short Credential Boundary |
ci.yml connect-short-credential-boundary |
Report-only | cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials; cargo check -p rustfs --release --features connect-e2e-short-credentials must fail |
| PR, non-doc change | Build RustFS Debug Binary |
ci.yml build-rustfs-debug-binary |
Report-only; prerequisite for the black-box jobs | cargo build -p rustfs --bins |
| PR, non-doc change | io_uring Integration (real) |
ci.yml uring-integration |
Report-only | cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture |
| PR, non-doc change | End-to-End Tests |
ci.yml e2e-tests |
Report-only | cargo nextest run --profile e2e-smoke -p e2e_test, then ./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>; membership guards scripts/check_test_wiring.py --check-profile e2e-smoke <listing.json> and scripts/check_security_smoke_count.sh check <listing.json> |
| PR, non-doc change | S3 Implemented Tests |
ci.yml s3-implemented-tests |
Report-only | build rustfs, then scripts/s3-tests/run.sh with the job's DEPLOY_MODE / TEST_MODE / MAXFAIL env |
| PR, non-doc change | S3 Lifecycle Behavior Tests |
ci.yml s3-lifecycle-behavior-tests |
Report-only | scripts/s3-tests/run.sh with the job's accelerated-scanner env |
PR touching paths in audit.yml |
Cargo Deny, Workflow Pin Report, Dependency Review |
audit.yml cargo-deny, workflow-pin-report, dependency-review |
Report-only | cargo deny check; scripts/security/check_workflow_pins.sh |
PR touching paths in architecture-migration-rules.yml |
Architecture Migration Rules |
architecture-migration-rules.yml architecture-migration-rules |
Report-only | scripts/check_architecture_migration_rules.sh |
PR touching paths in nix.yml |
Nix Build & Check |
nix.yml nix-validation |
Report-only | nix flake check |
PR touching paths in fuzz.yml |
Build Fuzz Harness, Smoke / <target> |
fuzz.yml fuzz-build, pr-fuzz-smoke |
Report-only | MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh |
PR touching paths in windows-filesystem.yml |
Rename Safety |
windows-filesystem.yml rename-safety |
Report-only | the cargo test -p rustfs-ecstore --lib <filter> commands in the job, on Windows |
PR touching paths in coverage.yml |
Workspace line coverage |
coverage.yml coverage |
Report-only | make coverage; python3 scripts/check_security_coverage.py target/llvm-cov/coverage.json |
PR touching paths in e2e-upgrade.yml |
Direct upgrade from the previous release, Mixed-version rolling upgrade from the previous release, Bucket configuration survives the upgrade, Rollback reads current bucket metadata |
e2e-upgrade.yml upgrade matrix |
Report-only | the cargo test --locked -p e2e_test command in the job with RUSTFS_UPGRADE_SOURCE_BINARY pointing at the pinned previous release (UPGRADE_SOURCE_VERSION) |
PR touching paths in oidc-keycloak.yml |
OIDC Keycloak live gate |
oidc-keycloak.yml oidc-keycloak-live |
Report-only | cargo build --locked -p rustfs --bin rustfs, then bash scripts/test/oidc_keycloak_live.sh ./target/debug/rustfs |
PR touching paths in targets-integration.yml |
PostgreSQL, MySQL, AMQP, and NATS |
targets-integration.yml targets-live |
Report-only | start the containers as in the job, export the RUSTFS_TEST_* DSNs, then the job's cargo test --locked -p rustfs-targets --test <name> -- --ignored --test-threads=1 commands |
| PR limited to main-CI-excluded paths | Quick Checks, Test and Lint |
ci-docs-only.yml quick-checks, test-and-lint |
Required | git diff --check; make doc-paths-check; scripts/check_no_planning_docs.sh |
merge_group; push to main |
End-to-End Tests (full merge gate) |
ci.yml e2e-full |
Report-only | cargo nextest run --profile e2e-full -p e2e_test |
e2e filters live in .config/nextest.toml; extend a profile instead of adding a second selector. Before a profile runs, scripts/check_test_wiring.py compares its listing to the committed digest in .config/e2e-<profile>-selection.txt, so a silent test drop fails closed.
Scanner usage and heal rebuild coverage are intentionally split by risk and
cost. data_usage_test runs in the PR e2e-smoke lane so changes that affect
authoritative scanner usage publication, quota-visible usage, or admin usage
snapshots get an end-to-end signal before merge review. heal_erasure_disk_rebuild_test
runs in e2e-full so core erasure heal rebuild regressions are caught no later
than the merge queue or main push lane; it also remains in e2e-nightly with
the serialized cluster fault-domain suites for scheduled soak signal.
Scheduled validation
Scheduled lanes never block a PR. Their workflow-local gate fails the run, scheduled failures route to the shared failure-issue action, and scheduled-validation-freshness.yml fails when a workflow listed in .github/scheduled-validations.json has not run within its max_age_hours (a never_ran_grace_until entry covers the window before a newly enabled cron's first slot). Cadence is qualitative here; the cron lives in each workflow's on.schedule.
| Workflow (cadence) | Jobs | Verdict and artifacts | In freshness list | Reproduce |
|---|---|---|---|---|
ci.yml (weekly) |
full matrix, including the schedule/dispatch-only rio-v2 jobs build-rustfs-debug-binary-rio-v2 and e2e-tests-rio-v2 |
per-job | yes | dispatch ci.yml |
build.yml (weekly) |
build-rustfs over the six-target platform matrix in prepare-platform-matrix (four Linux, macOS aarch64, Windows x86_64) |
build/package integrity | yes | dispatch build.yml with an exact platform set |
e2e-replication-nightly.yml (nightly) |
repl-nightly, cluster-nightly, protocols-nightly |
three independent gates; JUnit, membership listing, server logs | yes | cargo nextest run --profile e2e-repl-nightly -p e2e_test; --profile e2e-nightly; -j 1 --profile e2e-protocols |
e2e-distributed.yml (storage-sensitive PRs + nightly) |
distributed |
fail-closed 4-node 4-disk S3, durability, replication, movement, fault, and direct/rolling upgrade gate; JUnit, membership listing, per-node server logs | yes, with never_ran_grace_until |
download the pinned previous release as in the workflow, export RUSTFS_UPGRADE_SOURCE_BINARY, then cargo nextest run --profile e2e-distributed -p e2e_test |
e2e-s3tests.yml (weekly) |
s3tests (single and distributed, four shards each), upstream-head-canary |
compatibility gate; report, JUnit, node IDs, server logs | yes | scripts/s3-tests/run.sh against an existing single or distributed target |
fuzz.yml (nightly) |
nightly-fuzz-corpus per target |
gate; corpus and crash artifacts | yes | MAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh |
minio-interop.yml (nightly) |
minio-interop |
EC + SSE read-parity gate | yes, with never_ran_grace_until |
pinned Docker fixture steps in the workflow |
on-demand-migration-interop.yml (nightly) |
minio-source, cloud-source (aws, r2, gcs) |
report-only provider interop; one JSON report per provider naming cases, timings and source request counts, plus JUnit and MinIO logs. A cloud provider whose ODM_INTEROP_* secrets are absent is skipped with a summary note, not failed |
no | start the pinned MinIO container as in the job, export the RUSTFS_ODM_INTEROP_* variables, then cargo nextest run --profile e2e-odm-interop -p e2e_test |
performance-ab.yml (nightly) |
warp-ab |
regression-budget gate; A/B summaries and server logs | yes | bash scripts/run_hotpath_warp_abba.sh --help |
nightly-gnu.yml (nightly) |
build, kms-vault-lane, kms-vault-ha-failover |
build, live Vault, and HA failover gates | yes | commands and pinned Vault images in the workflow |
audit.yml (nightly) |
cargo-deny, workflow-pin-report |
dependency and workflow-pin gates | yes | cargo deny check; scripts/security/check_workflow_pins.sh |
mint.yml (weekly) |
mint |
report-only by design; per-suite PASS/FAIL/NA and raw log.json |
yes | pinned Docker sequence in the workflow |
coverage.yml (weekly) |
coverage |
report-only trend; lcov and JSON artifact | yes | make coverage |
runner-hygiene.yml (monthly) |
check-ephemerality |
runner ephemerality | yes | dispatch |
e2e-upgrade.yml (weekly) |
upgrade (4-case matrix) |
upgrade and rollback gate; server logs | no | see the PR row |
oidc-keycloak.yml (weekly) |
oidc-keycloak-live |
live OIDC gate | no | see the PR row |
targets-integration.yml (nightly) |
targets-live |
live target gate; container logs | no | see the PR row |
scheduled-validation-freshness.yml (nightly) |
check-freshness |
fails on a never-created or stale schedule | n/a | dispatch |
Manual workflow_dispatch runs are debugging evidence and do not open scheduled-failure issues. A manual performance run may explicitly allow a known regression; that override is not a passing baseline.
Packaged functional acceptance
rustfs-functional-chain.yml dispatches the packaged-build suites in rustfs-*-test.yml on the shared lab runners. A failing suite step or job must fail its workflow. Report collection, cleanup, and dispatch of the next suite can still run with always(); continuing diagnostics does not make the failed suite successful.
Workflow status preserves errors that the test scripts report. It does not establish complete execution or a common package identity across the chain: inspect the current run's case results, package identity, and test-script revision as well. A script that returns zero after a failed tool invocation needs its own result check.
Release validation
Post-merge and tag-driven; not a substitute for a PR gate.
| Trigger | Workflow / job | Result |
|---|---|---|
Push to main, weekly schedule, dispatch |
build.yml build-rustfs (a development build on a main push restricts the matrix to the Linux targets) |
build artifacts; no release publication |
| Valid release or preview tag | build.yml build-rustfs, create-release, upload-release-assets, publish-release |
draft release, checksummed assets, publish |
Successful non-preview release-tag build (workflow_run) |
docker.yml build-docker, scan-docker-image |
multi-architecture images and vulnerability report |
Successful release-tag build (workflow_run) |
package.yml package |
DEB/RPM packages and checksums uploaded to the release |
Successful non-preview release-tag build (workflow_run) |
helm-package.yml build-helm-package, publish-helm-package |
versioned chart and repository index |
| Final tag's release published | build.yml cleanup-preview-releases |
deletes every <target>-preview.<N> Release for that target; the tags are kept |
Use an exact preview tag for an end-to-end release rehearsal. Manual dispatches are backfill/debug paths and do not prove the automatic workflow_run chain.
Change checklist
Update this file in the same PR when a job or check name changes, a workflow gains or loses a pull_request or schedule trigger, required contexts or strict/merge-queue policy change, report-only vs gating semantics change, or .github/scheduled-validations.json membership changes. Do not copy timeouts, crons, or test counts here.
ECStore invariant selection
The existing ci.yml test-and-lint job runs the ordinary ECStore and filemeta tests. After that run, scripts/check_test_wiring.py --check-core checks the same nextest profile and package selection against .config/ecstore-required-tests.json. Every named test must exist, match the filter, and be non-ignored; the job also requires a nonempty JUnit report. This checks membership without running the tests twice. core-test-listing.json, JUnit, and the run log are retained in the existing test-and-lint artifact.
The manifest records a minimum set of invariants: write quorum, metadata rollback, stale-writer lock loss, plaintext Range content, multipart cancellation, hiding uncommitted LIST versions, real MinIO metadata, corrupt part arrays, and the shared on-demand-migration source-backend contract for each provider dialect (S3, Azure, native GCS). The three contract entries live in the rustfs suite and reach the lane through that package's default features, so dropping gcs from rustfs's defaults fails this check instead of silently deselecting the GCS contract (rustfs/backlog#2323). Renaming or moving a required test must update the manifest in the same change after checking the compiled listing. Extend this list as new deterministic regressions land; it is not a claim that all storage invariants are covered.
The checked-in MinIO corpus is pinned by file SHA256 and its documented source release. The static wiring guard and the CI selection check both reject missing or changed fixtures. These are metadata fixtures, not a legacy shard-body corpus or proof of crash durability. Optional legacy_bitrot_read_test runs may still skip when their external corpus is absent; they do not satisfy a required compatibility lane. Real encrypted fixture reads remain in minio-interop.yml, and multi-node fault schedules remain in the existing nightly cluster lane. In-process reopen tests do not establish power-loss durability.
Run python3 scripts/check_test_wiring.py --self-test to exercise the negative cases: removed/ignored/filtered tests, malformed listing, absent fixtures, and wrong fixture hashes. Do not update hashes merely to silence the guard; a fixture change needs source/provenance and compatibility review.
Scanner/Heal Evidence Receipts
The existing scripts/check_test_wiring.py also validates Scanner/Heal case
evidence registered in .config/scanner-heal-required-tests.json. It records
already-built binaries and checks existing nextest output; it does not build,
run tests, deploy servers, inject faults, or start another CI lane.
The initial case is background-target-restart, emitted by
heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_remote_shards_after_background_target_restart.
That test already runs in e2e-nightly. When RUSTFS_SCANNER_HEAL_RUN_DIR is set,
it checks the actual server and test-executable hashes against run.json, pins
the same server binary for all node starts, and writes its oracle only after
the real assertions pass. The artifact contains the actual pre/post target
PIDs, per-node S3 listings, expected and downloaded complete-body hashes/lengths,
and target-disk VersionShardCensus fingerprints. Existing baseline objects
must match their pre-fault physical manifests; the object created during the
outage has no pre-fault target shard and is checked for complete physical parts
and exact S3 content.
This case is a four-node, one-drive-per-node process-restart test. It is not
power-loss validation, a 3x4 EC8+4 experiment, an all-version inventory, or proof
of scanner enumeration, exact MRF disposition, legacy migration, or rollback.
The schema 2 registry separates the implemented single-set restart lane from
structured release lanes for authority coverage, checkpoint/crash, status and
outcome, MRF responsibility, mixed-version rollback, scheduler pressure,
maintenance producers, and EC8+4 multi-set coverage. All G01-G14/P1-P4 and
R-E/R-D/R-L release requirements stay pending until their actual
feature-specific oracles, measurements and required topologies exist. Missing
cases cannot be supplied by synthetic W20 results. W20's bounded JSON and
file-hash helpers are reused; its ABBA performance contracts remain in
docs/operations/scanner-benchmark-runbook.md.
Measured ABBA manifests must also carry the runbook's release_evidence
contract. The runner rejects reports that cannot bind the exact 3x4 EC8+4
topology, multi-pool/multi-set shape, distributed same-window metrics endpoints,
restart/crash modes, mixed-version reader/writer/rollback participation, and
allocation/flamegraph/RSS/save-frequency profile artifact plan. Synthetic runs
and manifests missing that contract remain harness-only evidence.
Recording One Case
Use a committed source tree, independently built current binaries, sufficient
free disk space, and a task-owned artifact directory that does not yet exist.
Set SERVER_BINARY and TEST_BINARY to those exact executable paths. The begin
command requires the server's embedded --version commit to match the clean
checkout and its embedded Git status to be clean. The E2E crate's build script
embeds its build-time Git revision/dirty state, lockfile Git blob, enabled crate
features, target, profile and encoded Rust flags. It tracks the crate/dependency
trees, Cargo inputs and Git HEAD/ref/index, including common.rs restart logic.
The producer checks this compiled identity against the receipt; it does not
copy a current source revision into an older test binary's identity. The E2E
uses its existing temporary cluster directories and cleanup. CARGO_TARGET_DIR
controls compilation output; nextest's default report store remains the
workspace's target/nextest. Execute the existing selected case as follows:
CASE=background-target-restart
FILTER='test(test_cluster_root_heal_recovers_remote_shards_after_background_target_restart)'
RUN_DIR="$PWD/artifacts/scanner-heal-run"
export RUSTFS_E2E_EXPECTED_FEATURES=default
scripts/python_bin.sh scripts/check_test_wiring.py \
--begin-scanner-heal "$RUN_DIR" "$SERVER_BINARY" "$TEST_BINARY"
export RUSTFS_SCANNER_HEAL_RUN_DIR="$RUN_DIR"
export CARGO_BIN_EXE_rustfs="$SERVER_BINARY"
cargo nextest list --profile e2e-nightly -p e2e_test -E "$FILTER" \
--message-format json > "$RUN_DIR/listing.json"
rm -f target/nextest/e2e-nightly/junit.xml
set +e
cargo nextest run --profile e2e-nightly -p e2e_test -E "$FILTER"
test_exit=$?
set -e
cp target/nextest/e2e-nightly/junit.xml "$RUN_DIR/junit.xml"
scripts/python_bin.sh scripts/check_test_wiring.py --finish-scanner-heal "$RUN_DIR" "$test_exit"
scripts/python_bin.sh scripts/check_test_wiring.py --check-scanner-heal "$RUN_DIR" "$CASE"
Set RUSTFS_E2E_EXPECTED_FEATURES to the actual intended e2e crate feature set,
including default for a default-feature build, comma-separated for extra
features, or empty for --no-default-features. It is mandatory when beginning
a run. Crate features are distinct from the spawned server's build features.
Do not replace a nonzero command exit with zero. Missing JUnit or an oracle
emission failure also fails acceptance. Each retry needs a new run directory;
the producer refuses to overwrite an existing oracle. Keep failed-run logs and
artifacts. The receipt pins source revision, actual binary hashes, run identity,
start/finish times, and the artifact hashes. listing.json, junit.xml, and
each oracle are limited to 1 MiB; object evidence has the fixture's 9..65 object
bound. Credentials are not included in the receipt.
The checker binds nextest's flattened suite binary-id/binary-path to the
actual test executable and requires the JUnit testcase's embedded execution
timestamp to fall inside the receipt window (with millisecond precision).
Copying an old JUnit file and refreshing its mtime does not make it new evidence.
Schema versions, topology counts, PIDs, EC geometry and shard indices require
actual integers: booleans and fractional values are rejected, and an index must
fit the physical data-plus-parity geometry.
The checker rejects unselected/ignored tests, zero/duplicate JUnit cases, failures, skipped tests, retry/flaky records, stale or changed artifacts, different builds or run IDs, unchanged process IDs, wrong topology, missing shard parts, and mismatched S3 content/listings. The raw oracle JSON is emitted by the real E2E producer, not accepted from an adapter copying expectations.
--check-scanner-heal "$RUN_DIR" release checks available case evidence and
returns nonzero for every pending release requirement. A focused case pass
does not approve release. In particular, R-E requires fixed-budget real
restarts without an unbudgeted final sweep, R-D requires the full
manager/event/ledger disposition chain, and R-L requires source-conflict and
crash/retirement evidence. Reader-only or unit fixtures cannot substitute for
these. The external rustfs/auto-testing functional workflows propagate suite
failures. Their workflow status does not establish this registry's required
case coverage, build provenance, or object-level oracles.
For automation, --check-scanner-heal-release "$RUN_DIR" emits one compact
JSON decision and exits nonzero while blocked. verified_cases contains only
cases that pass the complete receipt, build provenance, nextest/JUnit and real
oracle checks; rejected_cases names registered cases that do not, and
pending_gates names the unimplemented release requirements and
pending_lanes names the structured release lanes that still need real
evidence. Schema 1 is deliberately marked release_schema_capable: false
because it models only the single-version, unversioned-object restart/crash
cases. Schema 2 can describe the wider release matrix, but approval still
requires every registered case to verify and every required gate to leave
pending only after a future checker can bind it to real feature-specific
evidence. The current checker hard-rejects missing structured requirements and
pending gates mapped to an implemented lane, so clearing pending text cannot
become approval. A focused run, synthetic harness, compile-only result,
skipped/retried test, ordinary CI success, or unregistered mixed-version,
rollback, EC8+4 or performance claim therefore cannot become a release approval.
For high-risk rollback gates, evidence_fields records the specific proof
fields that a future real-evidence checker must bind before a pending gate can
move out of the blocked set. G03 keeps scoped ACK tied to durable root
publication, ACK request identity, participating peer capability snapshots, and
mixed-peer fallback oracles; G09 keeps mixed-version reader, writer, and rollback
payload evidence explicit. These fields are part of the release contract, not
evidence by themselves.
The upgrade compatibility E2E can emit raw G09 JSON artifacts when
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR points at a fresh, task-owned directory.
The rolling mixed-version test writes G09-mixed_version_reader_evidence.json
and G09-mixed_version_writer_evidence.json after the old/new reader and writer
assertions pass. The bucket-metadata rollback test writes
G09-rollback_payload_evidence.json after the current -> previous -> current
round trip has read back the known bucket configuration and objects. These
artifacts are measured inputs for a later release bundle; the bundle must still
record their relative paths, hashes, command provenance, timestamps, roles,
participating revisions, and case lists before
--check-scanner-heal-release-bundle can validate them.
For a release-candidate or PR-head Linux x86_64 host, run the full raw G09 artifact pass with:
scripts/run_scanner_heal_g09_upgrade_evidence.sh
The script mirrors the pinned previous-release asset used by the upgrade
workflow, builds the current checkout, runs the mixed-version and rollback E2E
lanes, and fails unless all three raw G09 artifacts are measured, revision-bound,
and role-bound. Use --source-binary for a custom previous-release binary on
another platform, or --test mixed-version|rollback while narrowing a failure.
It performs a free-space preflight before building so a saturated validation
host fails before producing partial evidence.
When the real release lanes have produced their dedicated artifacts, validate the complete hard-gate bundle with:
scripts/python_bin.sh scripts/check_test_wiring.py \
--check-scanner-heal-release-bundle /path/to/release-evidence.json
The bundle checker is intentionally stricter than the case checker. It requires
schema 2 registry metadata, evidence: measured, the current checkout revision,
all G01-G14/P1-P4/R-E/R-D/R-L gates, per-gate status: pass, lane identity,
relative artifact paths, matching SHA256 hashes, and non-empty summaries. It
also binds each evidence field to its own run provenance: source_revision,
run_id, measurement_window_id, timezone-qualified started_at and
finished_at, command arguments, and artifact format. The field
source_revision must match the bundle revision, and measured performance
duration cannot exceed the recorded run window.
When an evidence or profile artifact declares a JSON format, the checker also
opens that artifact and requires its payload to repeat the same measured
source_revision, run_id, measurement_window_id, gate and field identity;
profile sub-artifacts must additionally name their artifact kind. Updating only
the outer bundle hash cannot turn a stale JSON summary into current release
evidence.
The hard evidence shape remains claim-specific: mixed-version gates must name at
least two participating versions, crash/durable replay gates must include
crash-boundary evidence, G14 must record EC8+4 with at least three nodes and four
drives per node plus multi-set and multi-pool evidence, performance gates need
measured durations, P3's pressure run needs at least two hours, and P1 needs a
symbolized profile summary with resolved samples. Every G14 field and every
performance gate's fields must also share one measurement_window_id, so EC8+4,
multi-set/multi-pool, ABBA, throughput, and profiling artifacts cannot be
stitched together from unrelated runs. P1 profile_evidence must bind every
required profile artifact kind (allocation-profile, flamegraph,
rss-samples, and save-frequency) with a relative path, artifact format,
non-empty file, matching SHA256, and the same measurement window when a
per-artifact window is declared. Missing, synthetic, stale, tampered, undersized,
or topology-mismatched evidence returns a compact blocked or invalid JSON result
and a nonzero exit.
The scheduler-pressure lane must also carry the numbers needed to close W09, W10, and W11: bounded deferred item/byte/age limits, zero duplicate tasks, pressure pacing engagement, recovery and lock-hold timings, fixed offered load, foreground p95/p99 latency, throughput, error count, attempt-cost samples, and completed heal object counts.
This command validates the evidence package; it does not create evidence. A handwritten JSON file, a synthetic harness pass, a single focused case, or a local unit fixture still cannot satisfy the distributed, mixed-version, crash-restart, durable MRF replay, EC8+4, ABBA, or profiling gates.
Run parser/receipt regressions with
scripts/python_bin.sh scripts/check_test_wiring.py --self-test. Those fixtures
validate the checker only and produce no runtime or performance evidence.
For local bundle-shape dry runs, generate a task-owned fixture directory with:
scripts/python_bin.sh scripts/check_test_wiring.py \
--write-scanner-heal-release-bundle-fixture /path/to/fixture-dir
The generated file is marked fixture_only and is intentionally rejected by the
release bundle checker. Use it to rehearse field names, artifact paths, hashes,
profile artifact membership, mixed-version roles, and same-window provenance
before copying the shape into a real measured bundle. It is not ABBA, profile,
mixed-version, crash-restart, or release approval evidence.