mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-09 13:46:05 +00:00
houseme
5db0e14d04
* fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * fix(targets): reject trailing batch items (#7508) * test(scanner): preflight G09 evidence disk space Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: cui fliter <imcusg@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>
359 lines
30 KiB
Markdown
359 lines
30 KiB
Markdown
# CI gate matrix
|
|
|
|
**Use this when:** a check is red and you need to know whether it blocks the merge, which workflow and job produced it, and how to reproduce it locally.
|
|
**Source of truth:** the live `main` ruleset (command below) for required status; `.github/workflows/<file>.yml` for triggers, `paths`, `timeout-minutes`, and cron; `.config/nextest.toml` for e2e profile filters; `.github/scheduled-validations.json` for the freshness-watchdog list.
|
|
|
|
A job blocks a merge only when its exact check name is in the live `main` ruleset. A workflow name, a `merge_group` trigger, or a red PR check does not make a job required by itself.
|
|
|
|
## Required merge checks
|
|
|
|
The `main` ruleset (`6436880`) requires exactly these contexts, with `strict_required_status_checks_policy=false`:
|
|
|
|
| Required context | Producer | Validation |
|
|
|---|---|---|
|
|
| `CLA Check` | `cla.yml` | Contributor agreement |
|
|
| `Quick Checks` | `ci.yml` job `quick-checks` | Formatting and repository guard scripts |
|
|
| `Test and Lint` | `ci.yml` job `test-and-lint` | Clippy, workspace nextest (`ci` profile, excluding `e2e_test`), doctests, migration-gate count (`scripts/check_migration_gate_count.sh`) |
|
|
|
|
For PRs limited to the `paths-ignore` list in `ci.yml`, `ci-docs-only.yml` reports `Quick Checks` and `Test and Lint` under the same names; it runs the quick checks and `scripts/check_no_planning_docs.sh`, not a Rust build or tests. `scripts/check_ci_paths_sync.sh` keeps the two path lists aligned.
|
|
|
|
Verify the live rule before changing merge policy:
|
|
|
|
```bash
|
|
gh api repos/rustfs/rustfs/rulesets/6436880 \
|
|
--jq '.rules[] | select(.type == "required_status_checks") | .parameters'
|
|
```
|
|
|
|
Promotion rule: never promote a report-only lane to required from one green run. Require at least 14 days and 30 representative PRs with at least 99% complete execution, then update the ruleset and this file together.
|
|
|
|
## Pull request and merge matrix
|
|
|
|
"Report-only" means visible and actionable but not in the required list. Budgets are each job's `timeout-minutes` in the named workflow and are not copied here.
|
|
|
|
| Event | Check name | Workflow / job | Merge status | Reproduce |
|
|
|---|---|---|---|---|
|
|
| PR, non-doc change | `Quick Checks` | `ci.yml` `quick-checks` | Required | `make pre-commit` |
|
|
| PR, non-doc change | `Test and Lint` | `ci.yml` `test-and-lint` | Required | `cargo clippy --all-targets -- -D warnings`; `cargo nextest run --profile ci --all --exclude e2e_test`; `cargo test --all --doc`; `scripts/check_migration_gate_count.sh` |
|
|
| PR, non-doc change | `Typos` | `ci.yml` `typos` | Report-only | `typos` |
|
|
| PR, non-doc change | `ILM Integration (serial)` | `ci.yml` `test-ilm-integration-serial` | Report-only | exact command in the job |
|
|
| PR, non-doc change | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Report-only | `cargo nextest run` with the job's `--features` |
|
|
| PR, non-doc change | `Connect Short Credential Boundary` | `ci.yml` `connect-short-credential-boundary` | Report-only | `cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials`; `cargo check -p rustfs --release --features connect-e2e-short-credentials` must fail |
|
|
| PR, non-doc change | `Build RustFS Debug Binary` | `ci.yml` `build-rustfs-debug-binary` | Report-only; prerequisite for the black-box jobs | `cargo build -p rustfs --bins` |
|
|
| PR, non-doc change | `io_uring Integration (real)` | `ci.yml` `uring-integration` | Report-only | `cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture` |
|
|
| PR, non-doc change | `End-to-End Tests` | `ci.yml` `e2e-tests` | Report-only | `cargo nextest run --profile e2e-smoke -p e2e_test`, then `./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>`; membership guards `scripts/check_test_wiring.py --check-profile e2e-smoke <listing.json>` and `scripts/check_security_smoke_count.sh check <listing.json>` |
|
|
| PR, non-doc change | `S3 Implemented Tests` | `ci.yml` `s3-implemented-tests` | Report-only | build `rustfs`, then `scripts/s3-tests/run.sh` with the job's `DEPLOY_MODE` / `TEST_MODE` / `MAXFAIL` env |
|
|
| PR, non-doc change | `S3 Lifecycle Behavior Tests` | `ci.yml` `s3-lifecycle-behavior-tests` | Report-only | `scripts/s3-tests/run.sh` with the job's accelerated-scanner env |
|
|
| PR touching `paths` in `audit.yml` | `Cargo Deny`, `Workflow Pin Report`, `Dependency Review` | `audit.yml` `cargo-deny`, `workflow-pin-report`, `dependency-review` | Report-only | `cargo deny check`; `scripts/security/check_workflow_pins.sh` |
|
|
| PR touching `paths` in `architecture-migration-rules.yml` | `Architecture Migration Rules` | `architecture-migration-rules.yml` `architecture-migration-rules` | Report-only | `scripts/check_architecture_migration_rules.sh` |
|
|
| PR touching `paths` in `nix.yml` | `Nix Build & Check` | `nix.yml` `nix-validation` | Report-only | `nix flake check` |
|
|
| PR touching `paths` in `fuzz.yml` | `Build Fuzz Harness`, `Smoke / <target>` | `fuzz.yml` `fuzz-build`, `pr-fuzz-smoke` | Report-only | `MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh` |
|
|
| PR touching `paths` in `windows-filesystem.yml` | `Rename Safety` | `windows-filesystem.yml` `rename-safety` | Report-only | the `cargo test -p rustfs-ecstore --lib <filter>` commands in the job, on Windows |
|
|
| PR touching `paths` in `coverage.yml` | `Workspace line coverage` | `coverage.yml` `coverage` | Report-only | `make coverage`; `python3 scripts/check_security_coverage.py target/llvm-cov/coverage.json` |
|
|
| PR touching `paths` in `e2e-upgrade.yml` | `Direct upgrade from the previous release`, `Mixed-version rolling upgrade from the previous release`, `Bucket configuration survives the upgrade`, `Rollback reads current bucket metadata` | `e2e-upgrade.yml` `upgrade` matrix | Report-only | the `cargo test --locked -p e2e_test` command in the job with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release (`UPGRADE_SOURCE_VERSION`) |
|
|
| PR touching `paths` in `oidc-keycloak.yml` | `OIDC Keycloak live gate` | `oidc-keycloak.yml` `oidc-keycloak-live` | Report-only | `cargo build --locked -p rustfs --bin rustfs`, then `bash scripts/test/oidc_keycloak_live.sh ./target/debug/rustfs` |
|
|
| PR touching `paths` in `targets-integration.yml` | `PostgreSQL, MySQL, AMQP, and NATS` | `targets-integration.yml` `targets-live` | Report-only | start the containers as in the job, export the `RUSTFS_TEST_*` DSNs, then the job's `cargo test --locked -p rustfs-targets --test <name> -- --ignored --test-threads=1` commands |
|
|
| PR limited to main-CI-excluded paths | `Quick Checks`, `Test and Lint` | `ci-docs-only.yml` `quick-checks`, `test-and-lint` | Required | `git diff --check`; `make doc-paths-check`; `scripts/check_no_planning_docs.sh` |
|
|
| `merge_group`; push to `main` | `End-to-End Tests (full merge gate)` | `ci.yml` `e2e-full` | Report-only | `cargo nextest run --profile e2e-full -p e2e_test` |
|
|
|
|
e2e filters live in `.config/nextest.toml`; extend a profile instead of adding a second selector. Before a profile runs, `scripts/check_test_wiring.py` compares its listing to the committed digest in `.config/e2e-<profile>-selection.txt`, so a silent test drop fails closed.
|
|
|
|
Scanner usage and heal rebuild coverage are intentionally split by risk and
|
|
cost. `data_usage_test` runs in the PR `e2e-smoke` lane so changes that affect
|
|
authoritative scanner usage publication, quota-visible usage, or admin usage
|
|
snapshots get an end-to-end signal before merge review. `heal_erasure_disk_rebuild_test`
|
|
runs in `e2e-full` so core erasure heal rebuild regressions are caught no later
|
|
than the merge queue or `main` push lane; it also remains in `e2e-nightly` with
|
|
the serialized cluster fault-domain suites for scheduled soak signal.
|
|
|
|
## Scheduled validation
|
|
|
|
Scheduled lanes never block a PR. Their workflow-local gate fails the run, scheduled failures route to the shared failure-issue action, and `scheduled-validation-freshness.yml` fails when a workflow listed in `.github/scheduled-validations.json` has not run within its `max_age_hours` (a `never_ran_grace_until` entry covers the window before a newly enabled cron's first slot). Cadence is qualitative here; the cron lives in each workflow's `on.schedule`.
|
|
|
|
| Workflow (cadence) | Jobs | Verdict and artifacts | In freshness list | Reproduce |
|
|
|---|---|---|---|---|
|
|
| `ci.yml` (weekly) | full matrix, including the schedule/dispatch-only rio-v2 jobs `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` | per-job | yes | dispatch `ci.yml` |
|
|
| `build.yml` (weekly) | `build-rustfs` over the six-target platform matrix in `prepare-platform-matrix` (four Linux, macOS aarch64, Windows x86_64) | build/package integrity | yes | dispatch `build.yml` with an exact platform set |
|
|
| `e2e-replication-nightly.yml` (nightly) | `repl-nightly`, `cluster-nightly`, `protocols-nightly` | three independent gates; JUnit, membership listing, server logs | yes | `cargo nextest run --profile e2e-repl-nightly -p e2e_test`; `--profile e2e-nightly`; `-j 1 --profile e2e-protocols` |
|
|
| `e2e-distributed.yml` (storage-sensitive PRs + nightly) | `distributed` | fail-closed 4-node 4-disk S3, durability, replication, movement, fault, and direct/rolling upgrade gate; JUnit, membership listing, per-node server logs | yes, with `never_ran_grace_until` | download the pinned previous release as in the workflow, export `RUSTFS_UPGRADE_SOURCE_BINARY`, then `cargo nextest run --profile e2e-distributed -p e2e_test` |
|
|
| `e2e-s3tests.yml` (weekly) | `s3tests` (single and distributed, four shards each), `upstream-head-canary` | compatibility gate; report, JUnit, node IDs, server logs | yes | `scripts/s3-tests/run.sh` against an existing single or distributed target |
|
|
| `fuzz.yml` (nightly) | `nightly-fuzz-corpus` per target | gate; corpus and crash artifacts | yes | `MAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh` |
|
|
| `minio-interop.yml` (nightly) | `minio-interop` | EC + SSE read-parity gate | yes, with `never_ran_grace_until` | pinned Docker fixture steps in the workflow |
|
|
| `on-demand-migration-interop.yml` (nightly) | `minio-source`, `cloud-source` (`aws`, `r2`, `gcs`) | report-only provider interop; one JSON report per provider naming cases, timings and source request counts, plus JUnit and MinIO logs. A cloud provider whose `ODM_INTEROP_*` secrets are absent is skipped with a summary note, not failed | no | start the pinned MinIO container as in the job, export the `RUSTFS_ODM_INTEROP_*` variables, then `cargo nextest run --profile e2e-odm-interop -p e2e_test` |
|
|
| `performance-ab.yml` (nightly) | `warp-ab` | regression-budget gate; A/B summaries and server logs | yes | `bash scripts/run_hotpath_warp_abba.sh --help` |
|
|
| `nightly-gnu.yml` (nightly) | `build`, `kms-vault-lane`, `kms-vault-ha-failover` | build, live Vault, and HA failover gates | yes | commands and pinned Vault images in the workflow |
|
|
| `audit.yml` (nightly) | `cargo-deny`, `workflow-pin-report` | dependency and workflow-pin gates | yes | `cargo deny check`; `scripts/security/check_workflow_pins.sh` |
|
|
| `mint.yml` (weekly) | `mint` | report-only by design; per-suite PASS/FAIL/NA and raw `log.json` | yes | pinned Docker sequence in the workflow |
|
|
| `coverage.yml` (weekly) | `coverage` | report-only trend; lcov and JSON artifact | yes | `make coverage` |
|
|
| `runner-hygiene.yml` (monthly) | `check-ephemerality` | runner ephemerality | yes | dispatch |
|
|
| `e2e-upgrade.yml` (weekly) | `upgrade` (4-case matrix) | upgrade and rollback gate; server logs | no | see the PR row |
|
|
| `oidc-keycloak.yml` (weekly) | `oidc-keycloak-live` | live OIDC gate | no | see the PR row |
|
|
| `targets-integration.yml` (nightly) | `targets-live` | live target gate; container logs | no | see the PR row |
|
|
| `scheduled-validation-freshness.yml` (nightly) | `check-freshness` | fails on a never-created or stale schedule | n/a | dispatch |
|
|
|
|
Manual `workflow_dispatch` runs are debugging evidence and do not open scheduled-failure issues. A manual performance run may explicitly allow a known regression; that override is not a passing baseline.
|
|
|
|
## Packaged functional acceptance
|
|
|
|
`rustfs-functional-chain.yml` dispatches the packaged-build suites in `rustfs-*-test.yml` on the shared lab runners. A failing suite step or job must fail its workflow. Report collection, cleanup, and dispatch of the next suite can still run with `always()`; continuing diagnostics does not make the failed suite successful.
|
|
|
|
Workflow status preserves errors that the test scripts report. It does not establish complete execution or a common package identity across the chain: inspect the current run's case results, package identity, and test-script revision as well. A script that returns zero after a failed tool invocation needs its own result check.
|
|
|
|
## Release validation
|
|
|
|
Post-merge and tag-driven; not a substitute for a PR gate.
|
|
|
|
| Trigger | Workflow / job | Result |
|
|
|---|---|---|
|
|
| Push to `main`, weekly schedule, dispatch | `build.yml` `build-rustfs` (a development build on a main push restricts the matrix to the Linux targets) | build artifacts; no release publication |
|
|
| Valid release or preview tag | `build.yml` `build-rustfs`, `create-release`, `upload-release-assets`, `publish-release` | draft release, checksummed assets, publish |
|
|
| Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `build-docker`, `scan-docker-image` | multi-architecture images and vulnerability report |
|
|
| Successful release-tag build (`workflow_run`) | `package.yml` `package` | DEB/RPM packages and checksums uploaded to the release |
|
|
| Successful non-preview release-tag build (`workflow_run`) | `helm-package.yml` `build-helm-package`, `publish-helm-package` | versioned chart and repository index |
|
|
| Final tag's release published | `build.yml` `cleanup-preview-releases` | deletes every `<target>-preview.<N>` Release for that target; the tags are kept |
|
|
|
|
Use an exact preview tag for an end-to-end release rehearsal. Manual dispatches are backfill/debug paths and do not prove the automatic `workflow_run` chain.
|
|
|
|
## Change checklist
|
|
|
|
Update this file in the same PR when a job or check name changes, a workflow gains or loses a `pull_request` or `schedule` trigger, required contexts or strict/merge-queue policy change, report-only vs gating semantics change, or `.github/scheduled-validations.json` membership changes. Do not copy timeouts, crons, or test counts here.
|
|
|
|
## ECStore invariant selection
|
|
|
|
The existing `ci.yml` test-and-lint job runs the ordinary ECStore and filemeta tests. After that run, `scripts/check_test_wiring.py --check-core` checks the same nextest profile and package selection against `.config/ecstore-required-tests.json`. Every named test must exist, match the filter, and be non-ignored; the job also requires a nonempty JUnit report. This checks membership without running the tests twice. `core-test-listing.json`, JUnit, and the run log are retained in the existing test-and-lint artifact.
|
|
|
|
The manifest records a minimum set of invariants: write quorum, metadata rollback, stale-writer lock loss, plaintext Range content, multipart cancellation, hiding uncommitted LIST versions, real MinIO metadata, corrupt part arrays, and the shared on-demand-migration source-backend contract for each provider dialect (S3, Azure, native GCS). The three contract entries live in the `rustfs` suite and reach the lane through that package's default features, so dropping `gcs` from `rustfs`'s defaults fails this check instead of silently deselecting the GCS contract (rustfs/backlog#2323). Renaming or moving a required test must update the manifest in the same change after checking the compiled listing. Extend this list as new deterministic regressions land; it is not a claim that all storage invariants are covered.
|
|
|
|
The checked-in MinIO corpus is pinned by file SHA256 and its documented source release. The static wiring guard and the CI selection check both reject missing or changed fixtures. These are metadata fixtures, not a legacy shard-body corpus or proof of crash durability. Optional `legacy_bitrot_read_test` runs may still skip when their external corpus is absent; they do not satisfy a required compatibility lane. Real encrypted fixture reads remain in `minio-interop.yml`, and multi-node fault schedules remain in the existing nightly cluster lane. In-process reopen tests do not establish power-loss durability.
|
|
|
|
Run `python3 scripts/check_test_wiring.py --self-test` to exercise the negative cases: removed/ignored/filtered tests, malformed listing, absent fixtures, and wrong fixture hashes. Do not update hashes merely to silence the guard; a fixture change needs source/provenance and compatibility review.
|
|
## Scanner/Heal Evidence Receipts
|
|
|
|
The existing `scripts/check_test_wiring.py` also validates Scanner/Heal case
|
|
evidence registered in `.config/scanner-heal-required-tests.json`. It records
|
|
already-built binaries and checks existing nextest output; it does not build,
|
|
run tests, deploy servers, inject faults, or start another CI lane.
|
|
|
|
The initial case is `background-target-restart`, emitted by
|
|
`heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_remote_shards_after_background_target_restart`.
|
|
That test already runs in `e2e-nightly`. When `RUSTFS_SCANNER_HEAL_RUN_DIR` is set,
|
|
it checks the actual server and test-executable hashes against `run.json`, pins
|
|
the same server binary for all node starts, and writes its oracle only after
|
|
the real assertions pass. The artifact contains the actual pre/post target
|
|
PIDs, per-node S3 listings, expected and downloaded complete-body hashes/lengths,
|
|
and target-disk `VersionShardCensus` fingerprints. Existing baseline objects
|
|
must match their pre-fault physical manifests; the object created during the
|
|
outage has no pre-fault target shard and is checked for complete physical parts
|
|
and exact S3 content.
|
|
|
|
This case is a **four-node, one-drive-per-node process-restart test**. It is not
|
|
power-loss validation, a 3x4 EC8+4 experiment, an all-version inventory, or proof
|
|
of scanner enumeration, exact MRF disposition, legacy migration, or rollback.
|
|
The schema 2 registry separates the implemented single-set restart lane from
|
|
structured release lanes for authority coverage, checkpoint/crash, status and
|
|
outcome, MRF responsibility, mixed-version rollback, scheduler pressure,
|
|
maintenance producers, and EC8+4 multi-set coverage. All G01-G14/P1-P4 and
|
|
R-E/R-D/R-L release requirements stay `pending` until their actual
|
|
feature-specific oracles, measurements and required topologies exist. Missing
|
|
cases cannot be supplied by synthetic W20 results. W20's bounded JSON and
|
|
file-hash helpers are reused; its ABBA performance contracts remain in
|
|
`docs/operations/scanner-benchmark-runbook.md`.
|
|
Measured ABBA manifests must also carry the runbook's `release_evidence`
|
|
contract. The runner rejects reports that cannot bind the exact 3x4 EC8+4
|
|
topology, multi-pool/multi-set shape, distributed same-window metrics endpoints,
|
|
restart/crash modes, mixed-version reader/writer/rollback participation, and
|
|
allocation/flamegraph/RSS/save-frequency profile artifact plan. Synthetic runs
|
|
and manifests missing that contract remain harness-only evidence.
|
|
|
|
### Recording One Case
|
|
|
|
Use a committed source tree, independently built current binaries, sufficient
|
|
free disk space, and a task-owned artifact directory that does not yet exist.
|
|
Set `SERVER_BINARY` and `TEST_BINARY` to those exact executable paths. The begin
|
|
command requires the server's embedded `--version` commit to match the clean
|
|
checkout and its embedded Git status to be clean. The E2E crate's build script
|
|
embeds its build-time Git revision/dirty state, lockfile Git blob, enabled crate
|
|
features, target, profile and encoded Rust flags. It tracks the crate/dependency
|
|
trees, Cargo inputs and Git HEAD/ref/index, including `common.rs` restart logic.
|
|
The producer checks this compiled identity against the receipt; it does not
|
|
copy a current source revision into an older test binary's identity. The E2E
|
|
uses its existing temporary cluster directories and cleanup. `CARGO_TARGET_DIR`
|
|
controls compilation output; nextest's default report store remains the
|
|
workspace's `target/nextest`. Execute the existing selected case as follows:
|
|
|
|
```bash
|
|
CASE=background-target-restart
|
|
FILTER='test(test_cluster_root_heal_recovers_remote_shards_after_background_target_restart)'
|
|
RUN_DIR="$PWD/artifacts/scanner-heal-run"
|
|
export RUSTFS_E2E_EXPECTED_FEATURES=default
|
|
scripts/python_bin.sh scripts/check_test_wiring.py \
|
|
--begin-scanner-heal "$RUN_DIR" "$SERVER_BINARY" "$TEST_BINARY"
|
|
export RUSTFS_SCANNER_HEAL_RUN_DIR="$RUN_DIR"
|
|
export CARGO_BIN_EXE_rustfs="$SERVER_BINARY"
|
|
cargo nextest list --profile e2e-nightly -p e2e_test -E "$FILTER" \
|
|
--message-format json > "$RUN_DIR/listing.json"
|
|
rm -f target/nextest/e2e-nightly/junit.xml
|
|
set +e
|
|
cargo nextest run --profile e2e-nightly -p e2e_test -E "$FILTER"
|
|
test_exit=$?
|
|
set -e
|
|
cp target/nextest/e2e-nightly/junit.xml "$RUN_DIR/junit.xml"
|
|
scripts/python_bin.sh scripts/check_test_wiring.py --finish-scanner-heal "$RUN_DIR" "$test_exit"
|
|
scripts/python_bin.sh scripts/check_test_wiring.py --check-scanner-heal "$RUN_DIR" "$CASE"
|
|
```
|
|
|
|
Set `RUSTFS_E2E_EXPECTED_FEATURES` to the actual intended e2e crate feature set,
|
|
including `default` for a default-feature build, comma-separated for extra
|
|
features, or empty for `--no-default-features`. It is mandatory when beginning
|
|
a run. Crate features are distinct from the spawned server's build features.
|
|
|
|
Do not replace a nonzero command exit with zero. Missing JUnit or an oracle
|
|
emission failure also fails acceptance. Each retry needs a new run directory;
|
|
the producer refuses to overwrite an existing oracle. Keep failed-run logs and
|
|
artifacts. The receipt pins source revision, actual binary hashes, run identity,
|
|
start/finish times, and the artifact hashes. `listing.json`, `junit.xml`, and
|
|
each oracle are limited to 1 MiB; object evidence has the fixture's 9..65 object
|
|
bound. Credentials are not included in the receipt.
|
|
|
|
The checker binds nextest's flattened suite `binary-id`/`binary-path` to the
|
|
actual test executable and requires the JUnit testcase's embedded execution
|
|
timestamp to fall inside the receipt window (with millisecond precision).
|
|
Copying an old JUnit file and refreshing its mtime does not make it new evidence.
|
|
Schema versions, topology counts, PIDs, EC geometry and shard indices require
|
|
actual integers: booleans and fractional values are rejected, and an index must
|
|
fit the physical data-plus-parity geometry.
|
|
|
|
The checker rejects unselected/ignored tests, zero/duplicate JUnit cases,
|
|
failures, skipped tests, retry/flaky records, stale or changed artifacts,
|
|
different builds or run IDs, unchanged process IDs, wrong topology, missing
|
|
shard parts, and mismatched S3 content/listings. The raw oracle JSON is emitted
|
|
by the real E2E producer, not accepted from an adapter copying expectations.
|
|
|
|
`--check-scanner-heal "$RUN_DIR" release` checks available case evidence and
|
|
returns nonzero for every pending release requirement. A focused case pass
|
|
does not approve release. In particular, R-E requires fixed-budget real
|
|
restarts without an unbudgeted final sweep, R-D requires the full
|
|
manager/event/ledger disposition chain, and R-L requires source-conflict and
|
|
crash/retirement evidence. Reader-only or unit fixtures cannot substitute for
|
|
these. The external `rustfs/auto-testing` functional workflows propagate suite
|
|
failures. Their workflow status does not establish this registry's required
|
|
case coverage, build provenance, or object-level oracles.
|
|
|
|
For automation, `--check-scanner-heal-release "$RUN_DIR"` emits one compact
|
|
JSON decision and exits nonzero while blocked. `verified_cases` contains only
|
|
cases that pass the complete receipt, build provenance, nextest/JUnit and real
|
|
oracle checks; `rejected_cases` names registered cases that do not, and
|
|
`pending_gates` names the unimplemented release requirements and
|
|
`pending_lanes` names the structured release lanes that still need real
|
|
evidence. Schema 1 is deliberately marked `release_schema_capable: false`
|
|
because it models only the single-version, unversioned-object restart/crash
|
|
cases. Schema 2 can describe the wider release matrix, but approval still
|
|
requires every registered case to verify and every required gate to leave
|
|
`pending` only after a future checker can bind it to real feature-specific
|
|
evidence. The current checker hard-rejects missing structured requirements and
|
|
pending gates mapped to an implemented lane, so clearing pending text cannot
|
|
become approval. A focused run, synthetic harness, compile-only result,
|
|
skipped/retried test, ordinary CI success, or unregistered mixed-version,
|
|
rollback, EC8+4 or performance claim therefore cannot become a release approval.
|
|
For high-risk rollback gates, `evidence_fields` records the specific proof
|
|
fields that a future real-evidence checker must bind before a pending gate can
|
|
move out of the blocked set. G03 keeps scoped ACK tied to durable root
|
|
publication, ACK request identity, participating peer capability snapshots, and
|
|
mixed-peer fallback oracles; G09 keeps mixed-version reader, writer, and rollback
|
|
payload evidence explicit. These fields are part of the release contract, not
|
|
evidence by themselves.
|
|
|
|
The upgrade compatibility E2E can emit raw G09 JSON artifacts when
|
|
`RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR` points at a fresh, task-owned directory.
|
|
The rolling mixed-version test writes `G09-mixed_version_reader_evidence.json`
|
|
and `G09-mixed_version_writer_evidence.json` after the old/new reader and writer
|
|
assertions pass. The bucket-metadata rollback test writes
|
|
`G09-rollback_payload_evidence.json` after the current -> previous -> current
|
|
round trip has read back the known bucket configuration and objects. These
|
|
artifacts are measured inputs for a later release bundle; the bundle must still
|
|
record their relative paths, hashes, command provenance, timestamps, roles,
|
|
participating revisions, and case lists before
|
|
`--check-scanner-heal-release-bundle` can validate them.
|
|
|
|
For a release-candidate or PR-head Linux x86_64 host, run the full raw G09
|
|
artifact pass with:
|
|
|
|
```bash
|
|
scripts/run_scanner_heal_g09_upgrade_evidence.sh
|
|
```
|
|
|
|
The script mirrors the pinned previous-release asset used by the upgrade
|
|
workflow, builds the current checkout, runs the mixed-version and rollback E2E
|
|
lanes, and fails unless all three raw G09 artifacts are measured, revision-bound,
|
|
and role-bound. Use `--source-binary` for a custom previous-release binary on
|
|
another platform, or `--test mixed-version|rollback` while narrowing a failure.
|
|
It performs a free-space preflight before building so a saturated validation
|
|
host fails before producing partial evidence.
|
|
|
|
When the real release lanes have produced their dedicated artifacts, validate
|
|
the complete hard-gate bundle with:
|
|
|
|
```bash
|
|
scripts/python_bin.sh scripts/check_test_wiring.py \
|
|
--check-scanner-heal-release-bundle /path/to/release-evidence.json
|
|
```
|
|
|
|
The bundle checker is intentionally stricter than the case checker. It requires
|
|
schema 2 registry metadata, `evidence: measured`, the current checkout revision,
|
|
all G01-G14/P1-P4/R-E/R-D/R-L gates, per-gate `status: pass`, lane identity,
|
|
relative artifact paths, matching SHA256 hashes, and non-empty summaries. It
|
|
also binds each evidence field to its own run provenance: `source_revision`,
|
|
`run_id`, `measurement_window_id`, timezone-qualified `started_at` and
|
|
`finished_at`, command arguments, and artifact format. The field
|
|
`source_revision` must match the bundle revision, and measured performance
|
|
duration cannot exceed the recorded run window.
|
|
When an evidence or profile artifact declares a JSON format, the checker also
|
|
opens that artifact and requires its payload to repeat the same measured
|
|
`source_revision`, `run_id`, `measurement_window_id`, gate and field identity;
|
|
profile sub-artifacts must additionally name their artifact kind. Updating only
|
|
the outer bundle hash cannot turn a stale JSON summary into current release
|
|
evidence.
|
|
|
|
The hard evidence shape remains claim-specific: mixed-version gates must name at
|
|
least two participating versions, crash/durable replay gates must include
|
|
crash-boundary evidence, G14 must record EC8+4 with at least three nodes and four
|
|
drives per node plus multi-set and multi-pool evidence, performance gates need
|
|
measured durations, P3's pressure run needs at least two hours, and P1 needs a
|
|
symbolized profile summary with resolved samples. Every G14 field and every
|
|
performance gate's fields must also share one `measurement_window_id`, so EC8+4,
|
|
multi-set/multi-pool, ABBA, throughput, and profiling artifacts cannot be
|
|
stitched together from unrelated runs. P1 `profile_evidence` must bind every
|
|
required profile artifact kind (`allocation-profile`, `flamegraph`,
|
|
`rss-samples`, and `save-frequency`) with a relative path, artifact format,
|
|
non-empty file, matching SHA256, and the same measurement window when a
|
|
per-artifact window is declared. Missing, synthetic, stale, tampered, undersized,
|
|
or topology-mismatched evidence returns a compact blocked or invalid JSON result
|
|
and a nonzero exit.
|
|
|
|
The scheduler-pressure lane must also carry the numbers needed to close W09,
|
|
W10, and W11: bounded deferred item/byte/age limits, zero duplicate tasks,
|
|
pressure pacing engagement, recovery and lock-hold timings, fixed offered load,
|
|
foreground p95/p99 latency, throughput, error count, attempt-cost samples, and
|
|
completed heal object counts.
|
|
|
|
This command validates the evidence package; it does not create evidence. A
|
|
handwritten JSON file, a synthetic harness pass, a single focused case, or a
|
|
local unit fixture still cannot satisfy the distributed, mixed-version,
|
|
crash-restart, durable MRF replay, EC8+4, ABBA, or profiling gates.
|
|
|
|
Run parser/receipt regressions with
|
|
`scripts/python_bin.sh scripts/check_test_wiring.py --self-test`. Those fixtures
|
|
validate the checker only and produce no runtime or performance evidence.
|
|
|
|
For local bundle-shape dry runs, generate a task-owned fixture directory with:
|
|
|
|
```bash
|
|
scripts/python_bin.sh scripts/check_test_wiring.py \
|
|
--write-scanner-heal-release-bundle-fixture /path/to/fixture-dir
|
|
```
|
|
|
|
The generated file is marked `fixture_only` and is intentionally rejected by the
|
|
release bundle checker. Use it to rehearse field names, artifact paths, hashes,
|
|
profile artifact membership, mixed-version roles, and same-window provenance
|
|
before copying the shape into a real measured bundle. It is not ABBA, profile,
|
|
mixed-version, crash-restart, or release approval evidence.
|