Chris
4e16705873
* fix(ecstore): stop pruning at nonempty directories (#7616) * fix(ecstore): stop pruning at nonempty directories * test(ecstore): release pruning fixtures before temp cleanup (cherry picked from commit8f150d1d8e) * fix(heal): preserve retryable batch failures during recovery (#7642) * fix(heal): preserve retryable batch failures during recovery * test(heal): pin prebuilt hooks binaries in ci (cherry picked from commit5cd58319ed) * fix(s3): reject oversize single PUT early and map body errors to 4xx (#7635) * fix(s3): reject oversize single PUT early and map body errors to 4xx A single PutObject above the 5 GiB single-request ceiling was only rejected after the client had streamed 5 GiB into s3s's read-time body budget, and the resulting BodySizeLimitExceeded surfaced from the erasure writer as 500 InternalError. A body whose connection hit EOF before Content-Length bytes arrived (hyper's IncompleteBody) was also a 500. SDKs retry 500s, so one oversize upload was resent from offset 0 five times. - PutObject and UploadPart reject a declared length above MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the body; the constant moves to rustfs_config so the s3s limit and the admission check share one value. - ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body EOF to IncompleteBody across both io::Error conversions. Fixes #7596. * test(s3): cover UploadPart admission, aws-chunked length, real s3s limit - Poll-counting test body proves PutObject and UploadPart reject a declared size above the ceiling with zero body polls; exact-cap and zero-length parts pass admission. - A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the cap is admitted when the decoded length is within it and rejected when the decoded length is over it. - The display-based BodySizeLimitExceeded matcher is checked against the real error produced by the pinned s3s Body budget. (cherry picked from commit50b31bc75b) * fix(ecstore): make directory mtime fixture portable (#7623) * fix(ecstore): make directory mtime fixture portable * style(ecstore): format mtime fixture assertion --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> (cherry picked from commitb1cc286cac) * fix(storage): prevent readiness after native migration failures (#7652) * fix(storage): prevent readiness after native migration failures * fix(storage): skip unsupported IAM records before reading * fix(storage): use stable typed migration metadata errors * fix(storage): include migration record in startup errors * test(storage): cover native migration startup failures * test(storage): use array chunks in migration fixture --------- Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com> Co-authored-by: cxymds <cxymds@gmail.com> (cherry picked from commit0cbc3ffe61) * fix(admin): expose OIDC account display fields (#7654) Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers. (cherry picked from commitf02bc947cd) * fix(replication): correct peer joins and remote-state reporting (#7650) * fix(replication): propagate verified peer deployment identities * fix(replication): report actual remote peer state * fix(replication): defer initial sync until all peers join * test(replication): shut down TLS fixtures cleanly --------- Co-authored-by: houseme <housemecn@gmail.com> (cherry picked from commit853ae63b6a) * fix(s3): bound stalled UploadPart request bodies (#7659) * fix(s3): bound stalled UploadPart request bodies * fix(ci): preserve the S3S footprint ratchet (cherry picked from commit666dfd9f9f) * fix(tables): reject reserved warehouse locations (#7671) Co-authored-by: cxymds <cxymds@gmail.com> (cherry picked from commit414176c47f) * fix(ci): bind nightly lanes to one resolved source (#7688) (cherry picked from commit01d8e4347f) * fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626) * fix(replication): total-order rule sort and honor V1 top-level Prefix Rule matching had two defects from the pre-GA replication audit (rustfs/backlog#2367 C-1 and C-2): - The actionable-rule sort compared same-destination rules by priority but answered Equal for any other pair, which is not a total order; the standard library sort panics on such comparators once a slice exceeds the insertion-sort threshold, so an object matching more than 20 enabled rules across two or more targets could panic the PUT or DELETE task. Rules now sort by priority descending with destination and id as tie-breakers, and filter_target_arns preserves that order instead of draining a HashSet. - A V1 rule written without a <Filter> carries its prefix at the top level; that field was never read, so <Prefix>logs/</Prefix> matched every object. ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping precedence. The existing prefix fixtures were built this way and had been asserting nothing. * fix(admin): advertise data-usage and listen capabilities to rc The rc client gated `rc du` and `rc watch` on a pinned contract that matched server versions by the string prefix `1.0.0-rc.`; a server that reports `1.0.0` no longer matches, and the dynamic `advertised` list did not carry either name, so `rc du` against a GA server fails with an unsupported-capability error (rustfs/backlog#2367 E-2). Advertise `admin.data-usage` from the admin route inventory like the IAM entries, and `listen_notification` for the bucket `?events=` extension route the admin router dispatches. The client merges advertised entries ahead of its pinned contract, so no version sniffing is needed. * fix(site-replication): stop notifying the local site on remove and rotate The pending-remove and pending-rotation notification loops skipped the local site by endpoint only, while finalization identifies it by deployment id or endpoint. The reconcile tick resolves the local peer from the node's own listen address (and a handler from the request Host), so `remove --all` dialed the site's registered endpoint, waited out the request timeout against the lifecycle lock it was holding, and answered `Partial: failed to notify 1 peer(s)` for a removal that had succeeded (rustfs/backlog#2367 A-4, backlog#2195 item 3). Both loops now iterate the peers still awaiting notification through one helper that applies the finalization identity. * fix(site-replication): promote and settle IAM retries without a tick of slack Two retry-queue behaviours kept an IAM change from converging for ten to twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3, backlog#2305): - The lightweight 30-second pass filtered its reachability probe to bucket ops, so a backed-off IAM or bucket-metadata snapshot waited for the 600-second tick to notice the peer. It now probes every backed-off class and still replays only bounded bucket ops; promotion is a state flip the heavyweight tick acts on. - Backoffs are multiples of the tick interval, so a failure stamped δ seconds after a tick was 600 − δ old at the next tick and slipped a whole extra interval. The heavyweight drain now evaluates backoff halfway to its next tick. - An IAM entry first created by a non-deletion failure (the add bootstrap's snapshot send, the drain's own replay, an import-iam schedule) was never stamped `deletions_recorded`, so a later recorded deletion could not settle it and it escalated to the marker only `replicate repair` clears. Entries created by this binary now start recorded; a row persisted by an older binary keeps the escalation semantics. * fix(site-replication): reload peer node caches after bucket wiring writes Every S3 bucket-config write ends by asking the other nodes of the cluster to reload the bucket's metadata; the site-replication writers never did. On a multi-node site the node that ran the pairing (or applied a peer's bucket-meta item) rewrote the bucket targets and the derived replication rules on disk, while every other node kept serving its cached copy for up to the 15-minute refresh. A `resync start` routed to such a node reported every freshly wired bucket as `Config not found` and a bucket whose operator target the pairing had replaced as `recorded remote target no longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional SITE-105). Add one best-effort reload helper in the site-replication hooks and call it after the bucket setup, versioning, peer bucket-meta apply, removed-peer cleanup, make-with-versioning, and endpoint-refresh writes; the ensure helpers now report whether they wrote so unchanged passes stay silent. The resync manifest and start now read the persisted wiring instead of the node-local cache, matching the target read the start path already did. The new four-node e2e pairs two clusters and starts a resync through a non-coordinator node right after pairing; it also covers an IAM user created on a non-coordinator node converging to the peer site. * test(e2e): cover delete-marker replication from a multi-node source The functional suite reported delete markers created on a 3-node source never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report was a probe defect, but the shape had no coverage: the existing delete-marker e2e runs a single-node source. Pin it against a four-node source replicating to a four-node peer and to a single-node target, with the write and the delete issued through different nodes. * ci(e2e): refresh the distributed selection for the new replication cases Four distributed cases were added (two site-replication, two delete-marker replication). The linux digest is derived from the last CI listing of the lane (34 cases, matching the previous pin) plus the four new names; the darwin digest is the local listing, which selects the same 38 cases. (cherry picked from commitaeaba86d73) * fix(e2e): require a verified server binary for every e2e run (#7687) * fix(ci): share quick checks and lint workflows * fix(ci): install actionlint from its verified release * fix(ci): reject dependencies on required quick checks * feat(test): verify the E2E server build and source identity * test(e2e): register verified Darwin test membership * test(e2e): record verified Linux receipt test membership * test(e2e): record compiled Darwin receipt test membership * test(e2e): record compiled Linux receipt test membership * test(e2e): record Darwin e2e-full membership after merging main * fix(test): route scanner/heal evidence E2E runs through the verified server binary The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed. * docs(e2e): run server-backed e2e commands through the verified binary wrapper * test(e2e): record Linux e2e-full membership from the branch CI listing (cherry picked from commit2909b1bfe1) * fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697) * fix(sse): classify bare SSE-KMS writes when no KMS is available A `aws:kms` request without a key id, on a bucket without a default key, returned `500 InternalError` whenever no KMS service was running: the "no KMS key available" branch exited with an untyped storage error before the availability classification that the keyed form already received. Route that branch through the same split: `503 ServiceUnavailable` while a configured KMS is stopped, `400 InvalidRequest` when KMS was never configured, and `400 InvalidRequest` naming the missing key id when a running KMS has no default key. `CreateMultipartUpload` shares the path. Adds a unit test for the bare form and an e2e module that stops KMS through the admin API, runs a master-key-only node, and runs a Local KMS without a default key; refreshes the e2e-full selection digests. (cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8) * fix(sse): keep KMS error classes on the encrypted read path GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key no longer exists answered `500 InternalError` ("KMS key not found") while PutObject under the same key already answered `400 KMS.NotFoundException`. The read path carries its classification through ecstore's `EncryptionResolutionErrorKind`, which had no kind for a missing key, a denied KMS grant or a missing backend capability, so all three folded onto `DecryptionFailed` and the S3 layer reported an internal fault. Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on both sides of the boundary, and give an envelope the configured backend cannot unwrap a diagnosable message while keeping its `500`. Unit tests cover the kind round trip and the reader wrapping; a new e2e test deletes a key immediately and checks GET/Copy return 400 with `KMS.NotFoundException` while HEAD stays 200. The e2e-full selection digests are refreshed from the current listing (the previous digests predated the delete-authorization tests) and the e2e `create_default_key` helper is updated to the accepted `EncryptDecrypt` spelling. (cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2) * fix(kms): classify key-management errors on the admin routes `POST /kms/keys`, the legacy `create-key` alias and `generate-data-key` reported every backend refusal as `500`: a blank key name (which each backend failed on differently, the Local backend by writing a key file with an empty stem), a name already taken, an unknown key, a disabled key and a capability the backend lacks. `delete` and the lifecycle routes already classified the same errors. Refuse a blank or whitespace name in `KmsManager::create_key` before any backend sees it, and share one `KmsError` to status mapping across create, delete and generate-data-key (400 for validation and key state, 404 for an unknown key, 409 for a taken name, 501 for a missing capability, 500 only for damaged material). The XML-error routes carry the same status explicitly since s3s derives none for a custom code. The read-only Static backend now reports create, delete and cancel-deletion as `UnsupportedCapability`, matching its rotate and enable/disable answers, so the admin API returns 501 for all of them. (cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b) * fix(sse): stop SSE-S3 responses from naming the wrapping KMS key `x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms` objects only, but PutObject, CopyObject, CreateMultipartUpload and GetObject returned it for `AES256` objects too, carrying the KMS key that wraps the SSE-S3 data key (the service default, or the literal `default` on a node without KMS). The write paths copied `kms_key_id` from the encryption material unconditionally, and the single-decrypt GET classification did the same after resolving the key for authorization. Add `EncryptionMaterial::response_kms_key_id`, which yields the id only for SSE-KMS, use it at the four write-response sites, and gate the GET classification the same way. CompleteMultipartUpload and HeadObject already omitted the header. Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy and CreateMultipartUpload for AES256 with an aws:kms control. The e2e-full selection digests are refreshed from the current listing. (cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb) * fix(s3): validate PutBucketEncryption rules before storing them A default-encryption rule naming an unknown `SSEAlgorithm` (for example `AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as written: the only algorithm check on the route decided whether to fill in the default KMS key. `GetBucketEncryption` then advertised that configuration while the write path encrypted header-less writes under its `AES256` fallback, so the bucket's declared and actual schemes disagreed. Two comments claimed the route already refused unknown algorithms. Validate the configuration before any of it is applied: `MalformedXML` for a malformed rule set or unknown algorithm, `InvalidArgument` for a key id on a non-KMS rule, and nothing stored on refusal. Correct the two comments to describe when the AES256 fallback is still reachable. Unit tests cover every refusal and the accepted shapes; an e2e test checks the refusals leave the previous configuration in place. The e2e-full selection digests are refreshed from the current listing. (cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb) * test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes * test: align two unit tests with the new KMS and bucket-encryption contracts `scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still expects the state error (`InvalidOperation`) for cancelling a key that is not pending deletion; only the Static backend's mutations moved to `UnsupportedCapability`. The uninitialized-store PutBucketEncryption test now sends a well-formed AES256 rule so it reaches the store lookup instead of the new configuration validation. (cherry picked from commite2e6a2535a) * fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over (#7709) * fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over Site replication wired each bucket by looking for an existing replication target "to the same peer" and rewriting the first match in place as its own same-name target. An operator's bucket-level target that happened to point at that site (different target bucket, operator credentials) was the first match whenever it pre-dated the join, and the reconciler repeats the pass every 600s, so the takeover also depended on target order afterwards. The operator's rule then named an ARN no target backed and their bucket replication stopped silently, while the inherited bucket-level reset id made every site resync report the bucket as owned by another resync (rustfs/backlog#2479, rustfs/backlog#2489). Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead: - Wiring updates a target in place only under the same ARN, or when it is recognisably the site's own under an older ARN shape (same peer, same-name target bucket, site replication service account). Anything else gets the site target added next to it. - The site resync manifest takes the target the derived `site-repl-<deployment id>` rule names (same-name shape as fallback), so an operator target to the peer neither aborts the bucket as "multiple remote targets matched peer" nor gets resynced into. - Peer removal prunes only targets a pruned derived rule names or the same-name target bucket; operator targets stamped with the peer's deployment id survive together with their rules. Unit tests cover the three predicates. e2e `test_site_replication_keeps_operator_bucket_target_to_peer` runs a bucket-level replication plus `replication-reset` to the future peer, joins the sites, and requires the operator target untouched, both paths delivering, the site resync completing against the site target, and the operator target and rule surviving `replicate remove --all`; without the fix it fails at the join with the operator target gone. The repl-nightly selection digest is refreshed for the new case. * test(site-replication): drop a redundant clone flagged by clippy The reconcile unit test cloned the remote peer into the state map although the binding is not used afterwards; workspace clippy (-D warnings) rejects that as redundant_clone. (cherry picked from commitecdc55fa4b) * fix: enforce S3 permissions for recursive force deletion (#7661) * fix: enforce S3 authorization for recursive deletion * fix: satisfy the s3s footprint guard * fix: restore list versions policy compatibility (#7686) (cherry picked from commit3fd1ce414d) * fix(ci): repair functional defaults and chain regression checks (#7664) * fix(ci): default functional suites to nightly packages * test(ci): follow the fault-tolerance chain handoff (cherry picked from commit509a0fa90c) * fix(ci): align security workflow tests with chain (#7679) (cherry picked from commitd9e47d2813) * test(ecstore): keep tier cleanup tests stable after immediate receipt queueing Release PR #7766 made PUT/CopyObject overwrites queue the tier free-version cleanup receipt immediately, which broke two ecstore tests on release CI. In tier_overwrite_put_and_self_copy_recover_persisted_cleanup_owners the restarted store already runs expiry workers from the second iteration on, so they deleted the remote bytes before the test could assert that the commit leaves them in place; the test now fails the first remote DELETE via set_remove_failure(true) so the cleanup owner stays durable and the later restart still has to rediscover it from xl.meta (the failed remove does not bump remove_count, and the test re-enables removes before the recovery wait). In batch_transitioned_delete_post_commit_failures_roll_back_without_free_version_receipt the convergence loop now treats a transient InsufficientReadQuorum as "not yet converged", because cleanup rewrites xl.meta disk by disk and a racing read can briefly miss quorum (seen in the rio-v2 lane); any other error still panics. * Revert "fix(ci): align security workflow tests with chain (#7679)" This reverts commit4544359f6d. * Revert "fix(ci): repair functional defaults and chain regression checks (#7664)" This reverts commit5ba7ec0291. * fix(ecstore): pass shard integrity to backported ingest-mode test The stalled-reader test backported with #7659 used main's four-argument encode_with_ingest_mode, but release's signature takes an optional IntegrityBuilder, so pass None to keep the test focused on ingest-mode cleanup. --------- Co-authored-by: Henry Guo <marshawcoco@gmail.com> Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: RJ Regenold <rregenold@teamraft.com> Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com> Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: Jason Kossis <jkossis@gmail.com>
37 KiB
CI gate matrix
Use this when: a check is red and you need to know whether it blocks the merge, which workflow and job produced it, and how to reproduce it locally.
Source of truth: the live main ruleset (command below) for required status; .github/workflows/<file>.yml for triggers, paths, timeout-minutes, and cron; .config/nextest.toml for e2e profile filters; .github/scheduled-validations.json for the freshness-watchdog list.
A job blocks a merge when its exact check name is required by the live main ruleset, or when its result is required by the Test and Lint aggregate. A workflow name, a merge_group trigger, or an unrelated red PR check does not make a job required by itself.
Required merge checks
The main ruleset (6436880) requires exactly these contexts, with strict_required_status_checks_policy=false:
| Required context | Producer | Validation |
|---|---|---|
CLA Check |
cla.yml |
Contributor agreement |
Quick Checks |
ci.yml job quick-checks |
Formatting and repository guard scripts |
Test and Lint |
ci.yml job required-checks |
Exact expected results for every CI validation job, including workspace checks, critical E2E, feature lanes, and event-specific full suites |
Every PR enters ci.yml. The classify-changes job uses the base revision of scripts/ci_gate.py to select a conservative documentation-only path: root Markdown/licenses, AGENTS.md, Markdown under docs/ or .agents/skills/, and documentation images. Unknown paths, unavailable Git history, an empty diff, or a missing base policy select the full matrix. Renames include their deleted source path. Documentation-only PRs still run Quick Checks and Typos; the aggregate requires the expensive jobs to be skipped exactly as selected.
required-checks runs even after failed or skipped dependencies. scripts/ci_gate.py verify rejects missing jobs, unexpected jobs, failure, cancellation, and unexpected skips; optional lanes are required only on their declared events. Workspace Test and Lint is the ordinary Rust job, while Test and Lint uniquely names the aggregate. New validation jobs must update both its direct dependencies and the script contract. Test this wiring and its failure cases with python3 scripts/ci_gate.py --self-test.
Verify the live rule before changing merge policy:
gh api repos/rustfs/rustfs/rulesets/6436880 \
--jq '.rules[] | select(.type == "required_status_checks") | .parameters'
The aggregate requires the validation lanes already selected by ci.yml; this closes the gap where a failing critical lane left the required workspace check green. Independent workflows remain report-only unless separately required. Before adding a new expensive lane or moving existing PR coverage to a schedule, collect representative execution and regression evidence, establish ownership and a working scheduled replacement, and update this reference with the resulting policy.
Pull request and merge matrix
"Via aggregate" means a wrong result fails the required Test and Lint check. "Report-only" means visible and actionable but outside both the required list and aggregate. Budgets are each job's timeout-minutes in the named workflow and are not copied here.
| Event | Check name | Workflow / job | Merge status | Reproduce |
|---|---|---|---|---|
| PR, non-doc change | Quick Checks |
ci.yml quick-checks |
Required | make pre-commit |
| PR, non-doc change | Workspace Test and Lint |
ci.yml test-and-lint |
Via aggregate | cargo clippy --all-targets -- -D warnings; cargo nextest run --profile ci --all --exclude e2e_test; cargo test --all --doc; scripts/check_migration_gate_count.sh |
| PR, non-doc change | Typos |
ci.yml typos |
Via aggregate | typos |
| PR, non-doc change | ILM Integration (serial) |
ci.yml test-ilm-integration-serial |
Via aggregate | exact command in the job |
| PR, non-doc change | Test and Lint (rio-v2), Test and Lint (swift), Test and Lint (sftp) |
ci.yml test-and-lint-rio-v2, test-and-lint-protocols |
Via aggregate | cargo nextest run with the job's --features |
| PR, non-doc change | Connect Short Credential Boundary |
ci.yml connect-short-credential-boundary |
Via aggregate | cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials; cargo check -p rustfs --release --features connect-e2e-short-credentials must fail |
| PR, non-doc change | Build RustFS Debug Binary |
ci.yml build-rustfs-debug-binary |
Via aggregate; prerequisite for black-box jobs | python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks (binary plus its rustfs.e2e.json sidecar) |
| PR, non-doc change | io_uring Integration (real) |
ci.yml uring-integration |
Via aggregate | cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture |
| PR, non-doc change | End-to-End Tests |
ci.yml e2e-tests |
Via aggregate | python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test, then ./scripts/e2e-run.sh ./target/debug/rustfs <data-dir> under the same wrapper; membership guards scripts/check_test_wiring.py --check-profile e2e-smoke <listing.json> and scripts/check_security_smoke_count.sh check <listing.json> |
| PR, non-doc change | S3 Implemented Tests |
ci.yml s3-implemented-tests |
Via aggregate | build rustfs, then scripts/s3-tests/run.sh with the job's DEPLOY_MODE / TEST_MODE / MAXFAIL env |
| PR, non-doc change | S3 Lifecycle Behavior Tests |
ci.yml s3-lifecycle-behavior-tests |
Via aggregate | scripts/s3-tests/run.sh with the job's accelerated-scanner env |
PR touching paths in audit.yml |
Cargo Deny, Workflow Pin Report, Dependency Review |
audit.yml cargo-deny, workflow-pin-report, dependency-review |
Report-only | cargo deny check; scripts/security/check_workflow_pins.sh |
PR touching paths in architecture-migration-rules.yml |
Architecture Migration Rules |
architecture-migration-rules.yml architecture-migration-rules |
Report-only | scripts/check_architecture_migration_rules.sh |
PR touching paths in nix.yml |
Nix Build & Check |
nix.yml nix-validation |
Report-only | nix flake check |
PR touching paths in fuzz.yml |
Build Fuzz Harness, Smoke / <target> |
fuzz.yml fuzz-build, pr-fuzz-smoke |
Report-only | MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh |
PR touching paths in windows-filesystem.yml |
Rename Safety |
windows-filesystem.yml rename-safety |
Report-only | the cargo test -p rustfs-ecstore --lib <filter> commands in the job, on Windows |
PR touching paths in coverage.yml |
Workspace line coverage |
coverage.yml coverage |
Report-only | make coverage; python3 scripts/check_security_coverage.py target/llvm-cov/coverage.json |
PR touching paths in e2e-upgrade.yml |
Direct upgrade from the previous release, Mixed-version rolling upgrade from the previous release, Bucket configuration survives the upgrade, Rollback reads current bucket metadata |
e2e-upgrade.yml upgrade matrix |
Report-only | python3 scripts/e2e_binary.py build, then the job's python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test command with RUSTFS_UPGRADE_SOURCE_BINARY pointing at the pinned previous release (UPGRADE_SOURCE_VERSION) |
PR touching paths in oidc-keycloak.yml |
OIDC Keycloak live gate |
oidc-keycloak.yml oidc-keycloak-live |
Report-only | cargo build --locked -p rustfs --bin rustfs, then bash scripts/test/oidc_keycloak_live.sh ./target/debug/rustfs |
PR touching paths in targets-integration.yml |
PostgreSQL, MySQL, AMQP, and NATS |
targets-integration.yml targets-live |
Report-only | start the containers as in the job, export the RUSTFS_TEST_* DSNs, then the job's cargo test --locked -p rustfs-targets --test <name> -- --ignored --test-threads=1 commands |
| PR, documentation-only selection | Quick Checks, Typos, Test and Lint |
ci.yml quick-checks, typos, required-checks |
Required directly or via aggregate | Quick Checks commands; python3 scripts/ci_gate.py --self-test |
merge_group; push to main |
End-to-End Tests (full merge gate) |
ci.yml e2e-full |
Via aggregate on these events | python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-full -p e2e_test (CI adds --binary "$RUSTFS_E2E_STARTUP_CAS_BINARY" for the downloaded binary and sidecar) |
e2e filters live in .config/nextest.toml; extend a profile instead of adding a second selector. Before a profile runs, scripts/check_test_wiring.py compares its listing to the committed digest in .config/e2e-<profile>-selection.txt, so a silent test drop fails closed.
Scanner usage and heal rebuild coverage are intentionally split by risk and
cost. data_usage_test runs in the PR e2e-smoke lane so changes that affect
authoritative scanner usage publication, quota-visible usage, or admin usage
snapshots get an end-to-end signal before merge review. heal_erasure_disk_rebuild_test
runs in e2e-full so core erasure heal rebuild regressions are caught no later
than the merge queue or main push lane; it also remains in e2e-nightly with
the serialized cluster fault-domain suites for scheduled soak signal.
Scheduled validation
Scheduled lanes never block a PR. Their workflow-local gate fails the run, scheduled failures route to the shared failure-issue action, and scheduled-validation-freshness.yml fails when a workflow listed in .github/scheduled-validations.json has no recent attempt or completed successful scheduled run within its max_age_hours (a never_ran_grace_until entry covers the window before a newly enabled cron's first slot). Cadence is qualitative here; the cron lives in each workflow's on.schedule.
| Workflow (cadence) | Jobs | Verdict and artifacts | In freshness list | Reproduce |
|---|---|---|---|---|
ci.yml (weekly) |
full matrix, including the schedule/dispatch-only rio-v2 jobs build-rustfs-debug-binary-rio-v2 and e2e-tests-rio-v2 |
strict aggregate; the full E2E lane runs on dispatch, merge groups, and main pushes | yes | dispatch ci.yml |
build.yml (weekly) |
build-rustfs over the six-target platform matrix in prepare-platform-matrix (four Linux, macOS aarch64, Windows x86_64) |
build/package integrity | yes | dispatch build.yml with an exact platform set |
e2e-replication-nightly.yml (nightly) |
repl-nightly, cluster-nightly, protocols-nightly |
three independent gates; JUnit, membership listing, server logs | yes | python3 scripts/e2e_binary.py build --bins, then python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-repl-nightly -p e2e_test; --features e2e-test-hooks on both steps with --profile e2e-nightly; --features ftps,webdav,sftp on both steps with -j 1 --profile e2e-protocols |
e2e-distributed.yml (storage-sensitive PRs + nightly) |
distributed |
fail-closed 4-node 4-disk S3, durability, replication, movement, fault, and direct/rolling upgrade gate; JUnit, membership listing, per-node server logs | yes, with never_ran_grace_until |
download the pinned previous release as in the workflow, export RUSTFS_UPGRADE_SOURCE_BINARY, run python3 scripts/e2e_binary.py build --bins, then python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test |
e2e-s3tests.yml (weekly) |
s3tests (single and distributed, four shards each), upstream-head-canary |
compatibility gate; report, JUnit, node IDs, server logs | yes | scripts/s3-tests/run.sh against an existing single or distributed target |
fuzz.yml (nightly) |
nightly-fuzz-corpus per target |
gate; corpus and crash artifacts | yes | MAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh |
minio-interop.yml (nightly) |
minio-interop |
EC + SSE read-parity gate | yes, with never_ran_grace_until |
pinned Docker fixture steps in the workflow |
on-demand-migration-interop.yml (nightly) |
minio-source, cloud-source (aws, r2, gcs) |
report-only provider interop; one JSON report per provider naming cases, timings and source request counts, plus JUnit and MinIO logs. A cloud provider whose ODM_INTEROP_* secrets are absent is skipped with a summary note, not failed |
no | start the pinned MinIO container as in the job, export the RUSTFS_ODM_INTEROP_* variables, run python3 scripts/e2e_binary.py build --bins, then python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-odm-interop -p e2e_test |
performance-ab.yml (nightly) |
warp-ab |
regression-budget gate; A/B summaries and server logs | yes | bash scripts/run_hotpath_warp_abba.sh --help |
nightly-gnu.yml (nightly) |
build, kms-vault-lane, kms-vault-ha-failover |
build, live Vault, and HA failover gates | yes | commands and pinned Vault images in the workflow |
audit.yml (nightly) |
cargo-deny, workflow-pin-report |
dependency and workflow-pin gates | yes | cargo deny check; scripts/security/check_workflow_pins.sh |
mint.yml (weekly) |
mint |
report-only by design; per-suite PASS/FAIL/NA and raw log.json |
yes | pinned Docker sequence in the workflow |
coverage.yml (weekly) |
coverage |
report-only trend; lcov and JSON artifact | yes | make coverage |
runner-hygiene.yml (monthly) |
check-ephemerality |
runner ephemerality | yes | dispatch |
e2e-upgrade.yml (weekly) |
upgrade (4-case matrix) |
upgrade and rollback gate; server logs | no | see the PR row |
oidc-keycloak.yml (weekly) |
oidc-keycloak-live |
live OIDC gate | no | see the PR row |
targets-integration.yml (nightly) |
targets-live |
live target gate; container logs | no | see the PR row |
scheduled-validation-freshness.yml (nightly) |
check-freshness |
fails on missing or stale attempts or completed successes | n/a | dispatch |
Manual workflow_dispatch runs are debugging evidence and do not open scheduled-failure issues. A manual performance run may explicitly allow a known regression; that override is not a passing baseline.
Packaged functional acceptance
rustfs-functional-chain.yml dispatches the packaged-build suites in rustfs-*-test.yml on the shared lab runners. A failing suite step or job must fail its workflow. Report collection, cleanup, and dispatch of the next suite can still run with always(); continuing diagnostics does not make the failed suite successful.
Workflow status preserves errors that the test scripts report. It does not establish complete execution or a common package identity across the chain: inspect the current run's case results, package identity, and test-script revision as well. A script that returns zero after a failed tool invocation needs its own result check.
Release validation
Post-merge and tag-driven; not a substitute for a PR gate.
| Trigger | Workflow / job | Result |
|---|---|---|
Push to main, weekly schedule, dispatch |
build.yml build-rustfs (a development build on a main push restricts the matrix to the Linux targets) |
build artifacts; no release publication |
| Valid release or preview tag | build.yml build-rustfs, create-release, upload-release-assets, publish-release |
draft release, checksummed assets, publish |
Successful non-preview release-tag build (workflow_run) |
docker.yml build-docker, scan-docker-image |
multi-architecture images and vulnerability report |
Successful release-tag build (workflow_run) |
package.yml package |
DEB/RPM packages and checksums uploaded to the release |
Successful non-preview release-tag build (workflow_run) |
helm-package.yml build-helm-package, publish-helm-package |
versioned chart and repository index |
| Final tag's release published | build.yml cleanup-preview-releases |
deletes every <target>-preview.<N> Release for that target; the tags are kept |
Use an exact preview tag for an end-to-end release rehearsal. Manual dispatches are backfill/debug paths and do not prove the automatic workflow_run chain.
Change checklist
Update this file in the same PR when a job or check name changes, a workflow gains or loses a pull_request or schedule trigger, required contexts or strict/merge-queue policy change, report-only vs gating semantics change, or .github/scheduled-validations.json membership changes. Do not copy timeouts, crons, or test counts here.
ECStore invariant selection
The existing ci.yml test-and-lint job runs the ordinary ECStore and filemeta tests. After that run, scripts/check_test_wiring.py --check-core checks the same nextest profile and package selection against .config/ecstore-required-tests.json. Every named test must exist, match the filter, and be non-ignored; the job also requires a nonempty JUnit report. This checks membership without running the tests twice. core-test-listing.json, JUnit, and the run log are retained in the existing test-and-lint artifact.
The manifest records a minimum set of invariants: write quorum, metadata rollback, stale-writer lock loss, plaintext Range content, multipart cancellation, hiding uncommitted LIST versions, real MinIO metadata, corrupt part arrays, and the shared on-demand-migration source-backend contract for each provider dialect (S3, Azure, native GCS). The three contract entries live in the rustfs suite and reach the lane through that package's default features, so dropping gcs from rustfs's defaults fails this check instead of silently deselecting the GCS contract (rustfs/backlog#2323). Renaming or moving a required test must update the manifest in the same change after checking the compiled listing. Extend this list as new deterministic regressions land; it is not a claim that all storage invariants are covered.
The checked-in MinIO corpus is pinned by file SHA256 and its documented source release. The static wiring guard and the CI selection check both reject missing or changed fixtures. These are metadata fixtures, not a legacy shard-body corpus or proof of crash durability. Optional legacy_bitrot_read_test runs may still skip when their external corpus is absent; they do not satisfy a required compatibility lane. Real encrypted fixture reads remain in minio-interop.yml, and multi-node fault schedules remain in the existing nightly cluster lane. In-process reopen tests do not establish power-loss durability.
Run python3 scripts/check_test_wiring.py --self-test to exercise the negative cases: removed/ignored/filtered tests, malformed listing, absent fixtures, and wrong fixture hashes. Do not update hashes merely to silence the guard; a fixture change needs source/provenance and compatibility review.
Scanner/Heal Evidence Receipts
The existing scripts/check_test_wiring.py also validates Scanner/Heal case
evidence registered in .config/scanner-heal-required-tests.json. It records
already-built binaries and checks existing nextest output; it does not build,
run tests, deploy servers, inject faults, or start another CI lane.
The registered cases are emitted by existing E2E tests. The original
background-target-restart / background-target-crash cases run in
e2e-nightly on a four-node, one-drive-per-node topology. The
ec84-target-drive-restart case runs in e2e-distributed on a three-node,
four-drive EC8+4 topology. When RUSTFS_SCANNER_HEAL_RUN_DIR is set, the
producer checks the actual server and test-executable hashes against run.json,
pins the same server binary for all node starts, and writes its oracle only
after the real assertions pass. The artifact contains the actual pre/post target
PIDs, per-node S3 listings, expected and downloaded complete-body hashes/lengths,
and target-disk VersionShardCensus fingerprints. Existing baseline objects
must match their pre-fault physical manifests; the object created during the
outage has no pre-fault target shard and is checked for complete physical parts
and exact S3 content.
These cases are still restart-focused evidence slices. They are not power-loss
validation, an all-version inventory, or proof of scanner enumeration, exact MRF
disposition, legacy migration, multi-pool/multi-set release coverage, or
rollback.
The schema 2 registry separates the implemented single-set restart lane from
structured release lanes for authority coverage, checkpoint/crash, status and
outcome, MRF responsibility, mixed-version rollback, scheduler pressure,
maintenance producers, and EC8+4 multi-set coverage. All G01-G14/P1-P4 and
R-E/R-D/R-L release requirements stay pending until their actual
feature-specific oracles, measurements and required topologies exist. Missing
cases cannot be supplied by synthetic W20 results. W20's bounded JSON and
file-hash helpers are reused; its ABBA performance contracts remain in
docs/operations/scanner-benchmark-runbook.md.
Measured ABBA manifests must also carry the runbook's release_evidence
contract. The runner rejects reports that cannot bind the exact 3x4 EC8+4
topology, multi-pool/multi-set shape, distributed same-window metrics endpoints,
restart/crash modes, mixed-version reader/writer/rollback participation, and
allocation/flamegraph/RSS/save-frequency profile artifact plan. Synthetic runs
and manifests missing that contract remain harness-only evidence.
Recording One Case
Use a committed source tree, independently built current binaries, sufficient
free disk space, and a task-owned artifact directory that does not yet exist.
Set SERVER_BINARY and TEST_BINARY to those exact executable paths. The begin
command requires the server's embedded --version commit to match the clean
checkout and its embedded Git status to be clean. The E2E crate's build script
embeds its build-time Git revision/dirty state, lockfile Git blob, enabled crate
features, target, profile and encoded Rust flags. It tracks the crate/dependency
trees, Cargo inputs and Git HEAD/ref/index, including common.rs restart logic.
The producer checks this compiled identity against the receipt; it does not
copy a current source revision into an older test binary's identity. The E2E
uses its existing temporary cluster directories and cleanup. CARGO_TARGET_DIR
controls compilation output; nextest's default report store remains the
workspace's target/nextest. Prefer the registry-aware runner for concrete
cases:
scripts/run_scanner_heal_evidence_case.sh --case background-target-restart
scripts/run_scanner_heal_evidence_case.sh --case ec84-target-drive-restart
Set RUSTFS_E2E_EXPECTED_FEATURES to the actual intended e2e crate feature set,
including default for a default-feature build, comma-separated for extra
features, or empty for --no-default-features. It is mandatory when beginning
a run. Crate features are distinct from the spawned server's build features.
Do not replace a nonzero command exit with zero. Missing JUnit or an oracle
emission failure also fails acceptance. Each retry needs a new run directory;
the producer refuses to overwrite an existing oracle. Keep failed-run logs and
artifacts. The receipt pins source revision, actual binary hashes, run identity,
start/finish times, and the artifact hashes. listing.json, junit.xml, and
each oracle are limited to 1 MiB; object evidence has the fixture's 9..65 object
bound. Credentials are not included in the receipt.
The checker binds nextest's flattened suite binary-id/binary-path to the
actual test executable and requires the JUnit testcase's embedded execution
timestamp to fall inside the receipt window (with millisecond precision).
Copying an old JUnit file and refreshing its mtime does not make it new evidence.
Schema versions, topology counts, PIDs, EC geometry and shard indices require
actual integers: booleans and fractional values are rejected, and an index must
fit the physical data-plus-parity geometry.
The checker rejects unselected/ignored tests, zero/duplicate JUnit cases, failures, skipped tests, retry/flaky records, stale or changed artifacts, different builds or run IDs, unchanged process IDs, wrong topology, missing shard parts, and mismatched S3 content/listings. The raw oracle JSON is emitted by the real E2E producer, not accepted from an adapter copying expectations.
--check-scanner-heal "$RUN_DIR" release checks available case evidence and
returns nonzero for every pending release requirement. A focused case pass
does not approve release. In particular, R-E requires fixed-budget real
restarts without an unbudgeted final sweep, R-D requires the full
manager/event/ledger disposition chain, and R-L requires source-conflict and
crash/retirement evidence. Reader-only or unit fixtures cannot substitute for
these. The external rustfs/auto-testing functional workflows propagate suite
failures. Their workflow status does not establish this registry's required
case coverage, build provenance, or object-level oracles.
For automation, --check-scanner-heal-release "$RUN_DIR" emits one compact
JSON decision and exits nonzero while blocked. verified_cases contains only
cases that pass the complete receipt, build provenance, nextest/JUnit and real
oracle checks; rejected_cases names registered cases that do not, and
pending_gates names the unimplemented release requirements and
pending_lanes names the structured release lanes that still need real
evidence. Schema 1 is deliberately marked release_schema_capable: false
because it models only the single-version, unversioned-object restart/crash
cases. Schema 2 can describe the wider release matrix, but approval still
requires every registered case to verify and every required gate to leave
pending only after a future checker can bind it to real feature-specific
evidence. The current checker hard-rejects missing structured requirements and
pending gates mapped to an implemented lane, so clearing pending text cannot
become approval. A focused run, synthetic harness, compile-only result,
skipped/retried test, ordinary CI success, or unregistered mixed-version,
rollback, EC8+4 or performance claim therefore cannot become a release approval.
For high-risk rollback gates, evidence_fields records the specific proof
fields that a future real-evidence checker must bind before a pending gate can
move out of the blocked set. G03 keeps scoped ACK tied to durable root
publication, ACK request identity, participating peer capability snapshots, and
mixed-peer fallback oracles; G09 keeps mixed-version reader, writer, and rollback
payload evidence explicit. These fields are part of the release contract, not
evidence by themselves.
The upgrade compatibility E2E can emit raw G09 JSON artifacts when
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR points at a fresh, task-owned directory.
The rolling mixed-version test writes G09-mixed_version_reader_evidence.json
and G09-mixed_version_writer_evidence.json after the old/new reader and writer
assertions pass. The bucket-metadata rollback test writes
G09-rollback_payload_evidence.json after the current -> previous -> current
round trip has read back the known bucket configuration and objects. These
artifacts are measured inputs for a later release bundle; the bundle must still
record their relative paths, hashes, command provenance, timestamps, roles,
participating revisions, and case lists before
--check-scanner-heal-release-bundle can validate them.
For a release-candidate or PR-head Linux x86_64 host, run the full raw G09 artifact pass with:
scripts/run_scanner_heal_g09_upgrade_evidence.sh
The script mirrors the pinned previous-release asset used by the upgrade
workflow, builds the current checkout, runs the mixed-version and rollback E2E
lanes, and fails unless all three raw G09 artifacts are measured, revision-bound,
and role-bound. Use --source-binary for a custom previous-release binary on
another platform, or --test mixed-version|rollback while narrowing a failure.
It performs a free-space preflight before building so a saturated validation
host fails before producing partial evidence.
Operator-collected G01 root/quota authority evidence can be packaged with:
scripts/run_scanner_heal_authority_evidence.py \
--root-authority-json /path/to/root-authority.json \
--quota-authority-json /path/to/quota-authority.json \
--out-dir /path/to/authority-descriptor
The producer rejects fixture, dry-run, synthetic, stale-revision, incomplete
root authority, and incomplete quota authority inputs before writing
release-bundle-authority.json. The descriptor validates only G01; it still
needs the full bundle assembler and the remaining release lanes before a release
can be approved.
Operator-collected G03 scoped ACK evidence can be packaged with:
scripts/python_bin.sh scripts/run_scanner_heal_scoped_ack_evidence.py \
--proof-json /path/to/scoped-ack-proof.json \
--out-dir /path/to/scoped-ack-descriptor
The producer requires durable root publication, exact request identity,
participating peer capability, mixed-peer fallback, and mixed-version
provenance before writing release-bundle-scoped-ack.json.
The W16 recovery-intent and quota-authority lanes can emit raw G04/G12 JSON artifacts with:
scripts/run_scanner_heal_w16_recovery_evidence.sh
The runner builds the current checkout, runs the scanner recovery-intent and
disabled-startup crash-boundary tests, runs the scanner quota reset-preservation
tests, and runs the distributed hard-quota admission E2E. A full run writes
release-bundle-w16.json and validates the G04 and G12 gates with
--check-scanner-heal-release-bundle-gate. Use --test g04|g12 while narrowing
a failure; a single gate descriptor still does not approve the complete release
bundle.
The W13 durable MRF replay lanes can emit raw G07/G08/P4 JSON artifacts with:
scripts/run_scanner_heal_w13_mrf_evidence.sh
The runner builds the current checkout, runs the ignored MRF evidence test, and
writes release-bundle-w13.json for --check-scanner-heal-release-bundle-gate.
Use --test g07|g08|p4 while narrowing a failure. G08 disk-full evidence must
run against a real fillable filesystem: on Linux as root the runner mounts a
small tmpfs automatically, otherwise pass --enospc-root pointing at a
pre-mounted small filesystem. P4 is release evidence only when it completes the
default two-hour soak; --allow-short-soak is diagnostic and skips P4 bundle
gate validation.
Already collected W13 raw MRF artifacts can be re-packaged and checked without rerunning the Rust test with:
scripts/python_bin.sh scripts/run_scanner_heal_mrf_evidence.py \
--run-dir /path/to/scanner-heal-w13-evidence-run
The producer rejects fixture, dry-run, synthetic, stale-revision, incomplete
MRF responsibility, missing disk-full ENOSPC observations, and short P4 cleanup
soaks before writing release-bundle-w13.json.
Legacy rollback evidence for R-L is assembled from a measured proof JSON:
scripts/python_bin.sh scripts/run_scanner_heal_legacy_rollback_evidence.py \
--proof-json /path/to/legacy-rollback-proof.json \
--out-dir /path/to/legacy-rollback-descriptor
When the real release lanes have produced their dedicated artifacts, validate the complete hard-gate bundle with:
scripts/python_bin.sh scripts/check_test_wiring.py \
--check-scanner-heal-release-bundle /path/to/release-evidence.json
For a single Linux handoff checklist that keeps the measured runners in a stable order, generate the Scanner/Heal Linux evidence plan:
scripts/python_bin.sh scripts/run_scanner_heal_linux_evidence_plan.py \
--write-plan --out-dir /path/to/plan-dir
The plan is only an execution manifest. Its evidence_type is plan_only, and
it cannot satisfy any Gxx/Wxx/Rxx gate. Use --run-preflight only for the
lightweight registry and runner self-tests before starting a long Linux run.
After or during a Linux run, check which planned artifacts are still missing
without approving the release bundle:
scripts/python_bin.sh scripts/run_scanner_heal_linux_evidence_plan.py \
--status-root /path/to/run-root --format json
The status command exits nonzero while evidence is missing or malformed and
keeps release_approved: false; use its pending_gates and next_step fields
for issue writeback and failure triage.
Lane descriptors can be assembled into that bundle with:
scripts/python_bin.sh scripts/check_test_wiring.py \
--assemble-scanner-heal-release-bundle \
/path/to/release-bundle-g14.json \
/path/to/release-bundle-w16.json \
/path/to/assembled-release-bundle
The final argument is the new output directory. Every preceding argument is a measured descriptor, and each descriptor gate must keep its own passing status, lane identity, measured evidence type, complete required fields, relative artifact paths, and matching SHA256 hashes. The assembler may verify only the lanes already present; a partial assembled bundle remains blocked until all release gates are supplied.
The bundle checker is intentionally stricter than the case checker. It requires
schema 2 registry metadata, evidence: measured, the current checkout revision,
all G01-G14/P1-P4/R-E/R-D/R-L gates, per-gate status: pass, lane identity,
relative artifact paths, matching SHA256 hashes, and non-empty summaries. It
also binds each evidence field to its own run provenance: source_revision,
run_id, measurement_window_id, timezone-qualified started_at and
finished_at, command arguments, and artifact format. The field
source_revision must match the bundle revision, and measured performance
duration cannot exceed the recorded run window.
When an evidence or profile artifact declares a JSON format, the checker also
opens that artifact and requires its payload to repeat the same measured
source_revision, run_id, measurement_window_id, gate and field identity;
profile sub-artifacts must additionally name their artifact kind. Updating only
the outer bundle hash cannot turn a stale JSON summary into current release
evidence.
The hard evidence shape remains claim-specific: mixed-version gates must name at
least two participating versions, crash/durable replay gates must include
crash-boundary evidence, G14 must record EC8+4 with at least three nodes and four
drives per node plus multi-set and multi-pool evidence, performance gates need
measured durations, P3's pressure run needs at least two hours, and P1 needs a
symbolized profile summary with resolved samples. Every G14 field and every
performance gate's fields must also share one measurement_window_id, so EC8+4,
multi-set/multi-pool, ABBA, throughput, and profiling artifacts cannot be
stitched together from unrelated runs. P1 profile_evidence must bind every
required profile artifact kind (allocation-profile, flamegraph,
rss-samples, and save-frequency) with a relative path, artifact format,
non-empty file, matching SHA256, and descriptor-level source_revision,
run_id, and measurement_window_id values that match the parent profile
evidence. JSON profile artifact wrappers must also carry a bundled raw profile
path with matching raw SHA256 and byte count plus the relevant allocation, RSS,
sample, or save-frequency cost counters. Missing, synthetic, stale, tampered,
undersized, cross-run, or topology-mismatched evidence returns a compact blocked
or invalid JSON result and a nonzero exit.
The status-and-outcome raw collector normalizes live observations into the three
measured raw JSON artifacts for G05, G06, and R-D. The descriptor producer then
consumes those artifacts. The inputs must all carry schema 1, measured evidence,
matching source_revision, a shared run_id, a shared
measurement_window_id, matching started_at/finished_at timestamps, and
non-empty command provenance. The collector and producer reject synthetic input,
missing required cases, and command-line run/window/time overrides that would
relabel raw artifacts from another status-and-outcome run.
The scheduler-pressure lane must also carry the numbers needed to close W09, W10, and W11: bounded deferred item/byte/age limits, zero duplicate tasks, pressure pacing engagement, recovery and lock-hold timings, fixed offered load, foreground p95/p99 latency, throughput, error count, attempt-cost samples, and completed heal object counts.
This command validates the evidence package; it does not create evidence. A handwritten JSON file, a synthetic harness pass, a single focused case, or a local unit fixture still cannot satisfy the distributed, mixed-version, crash-restart, durable MRF replay, EC8+4, ABBA, or profiling gates.
Run parser/receipt regressions with
scripts/python_bin.sh scripts/check_test_wiring.py --self-test. Those fixtures
validate the checker only and produce no runtime or performance evidence.
For local bundle-shape dry runs, generate a task-owned fixture directory with:
scripts/python_bin.sh scripts/check_test_wiring.py \
--write-scanner-heal-release-bundle-fixture /path/to/fixture-dir
The generated file is marked fixture_only and is intentionally rejected by the
release bundle checker. Use it to rehearse field names, artifact paths, hashes,
profile artifact membership, mixed-version roles, and same-window provenance
before copying the shape into a real measured bundle. It is not ABBA, profile,
mixed-version, crash-restart, or release approval evidence.