mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-06 21:23:11 +00:00
2bdec2fdcd
* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593) feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592) Replace the assets-branch scheme with a proper GitHub Release on rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets are replaced in place on every build. This is the standard channel — visible on the repo's Releases page, stable download URLs, no git history growth (release assets live outside the repository). - New scripts/release/publish_nightly_assets.sh: resolves-or-creates the 'nightly' release via the REST API, deletes same-name assets, uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release body with the build provenance (ref@sha, run link, sizes, SHA256). Plain curl + python3, no gh CLI (the build fleet has none — #7586). - The workflow step shrinks to invoking the script; full flow exercised end-to-end against the real release with probe files (create / upload / overwrite / download round-trip / body update). * fix(ecstore): stop pruning at nonempty directories (#7616) * fix(ecstore): stop pruning at nonempty directories * test(ecstore): release pruning fixtures before temp cleanup * fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626) * fix(replication): total-order rule sort and honor V1 top-level Prefix Rule matching had two defects from the pre-GA replication audit (rustfs/backlog#2367 C-1 and C-2): - The actionable-rule sort compared same-destination rules by priority but answered Equal for any other pair, which is not a total order; the standard library sort panics on such comparators once a slice exceeds the insertion-sort threshold, so an object matching more than 20 enabled rules across two or more targets could panic the PUT or DELETE task. Rules now sort by priority descending with destination and id as tie-breakers, and filter_target_arns preserves that order instead of draining a HashSet. - A V1 rule written without a <Filter> carries its prefix at the top level; that field was never read, so <Prefix>logs/</Prefix> matched every object. ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping precedence. The existing prefix fixtures were built this way and had been asserting nothing. * fix(admin): advertise data-usage and listen capabilities to rc The rc client gated `rc du` and `rc watch` on a pinned contract that matched server versions by the string prefix `1.0.0-rc.`; a server that reports `1.0.0` no longer matches, and the dynamic `advertised` list did not carry either name, so `rc du` against a GA server fails with an unsupported-capability error (rustfs/backlog#2367 E-2). Advertise `admin.data-usage` from the admin route inventory like the IAM entries, and `listen_notification` for the bucket `?events=` extension route the admin router dispatches. The client merges advertised entries ahead of its pinned contract, so no version sniffing is needed. * fix(site-replication): stop notifying the local site on remove and rotate The pending-remove and pending-rotation notification loops skipped the local site by endpoint only, while finalization identifies it by deployment id or endpoint. The reconcile tick resolves the local peer from the node's own listen address (and a handler from the request Host), so `remove --all` dialed the site's registered endpoint, waited out the request timeout against the lifecycle lock it was holding, and answered `Partial: failed to notify 1 peer(s)` for a removal that had succeeded (rustfs/backlog#2367 A-4, backlog#2195 item 3). Both loops now iterate the peers still awaiting notification through one helper that applies the finalization identity. * fix(site-replication): promote and settle IAM retries without a tick of slack Two retry-queue behaviours kept an IAM change from converging for ten to twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3, backlog#2305): - The lightweight 30-second pass filtered its reachability probe to bucket ops, so a backed-off IAM or bucket-metadata snapshot waited for the 600-second tick to notice the peer. It now probes every backed-off class and still replays only bounded bucket ops; promotion is a state flip the heavyweight tick acts on. - Backoffs are multiples of the tick interval, so a failure stamped δ seconds after a tick was 600 − δ old at the next tick and slipped a whole extra interval. The heavyweight drain now evaluates backoff halfway to its next tick. - An IAM entry first created by a non-deletion failure (the add bootstrap's snapshot send, the drain's own replay, an import-iam schedule) was never stamped `deletions_recorded`, so a later recorded deletion could not settle it and it escalated to the marker only `replicate repair` clears. Entries created by this binary now start recorded; a row persisted by an older binary keeps the escalation semantics. * fix(site-replication): reload peer node caches after bucket wiring writes Every S3 bucket-config write ends by asking the other nodes of the cluster to reload the bucket's metadata; the site-replication writers never did. On a multi-node site the node that ran the pairing (or applied a peer's bucket-meta item) rewrote the bucket targets and the derived replication rules on disk, while every other node kept serving its cached copy for up to the 15-minute refresh. A `resync start` routed to such a node reported every freshly wired bucket as `Config not found` and a bucket whose operator target the pairing had replaced as `recorded remote target no longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional SITE-105). Add one best-effort reload helper in the site-replication hooks and call it after the bucket setup, versioning, peer bucket-meta apply, removed-peer cleanup, make-with-versioning, and endpoint-refresh writes; the ensure helpers now report whether they wrote so unchanged passes stay silent. The resync manifest and start now read the persisted wiring instead of the node-local cache, matching the target read the start path already did. The new four-node e2e pairs two clusters and starts a resync through a non-coordinator node right after pairing; it also covers an IAM user created on a non-coordinator node converging to the peer site. * test(e2e): cover delete-marker replication from a multi-node source The functional suite reported delete markers created on a 3-node source never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report was a probe defect, but the shape had no coverage: the existing delete-marker e2e runs a single-node source. Pin it against a four-node source replicating to a four-node peer and to a single-node target, with the write and the delete issued through different nodes. * ci(e2e): refresh the distributed selection for the new replication cases Four distributed cases were added (two site-replication, two delete-marker replication). The linux digest is derived from the last CI listing of the lane (34 cases, matching the previous pin) plus the four new names; the darwin digest is the local listing, which selects the same 38 cases. * feat(console): support a configurable console URL prefix (#7634) Co-authored-by: houseme <housemecn@gmail.com> * chore(release): merge release into main for rc.6 (#7638) * test(heal): cover MRF manifest CAS legacy transition (#7443) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ci): enable release branch checks and repair test imports (#7441) * test(scanner): structure heal release evidence lanes (#7442) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): require segment producer identities (#7444) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): expose recovery intent identity (#7445) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): summarize failed heal perf reports (#7446) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(common): retain unmatched MRF repair proofs (#7447) Add a recorded verified-repair consumer that discharges only exact retained anchors for the requested bucket while leaving unmatched proofs in the event ring. This gives the future durable successor writer a fail-closed primitive before any tombstone or GC path is enabled. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ci): route replication read plan through boundary (#7448) * test(scanner): cover flat-bucket quantum fairness (#7449) Add a production-entry scanner cohort regression that combines a wide flat bucket with a small bucket under a fixed object budget. The test keeps partial budgeted rounds unpublished, verifies small buckets are only marked after real execution, and confirms a later unbudgeted round can publish the complete aggregate. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): reuse same admission request id (#7451) Keep a retried heal start with the same request id from bypassing admission deduplication when the transport replay cache is unavailable but the manager still owns the task. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require perf summary evidence fields (#7453) Fail closed when measured passing Scanner/Heal ABBA summaries omit W10/W11 foreground pressure, lock wait, or attempt-cost evidence. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind release evidence fields (#7452) Require explicit scanner/heal release evidence field contracts for the scoped ACK and mixed-version rollback gates. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): tag dirty usage producer identities (#7454) Record production-facing segment invalidation producer identities when existing object-level dirty usage hooks observe PUT, CopyObject, DeleteObject/DeleteMarker, and CompleteMultipartUpload mutations. Keep the data non-authoritative and process-local so segment reuse activation still requires durable generation-window proof. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): gate release evidence bundles (#7457) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require hard evidence ABBA manifest (#7459) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(heal): add MRF committed snapshot writer (#7458) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover EC8+4 restart shard rebuild (#7461) Retry heal-control RPCs once after transport auth rejects a stale replay-scope epoch, and add a distributed EC8+4 restart heal evidence case that rebuilds a replaced drive with exact shard/body assertions. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): echo measured release evidence in ABBA fake adapter (#7460) Keep the synthetic ABBA test adapter aligned with the measured release evidence contract so result validation covers release_evidence drift. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require complete ABBA summary matrix (#7462) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * heal: retain MRF replay journal after accepted/merged replays Keep startup replay journal until a durable successor snapshot exists after Accepted/Merged admission. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: reclaim superseded MRF snapshots after readback Add a committed snapshot predecessor cleanup primitive that deletes only an older slot after the successor is read back as the current committed snapshot. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: require exact owner for object receipts Require object heal receipts to match the expected bucket incarnation before they can be recorded as positive repair evidence. This prevents stale or cross-incarnation receipts from clearing the wrong heal responsibility. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: latch object receipt owner before repair Capture the expected bucket incarnation before invoking object repair so a post-repair owner change cannot rewrite the responsibility that a storage receipt is allowed to prove. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: replay committed MRF checkpoints durably (#7465) Prefer committed MRF checkpoints during startup replay, retain accepted replay responsibilities until exact verified repair proofs arrive, and reclaim committed manifests only after discharge. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind hard evidence bundle provenance (#7467) Require Scanner/Heal release bundle fields to carry source, run, window, timestamp, command, and artifact format provenance before a measured gate can pass. Keep EC8+4 and performance gate fields tied to a single measurement window so unrelated artifacts cannot be stitched into a release approval. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * heal: reject dry-run object repair receipts Do not record positive storage repair receipts for dry-run object heal tasks, even if a producer accidentally returns a matching receipt. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: reject cancelled object repair receipts Do not record positive storage repair receipts once an object heal task has been cancelled, even if the receipt still matches the requested owner and object identity. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): reject failed object repair receipts Cover the receipt consumer path where a storage repair result carries both an error and a matching positive receipt. The failure may be recorded, but the receipt must not create repaired, healthy, or absent proof. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): cover bucket object repair receipts (#7468) Cover bucket and root heal sweeps recording authoritative object outcomes only when storage receipts match the latched bucket incarnation. Verify unavailable or stale receipt ownership keeps object repair execution intact while leaving canonical outcome proof as Unknown. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add EC8+4 heal restart evidence (#7469) Register and wire Scanner/Heal background target restart and crash evidence cases for the 3x4 EC8+4 topology. Validate the observed data/parity geometry in scanner-heal evidence receipts so multi-drive runs cannot satisfy the gate without proving EC8+4 metadata. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * heal: verify replacement pool metadata repair (#7471) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover quorum and mixed repair receipts Add focused oracles for transient quorum results that carry a matching receipt and for mixed grace plus repaired receipt batches. Only the repaired object may produce positive proof. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * heal: verify admin recreate pool metadata (#7474) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * heal: preserve merged result for duplicate submits Keep same-request-id replay receipts accepted for the receipt API, but preserve the legacy submit_heal_request duplicate admission result as Merged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): preserve EC84 restart semantics (#7478) Keep the EC8+4 background restart lane on the graceful-stop path and assert clean-restart marker absence only for restart scenarios. This prevents the hard evidence gate from silently exercising the crash path when it claims restart coverage. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF snapshot torn successor recovery Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): write EC84 distributed restart oracle (#7481) Bind the distributed EC8+4 restart evidence lane to a scanner/heal oracle artifact so release validation can consume the real nextest run instead of accepting only a passing test. Require the registry to assert 8+4 erasure geometry for the three-node, four-drive case. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): harden MRF replay boundaries (#7483) Reject journal records with unknown version-presence flags even when their CRC is valid, so rollback/future payloads cannot be accepted as known records. Gate committed checkpoint cleanup by the writer owner captured from the replay source, preserving retained manifests from other owners inside the same sequence window. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(ecstore): bind MRF manifest CAS dirsync recovery (#7482) Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): derive evidence runner profile from registry (#7484) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require mixed-version evidence roles (#7485) Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role. Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval. The release gate remains blocked until measured field evidence is present. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): publish committed MRF runtime checkpoints (#7490) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind profile artifacts in release evidence (#7487) Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks. Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): harden measured ABBA evidence claims (#7491) Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash. Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values. Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(e2e): group scanner heal evidence payload (#7494) Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint. The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF replay bundle fields (#7486) Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): profile EC84 evidence case runs (#7495) Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions. Expose the runtime profile in plan-only output and cover every registry case in the script self-test. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): reject empty release evidence artifacts (#7496) Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them. Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require G14 same-window field coverage (#7489) Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window. Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require two-hour measured ABBA windows (#7493) Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF idle checkpoint cleanup (#7497) Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(e2e): ignore untracked files in build identity (#7498) Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty. This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF disk-full evidence fields (#7499) Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): bind release JSON artifact provenance (#7500) Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF rollback mirror filtering (#7501) Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require segment activation evidence (#7504) * test(scanner): require segment activation evidence Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): cover activation proof inputs Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF crash matrix cases (#7505) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require release bundle domain evidence (#7502) Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF retention GC evidence (#7506) Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF cleanup GC soak evidence (#7507) Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF disk-full commit anchors (#7509) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): report port allocator bind errors (#7510) Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511) * fix(sse): resolve bucket default encryption per request PUT and the POST-object/extract path resolved a bucket's default encryption with a hard-coded "no explicit SSE-C" flag, so the default was layered onto a request that already carried an SSE-C header triple and then tripped that request's own mutual-exclusion check. Every bucket with default encryption refused SSE-C single PUTs with 400 InvalidArgument, while CreateMultipartUpload on the same bucket succeeded because it resolves SSE elsewhere. Both call sites now derive the flag from the request headers, as COPY already did. The bucket default's KMS key id was also inherited independently of the effective algorithm, so an explicit AES256 request against an aws:kms default bucket produced a self-contradictory algorithm/key-id pair and was rejected. The key id is now inherited only when the effective algorithm is aws:kms, matching the storage-layer resolver. Refs backlog#2368 B1, B2. * fix(sse): refuse SSE-KMS without a running KMS service A write requesting aws:kms on a node with no KMS service fell back to the node-local SSE-S3 provider: the data key was wrapped with RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded aws:kms and the requested KMS key id. The stored object claimed a KMS protection it never had, under a key that was never consulted, and no signal distinguished it from a genuine SSE-KMS object. The managed-encryption path now asks the resolved DEK provider whether it wraps with a node-local master key and refuses SSE-KMS in that case: InvalidRequest when KMS was never configured, ServiceUnavailable when a configured service is not running. The check sits after the per-key authorization gate so an unauthorized caller still receives AccessDenied whatever the KMS runtime state is, and asks the provider rather than a parallel availability signal because the provider is what actually wraps the key. A missing master key no longer answers an SSE-KMS request with an SSE-S3-worded configuration error. The SSE-S3 local fallback is unchanged. Refs backlog#2368 B4. * fix(ecstore): restore and archive tiers in stored coordinates Multipart restore addressed the remote tier in plaintext coordinates while the copy-back reads the stored representation. Each part received a misaligned slice of the remote object whose length still satisfied the range, the hash reader and the completion size check, so the restore reported success and silently replaced the object's bytes. Encrypted and compressed multipart objects were both affected. Restore now accumulates stored part sizes, passes the stored length to the hash reader alongside the plaintext length, and validates against the stored size. The copy-back digests stored bytes, so its computed MD5 is not the object's public ETag. Restore now preserves the object ETag on both the single-part and multipart paths, and gives each restored part its own recorded part ETag rather than the object-level value. Transition also handed the tier the object's SSE headers and its RustFS-wrapped data key as request headers. Any S3 target rejected an SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a second time under a key id it does not own, and the wrapped DEK left the cluster. The archive request now strips every SSE header and encryption marker with the predicate the replication path already uses; the local xl.meta keeps all of it, so read-through and restore are unaffected. Objects restored by an affected release are not detected or repaired retroactively and must be re-restored from the tier. Refs backlog#2368 B3, B5; backlog#2369 P7.1. * fix(rio): lock the v1 nonce layout within a segment Decrypting a v1 segment tried three historical nonce layouts per frame, independently for every frame. The last of them exists for streams written before 1.0.0-alpha.91, which reused a segment's part nonce for every block in it; because block zero's derived nonce equals that base nonce, a frame encrypted at index zero authenticated at any position. An attacker able to rewrite the underlying shards could replay it and have the forged plaintext returned with 200 and an unchanged length. Shard integrity uses a keyed-hash-free checksum, which such an attacker can recompute, so it is not a barrier. A segment now locks onto whichever layout decoded its first non-zero-index frame and rejects any later frame needing a different one. That leaves one residual shape: a stream built purely from repeats of frame zero has no later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK (default true, so pre-alpha.91 objects keep decrypting) drops the third layout entirely when set to false, which closes it. Turning it off refuses pre-alpha.91 objects, so migrate them first by rewriting in place. Refs backlog#2369 P2. * fix(kms): reload a service that failed to start POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted configuration matched the in-memory one byte for byte. A node whose KMS failed to start keeps that configuration and sits in Error, so the documented recovery call returned "reloaded successfully" while leaving the node down. Peers reached the same path through the reload broadcast, so a cluster that lost Vault during a rolling restart had no working recovery route other than the node-local start endpoint. Reload now short-circuits only for a service that is actually running, and otherwise reconfigures, which starts a service that is not running. The AWS backend also advertised key-version enumeration through kms/status, which its own documentation says it cannot do; the capability and its golden snapshot now say false. Refs backlog#2369 P1, P7.3. * docs: record the SSE and KMS changes for 1.0.0 The Unreleased changelog section carried no entry for any encryption work merged since 1.0.0-rc.5, including three items with operational impact: the config-secret variable whose absence persists secrets in cleartext with only a warning, the v2 frame write switch and its rolling-upgrade constraint, and per-key authorization making a public bucket incompatible with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS refusal as a breaking change with both routes out. Also corrects four places where documentation contradicted the code: the cleanup register still called encrypted range seek opt-in after its default flipped, the Helm README claimed vault_mount_path only applies to Transit while the template also feeds the KV2 mount, the disaster-recovery drill listed bundle contents for backends whose export is refused with 501, and the Chinese README capability table predated most of the feature set. Documents the SSE-S3 local master key as a first-class operational mode with its rotation dead end, and what the v1 frame layout does and does not authenticate. Refs backlog#2369 P5. * fix(kms): classify data-path KMS failures by what the caller can do Only "key not found" and a backend outage were classified; every other KMS failure that reached the S3 data path fell through to 500 InternalError with a generic message. A disabled or pending-deletion key, a denied KMS grant, an encryption-context mismatch, an unsupported algorithm, a credential or timeout failure, and a capability the configured backend does not have all looked identical to a server fault. SDKs therefore applied exponential backoff to configuration errors no retry can fix, and monitoring counted every one of them against the server's own error rate. Unusable-key and request-side failures now answer 400, a denied grant 403, transient backend failures 503, and a missing backend capability 501. Damaged, unreadable, or unknown-format key material keeps its 500: it is a server-side integrity fault, and existing tests pin it. The classifier is deliberately separate from the admin lifecycle mapping, which answers 404 for a missing key because there a key id is the resource being addressed; on the data path it arrives inside a request header or a bucket default. Messages either name what the caller asked for or stay generic, with deployment-side detail left on the error source the way the storage-IO mapping already does. Refs backlog#2368 B6. * fix(kms): track and renew static Vault tokens Token authentication hard-coded "this token carries no lease", so the renewal task never started, the remaining-TTL gauge was never published, and nothing looked wrong. `vault token create` grants a 768-hour TTL by default, so a cluster that had been healthy for a month turned every KMS call into a 403 and could not recover without a restart or a reconfigure. Production configuration validation only rejects the literal dev-token, so an ordinary expiring token reaches a whole cluster. The source now reads `auth/token/lookup-self` at login and adopts what Vault reports. A token with no expiry behaves exactly as before. An expiring renewable one is picked up by the existing renewal loop and renewed at half TTL like every other auth method. An expiring non-renewable one warns with its remaining lifetime and publishes the gauge, so the fail-closed window is visible before it arrives. The probe never fails the login: a policy that omits lookup-self, or a Vault that is briefly unreachable, warns and falls back to exactly the previous behaviour rather than taking down a deployment that works today. The scripted Vault test double answers the lookup out of band so existing scripts keep describing only the protocol under test. Refs backlog#2369 P3. * feat(sse): report SSE-C requests that arrive without TLS An SSE-C request carries the customer's AES key in a request header, so AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS accepted them on any transport: a plaintext hop hands the key to anyone on the path, and since the object cannot be read without that same key, the exposure lasts as long as the object does. Refusing outright is the correct end state but not a safe default to adopt inside a release window, because the project's own s3-tests and e2e lanes and most staging deployments speak plain HTTP. This release reports instead: each such request increments rustfs_ssec_plaintext_requests_total and logs one warning per process, so an operator can confirm nothing would break before the default flips. RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now. The verdict is per connection rather than per deployment: the layer is built with whether this listener terminated TLS, and additionally accepts an https protocol forwarded by a proxy the trusted-proxy configuration already vetted. It sits beside the rate limiter, after the layer that makes a forwarded protocol trustworthy and after the request context, so a rejection can echo the request id. Refs backlog#2369 P7.2. * fix(kms): say what a node-local backend means for a cluster The Local backend keeps key material on each node's own disk and generates its Argon2id salt per node, so two nodes derive different keys from the same master_key and an object encrypted on one node cannot be decrypted on another. Behind a load balancer that surfaces as intermittent 500s on reads that succeeded moments earlier, with nothing tying the symptom to the cause: the only signal was a generic "development, testing and demos only" positioning warning that says nothing about what actually breaks. Configuring or reconfiguring Local while the deployment is distributed now logs a dedicated event and appends the consequence to the configure response, so the operator who made the change sees it. The product decision to warn rather than refuse is unchanged. Refs backlog#2369 P7.4. * docs: record the remaining SSE and KMS changes for 1.0.0 Adds changelog entries for the KMS data-path status classification, the Vault static-token lease probe, the SSE-C plaintext-transport report and its switch, and the node-local backend warning. Documents two things the backend security guide never stated: that SSE-C belongs on a secure transport, with the counter and switch to plan the change around, and that the Local backend cannot be shared by a multi-node deployment because each node derives different keys from the same master key. Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4. * fix(kms): report an unreadable key store as an outage on the S3 path A backend now distinguishes a key store it could not read from a key that is genuinely absent, but the S3 boundary collapsed the first one back onto 500 InternalError through the fallthrough for integrity faults. The distinction was therefore invisible to the client: a temporary key-directory outage looked exactly like a permanently damaged key record, and neither the status nor the metric said the request was worth retrying. An unreadable key store joins the retryable class and answers 503, next to a backend error and a credential failure. Damaged, unreadable or unknown-format key material keeps its 500. Refs backlog#2368 B6; builds on rustfs/rustfs#7470. * test(storage): cover MRF cleanup delete ENOSPC anchor (#7513) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require heal retry stats evidence (#7514) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add release bundle dry-run fixture (#7515) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): confirm recovery intent accept readback (#7516) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require MRF cleanup delete ENOSPC evidence (#7518) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): require bounded retry window evidence (#7517) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): emit scanner heal G09 upgrade evidence (#7519) Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): persist MRF rollback mirror boundary (#7520) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add G09 upgrade evidence runner (#7522) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * test(scanner): add G09 upgrade evidence runner Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes. The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(scanner): implement V2 evidence preflights (#7523) * feat(scanner): wire dirty usage producer identities Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): add segment activation preflight proof Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(scanner): expose distributed invalidation evidence Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds. Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): assemble release evidence bundles (#7526) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add G09 upgrade evidence runner (#7524) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * test(scanner): add G09 upgrade evidence runner Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): replay recovery intents while disabled (#7521) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): preflight G09 evidence disk space (#7527) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * fix(targets): reject trailing batch items (#7508) * test(scanner): preflight G09 evidence disk space Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: cui fliter <imcusg@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): respect cargo target dir in G09 runner (#7528) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the… * feat: configure the console base path at build time (#7636) Embed RUSTFS_CONSOLE_BASE_PATH into console routes, browser redirects, and startup URLs so OEM builds can use their own namespace. Preserve the default for an unset or empty build variable and keep route tests prefix-aware. * chore(release): prepare 1.0.0-rc.6 (#7641) * chore(release): prepare 1.0.0-rc.6 * ci: run offline enrollment boundary checks independently * fix(heal): preserve retryable batch failures during recovery (#7642) * fix(heal): preserve retryable batch failures during recovery * test(heal): pin prebuilt hooks binaries in ci * fix(s3): reject oversize single PUT early and map body errors to 4xx (#7635) * fix(s3): reject oversize single PUT early and map body errors to 4xx A single PutObject above the 5 GiB single-request ceiling was only rejected after the client had streamed 5 GiB into s3s's read-time body budget, and the resulting BodySizeLimitExceeded surfaced from the erasure writer as 500 InternalError. A body whose connection hit EOF before Content-Length bytes arrived (hyper's IncompleteBody) was also a 500. SDKs retry 500s, so one oversize upload was resent from offset 0 five times. - PutObject and UploadPart reject a declared length above MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the body; the constant moves to rustfs_config so the s3s limit and the admission check share one value. - ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body EOF to IncompleteBody across both io::Error conversions. Fixes #7596. * test(s3): cover UploadPart admission, aws-chunked length, real s3s limit - Poll-counting test body proves PutObject and UploadPart reject a declared size above the ceiling with zero body polls; exact-cap and zero-length parts pass admission. - A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the cap is admitted when the decoded length is within it and rejected when the decoded length is over it. - The display-based BodySizeLimitExceeded matcher is checked against the real error produced by the pinned s3s Body budget. * fix(ecstore): make directory mtime fixture portable (#7623) * fix(ecstore): make directory mtime fixture portable * style(ecstore): format mtime fixture assertion --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> * fix(storage): prevent readiness after native migration failures (#7652) * fix(storage): prevent readiness after native migration failures * fix(storage): skip unsupported IAM records before reading * fix(storage): use stable typed migration metadata errors * fix(storage): include migration record in startup errors * test(storage): cover native migration startup failures * test(storage): use array chunks in migration fixture --------- Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com> Co-authored-by: cxymds <cxymds@gmail.com> * fix(admin): expose OIDC account display fields (#7654) Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers. * fix(replication): correct peer joins and remote-state reporting (#7650) * fix(replication): propagate verified peer deployment identities * fix(replication): report actual remote peer state * fix(replication): defer initial sync until all peers join * test(replication): shut down TLS fixtures cleanly --------- Co-authored-by: houseme <housemecn@gmail.com> * feat(ci): add fault-tolerance degradation suite to the functional chain (#7663) Adds a fault-tolerance suite that verifies read/write behavior under drive and node loss against the erasure-coding contract and snapshots health-endpoint responses at every degradation tier. Scenarios derived from product source (default_parity_count, erasure set sizing): - A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives - B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes - C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down lands exactly on the read-quorum boundary; 2 nodes down breaks it - C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads inside the reported divergence window (read quorum met while the lock majority is broken) By default a reads-refused-despite-met-read-quorum observation is reported as known-divergence without failing the suite; the strict input escalates it. Chain order becomes: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security -> replication -> fault-tolerance -> performance. * fix(ci): repair functional defaults and chain regression checks (#7664) * fix(ci): default functional suites to nightly packages * test(ci): follow the fault-tolerance chain handoff * "feat(ci): add fault-tolerance degradation suite to the functional chain" (#7667) Revert "feat(ci): add fault-tolerance degradation suite to the functional chain (#7663)" This reverts commit7c47c48e85. * fix(ci): align security workflow tests with chain (#7679) * test(kms): cover non-default Vault Transit paths (#7657) * test(kms): cover custom Vault Transit metadata locations * test(kms): include a non-default Transit mount in live coverage * test(ecstore): tolerate cleanup quorum transition * test(ci): restore ten-suite chain expectations * test(ecstore): tolerate cleanup quorum transition in both causal polls The unversioned duplicate-delete poll in batch_transitioned_delete_uses_free_version_per_item reads exact metadata while the same asynchronous free-version cleanup removes per-disk copies, so it can hit the same transient InsufficientReadQuorum as the versioned poll. Share one helper between both polls; every other error still fails immediately. --------- Co-authored-by: overtrue <anzhengchao@gmail.com> * fix(s3): bound stalled UploadPart request bodies (#7659) * fix(s3): bound stalled UploadPart request bodies * fix(ci): preserve the S3S footprint ratchet * fix: enforce S3 permissions for recursive force deletion (#7661) * fix: enforce S3 authorization for recursive deletion * fix: satisfy the s3s footprint guard * fix: restore list versions policy compatibility (#7686) * fix(tables): reject reserved warehouse locations (#7671) Co-authored-by: cxymds <cxymds@gmail.com> * fix(e2e): require a verified server binary for every e2e run (#7687) * fix(ci): share quick checks and lint workflows * fix(ci): install actionlint from its verified release * fix(ci): reject dependencies on required quick checks * feat(test): verify the E2E server build and source identity * test(e2e): register verified Darwin test membership * test(e2e): record verified Linux receipt test membership * test(e2e): record compiled Darwin receipt test membership * test(e2e): record compiled Linux receipt test membership * test(e2e): record Darwin e2e-full membership after merging main * fix(test): route scanner/heal evidence E2E runs through the verified server binary The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed. * docs(e2e): run server-backed e2e commands through the verified binary wrapper * test(e2e): record Linux e2e-full membership from the branch CI listing * fix(ci): bind nightly lanes to one resolved source (#7688) * feat(ci): measure queue and execution time by run attempt (#7689) * feat(ci): measure queue and execution time by run attempt * fix(ci): bound timing samples and recognize the old workspace lane * fix(ci): sample completed runs for stable timing comparisons * fix(ci): verify complete functional chain evidence (#7690) * fix(ci): bind nightly lanes to one resolved source * fix(ci): verify complete functional chain evidence * fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697) * fix(sse): classify bare SSE-KMS writes when no KMS is available A `aws:kms` request without a key id, on a bucket without a default key, returned `500 InternalError` whenever no KMS service was running: the "no KMS key available" branch exited with an untyped storage error before the availability classification that the keyed form already received. Route that branch through the same split: `503 ServiceUnavailable` while a configured KMS is stopped, `400 InvalidRequest` when KMS was never configured, and `400 InvalidRequest` naming the missing key id when a running KMS has no default key. `CreateMultipartUpload` shares the path. Adds a unit test for the bare form and an e2e module that stops KMS through the admin API, runs a master-key-only node, and runs a Local KMS without a default key; refreshes the e2e-full selection digests. (cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8) * fix(sse): keep KMS error classes on the encrypted read path GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key no longer exists answered `500 InternalError` ("KMS key not found") while PutObject under the same key already answered `400 KMS.NotFoundException`. The read path carries its classification through ecstore's `EncryptionResolutionErrorKind`, which had no kind for a missing key, a denied KMS grant or a missing backend capability, so all three folded onto `DecryptionFailed` and the S3 layer reported an internal fault. Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on both sides of the boundary, and give an envelope the configured backend cannot unwrap a diagnosable message while keeping its `500`. Unit tests cover the kind round trip and the reader wrapping; a new e2e test deletes a key immediately and checks GET/Copy return 400 with `KMS.NotFoundException` while HEAD stays 200. The e2e-full selection digests are refreshed from the current listing (the previous digests predated the delete-authorization tests) and the e2e `create_default_key` helper is updated to the accepted `EncryptDecrypt` spelling. (cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2) * fix(kms): classify key-management errors on the admin routes `POST /kms/keys`, the legacy `create-key` alias and `generate-data-key` reported every backend refusal as `500`: a blank key name (which each backend failed on differently, the Local backend by writing a key file with an empty stem), a name already taken, an unknown key, a disabled key and a capability the backend lacks. `delete` and the lifecycle routes already classified the same errors. Refuse a blank or whitespace name in `KmsManager::create_key` before any backend sees it, and share one `KmsError` to status mapping across create, delete and generate-data-key (400 for validation and key state, 404 for an unknown key, 409 for a taken name, 501 for a missing capability, 500 only for damaged material). The XML-error routes carry the same status explicitly since s3s derives none for a custom code. The read-only Static backend now reports create, delete and cancel-deletion as `UnsupportedCapability`, matching its rotate and enable/disable answers, so the admin API returns 501 for all of them. (cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b) * fix(sse): stop SSE-S3 responses from naming the wrapping KMS key `x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms` objects only, but PutObject, CopyObject, CreateMultipartUpload and GetObject returned it for `AES256` objects too, carrying the KMS key that wraps the SSE-S3 data key (the service default, or the literal `default` on a node without KMS). The write paths copied `kms_key_id` from the encryption material unconditionally, and the single-decrypt GET classification did the same after resolving the key for authorization. Add `EncryptionMaterial::response_kms_key_id`, which yields the id only for SSE-KMS, use it at the four write-response sites, and gate the GET classification the same way. CompleteMultipartUpload and HeadObject already omitted the header. Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy and CreateMultipartUpload for AES256 with an aws:kms control. The e2e-full selection digests are refreshed from the current listing. (cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb) * fix(s3): validate PutBucketEncryption rules before storing them A default-encryption rule naming an unknown `SSEAlgorithm` (for example `AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as written: the only algorithm check on the route decided whether to fill in the default KMS key. `GetBucketEncryption` then advertised that configuration while the write path encrypted header-less writes under its `AES256` fallback, so the bucket's declared and actual schemes disagreed. Two comments claimed the route already refused unknown algorithms. Validate the configuration before any of it is applied: `MalformedXML` for a malformed rule set or unknown algorithm, `InvalidArgument` for a key id on a non-KMS rule, and nothing stored on refusal. Correct the two comments to describe when the AES256 fallback is still reachable. Unit tests cover every refusal and the accepted shapes; an e2e test checks the refusals leave the previous configuration in place. The e2e-full selection digests are refreshed from the current listing. (cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb) * test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes * test: align two unit tests with the new KMS and bucket-encryption contracts `scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still expects the state error (`InvalidOperation`) for cancelling a key that is not pending deletion; only the Static backend's mutations moved to `UnsupportedCapability`. The uninitialized-store PutBucketEncryption test now sends a well-formed AES256 rule so it reaches the store lookup instead of the new configuration validation. * Collect bounded Connect environment inventory (#7710) feat(connect): collect environment inventory * Add Connect service license commands (#7711) feat: add Connect service license client * Add consent-bound diagnostic scheduling (#7713) feat(connect): schedule consent-bound diagnostics * feat(connect): advertise environment diagnostics (#7714) * feat(connect): add consent-bound profile exports (#7716) feat: add consent-bound profile exports * feat(connect): add bounded local log capture (#7718) * feat(connect): add bounded network performance producer (#7719) * Add bounded local telemetry export commands (#7720) feat(connect): add bounded telemetry producers * Add bounded drive performance command (#7721) feat(connect): add bounded drive performance producer * Add bounded top diagnostic captures (#7722) feat(connect): add bounded top diagnostic producers * Add client-to-deployment performance diagnostics (#7726) feat: add client performance diagnostics * Add a local environment inventory command (#7728) feat(connect): add local environment inventory command * feat(connect): add bounded object performance diagnostics (#7734) feat(connect): add object performance producer * Add authenticated inter-node network performance probes (#7739) * feat(connect): add authenticated inter-node network probes * fix(connect): respect storage facade boundary * fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over (#7709) * fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over Site replication wired each bucket by looking for an existing replication target "to the same peer" and rewriting the first match in place as its own same-name target. An operator's bucket-level target that happened to point at that site (different target bucket, operator credentials) was the first match whenever it pre-dated the join, and the reconciler repeats the pass every 600s, so the takeover also depended on target order afterwards. The operator's rule then named an ARN no target backed and their bucket replication stopped silently, while the inherited bucket-level reset id made every site resync report the bucket as owned by another resync (rustfs/backlog#2479, rustfs/backlog#2489). Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead: - Wiring updates a target in place only under the same ARN, or when it is recognisably the site's own under an older ARN shape (same peer, same-name target bucket, site replication service account). Anything else gets the site target added next to it. - The site resync manifest takes the target the derived `site-repl-<deployment id>` rule names (same-name shape as fallback), so an operator target to the peer neither aborts the bucket as "multiple remote targets matched peer" nor gets resynced into. - Peer removal prunes only targets a pruned derived rule names or the same-name target bucket; operator targets stamped with the peer's deployment id survive together with their rules. Unit tests cover the three predicates. e2e `test_site_replication_keeps_operator_bucket_target_to_peer` runs a bucket-level replication plus `replication-reset` to the future peer, joins the sites, and requires the operator target untouched, both paths delivering, the site resync completing against the site target, and the operator target and rule surviving `replicate remove --all`; without the fix it fails at the join with the operator target gone. The repl-nightly selection digest is refreshed for the new case. * test(site-replication): drop a redundant clone flagged by clippy The reconcile unit test cloned the remote peer into the state map although the binding is not used afterwards; workspace clippy (-D warnings) rejects that as redundant_clone. * chore(deps): update flake.lock (#7733) * chore(deps): update flake.lock Flake lock file updates: • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/17de0b9' (2026-09-04) → 'github:NixOS/nixpkgs/aff8a0b' (2026-09-10) • Updated input 'rust-overlay': 'github:oxalica/rust-overlay/c361047' (2026-09-05) → 'github:oxalica/rust-overlay/228ecef' (2026-09-12) * fix: satisfy Rust 1.98 clippy lints Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore(deps): update flake.lock (#7733) * chore(deps): update flake.lock Flake lock file updates: • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/17de0b9' (2026-09-04) → 'github:NixOS/nixpkgs/aff8a0b' (2026-09-10) • Updated input 'rust-overlay': 'github:oxalica/rust-overlay/c361047' (2026-09-05) → 'github:oxalica/rust-overlay/228ecef' (2026-09-12) * fix: satisfy Rust 1.98 clippy lints Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * feat(connect): support explicit enterprise proxies (#7745) feat(connect): add explicit proxy transport * feat(connect): emit drive unavailable log events (#7747) * feat(connect): add local inspect export producer (#7750) * feat(connect): add local inspect export producer * fix(connect): use the storage inspection facade * test(connect): cover nested allocator profile stats (#7717) * test(connect): cover nested allocator profile stats * test(connect): use async profile test lock * test(security): allow synthetic redaction fixture --------- Co-authored-by: Hauser <housemecn@gmail.com> * docs(security): add presigned copy advisory lesson (#7754) * fix(ecstore): refuse RMW and reads of unreadable bucket configs (#7759) An XML bucket sub-config whose stored bytes cannot be parsed was still folded into "absent" on two paths: update_config_with handed it to mutate, which rebuilt it from nothing and overwrote the only copy of the bytes, and the tagging/lifecycle/CORS/website/logging/accelerate/request-payment getters reported ConfigNotFound. Add an explicit ConfigState (Absent / Valid / Unreadable) over the existing retained parse failure, refuse the read-modify-write of an unreadable target config before mutate runs (per config, other configs stay writable), and make those getters fail closed. GetBucketLifecycle now only maps ConfigNotFound to NoSuchLifecycleConfiguration. The Swift tagging rewrite recognizes the refusal by type instead of a string sentinel. Refs rustfs/backlog#1734 * refactor(filemeta): own persisted metadata key authority (#7760) Add rustfs_filemeta::metadata_keys as the single source of the nine xl.meta meta_user keys, byte-identical to current literals, and migrate filemeta consumers off s3s::header and rustfs_utils header constants. Pin the literals, cross-check historical sources, and prove old xl.meta bytes still decode via a captured fixture and a per-character mutation sweep. Refs rustfs/backlog#1735 * Sync network performance protocol fixtures (#7761) test(connect): sync diagnostic protocol fixtures * Sync site replication result fixtures (#7762) test(protocol): sync site replication result fixtures * Sync telemetry producer fixtures (#7763) test(protocol): sync telemetry producer vectors * refactor(rio): own trailer source and drop s3s dependency (#7764) Move the aws-chunked trailing checksum handle behind a RustFS-owned rustfs_rio::TrailerSource (Pending/Missing/Present lookup with a documented EOF timing contract) and adapt s3s::TrailingHeaders once in the application crate. rio no longer depends on s3s; behaviour is unchanged. Refs rustfs/backlog#1735 * Deliver diagnostic scheduler receipts to Connect (#7767) feat(connect): deliver diagnostic scheduler receipts * fix(ecstore): refuse writes on unreadable bucket configs with 503 (#7768) Close the remaining paths where a stored bucket sub-configuration whose bytes cannot be parsed still read as absent (rustfs/backlog#1734, slice 2): - Object writes resolve versioning through BucketVersioningSys::get_for_write; RUSTFS_BUCKET_CONFIG_PARSE_MODE=strict refuses them, the default permissive mode keeps the historical unversioned write and records it. - Versioning, Object Lock, encryption, public access block and notification getters and the delete-time versioning check return the typed UnreadableBucketConfig refusal, which ApiError maps to 503 naming the bucket, config and stored length. StorageError::clone keeps it typed. - Object Lock checks stay on when the lock config is unreadable. - Notification setup isolates an unreadable config to its bucket instead of clearing its rules; GetBucketNotificationConfiguration reports it. - Add rustfs_bucket_metadata_parse_failed_total and rustfs_bucket_metadata_unparsable_current, an invalid-mode startup check, and the s3gate-parse-strict compat register entry. * feat(connect): add site replication performance producer (#7769) feat(connect): add site replication performance core * refactor(ecstore): read persisted object-metadata keys from the filemeta authority (A3b) (#7770) Migrate ecstore and rustfs-lifecycle consumers of the persisted xl.meta meta_user keys (object lock, restore, replication status, storage class, SSE read) from s3s::header / rustfs_utils header constants to rustfs_filemeta::metadata_keys. HTTP header production is unchanged and every lookup keeps its previous exact / case-insensitive mode. - SSE key: persisted only as lowercase; the mixed-case spelling is outbound replication user metadata. Keep the case-insensitive read and pin it against the pre-A3a fixture. - warm tier: strip promoted keys case-insensitively so the persisted X-Amz-Replication-Status is no longer forwarded to the tier. - fix stale HashReader::add_checksum_from_s3s call in ecstore tests. Refs rustfs/backlog#1735 * feat(connect): add device license renewal client (#7771) * feat(connect): add approved artifact relay (#7772) feat(connect): add approved artifact relay core * fix(connect): export relay CLI types (#7773) * feat(connect): add device report upload client (#7774) * feat(connect): add typed telemetry trace source (#7775) * fix(connect): avoid shadowing replication URL helper (#7777) * fix(connect): restore site replication test nonce (#7778) * fix(connect): restore rustfs build after site replication perf producer (#7779) #7769 left rustfs/src/connect/diagnostics/perf_site_replication.rs failing to compile: a local binding named object_url shadowed the object_url() helper in the same scope (E0618), and a test put the consent nonce on the request instead of LocalSiteReplicationConsent (E0063/E0560). Rename the binding to source_url and move the test nonce into the consent. * fix(rustfs): restore test compilation (#7780) * feat(connect): sign site replication target pair (#7781) * refactor(filemeta): own RestoreStatus and drop the s3s dependency (A3c) (#7782) Replace s3s::dto::{RestoreStatus, Timestamp} in filemeta with a filemeta-owned RestoreStatus (same field names, OffsetDateTime expiry). RestoreStatusOps and parse_restore_obj_status keep their signatures and persisted rendering; the ecstore restore finalize / lifecycle restore writers and rustfs RestoreObject now build the filemeta type. - crates/filemeta: drop s3s; tokio "time" becomes a dev-dependency (the metacache tests got it through s3s feature unification). - metadata_keys: drop the s3s half of the historical-source cross-check; PINNED, the pre-A3a fixture and per-character mutation still pin keys. - new test re-renders the pre-A3a fixture restore value byte-for-byte. - s3s footprint baselines 210 -> 209 files, ecstore 37 -> 36. Refs rustfs/backlog#1735 * feat(connect): add service license relay import (#7784) * feat(admin): add a gateway key inventory for the S3 stack switch (#7785) GET /rustfs/admin/v3/gateway-key-inventory (admin:InspectData) lists every stored object key the RustFS S3 gateway would refuse on every operation, so an operator can copy those objects to a safe key before switching stacks. The rules mirror the gateway key floor; . and .. segments never appear because ecstore refuses them on every write. Refs rustfs/gateway#754 * feat(server): add RUSTFS_S3_STACK with a gateway GetBucketLocation path (#7786) * fix(connect): restore the site replication producer build The producer shadowed its object_url helper with a local of the same name, and its test placed the consent nonce on the request, so neither the rustfs library nor its tests compiled on main. * feat(server): add RUSTFS_S3_STACK with a gateway GetBucketLocation path RUSTFS_S3_STACK=legacy (default) keeps the s3s service as the whole S3 entry. RUSTFS_S3_STACK=gateway routes GetBucketLocation through the RustFS Gateway pipeline pinned at rustfs/gateway@90b83a20 and falls back to the same s3s service for every other request, decided before the body is read. Refs rustfs/backlog#1752. * revert: remove M1 gateway git dependency until CI can access rustfs/gateway (#7787) Revert "feat(server): add RUSTFS_S3_STACK with a gateway GetBucketLocation path (#7786)" This reverts commitb5c2e1da90. * fix: retry IAM migration quorum failures on startup (#7788) * fix: retry interrupted uploads through proxies (#7789) fix: retry proxied upload interruptions * fix(connect): capture telemetry from server runtime (#7790) * fix: classify explicit proxy failures (#7791) * fix: skip IAM migration when legacy volume is absent (#7792) * fix(release): include CLI in Linux artifacts (#7794) * fix(release): serialize Linux binary links (#7797) * feat(connect): expose live lock count snapshots (#7801) * feat(connect): capture live API activity (#7804) * feat(connect): collect native Linux thread states (#7806) Collect bounded aggregate thread states from Linux procfs while preserving explicit unsupported outcomes for unavailable scopes. * feat(connect): capture live RPC activity (#7807) * test(connect): verify top.disk release artifacts (#7808) * test(connect): require native top.disk runner (#7809) * feat(connect): capture bounded local CPU profiles (#7811) * ci: retain top disk acceptance archive (#7812) * fix(connect): observe host traffic in top net (#7814) * ci: enable CPU profiling in supported release builds (#7815) * test(connect): verify profile cpu release artifacts (#7818) * test(connect): retain CPU profile verification key (#7819) * test(connect): retain CPU profile import bundle (#7820) * feat(connect): expose object inspect CLI (#7821) * feat(connect): wrap diagnostic reports for upload (#7822) * fix(connect): parse inspect paths as paths (#7826) * feat(connect): verify typed diagnostic jobs (#7827) * fix(connect): accept KMS report upload authorization (#7828) * Run signed diagnostic jobs in the RustFS service (#7829) feat(connect): run diagnostic jobs in service * fix(connect): emit canonical report manifest time (#7830) * Upload diagnostic job results through support bundles (#7831) feat(connect): upload diagnostic job results * fix(connect): emit canonical top envelope time (#7836) * fix(connect): publish diagnostic producer capabilities (#7837) * fix(io-metrics): drop the rustfs-common edge by injecting the S3 telemetry observer (#7795) * fix(io-metrics): drop the rustfs-common edge by injecting the S3 telemetry observerdc700d366(#7775) made the io-metrics leaf crate depend on rustfs-common to publish typed telemetry trace events from S3HttpRequestGuard, which breaks the backlog#1834 leaf-crate rule (io-metrics may only depend on rustfs-s3-ops). S3HttpRequestGuard now takes an optional fn-pointer completion observer (operation, latency, 2xx?) and knows nothing about the trace bus. The server builds guards through rustfs::server::s3_http_request_guard, which attaches the observer only while a telemetry subscriber exists and owns the S3Operation -> TelemetryTraceOperation mapping. * chore(guard): admit #7785 s3s ratchet growth (+1 file, +4 lines) #7785 landed the gateway key inventory admin handler after the baselines were verified, leaving check_s3s_footprint.sh red on main and every branch cut from it (measured 210 files / 1592 lines vs 209/1588 baselines). The handler follows the house admin convention whose Operation::call signature is s3s-typed (S3Request/S3Result), so it cannot route through a non-s3s seam until the s3gate admin migration replaces the admin router (rustfs/backlog#1677 F1); no local refactor can shed the file-level import. Record the measured growth with rationale: files 209 -> 210, s3_error! lines 1588 -> 1592. * fix * fix: resolve clippy gate failures in connect and CLI surfaces - Box the two large CommandResult performance variants and LicenseRenewalOutcome::Installed (clippy::large_enum_variant) - Return RelayError instead of () from RelayTransport::deliver and the test destination (clippy::result_unit_err) - Drop an unused mut on the protected relay file handle (unused_mut) - Pass the digest by value to base64 encoding and drop a redundant result_json clone (clippy::needless_borrows_for_generic_args, clippy::redundant_clone) * fix(connect): resolve the remaining site-replication clippy errors - Group the endpoint credential triple into SiteReplicationCredentials so SiteReplicationEndpoint::new takes 6 arguments (clippy::too_many_arguments) - Take the build-feature validator by &str (clippy::ptr_arg); the validator closure stays because &String needs the deref coercion * style: apply rustfmt to the site-replication credential grouping * fix(connect): keep the CPU profile fixture off the ecstore import scan #7811 used "rustfs_ecstore::disk::read_object" as the raw-symbol fixture of the profile redaction test, which the architecture guard counts as a direct rustfs_ecstore reference outside the compatibility boundaries (its scan deliberately includes inline test modules). The accumulator treats symbols opaquely, so use the internal rustfs::storage:: path that keeps the fixture realistic without matching the import scanner. * fix * fix * fix * fix --------- Co-authored-by: Hauser <housemecn@gmail.com> * fix(replication): count a bodiless 405 as a replicated delete marker (#7756) * fix(replication): accept a bodiless 405 as a replicated delete marker during resync A resync verifies each delete marker with `HEAD ?versionId=<marker>` on the target. S3 targets (RustFS, MinIO, AWS) answer that with 405 and no body, and the SDK only synthesizes an error code for 404, so the error arrived with `code() == None`. `is_retryable_delete_replication_head_error` then treated it as an ambiguous failure: every delete marker counted as a failed object with `target service error`, and a site resync over any bucket that holds a delete marker reported the whole bucket as failed (rustfs/backlog#2479, SITE-105 on rc.6 and nightly). Use the raw HTTP status the way the 404 path already does: a 405 without a code confirms the marker propagated. Ambiguous statuses still fail. - Unit tests drive `verify_resync_head_result` against a scripted target answering 405 (accepted) and 503 (still failed). - e2e `test_site_replication_resync_replicates_delete_marker` joins two sites, converges a live object and a delete marker, and requires the site resync to complete with zero failed objects; the repl-nightly selection digest is refreshed for the new case. * fix(replication): verify marker-version purges by absence during resync A delete-marker resync entry with a pending or failed version purge asks the target to remove the marker, so the bodiless 405 that proves a created marker propagated proves the purge did not happen. Accept the 405-as-success mapping only for marker creation (empty version_purge_status); a purge counts as replicated only when the target answers not found, and any other HEAD outcome stays failed. Unit tests drive both purge statuses against the 405 fixture and the 404 fixture. * fix(kms): refuse key ids that leave the key prefix on the Vault backends (#7727) Only the Local backend refused a key identifier containing a path separator. On Vault KV2 a create with the name bad/name succeeded and produced a nested KV2 path that the listing then reported as a directory rather than a key, and an identifier containing .. addressed a record outside the configured key prefix once the HTTP client normalised the URL; Vault Transit built its transit key name and its metadata path from the same unchecked identifier. Lift the Local backend's containment rule into a shared segment check (empty, /, backslash, NUL, . and ..) and apply it at the single point where each backend turns the identifier into a path or a Transit key name, so create, describe, encrypt, delete and the metadata writes all refuse with InvalidKey before any request reaches Vault. The admin API already maps that to 400. The AWS backend is untouched: it addresses keys by ARN and alias, both of which contain /. Refs rustfs/backlog#2474 (KMS-213 CreateNegatives on vault-kv2). Co-authored-by: Hauser <housemecn@gmail.com> * Preserve safe profile job failure reasons (#7860) fix(connect): preserve safe profile job failure reasons * Freeze the heartbeat producer capability registry (#7861) test(connect): freeze heartbeat producer capabilities * fix: resume GET body after peer short EOF (#7852) * fix: resume get body after short eof Treat mid-stream short EOF errors as resumable when the S3 GET body has an attached resume context. This lets peer-kill reads reopen the same object version at the emitted offset instead of aborting the 200 response body. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(connect): surface TLS peer bootstrap failures Map Connect registration TLS peer validation failures to the dedicated bootstrap error so wrong CA inputs fail closed with the sanitized TLS variant instead of the generic exchange error. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * ci: file scheduled-failure issues in rustfs/backlog (#7847) Scheduled pipeline failures currently open [scheduled-failure] tracking issues in rustfs/rustfs itself, adding bot noise to the code repository's issue tracker. Route that alerting to rustfs/backlog instead. - schedule-failure-issue gains target-repository (default rustfs/backlog); dedupe lookup, comment appends and issue creation target that repository. - New source-token input (default ${{ github.token }}) reads the failed-job list from the reported run; the run still lives in and links to rustfs/rustfs. The issue body now carries a '- Repository:' line. - All consumer workflows pass secrets.BACKLOG_ISSUE_TOKEN; the workflow-scoped GITHUB_TOKEN cannot open issues cross-repo. Alert job permissions blocks are unchanged. * fix(admin): keep the gateway key inventory off the s3s surface (#7817) The gateway key inventory handler added in #7785 imported s3s directly and built four errors with s3_error!, pushing the s3s footprint ratchet to 210 files and 1592 s3_error! lines on main. Route its S3 types and error construction through the crate::admin::storage_api::s3 facade like sibling admin handlers; the codes, HTTP statuses and messages are unchanged, and a new unit test pins them. Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Hauser <housemecn@gmail.com> * fix(s3): honor presigned UploadPart checksum queries (#7748) * fix(s3): honor presigned UploadPart checksum queries * fix(s3): keep checksum errors within gateway boundary --------- Co-authored-by: Hauser <housemecn@gmail.com> * fix(auth): reject unsigned x-amz headers on header-signed SigV4 requests (#7796) * fix(auth): reject unsigned x-amz headers on header-signed SigV4 requests A SigV4 request authenticated with an Authorization header only binds the headers named in its SignedHeaders list, but RustFS acted on every x-amz-* header that arrived, so a replayed header-signed PutObject carrying an unsigned x-amz-copy-source became a CopyObject run as the signer that could copy any object the signer can read (GHSA-xm99-m3gq-83g8). The presigned form was already closed by GHSA-g8w9-qw9q-fghr. reject_unsigned_amz_headers_on_sigv4_request now guards S3Access::check and S3Router::check_access: every SigV4 signed-header list the request carries must cover every x-amz-* header, the Authorization header is parsed with the verifier's own s3s-sigv4 parser and compared case-insensitively, the algorithm token is pinned to AWS4-HMAC-SHA256 because the upstream header path accepts any token, and the exempt set mirrors the upstream s3s fix (x-amz-content-sha256, x-amz-decoded-content-length, x-amz-trailer, x-amz-checksum-algorithm) plus x-amz-cf-id. Adds ghsa_xm99 unit, router and e2e regressions, raises the security smoke floor to 28, and records the advisory in docs/testing/security-regressions.md and CHANGELOG.md. * chore(deps): switch s3s and s3s-sigv4 to crates.io 0.16.0 * fix(server): enforce the SigV4 header guard ahead of s3s dispatch s3s 0.16 verifies the claimed algorithm as the first step of its own signature flow, so a request whose Authorization header swaps the AWS4-HMAC-SHA256 token was answered with 501 NotImplemented before RustFS's access layer could rule on the unsigned x-amz-copy-source (GHSA-xm99-m3gq-83g8). Add the SigV4HeaderGuardLayer as the innermost external stack layer, running reject_unsigned_amz_headers_on_sigv4_request in front of s3s and serializing its rejections as the same AccessDenied S3 error document the access layer produces. --------- Co-authored-by: Hauser <housemecn@gmail.com> * fix(table-catalog): map transient lock failures to unavailable (#7841) * Verify profile service jobs on native Linux (#7862) ci: verify profile service jobs on native Linux * Authorize the private Connect acceptance checkout (#7863) ci: authorize private Connect harness checkout * Run profile service acceptance directly (#7864) ci: run profile service acceptance directly * Retain profile service failure evidence (#7868) ci: retain profile service failure evidence * fix: update dependencies and preserve transport errors (#7798) * chore(deps): bump ed25519-dalek in the dependencies group Bumps the dependencies group with 1 update: [ed25519-dalek](https://github.com/dalek-cryptography/curve25519-dalek). Updates `ed25519-dalek` from 2.2.0 to 3.0.0 - [Release notes](https://github.com/dalek-cryptography/curve25519-dalek/releases) - [Changelog](https://github.com/dalek-cryptography/curve25519-dalek/blob/3.0.0/CHANGELOG.md) - [Commits](https://github.com/dalek-cryptography/curve25519-dalek/compare/ed25519-2.2.0...3.0.0) --- updated-dependencies: - dependency-name: ed25519-dalek dependency-version: 3.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: dependencies ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump jiff, aws-sdk-s3, datafusion and smallvec Bump the dependencies group: - jiff 0.2.35 -> 0.2.37 - aws-sdk-s3 1.146.0 -> 1.146.1 - datafusion 55.0.0 -> 55.1.0 - smallvec 1.16.0 -> 1.16.1 Update Cargo.lock for the matching transitive bumps (datafusion subcrates, hickory, cc, crc32fast, tinyvec, toml_edit and others); tinyvec_macros is no longer required. * chore(deps): use s3s 0.16.0 release and bump rust-version to 1.98.1 * fix: preserve ready reader errors during erasure write failures When erasure writers fail after the producer has already surfaced a reader error, return the reader error instead of aborting the producer and masking it with the write-side failure. This keeps UploadPart body read timeouts classified as RequestTimeout under loaded pipeline timing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: use current checksum API in multipart regressions * fix: stabilize connect dependency update tests Classify Connect proxy failures only when an explicit proxy is configured, preserve TLS classification across request error shapes, keep short object performance windows from spending too much budget on cleanup, and update multipart checksum tests for the current HashReader API. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Hauser <housemecn@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> * ci: expect BACKLOG_ISSUE_TOKEN in scheduled alert wiring checks (#7867) #7847 routed schedule-failure alerting to rustfs/backlog by switching the alert jobs' token to secrets.BACKLOG_ISSUE_TOKEN, but check_test_wiring.py still asserts the literal 'github-token: ${{ secrets.GITHUB_TOKEN }}', so Quick Checks fails on every open PR. Update the wiring assertions and the self-test fixtures to the new literal. Both `--self-test` (52 tests) and the full `validate()` pass locally. * Pace diagnostic job result redelivery (#7869) fix(connect): pace diagnostic result redelivery * Execute authenticated top.api service jobs (#7872) feat(connect): execute top.api service jobs * feat(connect): execute top.locks service jobs (#7874) * fix(ci): gate Connect acceptance on x86 build job (#7875) * fix(ci): repair E2E inventory and network CLI coverage (#7871) * ci: refresh Linux full E2E inventory after SigV4 tests * test(connect): drive traffic for top net CLI export coverage * feat(connect): execute performance network jobs (#7877) * feat(connect): execute bounded drive performance jobs (#7878) * Preserve the top API service job nonce (#7879) fix(connect): preserve top api job nonce * feat(connect): execute thread profile service jobs (#7880) * Execute authenticated top.rpc service jobs (#7881) feat(connect): execute top.rpc service jobs * ci: add service diagnostic acceptance workflows (#7882) * ci: add threads and RPC acceptance workflows (#7883) * fix(connect): deliver diagnostic results until expiry (#7886) * feat(connect): export signed environment inventory (#7887) feat: export signed environment inventory * fix: recover interrupted GETs and heal operations (#7876) * fix(heal): retry interrupted internode shard writes * fix(deps): update rustls for RUSTSEC-2026-0285 * fix(heal): retain RPC transport failures across metadata probes * test(heal): wait for replacement disk and coordinator readiness * test(connect): validate service-only diagnostic capabilities * fix(s3): resume GET bodies after erasure read quorum loss * fix: keep top locks within its capture window (#7888) * fix: recover heal after peer boot epoch changes (#7889) * test(heal): keep outage objects in the replacement erasure set * fix(rpc): retry authenticated requests after peer epoch changes * test(rpc): assert retryable rejection of stale boot epochs * ci: bound concurrent heal clusters in the full E2E gate * ci: register the table suite workflow on the default branch (#7893) * ci: register the table suite workflow on the default branch The table suite workflow (#7873) only exists on release so far, and GitHub registers workflow_dispatch targets exclusively from the default branch — the suite cannot be triggered until this file lands on main (same mechanism that keeps the fault-tolerance suite undispatchable). This adds the identical rustfs-table-test.yml (byte-for-byte the release version) plus the contract-test registration (JOBS / DIRECT_TESTS / upload allowlist), and nothing else from release. * ci: pin actions in the table workflow to full commit SHAs The default branch enforces check_workflow_pins.sh (unpinned third-party action refs fail CI). Use the same pinned SHAs as the other suite workflows on main: actions/checkout@9c091bb... (#v7) and actions/upload-artifact@b7c566a... (#v6). Note: the release copy of this workflow still uses @v4 tags; release does not run the pin gate. Release and main will converge on the pinned form during the full sync. * test(ecstore): synchronize causal cleanup observations (#7894) * fix(ecstore): create markers for repeated versioned deletes (#7884) Co-authored-by: cxymds <cxymds@gmail.com> * Preserve top.locks signed job nonce (#7898) fix(connect): preserve top locks job nonce * fix(ecstore): roll back failed delete marker writes (#7899) * chore(deps): refresh RustFS tokio tar dependency (#7907) * chore(deps): refresh RustFS tokio tar dependency Switch the astral-tokio-tar workspace alias to the released rustfs-tokio-tar 0.7.1 crate and remove the obsolete cargo-deny git source allowance. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): align AWS smithy dependency family Upgrade the AWS SDK and smithy crates together while patching aws-smithy-json 0.63.0 for the aws-smithy-types 1.7.0 DocumentObject API used by aws-config 1.12.0. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * revert(deps): use official AWS smithy compatible set Remove the local aws-smithy-json patch and roll the AWS smithy family back to the latest official combination that compiles with aws-config 1.12.0. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(architecture): retire tokio tar cleanup entry Remove the stale cleanup-register entry after switching back to the released rustfs-tokio-tar crate, leaving no unmatched compatibility marker for the architecture guard. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: restore table workflow failure contract Align the table functional workflow with the quick-check guard by letting suite failures stay red, generating PASS reports only from complete successful evidence, and preserving the shared backlog issue fallback for harness failures. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(deps): expose rustfs tokio tar dependency name Use the published rustfs-tokio-tar dependency key throughout the workspace while keeping source imports on the crate's tokio_tar lib target name. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ci): retain failed functional chain reports and check runners (#7906) --------- Signed-off-by: houseme <housemecn@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Hauser <housemecn@gmail.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> Co-authored-by: Henry Guo <marshawcoco@gmail.com> Co-authored-by: 唐小鸭 <tangtang1251@qq.com> Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: cui fliter <imcusg@gmail.com> Co-authored-by: RJ Regenold <rregenold@teamraft.com> Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: Jason Kossis <jkossis@gmail.com> Co-authored-by: Hiroaki KAWAI <hiroaki.kawai@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6232 lines
323 KiB
Python
Executable File
6232 lines
323 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""Fail when committed tests silently fall out of their execution wiring."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import json
|
|
import os
|
|
import re
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
import unittest
|
|
import uuid
|
|
import xml.etree.ElementTree as ET
|
|
from datetime import datetime, timedelta, timezone
|
|
from unittest import mock
|
|
from pathlib import Path
|
|
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
|
|
|
try:
|
|
import tomllib
|
|
except ModuleNotFoundError:
|
|
import tomli as tomllib
|
|
|
|
from scanner_abba import (
|
|
MAX_JSON_BYTES,
|
|
RELEASE_PROFILE_ARTIFACTS,
|
|
SCENARIOS,
|
|
digest,
|
|
number,
|
|
read_json,
|
|
require,
|
|
sha,
|
|
write_json,
|
|
)
|
|
|
|
|
|
ROOT = Path(__file__).resolve().parents[1]
|
|
SCANNER_HEAL_REGISTRY_SCHEMA_MAX = 2
|
|
SCANNER_HEAL_RELEASE_REQUIRED_GATES = (
|
|
"G01", "G02", "G03", "G04", "G05", "G06", "G07", "G08", "G09", "G10", "G11", "G12", "G13", "G14",
|
|
"P1", "P2", "P3", "P4", "R-E", "R-D", "R-L",
|
|
)
|
|
SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS = {
|
|
"G01": (
|
|
"root_authority_evidence",
|
|
"quota_authority_evidence",
|
|
),
|
|
"G02": (
|
|
"bounded_checkpoint_oracle",
|
|
"independent_version_inventory",
|
|
),
|
|
"G03": (
|
|
"durable_root_publication_proof",
|
|
"scoped_ack_request_identity",
|
|
"participating_peer_capability_snapshot",
|
|
"mixed_peer_ack_fallback_oracle",
|
|
),
|
|
"G04": (
|
|
"cache_boundary_crash_evidence",
|
|
"root_floor_intent_crash_evidence",
|
|
),
|
|
"G05": (
|
|
"per_object_outcome_oracle",
|
|
"terminal_retention_bounds",
|
|
),
|
|
"G06": (
|
|
"concurrent_status_evidence",
|
|
"legacy_client_compatibility",
|
|
"truncation_behavior",
|
|
),
|
|
"G07": (
|
|
"mrf_responsibility_oracle",
|
|
"commit_boundary_crash_matrix",
|
|
),
|
|
"G08": (
|
|
"mrf_capacity_evidence",
|
|
"disk_full_matrix",
|
|
"replica_loss_matrix",
|
|
),
|
|
"G09": (
|
|
"mixed_version_reader_evidence",
|
|
"mixed_version_writer_evidence",
|
|
"rollback_payload_evidence",
|
|
),
|
|
"G10": (
|
|
"scheduler_bound_evidence",
|
|
"pressure_recovery_evidence",
|
|
),
|
|
"G11": (
|
|
"maintenance_producer_matrix",
|
|
"complete_producer_inventory",
|
|
"segment_activation_preflight",
|
|
),
|
|
"G12": (
|
|
"reset_quota_path_evidence",
|
|
"settlement_quota_path_evidence",
|
|
),
|
|
"G13": (
|
|
"quorum_minus_one_matrix",
|
|
"unknown_disk_remount_matrix",
|
|
"object_lock_dry_run_grace_evidence",
|
|
),
|
|
"G14": (
|
|
"same_window_field_evidence",
|
|
"ec8_4_evidence",
|
|
"multi_set_evidence",
|
|
"multi_pool_evidence",
|
|
"distributed_segment_invalidation_evidence",
|
|
),
|
|
"P1": (
|
|
"cold_walk_share_measurement",
|
|
"foreground_latency_throughput_measurement",
|
|
"profile_evidence",
|
|
),
|
|
"P2": (
|
|
"post_stop_convergence_measurement",
|
|
"cold_segment_reuse_measurement",
|
|
),
|
|
"P3": (
|
|
"two_hour_pressure_measurement",
|
|
"heal_capacity_measurement",
|
|
"recovery_window_measurement",
|
|
),
|
|
"P4": (
|
|
"mrf_scale_measurement",
|
|
"mrf_replay_cost_measurement",
|
|
"retained_responsibility_evidence",
|
|
"mrf_cleanup_gc_soak_evidence",
|
|
),
|
|
"R-E": (
|
|
"fixed_budget_restart_evidence",
|
|
"enumeration_evidence",
|
|
"classification_evidence",
|
|
),
|
|
"R-D": (
|
|
"manager_disposition_evidence",
|
|
"event_disposition_evidence",
|
|
"ledger_disposition_evidence",
|
|
"grace_handling",
|
|
),
|
|
"R-L": (
|
|
"legacy_source_conflict_evidence",
|
|
"migration_gap_evidence",
|
|
"crash_safe_source_retirement_evidence",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS = {
|
|
"G01": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G01"],
|
|
"G02": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G02"],
|
|
"G03": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G03"],
|
|
"G04": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G04"],
|
|
"G05": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G05"],
|
|
"G06": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G06"],
|
|
"G07": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G07"],
|
|
"G08": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G08"],
|
|
"G09": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G09"],
|
|
"G10": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G10"],
|
|
"G11": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G11"],
|
|
"G12": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G12"],
|
|
"G13": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G13"],
|
|
"G14": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["G14"],
|
|
"P1": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["P1"],
|
|
"P2": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["P2"],
|
|
"P3": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["P3"],
|
|
"P4": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["P4"],
|
|
"R-E": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["R-E"],
|
|
"R-D": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["R-D"],
|
|
"R-L": SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS["R-L"],
|
|
}
|
|
SCANNER_HEAL_RELEASE_MIXED_VERSION_ROLES = {
|
|
("G03", "durable_root_publication_proof"): "durable-root-publication",
|
|
("G03", "scoped_ack_request_identity"): "scoped-ack-request",
|
|
("G03", "participating_peer_capability_snapshot"): "peer-capability-snapshot",
|
|
("G03", "mixed_peer_ack_fallback_oracle"): "mixed-peer-ack-fallback",
|
|
("G09", "mixed_version_reader_evidence"): "mixed-version-reader",
|
|
("G09", "mixed_version_writer_evidence"): "mixed-version-writer",
|
|
("G09", "rollback_payload_evidence"): "rollback-payload",
|
|
("R-L", "legacy_source_conflict_evidence"): "legacy-source-conflict",
|
|
("R-L", "migration_gap_evidence"): "migration-gap",
|
|
("R-L", "crash_safe_source_retirement_evidence"): "crash-safe-source-retirement",
|
|
}
|
|
SCANNER_HEAL_RELEASE_G01_ROOT_AUTHORITY_CASES = (
|
|
"root-cas-success",
|
|
"root-readback-success",
|
|
"incomplete-root-rejected",
|
|
"stale-root-rejected",
|
|
)
|
|
SCANNER_HEAL_RELEASE_G01_QUOTA_AUTHORITY_CASES = (
|
|
"quota-floor-readback",
|
|
"quota-over-limit-rejected",
|
|
"rejected-object-invisible",
|
|
"quota-fails-closed-without-authority",
|
|
)
|
|
SCANNER_HEAL_RELEASE_MRF_DURABLE_REPLAY_FIELDS = {
|
|
("G07", "mrf_responsibility_oracle"),
|
|
("G07", "commit_boundary_crash_matrix"),
|
|
("P4", "mrf_replay_cost_measurement"),
|
|
("P4", "retained_responsibility_evidence"),
|
|
("P4", "mrf_cleanup_gc_soak_evidence"),
|
|
}
|
|
SCANNER_HEAL_RELEASE_MRF_ARTIFACT_KINDS = {
|
|
("G07", "mrf_responsibility_oracle"): "mrf-durable-responsibility-oracle",
|
|
("G07", "commit_boundary_crash_matrix"): "mrf-commit-boundary-crash-matrix",
|
|
("G08", "mrf_capacity_evidence"): "mrf-capacity-boundary",
|
|
("G08", "disk_full_matrix"): "mrf-disk-full-enospc-matrix",
|
|
("G08", "replica_loss_matrix"): "mrf-replica-loss-matrix",
|
|
("P4", "mrf_scale_measurement"): "mrf-scale-measurement",
|
|
("P4", "mrf_replay_cost_measurement"): "mrf-replay-cost-measurement",
|
|
("P4", "retained_responsibility_evidence"): "mrf-retained-responsibility-soak",
|
|
("P4", "mrf_cleanup_gc_soak_evidence"): "mrf-cleanup-gc-soak",
|
|
}
|
|
SCANNER_HEAL_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS = (
|
|
"missing_producer_identity",
|
|
"missing_durable_journal_replay",
|
|
"restart_gap",
|
|
"generation_gap",
|
|
"overflow",
|
|
"stale_ack_generation",
|
|
"missing_cold_zero_walk_oracle",
|
|
"distributed_without_peer_invalidation",
|
|
)
|
|
SCANNER_HEAL_SEGMENT_ACTIVATION_PROOF_INPUTS = (
|
|
"source",
|
|
"bucket_incarnation",
|
|
"key_format",
|
|
"baseline_scan_plan_digest",
|
|
"process_epoch",
|
|
"generation_window",
|
|
"producer_identities",
|
|
"durable_dirty_producer_journal",
|
|
"ack_generation_guard",
|
|
)
|
|
SCANNER_HEAL_REQUIRED_PRODUCER_IDENTITIES = (
|
|
"put_object",
|
|
"delete_object",
|
|
"delete_marker",
|
|
"complete_multipart_upload",
|
|
"abort_multipart_upload",
|
|
"object_metadata",
|
|
"bucket_metadata",
|
|
"replication",
|
|
"tier_transition",
|
|
"tier_expiration",
|
|
"directory_object",
|
|
)
|
|
SCANNER_HEAL_REQUIRED_PRODUCER_FAMILIES = (
|
|
"put",
|
|
"delete",
|
|
"delete_marker",
|
|
"multipart",
|
|
"replication",
|
|
"tier",
|
|
"directory_object",
|
|
)
|
|
SCANNER_HEAL_DURABLE_JOURNAL_FAIL_CLOSED_CASES = (
|
|
"corrupt-record",
|
|
"mixed-version-producer",
|
|
"oversized-record",
|
|
"unsupported-producer",
|
|
"invalid-generation",
|
|
)
|
|
SCANNER_HEAL_RELEASE_G11_REQUIRED_CASES = {
|
|
"maintenance_producer_matrix": (
|
|
"object-mutation-producers",
|
|
"metadata-mutation-producers",
|
|
"replication-tier-producers",
|
|
"directory-object-producer",
|
|
),
|
|
"complete_producer_inventory": SCANNER_HEAL_REQUIRED_PRODUCER_IDENTITIES,
|
|
}
|
|
SCANNER_HEAL_RELEASE_G13_REQUIRED_CASES = {
|
|
"quorum_minus_one_matrix": (
|
|
"read-quorum-minus-one",
|
|
"write-quorum-minus-one",
|
|
"heal-quorum-minus-one",
|
|
"restart-quorum-minus-one",
|
|
),
|
|
"unknown_disk_remount_matrix": (
|
|
"unknown-disk-excluded-from-quorum",
|
|
"known-disk-remount-rejoins",
|
|
"stale-disk-remount-rejected",
|
|
),
|
|
"object_lock_dry_run_grace_evidence": (
|
|
"object-lock-delete-denied",
|
|
"dry-run-does-not-mutate",
|
|
"grace-window-retains-terminal-outcome",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_G07_REQUIRED_CASES = {
|
|
"mrf_responsibility_oracle": (
|
|
"legacy-journal-replay",
|
|
"scoped-journal-replay",
|
|
"committed-checkpoint-replay",
|
|
),
|
|
"commit_boundary_crash_matrix": (
|
|
"before-committed-payload",
|
|
"after-payload-before-manifest",
|
|
"after-manifest-before-cleanup",
|
|
"restart-replay-before-successor",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_G08_REQUIRED_CASES = {
|
|
"mrf_capacity_evidence": (
|
|
"queue-count-limit",
|
|
"journal-byte-limit",
|
|
"committed-payload-byte-limit",
|
|
),
|
|
"disk_full_matrix": (
|
|
"payload-write-enospc",
|
|
"manifest-write-enospc",
|
|
"journal-write-enospc",
|
|
"cleanup-delete-enospc",
|
|
),
|
|
"replica_loss_matrix": (
|
|
"single-replica-loss",
|
|
"quorum-minus-one",
|
|
"all-replicas-unavailable",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_MRF_CLEANUP_GC_SOAK_CASES = (
|
|
"retained-anchor-survives-restart",
|
|
"verified-successor-allows-idle-gc",
|
|
"stale-legacy-journal-cleanup",
|
|
"repeated-replay-no-resurrection",
|
|
)
|
|
SCANNER_HEAL_RELEASE_CRASH_BOUNDARY_FIELDS = {
|
|
("G04", "root_floor_intent_crash_evidence"): (
|
|
"persist-failure-no-202",
|
|
"same-key-retry-reuses-intent",
|
|
"different-params-conflict",
|
|
"process-restart-replay",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_G12_QUOTA_PATH_CASES = {
|
|
"reset_quota_path_evidence": (
|
|
"storage-owner-reconstruction",
|
|
"future-reservation-protocol-fail-closed",
|
|
"reservation-ledger-retained",
|
|
),
|
|
"settlement_quota_path_evidence": (
|
|
"distributed-hard-quota-admission",
|
|
"quota-stats-current-usage-observed",
|
|
"oversized-put-rejected",
|
|
"rejected-object-not-visible",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_G05_PER_OBJECT_OUTCOME_CASES = (
|
|
"object-repaired",
|
|
"object-already-healthy",
|
|
"object-skipped-by-policy",
|
|
"object-failed-and-retained",
|
|
)
|
|
SCANNER_HEAL_RELEASE_G05_TERMINAL_RETENTION_CASES = (
|
|
"finished-retained-until-window",
|
|
"failed-retained-until-window",
|
|
"canceled-retained-until-window",
|
|
"expired-terminal-pruned-after-window",
|
|
)
|
|
SCANNER_HEAL_RELEASE_G06_CONCURRENT_STATUS_CASES = (
|
|
"status-during-admin-heal",
|
|
"status-during-background-heal",
|
|
"status-while-peer-down",
|
|
"status-after-peer-rejoin",
|
|
)
|
|
SCANNER_HEAL_RELEASE_G06_LEGACY_CLIENT_CASES = (
|
|
"rustfs-admin-v3-background-heal-status",
|
|
"minio-admin-v3-background-heal-status",
|
|
"heal-client-token-empty-body",
|
|
"node-heal-status-v1-wire",
|
|
)
|
|
SCANNER_HEAL_RELEASE_G06_TRUNCATION_CASES = (
|
|
"oversize-node-status-reject",
|
|
"truncated-node-status-reject",
|
|
"trailing-data-node-status-reject",
|
|
)
|
|
SCANNER_HEAL_RELEASE_RD_MANAGER_CASES = (
|
|
"accepted",
|
|
"coalesced-duplicate",
|
|
"rejected-policy",
|
|
"terminal-retained",
|
|
)
|
|
SCANNER_HEAL_RELEASE_RD_EVENT_CASES = (
|
|
"event-repaired",
|
|
"event-failed",
|
|
"event-skipped",
|
|
"event-grace-retained",
|
|
)
|
|
SCANNER_HEAL_RELEASE_RD_LEDGER_CASES = (
|
|
"ledger-recorded",
|
|
"ledger-replayed",
|
|
"ledger-discharged",
|
|
"ledger-pruned-after-grace",
|
|
)
|
|
SCANNER_HEAL_RELEASE_RD_GRACE_CASES = (
|
|
"grace-open-retains-disposition",
|
|
"grace-expired-prunes-terminal",
|
|
"restart-preserves-grace-clock",
|
|
)
|
|
SCANNER_HEAL_RELEASE_RL_REQUIRED_CASES = {
|
|
"legacy_source_conflict_evidence": (
|
|
"data-movement-owned-target-equivalence",
|
|
"newer-target-conflict-rejected",
|
|
"legacy-part-checksum-gap-rejected",
|
|
),
|
|
"migration_gap_evidence": (
|
|
"encrypted-iam-decrypt-before-normalize",
|
|
"empty-legacy-source-inherits-prior-responsibilities",
|
|
"missing-corrupt-empty-sources-fail-closed",
|
|
),
|
|
"crash_safe_source_retirement_evidence": (
|
|
"source-change-capacity-failure-keeps-old-commit",
|
|
"torn-inactive-payload-keeps-previous-anchor",
|
|
"commit-boundary-lost-response-idempotent",
|
|
"successor-retry-validates-orphan",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_SCOPED_ACK_CASES = {
|
|
"durable_root_publication_proof": (
|
|
"root-cas-success",
|
|
"root-readback-success",
|
|
"dirty-fallback-on-missing-proof",
|
|
),
|
|
"scoped_ack_request_identity": (
|
|
"bucket-incarnation",
|
|
"exact-generation",
|
|
"scanner-instance",
|
|
"participating-peer-set",
|
|
"cleared-count-bound",
|
|
),
|
|
"participating_peer_capability_snapshot": (
|
|
"supports-scoped-ack",
|
|
"probe-only-fallback",
|
|
"missing-capability-dirty",
|
|
),
|
|
"mixed_peer_ack_fallback_oracle": (
|
|
"legacy-peer-fallback",
|
|
"truncated-token-reject",
|
|
"restarted-peer-reject",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_G03_REQUIRED_TRUE_FIELDS = {
|
|
"durable_root_publication_proof": (
|
|
"root_cas_observed",
|
|
"root_readback_observed",
|
|
"dirty_fallback_on_missing_proof_observed",
|
|
),
|
|
"scoped_ack_request_identity": (
|
|
"bucket_incarnation_observed",
|
|
"exact_generation_observed",
|
|
"scanner_instance_observed",
|
|
"participating_peer_set_observed",
|
|
"whole_cycle_fallback_observed",
|
|
"cleared_count_bound_observed",
|
|
),
|
|
"participating_peer_capability_snapshot": (
|
|
"capability_probe_observed",
|
|
"scoped_ack_capability_observed",
|
|
"probe_only_fallback_observed",
|
|
"missing_capability_kept_dirty",
|
|
),
|
|
"mixed_peer_ack_fallback_oracle": (
|
|
"legacy_peer_fallback_observed",
|
|
"truncated_token_rejected",
|
|
"restarted_peer_rejected",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_MIXED_VERSION_CASES = {
|
|
"mixed_version_reader_evidence": (
|
|
"old-writer-new-reader",
|
|
"new-writer-old-reader",
|
|
),
|
|
"mixed_version_writer_evidence": (
|
|
"old-reader-new-writer",
|
|
"new-reader-old-writer",
|
|
),
|
|
"rollback_payload_evidence": (
|
|
"rollback-to-old",
|
|
"rollback-to-new",
|
|
"unknown-field-retained",
|
|
),
|
|
}
|
|
SCANNER_HEAL_RELEASE_P4_RETAINED_RESPONSIBILITY_CASES = (
|
|
"retain-pending-replay-anchor",
|
|
"verified-proof-discharges-anchor",
|
|
"idle-cleanup-reclaims-runtime-checkpoint",
|
|
"idle-cleanup-reclaims-replay-source",
|
|
)
|
|
SCANNER_HEAL_RELEASE_SCHEDULER_BOUNDS = (
|
|
"admission-retry-idempotency",
|
|
"deadline-budget",
|
|
"lock-hold-bound",
|
|
"minimum-progress",
|
|
)
|
|
SCANNER_HEAL_RELEASE_PRESSURE_METRICS = (
|
|
"foreground_p95_ms",
|
|
"foreground_p99_ms",
|
|
"throughput_ops",
|
|
"error_rate",
|
|
"heal_lock_wait_p99_ms",
|
|
"attempt_cost_samples",
|
|
)
|
|
SCHEDULED_ALERT_WORKFLOWS = tuple(
|
|
item["workflow"]
|
|
for item in json.loads((ROOT / ".github/scheduled-validations.json").read_text())
|
|
)
|
|
|
|
|
|
def words(value: str) -> set[str]:
|
|
return {item.strip() for item in value.split(",") if item.strip()}
|
|
|
|
|
|
def rust_code_only(source: str) -> str:
|
|
"""Blank Rust comments and literals while preserving byte positions."""
|
|
def quoted_end(quote_index: int, delimiter: str) -> int:
|
|
end = quote_index + 1
|
|
while end < len(source):
|
|
if source[end] == "\\":
|
|
end += 2
|
|
elif source[end] == delimiter:
|
|
return end + 1
|
|
else:
|
|
end += 1
|
|
return end
|
|
|
|
code = list(source)
|
|
index = 0
|
|
while index < len(source):
|
|
if source.startswith("//", index):
|
|
end = source.find("\n", index)
|
|
end = len(source) if end < 0 else end
|
|
elif source.startswith("/*", index):
|
|
depth = 1
|
|
end = index + 2
|
|
while end < len(source) and depth:
|
|
if source.startswith("/*", end):
|
|
depth += 1
|
|
end += 2
|
|
elif source.startswith("*/", end):
|
|
depth -= 1
|
|
end += 2
|
|
else:
|
|
end += 1
|
|
else:
|
|
raw = re.match(r'(?:br|cr|r)(?P<hashes>#{0,})"', source[index:])
|
|
if raw:
|
|
marker = '"' + raw.group("hashes")
|
|
end = source.find(marker, index + raw.end())
|
|
end = len(source) if end < 0 else end + len(marker)
|
|
elif source[index] == '"' or source.startswith(('b"', 'c"'), index):
|
|
start_quote = index if source[index] == '"' else index + 1
|
|
end = quoted_end(start_quote, '"')
|
|
elif source[index] == "'" and index + 2 < len(source) and (
|
|
source[index + 1] == "\\" or source[index + 2] == "'"
|
|
):
|
|
end = quoted_end(index, "'")
|
|
elif source.startswith("b'", index):
|
|
end = quoted_end(index + 1, "'")
|
|
else:
|
|
index += 1
|
|
continue
|
|
|
|
for offset in range(index, end):
|
|
if code[offset] != "\n":
|
|
code[offset] = " "
|
|
index = end
|
|
return "".join(code)
|
|
|
|
|
|
def declared(parent: Path, module: str) -> bool:
|
|
source = parent.read_text()
|
|
code = rust_code_only(source)
|
|
pattern = re.compile(rf"^\s*(?:pub(?:\([^)]*\))?\s+)?mod\s+{re.escape(module)}\s*;", re.MULTILINE)
|
|
allowed_preambles = {
|
|
"#[cfg(test)]": "#[cfg(test)]",
|
|
"#[cfg(all(test,target_os=))]": '#[cfg(all(test,target_os="linux"))]',
|
|
}
|
|
for match in pattern.finditer(code):
|
|
prefix = code[: match.start()]
|
|
depths = {"(": 0, "[": 0, "{": 0}
|
|
pairs = {")": "(", "]": "[", "}": "{"}
|
|
for char in prefix:
|
|
if char in depths:
|
|
depths[char] += 1
|
|
elif char in pairs:
|
|
depths[pairs[char]] -= 1
|
|
if any(depths.values()):
|
|
continue
|
|
|
|
boundary = max(prefix.rfind(";"), prefix.rfind("{"), prefix.rfind("}"))
|
|
code_preamble = re.sub(r"\s+", "", prefix[boundary + 1 :])
|
|
if not code_preamble:
|
|
return True
|
|
if code_preamble in allowed_preambles:
|
|
attr_start = prefix.rfind("#[cfg", boundary + 1)
|
|
if attr_start >= 0 and re.sub(r"\s+", "", source[attr_start : match.start()]) == allowed_preambles[code_preamble]:
|
|
return True
|
|
return False
|
|
|
|
|
|
def module_source(src: Path, directory: Path) -> Path | None:
|
|
if not directory.parts:
|
|
return src / "lib.rs"
|
|
|
|
mod_file = src / directory / "mod.rs"
|
|
if mod_file.is_file():
|
|
return mod_file
|
|
|
|
sibling = src.joinpath(*directory.parts[:-1], f"{directory.name}.rs")
|
|
return sibling if sibling.is_file() else None
|
|
|
|
|
|
def check_e2e_modules(root: Path) -> list[str]:
|
|
src = root / "crates/e2e_test/src"
|
|
errors: list[str] = []
|
|
for test_file in sorted(src.rglob("*_test.rs")):
|
|
relative = test_file.relative_to(root).as_posix()
|
|
directory = test_file.relative_to(src).parent
|
|
parent = module_source(src, directory)
|
|
if parent is None:
|
|
errors.append(f"{relative}: no canonical parent module")
|
|
continue
|
|
|
|
if not declared(parent, test_file.stem):
|
|
errors.append(f"{relative}: not declared by {parent.relative_to(root).as_posix()}")
|
|
while directory.parts:
|
|
module = directory.name
|
|
directory = directory.parent
|
|
parent = module_source(src, directory)
|
|
if parent is None:
|
|
errors.append(f"{relative}: module {module} has no canonical parent")
|
|
break
|
|
if not declared(parent, module):
|
|
errors.append(f"{relative}: module {module} not declared by {parent.relative_to(root).as_posix()}")
|
|
return errors
|
|
|
|
|
|
def check_vault_test_groups(root: Path) -> list[str]:
|
|
src = root / "crates/e2e_test/src"
|
|
vault_modules = {
|
|
path.relative_to(src).with_suffix("").as_posix().replace("/", "::")
|
|
for path in src.rglob("*_test.rs")
|
|
if "VaultTestEnvironment" in rust_code_only(path.read_text())
|
|
}
|
|
config = tomllib.loads((root / ".config/nextest.toml").read_text())
|
|
errors: list[str] = []
|
|
for profile in ("default", "e2e-full"):
|
|
overrides = config.get("profile", {}).get(profile, {}).get("overrides", [])
|
|
filters = "\n".join(
|
|
override.get("filter", "")
|
|
for override in overrides
|
|
if override.get("test-group") == "e2e-vault"
|
|
)
|
|
for module in sorted(vault_modules):
|
|
if f"{module}::" not in filters:
|
|
errors.append(f".config/nextest.toml: profile.{profile} does not serialize Vault module {module}")
|
|
return errors
|
|
|
|
|
|
def check_ilm_build_budget(root: Path) -> list[str]:
|
|
workflow_path = root / ".github/workflows/ci.yml"
|
|
if not workflow_path.is_file():
|
|
return [".github/workflows/ci.yml: missing workflow"]
|
|
workflow = workflow_path.read_text()
|
|
job = re.search(
|
|
r"^ test-ilm-integration-serial:\n(?P<body>.*?)(?=^ [a-z0-9-]+:\n)",
|
|
workflow,
|
|
re.MULTILINE | re.DOTALL,
|
|
)
|
|
if job is None:
|
|
return [".github/workflows/ci.yml: missing test-ilm-integration-serial job"]
|
|
|
|
step = re.search(
|
|
r"^ - name: Run ignored ILM integration tests serially\n(?P<body>.*?)(?=^ - name:|\Z)",
|
|
job.group("body"),
|
|
re.MULTILINE | re.DOTALL,
|
|
)
|
|
if step is None:
|
|
return [".github/workflows/ci.yml: missing ignored ILM integration step"]
|
|
|
|
budget = (
|
|
"CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || "
|
|
"github.event_name == 'workflow_dispatch') && '3' || '2' }}"
|
|
)
|
|
if budget not in step.group("body"):
|
|
return [".github/workflows/ci.yml: ILM integration must keep the measured 3/2 Cargo build budget"]
|
|
return []
|
|
|
|
|
|
def check_fuzz_targets(root: Path) -> list[str]:
|
|
manifest = tomllib.loads((root / "fuzz/Cargo.toml").read_text())
|
|
expected = {item["name"] for item in manifest.get("bin", []) if "name" in item}
|
|
errors: list[str] = []
|
|
if not expected:
|
|
return ["fuzz/Cargo.toml: no [[bin]] fuzz targets found"]
|
|
|
|
runner = (root / "scripts/fuzz/run.sh").read_text()
|
|
match = re.search(r'^targets="([^"]+)"', runner, re.MULTILINE)
|
|
runner_targets = set(match.group(1).split()) if match else set()
|
|
if runner_targets != expected:
|
|
errors.append(f"scripts/fuzz/run.sh targets {sorted(runner_targets)} != manifest {sorted(expected)}")
|
|
|
|
workflow = (root / ".github/workflows/fuzz.yml").read_text()
|
|
matrices = [words(value) for value in re.findall(r"^\s*target:\s*\[([^]]+)]", workflow, re.MULTILINE)]
|
|
if len(matrices) != 2:
|
|
errors.append(f".github/workflows/fuzz.yml: expected smoke and nightly target matrices, found {len(matrices)}")
|
|
for index, matrix in enumerate(matrices, start=1):
|
|
if matrix != expected:
|
|
errors.append(f".github/workflows/fuzz.yml matrix {index} {sorted(matrix)} != manifest {sorted(expected)}")
|
|
|
|
runtime_targets = re.findall(r"^\s*FUZZ_TARGET:\s*(\S.*?)\s*$", workflow, re.MULTILINE)
|
|
if runtime_targets != ["${{ matrix.target }}", "${{ matrix.target }}"]:
|
|
errors.append(".github/workflows/fuzz.yml: smoke and nightly jobs must pass matrix.target to FUZZ_TARGET")
|
|
|
|
dependency_paths = {
|
|
f"{path.removeprefix('../')}/**"
|
|
for dependency in manifest.get("dependencies", {}).values()
|
|
if isinstance(dependency, dict)
|
|
and isinstance(path := dependency.get("path"), str)
|
|
and path.startswith("../crates/")
|
|
}
|
|
missing_paths = sorted(path for path in dependency_paths if f'"{path}"' not in workflow)
|
|
if missing_paths:
|
|
errors.append(f".github/workflows/fuzz.yml missing direct dependency paths: {', '.join(missing_paths)}")
|
|
|
|
staged_matches = re.findall(r"^\s*for target in ([^;]+); do", workflow, re.MULTILINE)
|
|
staged = set(staged_matches[0].split()) if staged_matches else set()
|
|
if staged != expected:
|
|
errors.append(f".github/workflows/fuzz.yml staged binaries {sorted(staged)} != manifest {sorted(expected)}")
|
|
|
|
return errors
|
|
|
|
|
|
def check_runner_selection(root: Path) -> list[str]:
|
|
runner = (root / "scripts/run_e2e_tests.sh").read_text()
|
|
errors: list[str] = []
|
|
if "--include-ignored" not in runner:
|
|
errors.append("scripts/run_e2e_tests.sh: runner must include default and ignored tests")
|
|
if "--test-threads=1" not in runner:
|
|
errors.append("scripts/run_e2e_tests.sh: runner must serialize fixed-port protocol tests")
|
|
if re.search(r"(?<!include-)--ignored\b", runner):
|
|
errors.append("scripts/run_e2e_tests.sh: bare --ignored silently excludes default tests")
|
|
if "--exact" in runner:
|
|
errors.append("scripts/run_e2e_tests.sh: --test is documented as a pattern and must not force exact matching")
|
|
if 'eval "$test_cmd"' in runner:
|
|
errors.append("scripts/run_e2e_tests.sh: command construction must not use eval")
|
|
start = re.search(r"start_rustfs\(\) \{(?P<body>.*?)\n\}\n\n# Function to run tests", runner, re.DOTALL)
|
|
start_body = start.group("body") if start else ""
|
|
if '"http://localhost:9000/health/ready"' not in start_body or not re.search(
|
|
r"curl [^\n]*(?:--fail|-f(?:\s|$))", start_body
|
|
):
|
|
errors.append("scripts/run_e2e_tests.sh: startup must require the ready endpoint to return HTTP success")
|
|
if start_body.count("return 0") != 1 or "nc -z" in start_body:
|
|
errors.append("scripts/run_e2e_tests.sh: startup readiness must not fall back to process or port liveness")
|
|
failed_start = re.search(r"if ! start_rustfs; then(?P<body>.*?)\n\s*fi", runner, re.DOTALL)
|
|
failed_start_commands = (
|
|
[line.strip() for line in failed_start.group("body").splitlines() if line.strip()] if failed_start else []
|
|
)
|
|
if failed_start_commands != ['print_error "Failed to start RustFS properly"', "exit 1"]:
|
|
errors.append("scripts/run_e2e_tests.sh: failed startup must not continue into tests")
|
|
return errors
|
|
|
|
|
|
def check_s3_tests_runner(root: Path) -> list[str]:
|
|
runner = (root / "scripts/s3-tests/run.sh").read_text()
|
|
errors: list[str] = []
|
|
if "--showlocals" in runner:
|
|
errors.append("scripts/s3-tests/run.sh: pytest failure diagnostics must not dump local values")
|
|
readiness = re.search(
|
|
r"test_s3_api_ready\(\) \{(?P<body>.*?)\n\}\n\n# First, wait",
|
|
runner,
|
|
re.DOTALL,
|
|
)
|
|
readiness_body = readiness.group("body") if readiness else ""
|
|
if (
|
|
'"http://${S3_HOST}:${S3_PORT}/health/ready"' not in readiness_body
|
|
or re.search(r"/health(?=[\"'\s])", readiness_body)
|
|
or '[ "${READY_CODE}" != "200" ]' not in readiness_body
|
|
):
|
|
errors.append("scripts/s3-tests/run.sh: startup must require the ready endpoint to return HTTP 200")
|
|
signed_probe = re.search(
|
|
r"if command -v awscurl\b[^\n]*; then(?P<body>.*?)\n\s*fi\s*"
|
|
r"(?:#[^\n]*\n\s*)*return 0\s*$",
|
|
readiness_body,
|
|
re.DOTALL,
|
|
)
|
|
signed_body = signed_probe.group("body") if signed_probe else ""
|
|
readiness_code = "\n".join(line.split("#", 1)[0] for line in readiness_body.splitlines())
|
|
signed_code = "\n".join(line.split("#", 1)[0] for line in signed_body.splitlines())
|
|
signed_commands = [
|
|
command
|
|
for line in signed_body.splitlines()
|
|
if (command := line.split("#", 1)[0].strip())
|
|
]
|
|
signed_success = re.search(
|
|
r'if echo "\$\{RESPONSE\}" \| grep -q "<ListAllMyBucketsResult"; then\s*'
|
|
r"(?:#[^\n]*\n\s*)*return 0\s*\n\s*fi",
|
|
signed_body,
|
|
)
|
|
response_capture = re.search(
|
|
r"^\s*RESPONSE=\$\((?P<command>.*?)\)\s*$",
|
|
signed_code,
|
|
re.DOTALL | re.MULTILINE,
|
|
)
|
|
response_command = response_capture.group("command") if response_capture else ""
|
|
response_command = response_command.replace("\\\n", " ").strip()
|
|
response_operators = re.search(
|
|
r";|\|\||&&|(?<![>|])\|(?!\|)|(?<![>&])&(?![>&0-9])|\$\(|[<>]\(|`",
|
|
response_command,
|
|
)
|
|
if (
|
|
len(re.findall(r"\breturn\s+0\b", readiness_code)) != 2
|
|
or len(re.findall(r"\breturn\s+0\b", signed_code)) != 1
|
|
or len(re.findall(r"(?<![A-Za-z0-9_])RESPONSE=", signed_code)) != 1
|
|
or not response_command.startswith("awscurl ")
|
|
or "\n" in response_command
|
|
or response_operators
|
|
or not signed_success
|
|
or not signed_commands
|
|
or signed_commands[-1] != "return 1"
|
|
):
|
|
errors.append("scripts/s3-tests/run.sh: readiness must bind success to the signed probe")
|
|
return errors
|
|
|
|
|
|
def check_workflow_readiness(root: Path) -> list[str]:
|
|
errors: list[str] = []
|
|
for relative in (".github/workflows/e2e-s3tests.yml", ".github/workflows/mint.yml"):
|
|
path = root / relative
|
|
try:
|
|
lines = path.read_text().splitlines()
|
|
except FileNotFoundError:
|
|
errors.append(f"{relative}: missing workflow")
|
|
continue
|
|
start = next(
|
|
(index for index, line in enumerate(lines) if line.strip() == "- name: Wait for RustFS ready"),
|
|
None,
|
|
)
|
|
if start is None:
|
|
errors.append(f"{relative}: missing RustFS readiness step")
|
|
continue
|
|
indent = len(lines[start]) - len(lines[start].lstrip())
|
|
end = next(
|
|
(
|
|
index
|
|
for index in range(start + 1, len(lines))
|
|
if lines[index].strip().startswith("- name:")
|
|
and len(lines[index]) - len(lines[index].lstrip()) <= indent
|
|
),
|
|
len(lines),
|
|
)
|
|
readiness_step = "\n".join(lines[start:end])
|
|
ready_branch = re.search(
|
|
r"if curl [^\n]*/health/ready[^\n]*; then(?P<body>.*?)\n\s*fi",
|
|
readiness_step,
|
|
re.DOTALL,
|
|
)
|
|
ready_body = ready_branch.group("body") if ready_branch else ""
|
|
ready_condition = ready_branch.group(0).splitlines()[0].rsplit("; then", 1)[0] if ready_branch else ""
|
|
step_commands = [
|
|
command
|
|
for line in readiness_step.splitlines()
|
|
if (command := line.split("#", 1)[0].strip())
|
|
]
|
|
step_code = "\n".join(step_commands)
|
|
ready_code = "\n".join(line.split("#", 1)[0] for line in ready_body.splitlines())
|
|
if (
|
|
"/health/ready" not in readiness_step
|
|
or re.search(r"/health(?=[\"'\s])", readiness_step)
|
|
or not re.search(r"curl [^\n]*(?:-sf|-fs|--fail)", readiness_step)
|
|
or not ready_branch
|
|
or re.search(r"\|\||&&|(?<![>|])\|(?!\|)|;|(?<![>&])&(?![>&])", ready_condition)
|
|
or len(re.findall(r"\bexit\s+0\b", step_code)) != 1
|
|
or not re.search(r"\bexit\s+0\b", ready_code)
|
|
or not step_commands
|
|
or step_commands[-1] != "exit 1"
|
|
):
|
|
errors.append(f"{relative}: RustFS readiness step must fail closed on /health/ready")
|
|
return errors
|
|
|
|
|
|
def profile_selection_entries(root: Path, profile: str) -> tuple[Path, list[str], dict[str, str]]:
|
|
if not re.fullmatch(r"e2e-[a-z0-9-]+", profile):
|
|
raise ValueError(f"invalid e2e profile name: {profile}")
|
|
path = root / f".config/{profile}-selection.txt"
|
|
lines = [line for line in path.read_text().splitlines() if line.strip()]
|
|
values = dict(line.split("=", 1) for line in lines if "=" in line)
|
|
if len(values) != len(lines) or any(not re.fullmatch(r"sha256(?:-[a-z0-9]+)?", key) for key in values):
|
|
raise ValueError(f"{path.relative_to(root).as_posix()}: invalid sha256 entry")
|
|
return path, lines, values
|
|
|
|
|
|
def profile_selection(root: Path, profile: str) -> str:
|
|
path, _, values = profile_selection_entries(root, profile)
|
|
key = f"sha256-{sys.platform}"
|
|
digest = values.get(key, values.get("sha256", ""))
|
|
if not re.fullmatch(r"[0-9a-f]{64}", digest):
|
|
raise ValueError(f"{path.relative_to(root).as_posix()}: missing sha256 for {sys.platform}")
|
|
return digest
|
|
|
|
|
|
def profile_listing_digest(listing: Path) -> tuple[int, str]:
|
|
data = json.loads(listing.read_text())
|
|
selected = sorted(
|
|
f"{suite_id}::{test_name}"
|
|
for suite_id, suite in data["rust-suites"].items()
|
|
for test_name, testcase in suite["testcases"].items()
|
|
if testcase.get("filter-match", {}).get("status") == "matches"
|
|
)
|
|
return len(selected), hashlib.sha256(("\n".join(selected) + "\n").encode()).hexdigest()
|
|
|
|
|
|
def update_profile_selection(root: Path, profile: str, listing: Path, platform: str) -> tuple[int, str, str]:
|
|
if not re.fullmatch(r"[a-z0-9]+", platform):
|
|
raise ValueError(f"invalid platform name: {platform}")
|
|
path, lines, values = profile_selection_entries(root, profile)
|
|
key = "sha256" if "sha256" in values else f"sha256-{platform}"
|
|
if key not in values:
|
|
raise ValueError(f"{path.relative_to(root).as_posix()}: missing {key} entry")
|
|
count, digest = profile_listing_digest(listing)
|
|
path.write_text("\n".join(f"{key}={digest}" if line.startswith(f"{key}=") else line for line in lines) + "\n")
|
|
return count, digest, key
|
|
|
|
|
|
def check_profile_definitions(root: Path) -> list[str]:
|
|
config = tomllib.loads((root / ".config/nextest.toml").read_text())
|
|
profiles = {
|
|
profile
|
|
for profile in config.get("profile", {})
|
|
if profile.startswith("e2e-")
|
|
}
|
|
selection_profiles = {
|
|
path.name.removesuffix("-selection.txt") for path in (root / ".config").glob("e2e-*-selection.txt")
|
|
}
|
|
errors: list[str] = []
|
|
for profile in sorted(profiles | selection_profiles):
|
|
if profile not in profiles:
|
|
errors.append(f".config/nextest.toml: missing profile.{profile}")
|
|
if profile not in selection_profiles:
|
|
errors.append(f".config/{profile}-selection.txt: missing expected profile selection")
|
|
continue
|
|
try:
|
|
profile_selection(root, profile)
|
|
except (FileNotFoundError, ValueError) as error:
|
|
errors.append(str(error))
|
|
return errors
|
|
|
|
|
|
def yaml_block(lines: list[str], key: str, indent: int) -> list[str] | None:
|
|
try:
|
|
start = lines.index(f"{' ' * indent}{key}:") + 1
|
|
except ValueError:
|
|
return None
|
|
end = next(
|
|
(
|
|
index
|
|
for index in range(start, len(lines))
|
|
if lines[index].strip()
|
|
and not lines[index].lstrip().startswith("#")
|
|
and len(lines[index]) - len(lines[index].lstrip()) <= indent
|
|
),
|
|
len(lines),
|
|
)
|
|
return lines[start:end]
|
|
|
|
|
|
def workflow_step_block(
|
|
job_lines: list[str], value: str, key: str = "uses", indent: int = 6
|
|
) -> tuple[int, list[str]] | None:
|
|
uses_index = next(
|
|
(
|
|
index
|
|
for index, line in enumerate(job_lines)
|
|
if (
|
|
line.split("#", 1)[0].strip() == f"- {key}: {value}"
|
|
and len(line) - len(line.lstrip()) == indent
|
|
)
|
|
or (
|
|
line.split("#", 1)[0].strip() == f"{key}: {value}"
|
|
and len(line) - len(line.lstrip()) == indent + 2
|
|
)
|
|
),
|
|
None,
|
|
)
|
|
if uses_index is None:
|
|
return None
|
|
start = next(
|
|
(
|
|
index
|
|
for index in range(uses_index, -1, -1)
|
|
if job_lines[index].lstrip().startswith("- ")
|
|
),
|
|
uses_index,
|
|
)
|
|
indent = len(job_lines[start]) - len(job_lines[start].lstrip())
|
|
end = next(
|
|
(
|
|
index
|
|
for index in range(start + 1, len(job_lines))
|
|
if len(job_lines[index]) - len(job_lines[index].lstrip()) == indent
|
|
and job_lines[index].lstrip().startswith("- ")
|
|
),
|
|
len(job_lines),
|
|
)
|
|
return start, job_lines[start:end]
|
|
|
|
|
|
def yaml_scalar_continues(lines: list[str], index: int, indent: int) -> bool:
|
|
following = next(
|
|
(line for line in lines[index + 1:] if line.strip() and not line.lstrip().startswith("#")), None
|
|
)
|
|
return following is not None and len(following) - len(following.lstrip()) > indent
|
|
|
|
|
|
def check_quick_checks(root: Path) -> list[str]:
|
|
errors: list[str] = []
|
|
bypass_key = r'''(?:if|continue-on-error|needs|"if"|"continue-on-error"|"needs"|'if'|'continue-on-error'|'needs')\s*:'''
|
|
for name in ("ci.yml",):
|
|
relative = f".github/workflows/{name}"
|
|
path = root / relative
|
|
job = yaml_block(path.read_text().splitlines(), "quick-checks", 2) if path.is_file() else None
|
|
if job is None:
|
|
errors.append(f"{relative}: missing Quick Checks job")
|
|
continue
|
|
conditions = [index for index, line in enumerate(job) if re.match(rf"^ {bypass_key}", line)]
|
|
expected = ["if: github.event_name != 'pull_request' || github.event.action != 'closed'"] if name == "ci.yml" else []
|
|
if [job[index].strip() for index in conditions] != expected or any(
|
|
yaml_scalar_continues(job, index, 4) for index in conditions
|
|
):
|
|
errors.append(f"{relative}: Quick Checks job must not add dependencies, bypass failures, or change its event condition")
|
|
checkout = workflow_step_block(job, "actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0")
|
|
action = workflow_step_block(job, "./.github/actions/quick-checks")
|
|
if checkout is None or action is None:
|
|
errors.append(f"{relative}: Quick Checks requires checkout and the shared quick-checks action")
|
|
continue
|
|
if checkout[0] >= action[0]:
|
|
errors.append(f"{relative}: checkout must run before shared Quick Checks")
|
|
if " persist-credentials: false" not in checkout[1]:
|
|
errors.append(f"{relative}: Quick Checks checkout must disable persisted credentials")
|
|
for step in (checkout, action):
|
|
if any(re.match(rf"^\s+(?:- )?{bypass_key}", line) for line in step[1]):
|
|
errors.append(f"{relative}: Quick Checks checkout and shared action must run without bypasses")
|
|
|
|
relative = ".github/actions/quick-checks/action.yml"
|
|
path = root / relative
|
|
runs = yaml_block(path.read_text().splitlines(), "runs", 0) if path.is_file() else None
|
|
if runs is None or " using: composite" not in runs:
|
|
errors.append(f"{relative}: missing composite action")
|
|
return errors
|
|
steps = yaml_block(runs, "steps", 2) or []
|
|
for command in ("shellcheck --version && actionlint", "./scripts/check_error_other_format_ratchet.sh", "make script-tests"):
|
|
step = workflow_step_block(steps, command, key="run", indent=4)
|
|
if step is None:
|
|
errors.append(f"{relative}: missing direct execution of {command}")
|
|
continue
|
|
if " shell: bash" not in step[1] or any(
|
|
re.match(rf"^\s+(?:- )?{bypass_key}", line) for line in step[1]
|
|
):
|
|
errors.append(f"{relative}: {command} must use bash without a condition or continue-on-error")
|
|
run_index = next(
|
|
index for index, line in enumerate(step[1])
|
|
if line.split("#", 1)[0].rstrip() in (f" run: {command}", f" - run: {command}")
|
|
)
|
|
if yaml_scalar_continues(step[1], run_index, 6):
|
|
errors.append(f"{relative}: {command} must remain a single-line run scalar")
|
|
return errors
|
|
|
|
|
|
def alert_step_errors(
|
|
job_lines: list[str],
|
|
expected_action_if: str | None,
|
|
required_permissions: tuple[str, ...],
|
|
required_action_tokens: tuple[str, ...],
|
|
) -> list[str]:
|
|
checkout = workflow_step_block(job_lines, "actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0")
|
|
action = workflow_step_block(job_lines, "./.github/actions/schedule-failure-issue")
|
|
errors: list[str] = []
|
|
permissions = yaml_block(job_lines, "permissions", 4)
|
|
permission_text = "\n".join(line.split("#", 1)[0] for line in permissions or [])
|
|
missing_permissions = [token for token in required_permissions if token not in permission_text]
|
|
if missing_permissions:
|
|
errors.append("alert job permissions missing " + ", ".join(missing_permissions))
|
|
if checkout is None:
|
|
errors.append("checkout step is missing")
|
|
if action is None:
|
|
errors.append("local alert action step is missing")
|
|
if checkout is None or action is None:
|
|
return errors
|
|
|
|
if checkout[0] >= action[0]:
|
|
errors.append("checkout must run before the local alert action")
|
|
checkout_ifs = [line.strip() for line in checkout[1] if line.strip().startswith("if:")]
|
|
if checkout_ifs:
|
|
errors.append("checkout step must not be conditional")
|
|
action_ifs = [line.strip() for line in action[1] if line.strip().startswith("if:")]
|
|
expected_ifs = [] if expected_action_if is None else [expected_action_if]
|
|
if action_ifs != expected_ifs:
|
|
errors.append("alert action has an invalid step condition")
|
|
action_text = "\n".join(line.split("#", 1)[0] for line in action[1])
|
|
missing_action_tokens = [token for token in required_action_tokens if token not in action_text]
|
|
if missing_action_tokens:
|
|
errors.append("alert action inputs missing " + ", ".join(missing_action_tokens))
|
|
return errors
|
|
|
|
|
|
def schedule_utc_slots(hour: int, minute: int, timezone_name: str | None) -> set[tuple[int, int]]:
|
|
if timezone_name is None:
|
|
return {(hour, minute)}
|
|
zone = ZoneInfo(timezone_name)
|
|
return {
|
|
(utc.hour, utc.minute)
|
|
for year in (2025, 2026)
|
|
for month in range(1, 13)
|
|
for utc in [datetime(year, month, 1, hour, minute, tzinfo=zone).astimezone(timezone.utc)]
|
|
}
|
|
|
|
|
|
def check_scheduled_alerts(root: Path) -> list[str]:
|
|
errors: list[str] = []
|
|
schedule_slots: dict[tuple[int, int], list[str]] = {}
|
|
for relative in SCHEDULED_ALERT_WORKFLOWS:
|
|
path = root / relative
|
|
try:
|
|
lines = path.read_text().splitlines()
|
|
except FileNotFoundError:
|
|
errors.append(f"{relative}: missing scheduled validation workflow")
|
|
continue
|
|
|
|
on_block = yaml_block(lines, "on", 0)
|
|
schedule_block = yaml_block(on_block or [], "schedule", 2)
|
|
schedule_lines = schedule_block or []
|
|
cron_indices = [index for index, line in enumerate(schedule_lines) if re.match(r"^\s*-\s+cron:", line)]
|
|
if not cron_indices:
|
|
errors.append(f"{relative}: missing simple numeric schedule")
|
|
else:
|
|
for position, cron_index in enumerate(cron_indices):
|
|
cron_line = schedule_lines[cron_index]
|
|
schedule = re.match(r"^\s*-\s+cron:\s*[\"']?(\d+)\s+(\d+)\s+", cron_line)
|
|
if not schedule:
|
|
errors.append(f"{relative}: missing simple numeric schedule")
|
|
continue
|
|
minute, hour = map(int, schedule.groups())
|
|
if minute == 0:
|
|
errors.append(f"{relative}: scheduled validation must avoid minute zero")
|
|
entry_end = cron_indices[position + 1] if position + 1 < len(cron_indices) else len(schedule_lines)
|
|
entry = "\n".join(schedule_lines[cron_index + 1 : entry_end])
|
|
timezone_match = re.search(r"^\s*timezone:\s*[\"']?([^\"'\s]+)", entry, re.MULTILINE)
|
|
timezone_name = timezone_match.group(1) if timezone_match else None
|
|
try:
|
|
utc_slots = schedule_utc_slots(hour, minute, timezone_name)
|
|
except ZoneInfoNotFoundError:
|
|
errors.append(f"{relative}: unknown schedule timezone {timezone_name}")
|
|
continue
|
|
for slot in utc_slots:
|
|
schedule_slots.setdefault(slot, []).append(relative)
|
|
|
|
job_lines = yaml_block(lines, "alert-on-failure", 2)
|
|
if job_lines is None:
|
|
errors.append(f"{relative}: missing alert-on-failure job")
|
|
continue
|
|
job = "\n".join(line.split("#", 1)[0] for line in job_lines)
|
|
required = (
|
|
"always()",
|
|
"github.event_name == 'schedule'",
|
|
"contains(needs.*.result, 'failure')",
|
|
"issues: write",
|
|
"uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0",
|
|
"uses: ./.github/actions/schedule-failure-issue",
|
|
"github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}",
|
|
)
|
|
missing = [token for token in required if token not in job]
|
|
if missing:
|
|
errors.append(f"{relative}: alert-on-failure missing {', '.join(missing)}")
|
|
else:
|
|
errors.extend(
|
|
f"{relative}: {error}"
|
|
for error in alert_step_errors(job_lines, None, ("issues: write",), ("github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}",))
|
|
)
|
|
|
|
for (hour, minute), workflows in schedule_slots.items():
|
|
if len(workflows) > 1:
|
|
errors.append(
|
|
f"scheduled validations share {hour:02d}:{minute:02d} UTC: {', '.join(workflows)}"
|
|
)
|
|
|
|
watchdog_path = root / ".github/workflows/scheduled-validation-watchdog.yml"
|
|
try:
|
|
watchdog_lines = watchdog_path.read_text().splitlines()
|
|
except FileNotFoundError:
|
|
errors.append(".github/workflows/scheduled-validation-watchdog.yml: missing completion watchdog")
|
|
return errors
|
|
watchdog_on = yaml_block(watchdog_lines, "on", 0)
|
|
watchdog_run = yaml_block(watchdog_on or [], "workflow_run", 2)
|
|
watchdog_workflows = yaml_block(watchdog_run or [], "workflows", 4)
|
|
if watchdog_workflows is None:
|
|
errors.append(".github/workflows/scheduled-validation-watchdog.yml: missing workflow_run workflows")
|
|
return errors
|
|
watchdog_sources = "\n".join(line.split("#", 1)[0] for line in watchdog_workflows)
|
|
for relative in SCHEDULED_ALERT_WORKFLOWS:
|
|
path = root / relative
|
|
if not path.is_file():
|
|
continue
|
|
source = path.read_text()
|
|
match = re.search(r"^name:\s*[\"']?([^\"'\n]+)", source, re.MULTILINE)
|
|
if not match:
|
|
errors.append(f"{relative}: missing workflow name")
|
|
elif f'- "{match.group(1).strip()}"' not in watchdog_sources:
|
|
errors.append(f"{relative}: missing from scheduled completion watchdog")
|
|
watchdog_job_lines = yaml_block(watchdog_lines, "alert-on-incomplete-run", 2)
|
|
if watchdog_job_lines is None:
|
|
errors.append(".github/workflows/scheduled-validation-watchdog.yml: missing alert-on-incomplete-run job")
|
|
return errors
|
|
watchdog_job = "\n".join(line.split("#", 1)[0] for line in watchdog_job_lines)
|
|
required = (
|
|
"github.event.workflow_run.event == 'schedule'",
|
|
"github.event.workflow_run.conclusion != 'success'",
|
|
"github.event.workflow_run.conclusion != 'failure'",
|
|
"actions: read",
|
|
"issues: write",
|
|
"uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0",
|
|
"uses: ./.github/actions/schedule-failure-issue",
|
|
"github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}",
|
|
"workflow-name: ${{ github.event.workflow_run.name }}",
|
|
"source-run-id: ${{ github.event.workflow_run.id }}",
|
|
"source-run-attempt: ${{ github.event.workflow_run.run_attempt }}",
|
|
"source-event: ${{ github.event.workflow_run.event }}",
|
|
"source-ref-name: ${{ github.event.workflow_run.head_branch }}",
|
|
"source-sha: ${{ github.event.workflow_run.head_sha }}",
|
|
)
|
|
missing = [token for token in required if token not in watchdog_job]
|
|
if missing:
|
|
errors.append(
|
|
".github/workflows/scheduled-validation-watchdog.yml: missing " + ", ".join(missing)
|
|
)
|
|
else:
|
|
errors.extend(
|
|
".github/workflows/scheduled-validation-watchdog.yml: " + error
|
|
for error in alert_step_errors(
|
|
watchdog_job_lines,
|
|
None,
|
|
("actions: read", "issues: write"),
|
|
(
|
|
"github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}",
|
|
"workflow-name: ${{ github.event.workflow_run.name }}",
|
|
"source-run-id: ${{ github.event.workflow_run.id }}",
|
|
"source-run-attempt: ${{ github.event.workflow_run.run_attempt }}",
|
|
"source-event: ${{ github.event.workflow_run.event }}",
|
|
"source-ref-name: ${{ github.event.workflow_run.head_branch }}",
|
|
"source-sha: ${{ github.event.workflow_run.head_sha }}",
|
|
),
|
|
)
|
|
)
|
|
|
|
freshness_path = root / ".github/workflows/scheduled-validation-freshness.yml"
|
|
try:
|
|
freshness_lines = freshness_path.read_text().splitlines()
|
|
except FileNotFoundError:
|
|
errors.append(".github/workflows/scheduled-validation-freshness.yml: missing freshness check")
|
|
return errors
|
|
freshness_job_lines = yaml_block(freshness_lines, "check-freshness", 2)
|
|
if freshness_job_lines is None:
|
|
errors.append(".github/workflows/scheduled-validation-freshness.yml: missing check-freshness job")
|
|
return errors
|
|
freshness_job = "\n".join(line.split("#", 1)[0] for line in freshness_job_lines)
|
|
required = (
|
|
"python3 scripts/check_scheduled_validation_freshness.py",
|
|
"actions: read",
|
|
"issues: write",
|
|
"if: failure()",
|
|
"uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0",
|
|
"uses: ./.github/actions/schedule-failure-issue",
|
|
"github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}",
|
|
"details-file: ${{ runner.temp }}/scheduled-validation-freshness.md",
|
|
)
|
|
missing = [token for token in required if token not in freshness_job]
|
|
if missing:
|
|
errors.append(
|
|
".github/workflows/scheduled-validation-freshness.yml: missing " + ", ".join(missing)
|
|
)
|
|
else:
|
|
errors.extend(
|
|
".github/workflows/scheduled-validation-freshness.yml: " + error
|
|
for error in alert_step_errors(
|
|
freshness_job_lines,
|
|
"if: failure()",
|
|
("actions: read", "issues: write"),
|
|
(
|
|
"github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}",
|
|
"details-file: ${{ runner.temp }}/scheduled-validation-freshness.md",
|
|
),
|
|
)
|
|
)
|
|
if not (root / "scripts/check_scheduled_validation_freshness.py").is_file():
|
|
errors.append("scripts/check_scheduled_validation_freshness.py: missing freshness checker")
|
|
return errors
|
|
|
|
|
|
def check_profile_listing(root: Path, profile: str, listing: Path) -> list[str]:
|
|
try:
|
|
expected_digest = profile_selection(root, profile)
|
|
count, digest = profile_listing_digest(listing)
|
|
except (FileNotFoundError, KeyError, TypeError, ValueError, json.JSONDecodeError) as error:
|
|
return [f"cannot read {profile} nextest listing: {error}"]
|
|
if digest != expected_digest:
|
|
return [
|
|
f"{profile} selection changed: count={count} sha256={digest}; "
|
|
f"expected sha256={expected_digest}"
|
|
]
|
|
print(f"{profile} selection OK: {count} tests, sha256={digest}")
|
|
return []
|
|
|
|
|
|
def core_requirements(root: Path) -> dict:
|
|
data = json.loads((root / ".config/ecstore-required-tests.json").read_text())
|
|
if not data["tests"] or not data["fixtures"]:
|
|
raise ValueError("core test and fixture requirements must not be empty")
|
|
identities = [(test["suite"], test["name"]) for test in data["tests"]]
|
|
if len(set(identities)) != len(identities):
|
|
raise ValueError("duplicate core test requirement")
|
|
return data
|
|
|
|
|
|
def check_core_fixtures(root: Path) -> list[str]:
|
|
try:
|
|
fixtures = core_requirements(root)["fixtures"]
|
|
errors = []
|
|
for fixture in fixtures:
|
|
path = (root / fixture["path"]).resolve()
|
|
if not path.is_relative_to(root.resolve()):
|
|
raise ValueError("core fixture path escapes repository")
|
|
if not path.is_file():
|
|
errors.append(f"{fixture['path']}: required core fixture missing")
|
|
elif hashlib.sha256(path.read_bytes()).hexdigest() != fixture["sha256"]:
|
|
errors.append(f"{fixture['path']}: core fixture sha256 mismatch")
|
|
return errors
|
|
except (OSError, KeyError, TypeError, ValueError) as error:
|
|
return [f"cannot validate core fixtures: {error}"]
|
|
|
|
|
|
def check_core_listing(root: Path, listing: Path) -> list[str]:
|
|
"""Check the existing CI run's selection, not a second filtered test run."""
|
|
try:
|
|
required = core_requirements(root)["tests"]
|
|
suites = json.loads(listing.read_text())["rust-suites"]
|
|
if not isinstance(suites, dict):
|
|
raise ValueError("rust-suites must be an object")
|
|
errors = check_core_fixtures(root)
|
|
for test in required:
|
|
testcase = suites.get(test["suite"], {}).get("testcases", {}).get(test["name"], {})
|
|
if testcase.get("ignored") is not False or testcase.get("filter-match", {}).get("status") != "matches":
|
|
errors.append(f"{test['invariant']}: required test not selected: {test['suite']}::{test['name']}")
|
|
return errors
|
|
except (OSError, KeyError, TypeError, ValueError) as error:
|
|
return [f"cannot read core nextest listing: {error}"]
|
|
|
|
|
|
def evidence_integer(value: object, name: str, minimum: int, maximum: int) -> int:
|
|
require(type(value) is int and minimum <= value <= maximum, f"invalid integer {name}")
|
|
return value
|
|
|
|
|
|
def evidence_string(value: object, name: str, pattern: str | None = None) -> str:
|
|
require(isinstance(value, str) and value.strip(), f"invalid string {name}")
|
|
if pattern is not None:
|
|
require(re.fullmatch(pattern, value) is not None, f"invalid string {name}")
|
|
return value
|
|
|
|
|
|
def evidence_timestamp(value: object, name: str) -> datetime:
|
|
text = evidence_string(value, name)
|
|
try:
|
|
parsed = datetime.fromisoformat(text.replace("Z", "+00:00"))
|
|
except ValueError as error:
|
|
raise ValueError(f"invalid timestamp {name}") from error
|
|
require(parsed.tzinfo is not None, f"{name} must include timezone")
|
|
return parsed
|
|
|
|
|
|
def evidence_string_list(value: object, name: str) -> list[str]:
|
|
require(
|
|
isinstance(value, list) and value and all(isinstance(item, str) and item.strip() for item in value),
|
|
f"{name} must be a non-empty string list",
|
|
)
|
|
return value
|
|
|
|
|
|
def evidence_exact_strings(value: object, expected: tuple[str, ...], name: str) -> list[str]:
|
|
strings = evidence_string_list(value, name)
|
|
observed = set(strings)
|
|
require(len(observed) == len(strings), f"duplicate {name}")
|
|
missing = sorted(set(expected) - observed)
|
|
require(not missing, f"missing {name}: {', '.join(missing)}")
|
|
unknown = sorted(observed - set(expected))
|
|
require(not unknown, f"unknown {name}: {', '.join(unknown)}")
|
|
return strings
|
|
|
|
|
|
def scanner_heal_registry_schema(registry: dict[str, object]) -> int:
|
|
return evidence_integer(registry.get("schema"), "registry schema", 1, SCANNER_HEAL_REGISTRY_SCHEMA_MAX)
|
|
|
|
|
|
def scanner_heal_oracle_names(root: Path) -> tuple[str, ...]:
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
scanner_heal_registry_schema(registry)
|
|
cases = registry.get("cases")
|
|
require(isinstance(cases, dict) and cases, "invalid scanner/heal registry")
|
|
names = set()
|
|
for case_id, requirement in cases.items():
|
|
require(isinstance(case_id, str) and case_id, "invalid scanner/heal case identity")
|
|
lane = requirement.get("lane")
|
|
require(isinstance(lane, str) and re.fullmatch(r"[a-z0-9-]+", lane) is not None,
|
|
f"invalid nextest profile lane for {case_id}")
|
|
oracle = requirement.get("oracle")
|
|
require(isinstance(oracle, str) and oracle.endswith(".json"), f"invalid oracle for {case_id}")
|
|
path = Path(oracle)
|
|
require(not path.is_absolute() and ".." not in path.parts, f"oracle path escapes run directory for {case_id}")
|
|
require(requirement.get("evidence") in ("process-restart", "process-crash-restart"),
|
|
f"invalid evidence for {case_id}")
|
|
require(type(requirement.get("unclean_shutdown_marker")) is bool,
|
|
f"invalid unclean-shutdown marker expectation for {case_id}")
|
|
if "erasure" in requirement:
|
|
erasure = requirement["erasure"]
|
|
require(isinstance(erasure, dict), f"invalid erasure expectation for {case_id}")
|
|
data_blocks = evidence_integer(erasure.get("data_blocks"), f"{case_id} data_blocks", 1, 16)
|
|
parity_blocks = evidence_integer(erasure.get("parity_blocks"), f"{case_id} parity_blocks", 1, 16)
|
|
require(data_blocks >= parity_blocks, f"invalid erasure geometry for {case_id}")
|
|
expected_set_drives = requirement.get(
|
|
"erasure_set_drive_count",
|
|
requirement["topology"]["nodes"] * requirement["topology"]["drives_per_node"],
|
|
)
|
|
require(data_blocks + parity_blocks == expected_set_drives,
|
|
f"erasure geometry differs from topology for {case_id}")
|
|
if "erasure_set_drive_count" in requirement:
|
|
erasure_set_drive_count = evidence_integer(requirement.get("erasure_set_drive_count"),
|
|
f"{case_id} erasure_set_drive_count", 1, 64)
|
|
total_drives = requirement["topology"]["nodes"] * requirement["topology"]["drives_per_node"]
|
|
require(total_drives % erasure_set_drive_count == 0,
|
|
f"erasure set drive count does not divide topology for {case_id}")
|
|
if "sets" in requirement:
|
|
require(evidence_integer(requirement.get("sets"), f"{case_id} sets", 1, 1024)
|
|
== total_drives // erasure_set_drive_count, f"set count differs from topology for {case_id}")
|
|
if "pools" in requirement:
|
|
evidence_integer(requirement.get("pools"), f"{case_id} pools", 1, 1024)
|
|
names.add(oracle)
|
|
return tuple(sorted(names))
|
|
|
|
|
|
def scanner_heal_release_requirements(registry: dict[str, object]) -> tuple[dict[str, dict[str, object]], bool, list[str]]:
|
|
schema = scanner_heal_registry_schema(registry)
|
|
if schema == 1:
|
|
pending = registry.get("release_pending")
|
|
require(isinstance(pending, dict), "invalid scanner/heal release requirements")
|
|
requirements = {}
|
|
for gate, reason in pending.items():
|
|
require(isinstance(gate, str) and re.fullmatch(r"[A-Z][A-Z0-9-]*", gate) is not None,
|
|
"invalid scanner/heal release gate")
|
|
require(isinstance(reason, str) and reason.strip(), f"missing release requirement for {gate}")
|
|
requirements[gate] = {"gate": gate, "status": "pending", "lane": "schema-1-pending",
|
|
"description": reason, "requires": [reason]}
|
|
return requirements, False, ["schema-1-pending"] if requirements else []
|
|
|
|
lanes = registry.get("release_lanes")
|
|
cases = registry.get("cases")
|
|
require(isinstance(cases, dict) and cases, "invalid scanner/heal registry")
|
|
require(isinstance(lanes, dict) and lanes, "invalid scanner/heal release lanes")
|
|
lane_statuses = {}
|
|
lane_gates = {}
|
|
for lane_id, lane in lanes.items():
|
|
require(isinstance(lane_id, str) and re.fullmatch(r"[a-z0-9-]+", lane_id) is not None,
|
|
"invalid scanner/heal release lane")
|
|
require(isinstance(lane, dict), f"invalid release lane {lane_id}")
|
|
status = lane.get("status")
|
|
require(status in ("implemented", "pending"), f"invalid release lane status for {lane_id}")
|
|
lane_statuses[lane_id] = status
|
|
if status == "implemented":
|
|
lane_cases = lane.get("cases")
|
|
require(isinstance(lane_cases, list) and lane_cases and all(isinstance(case, str) and case for case in lane_cases),
|
|
f"implemented release lane {lane_id} has no cases")
|
|
require(all(case in cases for case in lane_cases), f"implemented release lane {lane_id} has unknown cases")
|
|
else:
|
|
gates = lane.get("gates")
|
|
require(isinstance(gates, list) and gates and all(isinstance(gate, str) and gate for gate in gates),
|
|
f"pending release lane {lane_id} has no gates")
|
|
lane_gates[lane_id] = set(gates)
|
|
|
|
raw_requirements = registry.get("release_requirements")
|
|
require(isinstance(raw_requirements, list) and raw_requirements, "invalid scanner/heal release requirements")
|
|
requirements: dict[str, dict[str, object]] = {}
|
|
for item in raw_requirements:
|
|
require(isinstance(item, dict), "invalid scanner/heal release requirement")
|
|
gate = item.get("gate")
|
|
require(isinstance(gate, str) and re.fullmatch(r"[A-Z][A-Z0-9-]*", gate) is not None,
|
|
"invalid scanner/heal release gate")
|
|
require(gate not in requirements, f"duplicate scanner/heal release gate {gate}")
|
|
status = item.get("status")
|
|
require(status == "pending", f"release gate {gate} must stay pending until real evidence is registered")
|
|
lane = item.get("lane")
|
|
require(isinstance(lane, str) and lane in lane_statuses, f"unknown release lane for {gate}")
|
|
require(lane_statuses[lane] == "pending", f"pending release gate {gate} mapped to non-pending lane {lane}")
|
|
description = item.get("description")
|
|
require(isinstance(description, str) and description.strip(), f"missing release requirement for {gate}")
|
|
requires = item.get("requires")
|
|
require(isinstance(requires, list) and requires and
|
|
all(isinstance(requirement, str) and requirement.strip() for requirement in requires),
|
|
f"missing concrete evidence requirements for {gate}")
|
|
evidence_fields = item.get("evidence_fields", [])
|
|
require(isinstance(evidence_fields, list) and
|
|
all(isinstance(field, str) and re.fullmatch(r"[a-z0-9][a-z0-9_]*", field) is not None
|
|
for field in evidence_fields),
|
|
f"invalid evidence fields for release gate {gate}")
|
|
required_fields = set(SCANNER_HEAL_RELEASE_REQUIRED_EVIDENCE_FIELDS.get(gate, ()))
|
|
missing_fields = sorted(required_fields - set(evidence_fields))
|
|
require(not missing_fields,
|
|
f"release gate {gate} missing required evidence fields: {', '.join(missing_fields)}")
|
|
requirements[gate] = item
|
|
|
|
missing = sorted(set(SCANNER_HEAL_RELEASE_REQUIRED_GATES) - set(requirements))
|
|
require(not missing, f"missing scanner/heal release requirements: {', '.join(missing)}")
|
|
for lane, gates in lane_gates.items():
|
|
unknown = sorted(gates - set(requirements))
|
|
require(not unknown, f"pending release lane {lane} has unknown gates: {', '.join(unknown)}")
|
|
mapped = {gate for gate, requirement in requirements.items() if requirement["lane"] == lane}
|
|
require(gates == mapped, f"pending release lane {lane} gates do not match release requirements")
|
|
pending_lanes = sorted(lane for lane, status in lane_statuses.items() if status == "pending")
|
|
return requirements, True, pending_lanes
|
|
|
|
|
|
def begin_scanner_heal_receipt(root: Path, directory: Path, binary: Path, test_binary: Path) -> None:
|
|
"""Record an existing build; this command never builds or runs a test."""
|
|
require(not directory.exists(), "scanner/heal run directory must be new")
|
|
require(not subprocess.check_output(["git", "status", "--porcelain", "--untracked-files=no"], cwd=root, text=True).strip(),
|
|
"commit tracked source changes before creating evidence")
|
|
builds = {}
|
|
for label, path in (("binary", binary), ("test_binary", test_binary)):
|
|
path = path.resolve(strict=True)
|
|
require(path.is_file() and os.access(path, os.X_OK), f"missing executable {label}")
|
|
builds[label] = {"path": str(path), "sha256": digest(path)}
|
|
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=root, text=True).strip()
|
|
require(re.fullmatch(r"[0-9a-f]{40}", revision), "invalid source revision")
|
|
version = subprocess.check_output([builds["binary"]["path"], "--version"], text=True, timeout=30)
|
|
embedded_revision = re.search(r"^git commit\s*:\s*([0-9a-f]{40})\s*$", version, re.MULTILINE)
|
|
embedded_status = re.search(r"^git status\s*:\s*(.*)\Z", version, re.MULTILINE | re.DOTALL)
|
|
require(embedded_revision is not None and embedded_revision[1] == revision, "server binary source revision mismatch")
|
|
require(embedded_status is not None and not embedded_status[1].strip(), "server binary was built from dirty/unknown source")
|
|
lock_blob = subprocess.check_output(["git", "hash-object", "Cargo.lock"], cwd=root, text=True).strip()
|
|
require(re.fullmatch(r"[0-9a-f]{40}", lock_blob), "invalid Cargo.lock identity")
|
|
features = os.environ.get("RUSTFS_E2E_EXPECTED_FEATURES")
|
|
require(features is not None, "set RUSTFS_E2E_EXPECTED_FEATURES to the compiled e2e crate feature set")
|
|
features = ",".join(sorted(set(filter(None, (feature.strip() for feature in features.split(","))))))
|
|
require(all(re.fullmatch(r"[a-z0-9-]+", feature) for feature in features.split(",") if feature), "invalid expected features")
|
|
directory.mkdir(parents=True)
|
|
write_json(directory / "run.json", {"schema": 1, "run_id": uuid.uuid4().hex,
|
|
"source_revision": revision,
|
|
"binary_source_revision": embedded_revision[1],
|
|
"test_build": {"source_revision": revision, "dirty": False,
|
|
"lock_blob": lock_blob, "features": features},
|
|
"started_at": datetime.now(timezone.utc).timestamp(), **builds})
|
|
|
|
|
|
def finish_scanner_heal_receipt(directory: Path, exit_code: int, root: Path = ROOT) -> None:
|
|
require(type(exit_code) is int and 0 <= exit_code <= 255, "invalid test exit code")
|
|
require(not (directory / "execution.json").exists(), "execution receipt already exists")
|
|
run = read_json(directory / "run.json")
|
|
artifacts = {}
|
|
oracle_names = scanner_heal_oracle_names(root)
|
|
for name in ("listing.json", "junit.xml"):
|
|
path = directory / name
|
|
if exit_code != 0 and not path.exists():
|
|
continue
|
|
require(path.is_file() and 0 < path.stat().st_size <= MAX_JSON_BYTES, f"missing/oversized {name}")
|
|
require(path.stat().st_mtime >= run["started_at"], f"stale {name}")
|
|
artifacts[name] = digest(path)
|
|
for name in oracle_names:
|
|
path = directory / name
|
|
if not path.exists():
|
|
continue
|
|
require(path.is_file() and 0 < path.stat().st_size <= MAX_JSON_BYTES, f"missing/oversized {name}")
|
|
require(path.stat().st_mtime >= run["started_at"], f"stale {name}")
|
|
artifacts[name] = digest(path)
|
|
if exit_code == 0:
|
|
require(any(name in artifacts for name in oracle_names), "missing scanner/heal case oracle")
|
|
write_json(directory / "execution.json", {"run_id": run["run_id"], "exit_code": exit_code,
|
|
"finished_at": datetime.now(timezone.utc).timestamp(),
|
|
"artifacts": artifacts})
|
|
|
|
|
|
def check_scanner_heal_evidence(root: Path, directory: Path, case_id: str) -> list[str]:
|
|
"""Validate one actual case, or fail the release while required lanes are pending."""
|
|
try:
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
scanner_heal_registry_schema(registry)
|
|
require(registry.get("cases"), "invalid scanner/heal registry")
|
|
selected = registry["cases"] if case_id == "release" else {case_id: registry["cases"][case_id]}
|
|
run = read_json(directory / "run.json")
|
|
execution = read_json(directory / "execution.json")
|
|
evidence_integer(run.get("schema"), "run schema", 1, 1)
|
|
require(re.fullmatch(r"[0-9a-f]{32}", run["run_id"]), "invalid run identity")
|
|
require(re.fullmatch(r"[0-9a-f]{40}", run["source_revision"]), "invalid source revision")
|
|
require(run.get("binary_source_revision") == run["source_revision"], "server source provenance missing")
|
|
expected_build = run["test_build"]
|
|
require(expected_build["source_revision"] == run["source_revision"] and expected_build["dirty"] is False,
|
|
"test source provenance missing")
|
|
require(re.fullmatch(r"[0-9a-f]{40}", expected_build["lock_blob"]), "invalid test lockfile identity")
|
|
require(isinstance(expected_build["features"], str), "missing test features")
|
|
require(execution.get("run_id") == run["run_id"], "execution belongs to another run")
|
|
require(type(execution.get("exit_code")) is int and execution["exit_code"] == 0, "test command failed or did not run")
|
|
number(run["started_at"], "started_at", 1)
|
|
number(execution["finished_at"], "finished_at", run["started_at"])
|
|
for label in ("binary", "test_binary"):
|
|
require(sha(run[label]["sha256"]) and digest(Path(run[label]["path"])) == run[label]["sha256"],
|
|
f"{label} changed or missing")
|
|
for name in ("listing.json", "junit.xml"):
|
|
path = directory / name
|
|
require(0 < path.stat().st_size <= MAX_JSON_BYTES, f"missing/oversized {name}")
|
|
require(run["started_at"] <= path.stat().st_mtime <= execution["finished_at"], f"{name} outside run window")
|
|
require(digest(path) == execution["artifacts"][name], f"{name} hash mismatch")
|
|
suites = read_json(directory / "listing.json")["rust-suites"]
|
|
xml = (directory / "junit.xml").read_bytes()
|
|
require(b"<!DOCTYPE" not in xml and b"<!ENTITY" not in xml, "JUnit entities are forbidden")
|
|
junit = ET.fromstring(xml)
|
|
cases = list(junit.iter("testcase"))
|
|
require(bool(cases), "JUnit has zero testcases")
|
|
for case in cases:
|
|
require(not any(child.tag in ("failure", "error", "skipped", "rerunFailure", "rerunError", "flakyFailure", "flakyError")
|
|
for child in case), "JUnit contains failed, skipped or retried tests")
|
|
errors = []
|
|
for name, requirement in selected.items():
|
|
suite, test = requirement["suite"], requirement["name"]
|
|
for key in ("nodes", "drives_per_node"):
|
|
evidence_integer(requirement["topology"][key], f"required {key}", 1, 16)
|
|
listing_suite = suites[suite]
|
|
require(listing_suite["binary-id"] == suite, "nextest suite binary identity mismatch")
|
|
listed_binary = Path(listing_suite["binary-path"]).resolve(strict=True)
|
|
require(listed_binary == Path(run["test_binary"]["path"]).resolve(strict=True) and
|
|
digest(listed_binary) == run["test_binary"]["sha256"], "nextest selected another test binary")
|
|
require(listing_suite["package-name"] == "e2e_test" and listing_suite["build-platform"] in ("host", "target"),
|
|
"unexpected nextest suite metadata")
|
|
listed = listing_suite.get("testcases", {}).get(test, {})
|
|
require(listed.get("ignored") is False and listed.get("filter-match", {}).get("status") == "matches",
|
|
f"required test not selected: {suite}::{test}")
|
|
matches = [case for case in cases if case.get("name") == test and case.get("classname") == suite]
|
|
require(len(matches) == 1, f"missing/duplicate JUnit case: {suite}::{test}")
|
|
started = datetime.fromisoformat(matches[0].attrib["timestamp"].replace("Z", "+00:00"))
|
|
require(started.tzinfo is not None, "JUnit timestamp must include timezone")
|
|
# quick-junit truncates timestamps to milliseconds.
|
|
require(run["started_at"] - 0.001 <= started.timestamp() <= execution["finished_at"],
|
|
"JUnit testcase executed outside this run")
|
|
path = directory / requirement["oracle"]
|
|
require(path.resolve().is_relative_to(directory.resolve()), "oracle path escapes run directory")
|
|
require(run["started_at"] <= path.stat().st_mtime <= execution["finished_at"], "oracle outside run window")
|
|
require(digest(path) == execution["artifacts"][requirement["oracle"]], "oracle hash mismatch")
|
|
oracle = read_json(path)
|
|
evidence_integer(oracle.get("schema"), "oracle schema", 1, 1)
|
|
require(oracle.get("evidence") == requirement["evidence"], f"not real {requirement['evidence']} evidence")
|
|
require(oracle.get("case") == name and oracle.get("run_id") == run["run_id"], "oracle belongs to another case/run")
|
|
require(oracle.get("source_revision") == run["source_revision"], "oracle source mismatch")
|
|
require(oracle.get("unclean_shutdown_marker") is requirement["unclean_shutdown_marker"],
|
|
"unclean-shutdown marker evidence mismatch")
|
|
built = oracle["test_build"]
|
|
for key in ("source_revision", "dirty", "lock_blob", "features"):
|
|
require(built[key] == expected_build[key], f"compiled test {key} mismatch")
|
|
require(built["dirty"] is False, "test binary was compiled from dirty source")
|
|
require(all(isinstance(built[key], str) and built[key] and built[key] != "unknown" for key in ("target", "profile")),
|
|
"missing compiled target/profile")
|
|
require(isinstance(built["rustflags_hex"], str) and re.fullmatch(r"(?:[0-9a-f]{2})*", built["rustflags_hex"]) is not None,
|
|
"invalid compiled rustflags")
|
|
for label in ("binary", "test_binary"):
|
|
require(oracle.get(f"{label}_sha256") == run[label]["sha256"], f"oracle {label} mismatch")
|
|
require(oracle.get("topology") == requirement["topology"], "oracle topology mismatch")
|
|
for key in ("nodes", "drives_per_node"):
|
|
evidence_integer(oracle["topology"][key], f"observed {key}", 1, 16)
|
|
expected_set_drives = requirement.get(
|
|
"erasure_set_drive_count",
|
|
oracle["topology"]["nodes"] * oracle["topology"]["drives_per_node"],
|
|
)
|
|
require(oracle.get("erasure_set_drive_count") == expected_set_drives,
|
|
"oracle erasure set drive count mismatch")
|
|
if "sets" in requirement:
|
|
require(oracle.get("sets") == requirement["sets"], "oracle set count mismatch")
|
|
if "pools" in requirement:
|
|
require(oracle.get("pools") == requirement["pools"], "oracle pool count mismatch")
|
|
expected_outage_target_required = requirement.get("outage_target_manifest_required", True)
|
|
require(oracle.get("outage_target_manifest_required", True) is expected_outage_target_required,
|
|
"oracle outage target-manifest contract mismatch")
|
|
outage_write = oracle.get("outage_write")
|
|
outage_write_deferred = oracle.get("outage_write_deferred_until_rejoin", False)
|
|
require(type(outage_write_deferred) is bool, "invalid outage-write deferred flag")
|
|
if not expected_outage_target_required and outage_write is not None:
|
|
require(isinstance(outage_write, dict), "invalid optional outage-write diagnostic")
|
|
require(outage_write.get("attempted") is True, "optional outage-write diagnostic was not attempted")
|
|
require(outage_write.get("required") is False, "optional outage-write diagnostic required flag mismatch")
|
|
require(outage_write.get("accepted") is True, "optional outage-write final S3 body was not accepted")
|
|
outage_write_attempts = evidence_integer(outage_write.get("attempts"), "optional outage-write attempts", 1, 1024)
|
|
outage_write_service_unavailable = evidence_integer(
|
|
outage_write.get("service_unavailable"), "optional outage-write ServiceUnavailable count", 0, 1024
|
|
)
|
|
if outage_write_deferred:
|
|
require(
|
|
outage_write_service_unavailable == outage_write_attempts,
|
|
"deferred outage write must account for every down-window attempt",
|
|
)
|
|
else:
|
|
require(outage_write_deferred is False, "outage-write deferred flag requires optional outage-write diagnostic")
|
|
if requirement.get("sets", 1) > 1 or requirement.get("pools", 1) > 1:
|
|
require(oracle.get("distributed_ec_invalidation") is True,
|
|
"oracle missing distributed EC invalidation proof")
|
|
evidence_integer(oracle.get("peer_count"), "oracle peer_count", 3, 64)
|
|
require(oracle.get("same_window_remote_proof") is True, "oracle missing same-window remote proof")
|
|
require(oracle.get("all_peers_bound_to_generation_window") is True,
|
|
"oracle missing peer generation-window binding")
|
|
expected_erasure = requirement.get("erasure")
|
|
if expected_erasure is not None:
|
|
require(isinstance(expected_erasure, dict), "invalid erasure expectation")
|
|
expected_data_blocks = evidence_integer(expected_erasure.get("data_blocks"), "expected EC data blocks", 1, 16)
|
|
expected_parity_blocks = evidence_integer(expected_erasure.get("parity_blocks"), "expected EC parity blocks", 1, 16)
|
|
require(
|
|
expected_data_blocks + expected_parity_blocks == expected_set_drives,
|
|
"expected EC geometry differs from topology",
|
|
)
|
|
else:
|
|
expected_data_blocks = None
|
|
expected_parity_blocks = None
|
|
evidence_integer(oracle.get("pid_before"), "pid_before", 1, 2**32 - 1)
|
|
evidence_integer(oracle.get("pid_after"), "pid_after", 1, 2**32 - 1)
|
|
require(oracle["pid_before"] != oracle["pid_after"], "no process restart witnessed")
|
|
objects = oracle["objects"]
|
|
require(isinstance(objects, list) and requirement["min_objects"] <= len(objects) <= requirement["max_objects"],
|
|
"incomplete/oversized object oracle")
|
|
require(len({obj["key"] for obj in objects}) == len(objects), "duplicate object identity")
|
|
outage_object_count = sum(obj["expected_physical"] is None for obj in objects)
|
|
require(outage_object_count == 1, "only the outage object may lack a pre-fault target manifest")
|
|
for obj in objects:
|
|
require(isinstance(obj["key"], str) and 0 < len(obj["key"].encode()) <= 1024, "invalid object identity")
|
|
require(obj["version_id"] is None, "this case only covers unversioned objects")
|
|
require(type(obj["expected_bytes"]) is int and obj["expected_bytes"] > 0, "missing expected bytes")
|
|
require(type(obj["actual_bytes"]) is int and obj["actual_bytes"] == obj["expected_bytes"], "S3 body length mismatch")
|
|
require(sha(obj["expected_sha256"]) and obj["actual_sha256"] == obj["expected_sha256"], "S3 body digest mismatch")
|
|
physical = obj["physical"]
|
|
if obj["expected_physical"] is not None:
|
|
require(physical == obj["expected_physical"], "target shard differs from pre-fault manifest")
|
|
for geometry in [physical] + ([obj["expected_physical"]] if obj["expected_physical"] is not None else []):
|
|
if not geometry["has_xl_meta"] and not expected_outage_target_required and obj["expected_physical"] is None:
|
|
continue
|
|
data = evidence_integer(geometry["data_blocks"], "EC data blocks", 1, 16)
|
|
parity = evidence_integer(geometry["parity_blocks"], "EC parity blocks", 1, 16)
|
|
require(data + parity == expected_set_drives,
|
|
"EC geometry differs from this case's erasure set")
|
|
if expected_data_blocks is not None:
|
|
require(data == expected_data_blocks and parity == expected_parity_blocks,
|
|
"EC data/parity geometry differs from the required case")
|
|
evidence_integer(geometry["erasure_index"], "target erasure index", 1, data + parity)
|
|
if not expected_outage_target_required and obj["expected_physical"] is None and not physical["has_xl_meta"]:
|
|
continue
|
|
require(physical["has_xl_meta"] is True and physical["version_id"] is None, "missing target metadata")
|
|
parts = physical["expected_part_numbers"]
|
|
require(isinstance(parts, list) and 0 < len(parts) <= 10000, "no physical part coverage")
|
|
require(all(type(part) is int and part > 0 for part in parts) and len(set(parts)) == len(parts),
|
|
"invalid physical part identity")
|
|
require({str(part) for part in parts} == set(physical["present_part_fingerprints"]), "target shard parts missing")
|
|
for part in physical["present_part_fingerprints"].values():
|
|
require(type(part["size"]) is int and part["size"] > 0 and sha(part["sha256"]), "invalid target shard fingerprint")
|
|
node_listings = oracle["node_listings"]
|
|
require(isinstance(node_listings, list) and len(node_listings) == requirement["topology"]["nodes"],
|
|
"missing per-node S3 listing")
|
|
require(all(keys == sorted(obj["key"] for obj in objects) for keys in node_listings),
|
|
"S3 listing differs from object oracle")
|
|
if case_id == "release":
|
|
release_requirements, _, _ = scanner_heal_release_requirements(registry)
|
|
errors.extend(
|
|
f"pending {gate}: {requirement['description']}" for gate, requirement in release_requirements.items()
|
|
)
|
|
return errors
|
|
except (OSError, KeyError, TypeError, ValueError, ET.ParseError) as error:
|
|
return [f"scanner/heal evidence rejected: {error}"]
|
|
|
|
|
|
def scanner_heal_release_status(root: Path, directory: Path) -> dict[str, object]:
|
|
"""Return a compact release decision without weakening case validation."""
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
scanner_heal_registry_schema(registry)
|
|
cases = registry.get("cases")
|
|
require(isinstance(cases, dict) and cases, "invalid scanner/heal registry")
|
|
release_requirements, release_schema_capable, pending_lanes = scanner_heal_release_requirements(registry)
|
|
|
|
verified_cases = []
|
|
rejected_cases = []
|
|
for case_id in sorted(cases):
|
|
if check_scanner_heal_evidence(root, directory, case_id):
|
|
rejected_cases.append(case_id)
|
|
else:
|
|
verified_cases.append(case_id)
|
|
|
|
return {
|
|
"schema": 1,
|
|
"decision": "blocked",
|
|
"release_approved": False,
|
|
"release_schema_capable": release_schema_capable,
|
|
"verified_cases": verified_cases,
|
|
"rejected_cases": rejected_cases,
|
|
"pending_gates": sorted(release_requirements),
|
|
"pending_lanes": pending_lanes,
|
|
}
|
|
|
|
|
|
def release_bundle_artifact_path(bundle_path: Path, raw_path: object, gate: str, field: str) -> Path:
|
|
require(isinstance(raw_path, str) and raw_path.strip(), f"{gate}.{field} missing artifact")
|
|
path = Path(raw_path)
|
|
require(not path.is_absolute() and ".." not in path.parts, f"{gate}.{field} artifact path escapes bundle directory")
|
|
resolved = (bundle_path.parent / path).resolve()
|
|
require(resolved.is_relative_to(bundle_path.parent.resolve()), f"{gate}.{field} artifact path escapes bundle directory")
|
|
require(resolved.is_file(), f"{gate}.{field} artifact is missing")
|
|
require(resolved.stat().st_size > 0, f"{gate}.{field} artifact is empty")
|
|
return resolved
|
|
|
|
|
|
def is_json_artifact_format(value: str) -> bool:
|
|
normalized = value.lower()
|
|
return normalized == "json" or normalized.endswith("+json")
|
|
|
|
|
|
def release_bundle_json_artifact_mirrored_fields(gate: str, field: str) -> tuple[str, ...]:
|
|
fields: list[str] = []
|
|
if gate == "G01":
|
|
if field == "root_authority_evidence":
|
|
fields.extend((
|
|
"root_authority_cases",
|
|
"root_cas_observed",
|
|
"root_readback_observed",
|
|
"incomplete_root_rejected",
|
|
"stale_root_rejected",
|
|
))
|
|
if field == "quota_authority_evidence":
|
|
fields.extend((
|
|
"quota_authority_cases",
|
|
"quota_floor_readback_observed",
|
|
"over_limit_put_rejected",
|
|
"rejected_object_invisible",
|
|
"quota_fails_closed_without_authority",
|
|
))
|
|
if gate in ("G03", "G09", "R-L"):
|
|
fields.extend(("versions", "mixed_version_role"))
|
|
if gate == "G02":
|
|
if field == "bounded_checkpoint_oracle":
|
|
fields.extend((
|
|
"checkpoint_progress_bounded",
|
|
"raw_entry_budget",
|
|
"max_raw_entries_per_round",
|
|
"max_objects_processed_per_round",
|
|
"bounded_work_quantum_observed",
|
|
"durable_checkpoint_committed",
|
|
"no_unbounded_tail",
|
|
))
|
|
if field == "independent_version_inventory":
|
|
fields.extend((
|
|
"independent_version_inventory_observed",
|
|
"objects_expected",
|
|
"objects_retained",
|
|
"versions_retained",
|
|
"bytes_retained",
|
|
))
|
|
if gate in ("G04", "G07", "R-E", "R-L"):
|
|
fields.append("crash_points")
|
|
if gate == "R-E":
|
|
if field == "fixed_budget_restart_evidence":
|
|
fields.extend((
|
|
"fixed_budget_restart_converged",
|
|
"restart_rounds",
|
|
"raw_entry_budget",
|
|
"no_unbudgeted_final_sweep",
|
|
))
|
|
if field == "enumeration_evidence":
|
|
fields.extend((
|
|
"raw_enumeration_observed",
|
|
"durable_raw_page_commit_observed",
|
|
"raw_page_index_complete",
|
|
"enumeration_frontier_retained",
|
|
))
|
|
if field == "classification_evidence":
|
|
fields.extend((
|
|
"classification_observed",
|
|
"objects_processed",
|
|
"objects_retained",
|
|
"versions_retained",
|
|
"bytes_retained",
|
|
))
|
|
if gate == "G03":
|
|
fields.append("scoped_ack_cases")
|
|
fields.extend(SCANNER_HEAL_RELEASE_G03_REQUIRED_TRUE_FIELDS[field])
|
|
if gate == "G04" and field == "root_floor_intent_crash_evidence":
|
|
fields.extend(("durable_intent_cases", "persist_failure_blocks_acceptance"))
|
|
if gate == "G12":
|
|
fields.append("quota_path_cases")
|
|
if gate == "G05":
|
|
if field == "per_object_outcome_oracle":
|
|
fields.extend(("per_object_outcome_cases", "outcome_counts", "status_matches_object_oracle"))
|
|
if field == "terminal_retention_bounds":
|
|
fields.extend((
|
|
"terminal_retention_cases",
|
|
"terminal_retention_window_seconds",
|
|
"max_terminal_record_age_seconds",
|
|
"terminal_records_pruned_after_window",
|
|
))
|
|
if gate == "G06":
|
|
if field == "concurrent_status_evidence":
|
|
fields.extend((
|
|
"concurrent_status_cases",
|
|
"status_samples",
|
|
"all_status_responses_http_success",
|
|
"partial_status_reports_degraded",
|
|
))
|
|
if field == "legacy_client_compatibility":
|
|
fields.extend((
|
|
"legacy_client_cases",
|
|
"rustfs_and_minio_paths_compatible",
|
|
"empty_body_status_requests_accepted",
|
|
))
|
|
if field == "truncation_behavior":
|
|
fields.extend(("truncation_cases", "truncated_payloads_rejected", "max_status_payload_bytes"))
|
|
if gate == "G07":
|
|
fields.append({
|
|
"mrf_responsibility_oracle": "mrf_responsibility_cases",
|
|
"commit_boundary_crash_matrix": "commit_crash_cases",
|
|
}[field])
|
|
if gate == "G08":
|
|
fields.append({
|
|
"mrf_capacity_evidence": "capacity_cases",
|
|
"disk_full_matrix": "disk_full_cases",
|
|
"replica_loss_matrix": "replica_loss_cases",
|
|
}[field])
|
|
if gate == "G09":
|
|
fields.append("mixed_version_cases")
|
|
if field == "rollback_payload_evidence":
|
|
fields.append("rollback_payload_replayed")
|
|
if gate == "G11":
|
|
fields.extend({
|
|
"maintenance_producer_matrix": (
|
|
"producer_identities",
|
|
"producer_families",
|
|
"matrix_cases",
|
|
"durable_identity_observed",
|
|
"durable_journal_replay_observed",
|
|
"durable_journal_hydration_bound",
|
|
"generation_window_bound",
|
|
"restart_gap_absent",
|
|
"overflow_absent",
|
|
"ack_clear_generation_observed",
|
|
"invalid_journal_inputs_rejected",
|
|
),
|
|
"complete_producer_inventory": (
|
|
"required_producer_identities",
|
|
"observed_producer_identities",
|
|
"required_producer_families",
|
|
"observed_producer_families",
|
|
"missing_producer_identities",
|
|
"unknown_producer_excluded",
|
|
),
|
|
"segment_activation_preflight": (
|
|
"production_activation",
|
|
"scanner_segment_reuse_activated",
|
|
"proof_inputs",
|
|
"fail_closed_checks",
|
|
),
|
|
}[field])
|
|
if gate == "G13":
|
|
fields.extend({
|
|
"quorum_minus_one_matrix": (
|
|
"quorum_cases",
|
|
"no_success_at_quorum_minus_one",
|
|
"exact_quorum_restored",
|
|
),
|
|
"unknown_disk_remount_matrix": (
|
|
"remount_cases",
|
|
"unknown_disks_excluded",
|
|
"remounted_disks_revalidated",
|
|
"stale_incarnation_rejected",
|
|
),
|
|
"object_lock_dry_run_grace_evidence": (
|
|
"grace_cases",
|
|
"object_lock_denials_preserved",
|
|
"dry_run_mutation_count",
|
|
"grace_outcomes_retained",
|
|
),
|
|
}[field])
|
|
if gate == "G14":
|
|
fields.append("case_evidence")
|
|
if field == "same_window_field_evidence":
|
|
fields.append("same_window_fields")
|
|
if field == "ec8_4_evidence":
|
|
fields.append("topology")
|
|
if field == "multi_set_evidence":
|
|
fields.append("sets")
|
|
if field == "multi_pool_evidence":
|
|
fields.append("pools")
|
|
if field == "distributed_segment_invalidation_evidence":
|
|
fields.extend((
|
|
"invalidation_domain",
|
|
"distributed_ec_invalidation",
|
|
"peer_count",
|
|
"same_window_remote_proof",
|
|
"all_peers_bound_to_generation_window",
|
|
))
|
|
if (gate, field) in SCANNER_HEAL_RELEASE_MRF_DURABLE_REPLAY_FIELDS:
|
|
fields.extend(("replayed_records", "responsibility_anchor_retained", "successor_snapshot_published"))
|
|
if gate == "P4" and field == "retained_responsibility_evidence":
|
|
fields.extend((
|
|
"duration_seconds",
|
|
"retained_responsibility_cases",
|
|
"retention_window_seconds",
|
|
"idle_cleanup_observed",
|
|
"verified_proof_discharge_observed",
|
|
))
|
|
if gate == "P4" and field == "mrf_cleanup_gc_soak_evidence":
|
|
fields.extend((
|
|
"duration_seconds",
|
|
"cleanup_gc_cases",
|
|
"verified_idle_gc_observed",
|
|
"pending_responsibilities_after_gc",
|
|
"stale_journals_after_gc",
|
|
))
|
|
if gate == "R-D":
|
|
if field == "manager_disposition_evidence":
|
|
fields.extend(("manager_disposition_cases", "manager_dispositions_are_terminal"))
|
|
if field == "event_disposition_evidence":
|
|
fields.extend(("event_disposition_cases", "events_correlate_to_manager_dispositions"))
|
|
if field == "ledger_disposition_evidence":
|
|
fields.extend((
|
|
"ledger_disposition_cases",
|
|
"ledger_correlates_to_events",
|
|
"ledger_replay_preserves_terminal_disposition",
|
|
))
|
|
if field == "grace_handling":
|
|
fields.extend((
|
|
"grace_cases",
|
|
"grace_window_seconds",
|
|
"grace_retention_observed",
|
|
"grace_expiry_pruned_terminal_records",
|
|
))
|
|
if gate == "R-L":
|
|
if field == "legacy_source_conflict_evidence":
|
|
fields.extend((
|
|
"legacy_source_conflict_cases",
|
|
"source_conflicts_rejected",
|
|
"takeover_identity_bound",
|
|
"legacy_checksum_gap_rejected",
|
|
))
|
|
if field == "migration_gap_evidence":
|
|
fields.extend((
|
|
"migration_gap_cases",
|
|
"migration_gap_closed",
|
|
"legacy_sources_fail_closed",
|
|
"prior_responsibilities_inherited",
|
|
))
|
|
if field == "crash_safe_source_retirement_evidence":
|
|
fields.extend((
|
|
"source_retirement_cases",
|
|
"source_retirement_is_crash_safe",
|
|
"old_source_retained_until_successor",
|
|
"recovered_pending_migration",
|
|
))
|
|
return tuple(dict.fromkeys(fields))
|
|
|
|
|
|
PROFILE_ARTIFACT_REQUIRED_METRICS = {
|
|
"allocation-profile": ("resolved_samples", "allocation_bytes"),
|
|
"flamegraph": ("resolved_samples",),
|
|
"rss-samples": ("resolved_samples", "rss_peak_bytes"),
|
|
"save-frequency": ("resolved_samples", "save_operations", "saved_bytes"),
|
|
}
|
|
|
|
|
|
def validate_profile_json_artifact_payload(bundle_path: Path, path: Path, payload: dict[str, object],
|
|
gate: str, field: str, artifact_kind: str) -> None:
|
|
prefix = f"{gate}.{field}.{artifact_kind}"
|
|
raw_profile = release_bundle_artifact_path(
|
|
bundle_path,
|
|
payload.get("raw_profile_artifact"),
|
|
gate,
|
|
f"{field}.{artifact_kind}.raw_profile",
|
|
)
|
|
require(sha(payload.get("raw_profile_sha256")) and digest(raw_profile) == payload["raw_profile_sha256"],
|
|
f"{prefix} raw profile hash mismatch")
|
|
raw_bytes = evidence_integer(payload.get("raw_profile_bytes"), f"{prefix}.raw_profile_bytes", 1, 2**63 - 1)
|
|
require(raw_bytes == raw_profile.stat().st_size, f"{prefix} raw profile bytes mismatch")
|
|
evidence_string(payload.get("raw_profile_format"), f"{prefix}.raw_profile_format",
|
|
r"[A-Za-z0-9][A-Za-z0-9._+:-]{1,63}")
|
|
require(raw_profile.resolve() != path.resolve(), f"{prefix} raw profile must be distinct from wrapper")
|
|
for metric in PROFILE_ARTIFACT_REQUIRED_METRICS[artifact_kind]:
|
|
evidence_integer(payload.get(metric), f"{prefix}.{metric}", 1, 2**63 - 1)
|
|
|
|
|
|
def validate_g14_json_artifact_payload(bundle_path: Path, payload: dict[str, object],
|
|
evidence: dict[str, object] | None, gate: str, field: str,
|
|
source_revision: str, window_id: str) -> None:
|
|
prefix = f"{gate}.{field}"
|
|
require(evidence is not None, f"{prefix} JSON artifact missing outer evidence context")
|
|
for mirror_field in release_bundle_json_artifact_mirrored_fields(gate, field):
|
|
require(payload.get(mirror_field) == evidence.get(mirror_field),
|
|
f"{prefix} JSON artifact {mirror_field} mismatch")
|
|
cases = payload.get("case_evidence")
|
|
require(isinstance(cases, list) and cases, f"{prefix} JSON artifact missing case_evidence")
|
|
seen_cases = set()
|
|
for index, item in enumerate(cases):
|
|
require(isinstance(item, dict), f"{prefix} JSON artifact case_evidence[{index}] must be an object")
|
|
case = evidence_string(
|
|
item.get("case"),
|
|
f"{prefix} JSON artifact case_evidence[{index}].case",
|
|
r"[A-Za-z0-9][A-Za-z0-9._:-]{1,127}",
|
|
)
|
|
require(case not in seen_cases, f"{prefix} JSON artifact case_evidence has duplicate case: {case}")
|
|
seen_cases.add(case)
|
|
artifact = release_bundle_artifact_path(
|
|
bundle_path,
|
|
item.get("artifact"),
|
|
gate,
|
|
f"{field}.case_evidence.{case}",
|
|
)
|
|
require(sha(item.get("sha256")) and digest(artifact) == item["sha256"],
|
|
f"{prefix} JSON artifact case_evidence[{index}] artifact hash mismatch")
|
|
require(item.get("source_revision") == source_revision,
|
|
f"{prefix} JSON artifact case_evidence[{index}].source_revision mismatch")
|
|
require(item.get("measurement_window_id") == window_id,
|
|
f"{prefix} JSON artifact case_evidence[{index}].measurement_window_id mismatch")
|
|
case_payload = read_json(artifact)
|
|
require(isinstance(case_payload, dict), f"{prefix} JSON artifact case_evidence[{index}] artifact must be a JSON object")
|
|
for marker in ("fixture", "fixture_only", "dry_run", "synthetic"):
|
|
require(case_payload.get(marker) is not True,
|
|
f"{prefix} JSON artifact case_evidence[{index}] artifact is {marker}")
|
|
require(case_payload.get("case") == case,
|
|
f"{prefix} JSON artifact case_evidence[{index}] artifact case mismatch")
|
|
if "source_revision" in case_payload:
|
|
require(case_payload["source_revision"] == source_revision,
|
|
f"{prefix} JSON artifact case_evidence[{index}] artifact source revision mismatch")
|
|
if "measurement_window_id" in case_payload:
|
|
require(case_payload["measurement_window_id"] == window_id,
|
|
f"{prefix} JSON artifact case_evidence[{index}] artifact measurement window mismatch")
|
|
|
|
|
|
def validate_release_bundle_json_artifact_payload(bundle_path: Path, path: Path, source_revision: str, gate: str, field: str,
|
|
run_id: str, window_id: str,
|
|
artifact_kind: str | None = None,
|
|
evidence: dict[str, object] | None = None) -> None:
|
|
payload = read_json(path)
|
|
prefix = f"{gate}.{field}"
|
|
for marker in ("fixture", "fixture_only", "dry_run", "synthetic"):
|
|
require(payload.get(marker) is not True, f"{prefix} JSON artifact is {marker}")
|
|
require(payload.get("evidence_type") == "measured", f"{prefix} JSON artifact must be measured")
|
|
require(payload.get("source_revision") == source_revision, f"{prefix} JSON artifact source revision mismatch")
|
|
require(payload.get("run_id") == run_id, f"{prefix} JSON artifact run_id mismatch")
|
|
require(payload.get("measurement_window_id") == window_id, f"{prefix} JSON artifact measurement window mismatch")
|
|
require(payload.get("gate") == gate, f"{prefix} JSON artifact gate mismatch")
|
|
require(payload.get("field") == field, f"{prefix} JSON artifact field mismatch")
|
|
if artifact_kind is not None:
|
|
require(payload.get("artifact_kind") == artifact_kind, f"{prefix} JSON artifact kind mismatch")
|
|
if field == "profile_evidence":
|
|
validate_profile_json_artifact_payload(bundle_path, path, payload, gate, field, artifact_kind)
|
|
mrf_artifact_kind = SCANNER_HEAL_RELEASE_MRF_ARTIFACT_KINDS.get((gate, field))
|
|
if mrf_artifact_kind is not None:
|
|
require(payload.get("artifact_kind") == mrf_artifact_kind,
|
|
f"{prefix} JSON artifact kind must be {mrf_artifact_kind}")
|
|
mirror_fields = release_bundle_json_artifact_mirrored_fields(gate, field)
|
|
for mirror_field in mirror_fields:
|
|
require(mirror_field in payload, f"{prefix} JSON artifact missing {mirror_field}")
|
|
if not mirror_fields:
|
|
return
|
|
validate_release_bundle_domain_evidence(gate, field, payload)
|
|
if gate == "G14":
|
|
validate_g14_json_artifact_payload(bundle_path, payload, evidence, gate, field, source_revision, window_id)
|
|
if gate in ("G03", "G09", "R-L"):
|
|
versions = payload.get("versions")
|
|
require(isinstance(versions, list) and
|
|
len(set(versions)) >= 2 and
|
|
all(isinstance(version, str) and re.fullmatch(r"[0-9a-f]{40}", version) is not None
|
|
for version in versions),
|
|
f"{prefix} JSON artifact requires mixed-version evidence")
|
|
require(source_revision in versions, f"{prefix} JSON artifact versions omit tested source revision")
|
|
expected_role = SCANNER_HEAL_RELEASE_MIXED_VERSION_ROLES[(gate, field)]
|
|
require(payload.get("mixed_version_role") == expected_role,
|
|
f"{prefix} JSON artifact mixed-version role must be {expected_role}")
|
|
if gate in ("G04", "G07", "R-E", "R-L"):
|
|
crash_points = payload.get("crash_points")
|
|
require(isinstance(crash_points, list) and crash_points,
|
|
f"{prefix} JSON artifact requires crash-boundary evidence")
|
|
if (gate, field) in SCANNER_HEAL_RELEASE_MRF_DURABLE_REPLAY_FIELDS:
|
|
evidence_integer(payload.get("replayed_records"), f"{prefix} JSON artifact replayed_records", 1, 2**63 - 1)
|
|
require(payload.get("responsibility_anchor_retained") is True,
|
|
f"{prefix} JSON artifact requires retained MRF responsibility anchors")
|
|
require(payload.get("successor_snapshot_published") is True,
|
|
f"{prefix} JSON artifact requires successor snapshot publication evidence")
|
|
if gate == "P4" and field == "mrf_cleanup_gc_soak_evidence":
|
|
release_bundle_exact_strings(
|
|
payload.get("cleanup_gc_cases"),
|
|
SCANNER_HEAL_RELEASE_MRF_CLEANUP_GC_SOAK_CASES,
|
|
f"{prefix} JSON artifact cleanup_gc_cases",
|
|
)
|
|
require(payload.get("verified_idle_gc_observed") is True,
|
|
f"{prefix} JSON artifact requires verified idle GC evidence")
|
|
require(payload.get("pending_responsibilities_after_gc") == 0,
|
|
f"{prefix} JSON artifact requires zero pending responsibilities after GC")
|
|
require(payload.get("stale_journals_after_gc") == 0,
|
|
f"{prefix} JSON artifact requires zero stale journals after GC")
|
|
if gate == "G07":
|
|
case_field = {
|
|
"mrf_responsibility_oracle": "mrf_responsibility_cases",
|
|
"commit_boundary_crash_matrix": "commit_crash_cases",
|
|
}[field]
|
|
cases = evidence_string_list(payload.get(case_field), f"{prefix} JSON artifact {case_field}")
|
|
missing_cases = sorted(set(SCANNER_HEAL_RELEASE_G07_REQUIRED_CASES[field]) - set(cases))
|
|
require(not missing_cases, f"{prefix} JSON artifact missing cases: {', '.join(missing_cases)}")
|
|
if gate == "G08":
|
|
case_field = {
|
|
"mrf_capacity_evidence": "capacity_cases",
|
|
"disk_full_matrix": "disk_full_cases",
|
|
"replica_loss_matrix": "replica_loss_cases",
|
|
}[field]
|
|
cases = evidence_string_list(payload.get(case_field), f"{prefix} JSON artifact {case_field}")
|
|
missing_cases = sorted(set(SCANNER_HEAL_RELEASE_G08_REQUIRED_CASES[field]) - set(cases))
|
|
require(not missing_cases, f"{prefix} JSON artifact missing cases: {', '.join(missing_cases)}")
|
|
|
|
|
|
def release_bundle_bool_true(value: object, name: str) -> None:
|
|
require(value is True, f"{name} must be true")
|
|
|
|
|
|
def release_bundle_exact_strings(value: object, expected: tuple[str, ...], name: str) -> None:
|
|
require(isinstance(value, list) and all(isinstance(item, str) and item.strip() for item in value),
|
|
f"{name} must list measured cases")
|
|
observed = set(value)
|
|
require(len(observed) == len(value), f"{name} has duplicate cases")
|
|
missing = sorted(set(expected) - observed)
|
|
require(not missing, f"{name} missing cases: {', '.join(missing)}")
|
|
unknown = sorted(observed - set(expected))
|
|
require(not unknown, f"{name} has unknown cases: {', '.join(unknown)}")
|
|
|
|
|
|
def release_bundle_number(value: object, name: str, minimum: int | float = 0) -> float:
|
|
return number(value, name, minimum)
|
|
|
|
|
|
def validate_release_bundle_domain_evidence(gate: str, field: str, evidence: dict[str, object]) -> None:
|
|
if gate == "G01":
|
|
if field == "root_authority_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("root_authority_cases"),
|
|
SCANNER_HEAL_RELEASE_G01_ROOT_AUTHORITY_CASES,
|
|
f"{gate}.{field}.root_authority_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("root_cas_observed"), f"{gate}.{field}.root_cas_observed")
|
|
release_bundle_bool_true(evidence.get("root_readback_observed"), f"{gate}.{field}.root_readback_observed")
|
|
release_bundle_bool_true(evidence.get("incomplete_root_rejected"),
|
|
f"{gate}.{field}.incomplete_root_rejected")
|
|
release_bundle_bool_true(evidence.get("stale_root_rejected"), f"{gate}.{field}.stale_root_rejected")
|
|
if field == "quota_authority_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("quota_authority_cases"),
|
|
SCANNER_HEAL_RELEASE_G01_QUOTA_AUTHORITY_CASES,
|
|
f"{gate}.{field}.quota_authority_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("quota_floor_readback_observed"),
|
|
f"{gate}.{field}.quota_floor_readback_observed")
|
|
release_bundle_bool_true(evidence.get("over_limit_put_rejected"),
|
|
f"{gate}.{field}.over_limit_put_rejected")
|
|
release_bundle_bool_true(evidence.get("rejected_object_invisible"),
|
|
f"{gate}.{field}.rejected_object_invisible")
|
|
release_bundle_bool_true(evidence.get("quota_fails_closed_without_authority"),
|
|
f"{gate}.{field}.quota_fails_closed_without_authority")
|
|
|
|
if gate == "G02":
|
|
if field == "bounded_checkpoint_oracle":
|
|
release_bundle_bool_true(evidence.get("checkpoint_progress_bounded"),
|
|
f"{gate}.{field}.checkpoint_progress_bounded")
|
|
budget = evidence_integer(evidence.get("raw_entry_budget"), f"{gate}.{field}.raw_entry_budget", 1, 4096)
|
|
max_raw = evidence_integer(evidence.get("max_raw_entries_per_round"),
|
|
f"{gate}.{field}.max_raw_entries_per_round", 1, 4096)
|
|
max_objects = evidence_integer(evidence.get("max_objects_processed_per_round"),
|
|
f"{gate}.{field}.max_objects_processed_per_round", 1, 4096)
|
|
require(max_raw <= budget, f"{gate}.{field} raw entries exceed fixed budget")
|
|
require(max_objects <= budget, f"{gate}.{field} processed objects exceed fixed budget")
|
|
release_bundle_bool_true(evidence.get("bounded_work_quantum_observed"),
|
|
f"{gate}.{field}.bounded_work_quantum_observed")
|
|
release_bundle_bool_true(evidence.get("durable_checkpoint_committed"),
|
|
f"{gate}.{field}.durable_checkpoint_committed")
|
|
release_bundle_bool_true(evidence.get("no_unbounded_tail"), f"{gate}.{field}.no_unbounded_tail")
|
|
if field == "independent_version_inventory":
|
|
release_bundle_bool_true(evidence.get("independent_version_inventory_observed"),
|
|
f"{gate}.{field}.independent_version_inventory_observed")
|
|
objects_expected = evidence_integer(evidence.get("objects_expected"),
|
|
f"{gate}.{field}.objects_expected", 1, 2**63 - 1)
|
|
objects_retained = evidence_integer(evidence.get("objects_retained"),
|
|
f"{gate}.{field}.objects_retained", 1, 2**63 - 1)
|
|
versions_retained = evidence_integer(evidence.get("versions_retained"),
|
|
f"{gate}.{field}.versions_retained", 1, 2**63 - 1)
|
|
bytes_retained = evidence_integer(evidence.get("bytes_retained"),
|
|
f"{gate}.{field}.bytes_retained", 1, 2**63 - 1)
|
|
require(objects_retained == objects_expected,
|
|
f"{gate}.{field} retained object inventory must match expected objects")
|
|
require(versions_retained == objects_expected,
|
|
f"{gate}.{field} retained version inventory must match expected objects")
|
|
require(bytes_retained == objects_expected,
|
|
f"{gate}.{field} retained byte inventory must match expected objects")
|
|
|
|
if gate == "G03":
|
|
release_bundle_exact_strings(
|
|
evidence.get("scoped_ack_cases"),
|
|
SCANNER_HEAL_RELEASE_SCOPED_ACK_CASES[field],
|
|
f"{gate}.{field}.scoped_ack_cases",
|
|
)
|
|
for required in SCANNER_HEAL_RELEASE_G03_REQUIRED_TRUE_FIELDS[field]:
|
|
release_bundle_bool_true(evidence.get(required), f"{gate}.{field}.{required}")
|
|
|
|
if gate == "G04" and field == "root_floor_intent_crash_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("durable_intent_cases"),
|
|
SCANNER_HEAL_RELEASE_CRASH_BOUNDARY_FIELDS[(gate, field)],
|
|
f"{gate}.{field}.durable_intent_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("persist_failure_blocks_acceptance"),
|
|
f"{gate}.{field}.persist_failure_blocks_acceptance")
|
|
|
|
if gate == "G12":
|
|
release_bundle_exact_strings(
|
|
evidence.get("quota_path_cases"),
|
|
SCANNER_HEAL_RELEASE_G12_QUOTA_PATH_CASES[field],
|
|
f"{gate}.{field}.quota_path_cases",
|
|
)
|
|
|
|
if gate == "G05":
|
|
if field == "per_object_outcome_oracle":
|
|
release_bundle_exact_strings(
|
|
evidence.get("per_object_outcome_cases"),
|
|
SCANNER_HEAL_RELEASE_G05_PER_OBJECT_OUTCOME_CASES,
|
|
f"{gate}.{field}.per_object_outcome_cases",
|
|
)
|
|
outcomes = evidence.get("outcome_counts")
|
|
require(isinstance(outcomes, dict), f"{gate}.{field} missing outcome counts")
|
|
for outcome in ("repaired", "healthy", "skipped", "failed"):
|
|
evidence_integer(outcomes.get(outcome), f"{gate}.{field}.outcome_counts.{outcome}", 1, 2**63 - 1)
|
|
release_bundle_bool_true(evidence.get("status_matches_object_oracle"),
|
|
f"{gate}.{field}.status_matches_object_oracle")
|
|
if field == "terminal_retention_bounds":
|
|
release_bundle_exact_strings(
|
|
evidence.get("terminal_retention_cases"),
|
|
SCANNER_HEAL_RELEASE_G05_TERMINAL_RETENTION_CASES,
|
|
f"{gate}.{field}.terminal_retention_cases",
|
|
)
|
|
retention_window = evidence_integer(
|
|
evidence.get("terminal_retention_window_seconds"),
|
|
f"{gate}.{field}.terminal_retention_window_seconds",
|
|
1,
|
|
86400,
|
|
)
|
|
max_age = evidence_integer(
|
|
evidence.get("max_terminal_record_age_seconds"),
|
|
f"{gate}.{field}.max_terminal_record_age_seconds",
|
|
0,
|
|
86400,
|
|
)
|
|
require(max_age <= retention_window, f"{gate}.{field}.max_terminal_record_age_seconds exceeds retention window")
|
|
evidence_integer(evidence.get("terminal_records_pruned_after_window"),
|
|
f"{gate}.{field}.terminal_records_pruned_after_window", 1, 2**63 - 1)
|
|
|
|
if gate == "G06":
|
|
if field == "concurrent_status_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("concurrent_status_cases"),
|
|
SCANNER_HEAL_RELEASE_G06_CONCURRENT_STATUS_CASES,
|
|
f"{gate}.{field}.concurrent_status_cases",
|
|
)
|
|
evidence_integer(evidence.get("status_samples"), f"{gate}.{field}.status_samples", 2, 2**63 - 1)
|
|
release_bundle_bool_true(evidence.get("all_status_responses_http_success"),
|
|
f"{gate}.{field}.all_status_responses_http_success")
|
|
release_bundle_bool_true(evidence.get("partial_status_reports_degraded"),
|
|
f"{gate}.{field}.partial_status_reports_degraded")
|
|
if field == "legacy_client_compatibility":
|
|
release_bundle_exact_strings(
|
|
evidence.get("legacy_client_cases"),
|
|
SCANNER_HEAL_RELEASE_G06_LEGACY_CLIENT_CASES,
|
|
f"{gate}.{field}.legacy_client_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("rustfs_and_minio_paths_compatible"),
|
|
f"{gate}.{field}.rustfs_and_minio_paths_compatible")
|
|
release_bundle_bool_true(evidence.get("empty_body_status_requests_accepted"),
|
|
f"{gate}.{field}.empty_body_status_requests_accepted")
|
|
if field == "truncation_behavior":
|
|
release_bundle_exact_strings(
|
|
evidence.get("truncation_cases"),
|
|
SCANNER_HEAL_RELEASE_G06_TRUNCATION_CASES,
|
|
f"{gate}.{field}.truncation_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("truncated_payloads_rejected"),
|
|
f"{gate}.{field}.truncated_payloads_rejected")
|
|
evidence_integer(evidence.get("max_status_payload_bytes"), f"{gate}.{field}.max_status_payload_bytes", 1, 2**20)
|
|
|
|
if gate == "R-E":
|
|
if field == "fixed_budget_restart_evidence":
|
|
release_bundle_bool_true(evidence.get("fixed_budget_restart_converged"),
|
|
f"{gate}.{field}.fixed_budget_restart_converged")
|
|
evidence_integer(evidence.get("restart_rounds"), f"{gate}.{field}.restart_rounds", 2, 64)
|
|
evidence_integer(evidence.get("raw_entry_budget"), f"{gate}.{field}.raw_entry_budget", 1, 4096)
|
|
release_bundle_bool_true(evidence.get("no_unbudgeted_final_sweep"),
|
|
f"{gate}.{field}.no_unbudgeted_final_sweep")
|
|
if field == "enumeration_evidence":
|
|
release_bundle_bool_true(evidence.get("raw_enumeration_observed"),
|
|
f"{gate}.{field}.raw_enumeration_observed")
|
|
release_bundle_bool_true(evidence.get("durable_raw_page_commit_observed"),
|
|
f"{gate}.{field}.durable_raw_page_commit_observed")
|
|
release_bundle_bool_true(evidence.get("raw_page_index_complete"),
|
|
f"{gate}.{field}.raw_page_index_complete")
|
|
release_bundle_bool_true(evidence.get("enumeration_frontier_retained"),
|
|
f"{gate}.{field}.enumeration_frontier_retained")
|
|
if field == "classification_evidence":
|
|
release_bundle_bool_true(evidence.get("classification_observed"),
|
|
f"{gate}.{field}.classification_observed")
|
|
objects_processed = evidence_integer(evidence.get("objects_processed"),
|
|
f"{gate}.{field}.objects_processed", 1, 2**63 - 1)
|
|
objects_retained = evidence_integer(evidence.get("objects_retained"),
|
|
f"{gate}.{field}.objects_retained", 1, 2**63 - 1)
|
|
versions_retained = evidence_integer(evidence.get("versions_retained"),
|
|
f"{gate}.{field}.versions_retained", 1, 2**63 - 1)
|
|
bytes_retained = evidence_integer(evidence.get("bytes_retained"),
|
|
f"{gate}.{field}.bytes_retained", 1, 2**63 - 1)
|
|
require(objects_processed <= objects_retained,
|
|
f"{gate}.{field} processed objects exceed retained objects")
|
|
require(versions_retained == objects_retained,
|
|
f"{gate}.{field} version inventory must match retained objects")
|
|
require(bytes_retained == objects_retained,
|
|
f"{gate}.{field} byte inventory must match retained objects")
|
|
|
|
if gate == "G09":
|
|
release_bundle_exact_strings(
|
|
evidence.get("mixed_version_cases"),
|
|
SCANNER_HEAL_RELEASE_MIXED_VERSION_CASES[field],
|
|
f"{gate}.{field}.mixed_version_cases",
|
|
)
|
|
if field == "rollback_payload_evidence":
|
|
release_bundle_bool_true(evidence.get("rollback_payload_replayed"),
|
|
f"{gate}.{field}.rollback_payload_replayed")
|
|
|
|
if gate == "G10":
|
|
if field == "scheduler_bound_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("scheduler_bounds"),
|
|
SCANNER_HEAL_RELEASE_SCHEDULER_BOUNDS,
|
|
f"{gate}.{field}.scheduler_bounds",
|
|
)
|
|
release_bundle_bool_true(evidence.get("duplicate_task_bound_observed"),
|
|
f"{gate}.{field}.duplicate_task_bound_observed")
|
|
if field == "pressure_recovery_evidence":
|
|
metrics = evidence.get("pressure_metrics")
|
|
require(isinstance(metrics, dict), f"{gate}.{field} missing pressure metrics")
|
|
for metric in SCANNER_HEAL_RELEASE_PRESSURE_METRICS:
|
|
release_bundle_number(metrics.get(metric), f"{gate}.{field}.{metric}")
|
|
evidence_integer(metrics.get("foreground_pressure_samples"),
|
|
f"{gate}.{field}.foreground_pressure_samples", 1, 2**63 - 1)
|
|
evidence_integer(metrics.get("foreground_pressure_high_samples"),
|
|
f"{gate}.{field}.foreground_pressure_high_samples", 1, 2**63 - 1)
|
|
|
|
if gate == "G11":
|
|
if field == "maintenance_producer_matrix":
|
|
release_bundle_exact_strings(
|
|
evidence.get("producer_identities"),
|
|
SCANNER_HEAL_REQUIRED_PRODUCER_IDENTITIES,
|
|
f"{gate}.{field}.producer_identities",
|
|
)
|
|
release_bundle_exact_strings(
|
|
evidence.get("producer_families"),
|
|
SCANNER_HEAL_REQUIRED_PRODUCER_FAMILIES,
|
|
f"{gate}.{field}.producer_families",
|
|
)
|
|
release_bundle_exact_strings(
|
|
evidence.get("matrix_cases"),
|
|
SCANNER_HEAL_RELEASE_G11_REQUIRED_CASES[field],
|
|
f"{gate}.{field}.matrix_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("durable_identity_observed"),
|
|
f"{gate}.{field}.durable_identity_observed")
|
|
release_bundle_bool_true(evidence.get("durable_journal_replay_observed"),
|
|
f"{gate}.{field}.durable_journal_replay_observed")
|
|
release_bundle_bool_true(evidence.get("durable_journal_hydration_bound"),
|
|
f"{gate}.{field}.durable_journal_hydration_bound")
|
|
release_bundle_bool_true(evidence.get("generation_window_bound"),
|
|
f"{gate}.{field}.generation_window_bound")
|
|
release_bundle_bool_true(evidence.get("restart_gap_absent"), f"{gate}.{field}.restart_gap_absent")
|
|
release_bundle_bool_true(evidence.get("overflow_absent"), f"{gate}.{field}.overflow_absent")
|
|
release_bundle_bool_true(evidence.get("ack_clear_generation_observed"),
|
|
f"{gate}.{field}.ack_clear_generation_observed")
|
|
release_bundle_exact_strings(
|
|
evidence.get("invalid_journal_inputs_rejected"),
|
|
SCANNER_HEAL_DURABLE_JOURNAL_FAIL_CLOSED_CASES,
|
|
f"{gate}.{field}.invalid_journal_inputs_rejected",
|
|
)
|
|
if field == "complete_producer_inventory":
|
|
release_bundle_exact_strings(
|
|
evidence.get("required_producer_identities"),
|
|
SCANNER_HEAL_REQUIRED_PRODUCER_IDENTITIES,
|
|
f"{gate}.{field}.required_producer_identities",
|
|
)
|
|
release_bundle_exact_strings(
|
|
evidence.get("observed_producer_identities"),
|
|
SCANNER_HEAL_REQUIRED_PRODUCER_IDENTITIES,
|
|
f"{gate}.{field}.observed_producer_identities",
|
|
)
|
|
release_bundle_exact_strings(
|
|
evidence.get("required_producer_families"),
|
|
SCANNER_HEAL_REQUIRED_PRODUCER_FAMILIES,
|
|
f"{gate}.{field}.required_producer_families",
|
|
)
|
|
release_bundle_exact_strings(
|
|
evidence.get("observed_producer_families"),
|
|
SCANNER_HEAL_REQUIRED_PRODUCER_FAMILIES,
|
|
f"{gate}.{field}.observed_producer_families",
|
|
)
|
|
require(evidence.get("missing_producer_identities") == [],
|
|
f"{gate}.{field} requires zero missing producer identities")
|
|
release_bundle_bool_true(evidence.get("unknown_producer_excluded"),
|
|
f"{gate}.{field}.unknown_producer_excluded")
|
|
|
|
if gate == "G13":
|
|
if field == "quorum_minus_one_matrix":
|
|
release_bundle_exact_strings(
|
|
evidence.get("quorum_cases"),
|
|
SCANNER_HEAL_RELEASE_G13_REQUIRED_CASES[field],
|
|
f"{gate}.{field}.quorum_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("no_success_at_quorum_minus_one"),
|
|
f"{gate}.{field}.no_success_at_quorum_minus_one")
|
|
release_bundle_bool_true(evidence.get("exact_quorum_restored"), f"{gate}.{field}.exact_quorum_restored")
|
|
if field == "unknown_disk_remount_matrix":
|
|
release_bundle_exact_strings(
|
|
evidence.get("remount_cases"),
|
|
SCANNER_HEAL_RELEASE_G13_REQUIRED_CASES[field],
|
|
f"{gate}.{field}.remount_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("unknown_disks_excluded"),
|
|
f"{gate}.{field}.unknown_disks_excluded")
|
|
release_bundle_bool_true(evidence.get("remounted_disks_revalidated"),
|
|
f"{gate}.{field}.remounted_disks_revalidated")
|
|
release_bundle_bool_true(evidence.get("stale_incarnation_rejected"),
|
|
f"{gate}.{field}.stale_incarnation_rejected")
|
|
if field == "object_lock_dry_run_grace_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("grace_cases"),
|
|
SCANNER_HEAL_RELEASE_G13_REQUIRED_CASES[field],
|
|
f"{gate}.{field}.grace_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("object_lock_denials_preserved"),
|
|
f"{gate}.{field}.object_lock_denials_preserved")
|
|
evidence_integer(evidence.get("dry_run_mutation_count"), f"{gate}.{field}.dry_run_mutation_count", 0, 0)
|
|
release_bundle_bool_true(evidence.get("grace_outcomes_retained"),
|
|
f"{gate}.{field}.grace_outcomes_retained")
|
|
|
|
if gate == "P1":
|
|
if field == "cold_walk_share_measurement":
|
|
share = release_bundle_number(evidence.get("cold_walk_share"), f"{gate}.{field}.cold_walk_share")
|
|
require(share <= 1, f"{gate}.{field}.cold_walk_share exceeds one")
|
|
evidence_integer(evidence.get("walk_objects"), f"{gate}.{field}.walk_objects", 1, 2**63 - 1)
|
|
evidence_integer(evidence.get("cold_walk_objects"), f"{gate}.{field}.cold_walk_objects", 0, 2**63 - 1)
|
|
if field == "foreground_latency_throughput_measurement":
|
|
for metric in ("foreground_p95_ms", "foreground_p99_ms", "throughput_ops", "error_rate"):
|
|
release_bundle_number(evidence.get(metric), f"{gate}.{field}.{metric}")
|
|
if field == "profile_evidence":
|
|
for metric in ("allocation_bytes", "rss_peak_bytes", "save_operations", "saved_bytes"):
|
|
evidence_integer(evidence.get(metric), f"{gate}.{field}.{metric}", 1, 2**63 - 1)
|
|
|
|
if gate == "P3":
|
|
if field == "two_hour_pressure_measurement":
|
|
release_bundle_exact_strings(evidence.get("abba_legs"), ("A1", "B1", "B2", "A2"),
|
|
f"{gate}.{field}.abba_legs")
|
|
release_bundle_exact_strings(evidence.get("scenarios"), SCENARIOS, f"{gate}.{field}.scenarios")
|
|
for metric in ("foreground_p95_ms", "foreground_p99_ms", "throughput_ops"):
|
|
release_bundle_number(evidence.get(metric), f"{gate}.{field}.{metric}", 1)
|
|
if field == "heal_capacity_measurement":
|
|
capacity = evidence.get("heal_capacity")
|
|
require(isinstance(capacity, dict), f"{gate}.{field} missing heal capacity")
|
|
for metric in ("objects", "versions", "bytes", "completed_objects"):
|
|
evidence_integer(capacity.get(metric), f"{gate}.{field}.{metric}", 1, 2**63 - 1)
|
|
if field == "recovery_window_measurement":
|
|
release_bundle_exact_strings(
|
|
evidence.get("fault_modes"),
|
|
("process-restart", "process-crash-restart"),
|
|
f"{gate}.{field}.fault_modes",
|
|
)
|
|
for metric in ("recovery_p95_ms", "recovery_p99_ms"):
|
|
release_bundle_number(evidence.get(metric), f"{gate}.{field}.{metric}", 1)
|
|
|
|
if gate == "P4" and field == "retained_responsibility_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("retained_responsibility_cases"),
|
|
SCANNER_HEAL_RELEASE_P4_RETAINED_RESPONSIBILITY_CASES,
|
|
f"{gate}.{field}.retained_responsibility_cases",
|
|
)
|
|
retention_window = evidence_integer(
|
|
evidence.get("retention_window_seconds"),
|
|
f"{gate}.{field}.retention_window_seconds",
|
|
7200,
|
|
86400,
|
|
)
|
|
duration = evidence_integer(evidence.get("duration_seconds"), f"{gate}.{field}.duration_seconds", 1, 86400)
|
|
require(duration >= retention_window, f"{gate}.{field} duration must cover retention window")
|
|
release_bundle_bool_true(evidence.get("idle_cleanup_observed"), f"{gate}.{field}.idle_cleanup_observed")
|
|
release_bundle_bool_true(
|
|
evidence.get("verified_proof_discharge_observed"),
|
|
f"{gate}.{field}.verified_proof_discharge_observed",
|
|
)
|
|
|
|
if gate == "R-D":
|
|
if field == "manager_disposition_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("manager_disposition_cases"),
|
|
SCANNER_HEAL_RELEASE_RD_MANAGER_CASES,
|
|
f"{gate}.{field}.manager_disposition_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("manager_dispositions_are_terminal"),
|
|
f"{gate}.{field}.manager_dispositions_are_terminal")
|
|
if field == "event_disposition_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("event_disposition_cases"),
|
|
SCANNER_HEAL_RELEASE_RD_EVENT_CASES,
|
|
f"{gate}.{field}.event_disposition_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("events_correlate_to_manager_dispositions"),
|
|
f"{gate}.{field}.events_correlate_to_manager_dispositions")
|
|
if field == "ledger_disposition_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("ledger_disposition_cases"),
|
|
SCANNER_HEAL_RELEASE_RD_LEDGER_CASES,
|
|
f"{gate}.{field}.ledger_disposition_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("ledger_correlates_to_events"),
|
|
f"{gate}.{field}.ledger_correlates_to_events")
|
|
release_bundle_bool_true(evidence.get("ledger_replay_preserves_terminal_disposition"),
|
|
f"{gate}.{field}.ledger_replay_preserves_terminal_disposition")
|
|
if field == "grace_handling":
|
|
release_bundle_exact_strings(
|
|
evidence.get("grace_cases"),
|
|
SCANNER_HEAL_RELEASE_RD_GRACE_CASES,
|
|
f"{gate}.{field}.grace_cases",
|
|
)
|
|
evidence_integer(evidence.get("grace_window_seconds"), f"{gate}.{field}.grace_window_seconds", 1, 86400)
|
|
release_bundle_bool_true(evidence.get("grace_retention_observed"),
|
|
f"{gate}.{field}.grace_retention_observed")
|
|
release_bundle_bool_true(evidence.get("grace_expiry_pruned_terminal_records"),
|
|
f"{gate}.{field}.grace_expiry_pruned_terminal_records")
|
|
|
|
if gate == "R-L":
|
|
if field == "legacy_source_conflict_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("legacy_source_conflict_cases"),
|
|
SCANNER_HEAL_RELEASE_RL_REQUIRED_CASES[field],
|
|
f"{gate}.{field}.legacy_source_conflict_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("source_conflicts_rejected"),
|
|
f"{gate}.{field}.source_conflicts_rejected")
|
|
release_bundle_bool_true(evidence.get("takeover_identity_bound"),
|
|
f"{gate}.{field}.takeover_identity_bound")
|
|
release_bundle_bool_true(evidence.get("legacy_checksum_gap_rejected"),
|
|
f"{gate}.{field}.legacy_checksum_gap_rejected")
|
|
if field == "migration_gap_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("migration_gap_cases"),
|
|
SCANNER_HEAL_RELEASE_RL_REQUIRED_CASES[field],
|
|
f"{gate}.{field}.migration_gap_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("migration_gap_closed"),
|
|
f"{gate}.{field}.migration_gap_closed")
|
|
release_bundle_bool_true(evidence.get("legacy_sources_fail_closed"),
|
|
f"{gate}.{field}.legacy_sources_fail_closed")
|
|
release_bundle_bool_true(evidence.get("prior_responsibilities_inherited"),
|
|
f"{gate}.{field}.prior_responsibilities_inherited")
|
|
if field == "crash_safe_source_retirement_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("source_retirement_cases"),
|
|
SCANNER_HEAL_RELEASE_RL_REQUIRED_CASES[field],
|
|
f"{gate}.{field}.source_retirement_cases",
|
|
)
|
|
release_bundle_bool_true(evidence.get("source_retirement_is_crash_safe"),
|
|
f"{gate}.{field}.source_retirement_is_crash_safe")
|
|
release_bundle_bool_true(evidence.get("old_source_retained_until_successor"),
|
|
f"{gate}.{field}.old_source_retained_until_successor")
|
|
release_bundle_bool_true(evidence.get("recovered_pending_migration"),
|
|
f"{gate}.{field}.recovered_pending_migration")
|
|
|
|
|
|
def validate_release_bundle_artifact(bundle_path: Path, source_revision: str, gate: str, field: str,
|
|
evidence: dict[str, object]) -> str:
|
|
require(evidence.get("evidence_type") == "measured", f"{gate}.{field} must be measured evidence")
|
|
require(evidence.get("source_revision") == source_revision, f"{gate}.{field} source revision mismatch")
|
|
run_id = evidence_string(evidence.get("run_id"), f"{gate}.{field}.run_id", r"[A-Za-z0-9][A-Za-z0-9._:-]{7,127}")
|
|
window_id = evidence_string(evidence.get("measurement_window_id"), f"{gate}.{field}.measurement_window_id",
|
|
r"[A-Za-z0-9][A-Za-z0-9._:-]{7,127}")
|
|
require(window_id != run_id, f"{gate}.{field} must separate run and measurement-window identities")
|
|
started = evidence_timestamp(evidence.get("started_at"), f"{gate}.{field}.started_at")
|
|
finished = evidence_timestamp(evidence.get("finished_at"), f"{gate}.{field}.finished_at")
|
|
require(started <= finished, f"{gate}.{field} evidence timestamps are inverted")
|
|
command = evidence.get("command")
|
|
require(isinstance(command, list) and command and
|
|
all(isinstance(part, str) and part.strip() for part in command),
|
|
f"{gate}.{field} missing command provenance")
|
|
artifact_format = evidence_string(evidence.get("artifact_format"), f"{gate}.{field}.artifact_format",
|
|
r"[A-Za-z0-9][A-Za-z0-9._+:-]{1,63}")
|
|
artifact = release_bundle_artifact_path(bundle_path, evidence.get("artifact"), gate, field)
|
|
require(sha(evidence.get("sha256")) and digest(artifact) == evidence["sha256"], f"{gate}.{field} artifact hash mismatch")
|
|
if is_json_artifact_format(artifact_format):
|
|
validate_release_bundle_json_artifact_payload(
|
|
bundle_path,
|
|
artifact,
|
|
source_revision,
|
|
gate,
|
|
field,
|
|
run_id,
|
|
window_id,
|
|
evidence=evidence,
|
|
)
|
|
summary = evidence.get("summary")
|
|
require(isinstance(summary, str) and summary.strip(), f"{gate}.{field} missing human summary")
|
|
if gate.startswith("P"):
|
|
duration = evidence_integer(evidence.get("duration_seconds"), f"{gate}.{field}.duration_seconds", 1, 86400)
|
|
require((finished - started).total_seconds() + 1 >= duration, f"{gate}.{field} duration exceeds run window")
|
|
require(duration >= 900, f"{gate}.{field} requires at least 900 seconds")
|
|
if gate == "P3" and field == "two_hour_pressure_measurement":
|
|
require(duration >= 7200, f"{gate}.{field} requires at least two hours")
|
|
if gate == "P4" and field == "mrf_cleanup_gc_soak_evidence":
|
|
require(duration >= 7200, f"{gate}.{field} requires at least two hours")
|
|
elif "duration_seconds" in evidence:
|
|
evidence_integer(evidence.get("duration_seconds"), f"{gate}.{field}.duration_seconds", 1, 86400)
|
|
if gate == "G10":
|
|
if field == "scheduler_bound_evidence":
|
|
evidence_integer(evidence.get("max_deferred_items"), f"{gate}.{field}.max_deferred_items", 1, 2**31 - 1)
|
|
evidence_integer(evidence.get("max_deferred_bytes"), f"{gate}.{field}.max_deferred_bytes", 1, 2**63 - 1)
|
|
evidence_integer(evidence.get("max_retry_age_seconds"), f"{gate}.{field}.max_retry_age_seconds", 1, 86400)
|
|
evidence_integer(evidence.get("duplicate_task_count"), f"{gate}.{field}.duplicate_task_count", 0, 0)
|
|
if field == "pressure_recovery_evidence":
|
|
require(evidence.get("pressure_pacing_engaged") is True,
|
|
f"{gate}.{field} requires pressure pacing engagement")
|
|
evidence_integer(evidence.get("recovery_window_seconds"), f"{gate}.{field}.recovery_window_seconds", 1, 86400)
|
|
evidence_integer(evidence.get("lock_hold_p95_ms"), f"{gate}.{field}.lock_hold_p95_ms", 0, 2**31 - 1)
|
|
evidence_integer(evidence.get("foreground_latency_p95_ms"),
|
|
f"{gate}.{field}.foreground_latency_p95_ms", 1, 2**31 - 1)
|
|
if gate in ("G01", "G11", "G12", "G13", "R-L"):
|
|
validate_release_bundle_domain_evidence(gate, field, evidence)
|
|
if gate == "P1" and field == "foreground_latency_throughput_measurement":
|
|
evidence_integer(evidence.get("foreground_latency_p95_ms"),
|
|
f"{gate}.{field}.foreground_latency_p95_ms", 1, 2**31 - 1)
|
|
evidence_integer(evidence.get("foreground_latency_p99_ms"),
|
|
f"{gate}.{field}.foreground_latency_p99_ms", 1, 2**31 - 1)
|
|
evidence_integer(evidence.get("throughput_ops_per_second"),
|
|
f"{gate}.{field}.throughput_ops_per_second", 1, 2**31 - 1)
|
|
evidence_integer(evidence.get("error_count"), f"{gate}.{field}.error_count", 0, 0)
|
|
if gate == "P3":
|
|
if field == "two_hour_pressure_measurement":
|
|
require(evidence.get("fixed_offered_load") is True, f"{gate}.{field} requires fixed offered load")
|
|
evidence_integer(evidence.get("foreground_latency_p99_ms"),
|
|
f"{gate}.{field}.foreground_latency_p99_ms", 1, 2**31 - 1)
|
|
evidence_integer(evidence.get("attempt_cost_samples"),
|
|
f"{gate}.{field}.attempt_cost_samples", 1, 2**63 - 1)
|
|
if field == "heal_capacity_measurement":
|
|
evidence_integer(evidence.get("completed_heal_objects"),
|
|
f"{gate}.{field}.completed_heal_objects", 1, 2**63 - 1)
|
|
evidence_integer(evidence.get("duplicate_task_count"), f"{gate}.{field}.duplicate_task_count", 0, 0)
|
|
if field == "recovery_window_measurement":
|
|
evidence_integer(evidence.get("pressure_recovery_window_seconds"),
|
|
f"{gate}.{field}.pressure_recovery_window_seconds", 1, 86400)
|
|
evidence_integer(evidence.get("lock_hold_p95_ms"), f"{gate}.{field}.lock_hold_p95_ms", 0, 2**31 - 1)
|
|
if gate in ("G03", "G09", "R-L"):
|
|
versions = evidence.get("versions")
|
|
require(isinstance(versions, list) and
|
|
len(set(versions)) >= 2 and
|
|
all(isinstance(version, str) and re.fullmatch(r"[0-9a-f]{40}", version) is not None
|
|
for version in versions),
|
|
f"{gate}.{field} requires mixed-version evidence")
|
|
require(source_revision in versions, f"{gate}.{field} versions omit tested source revision")
|
|
expected_role = SCANNER_HEAL_RELEASE_MIXED_VERSION_ROLES[(gate, field)]
|
|
require(evidence.get("mixed_version_role") == expected_role,
|
|
f"{gate}.{field} mixed-version role must be {expected_role}")
|
|
if gate in ("G04", "G07", "R-E", "R-L"):
|
|
crash_points = evidence.get("crash_points")
|
|
require(isinstance(crash_points, list) and crash_points,
|
|
f"{gate}.{field} requires crash-boundary evidence")
|
|
if (gate, field) in SCANNER_HEAL_RELEASE_MRF_DURABLE_REPLAY_FIELDS:
|
|
evidence_integer(evidence.get("replayed_records"), f"{gate}.{field}.replayed_records", 1, 2**63 - 1)
|
|
require(evidence.get("responsibility_anchor_retained") is True,
|
|
f"{gate}.{field} requires retained MRF responsibility anchors")
|
|
require(evidence.get("successor_snapshot_published") is True,
|
|
f"{gate}.{field} requires successor snapshot publication evidence")
|
|
if gate == "P4" and field == "mrf_cleanup_gc_soak_evidence":
|
|
release_bundle_exact_strings(
|
|
evidence.get("cleanup_gc_cases"),
|
|
SCANNER_HEAL_RELEASE_MRF_CLEANUP_GC_SOAK_CASES,
|
|
f"{gate}.{field}.cleanup_gc_cases",
|
|
)
|
|
require(evidence.get("verified_idle_gc_observed") is True,
|
|
f"{gate}.{field} requires verified idle GC evidence")
|
|
require(evidence.get("pending_responsibilities_after_gc") == 0,
|
|
f"{gate}.{field} requires zero pending responsibilities after GC")
|
|
require(evidence.get("stale_journals_after_gc") == 0,
|
|
f"{gate}.{field} requires zero stale journals after GC")
|
|
if field == "segment_activation_preflight":
|
|
require(evidence.get("production_activation") is True,
|
|
f"{gate}.{field} must prove production activation is enabled")
|
|
require(evidence.get("scanner_segment_reuse_activated") is True,
|
|
f"{gate}.{field} must prove the runtime activation gate is enabled")
|
|
evidence_exact_strings(evidence.get("proof_inputs"),
|
|
SCANNER_HEAL_SEGMENT_ACTIVATION_PROOF_INPUTS,
|
|
f"{gate}.{field}.proof_inputs")
|
|
evidence_exact_strings(evidence.get("fail_closed_checks"),
|
|
SCANNER_HEAL_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS,
|
|
f"{gate}.{field}.fail_closed_checks")
|
|
if field == "cold_segment_reuse_measurement":
|
|
evidence_integer(evidence.get("hot_walked_segments"), f"{gate}.{field}.hot_walked_segments", 1, 2**63 - 1)
|
|
evidence_integer(evidence.get("cold_walked_segments"), f"{gate}.{field}.cold_walked_segments", 0, 0)
|
|
require(evidence.get("full_walk_oracle_equivalent") is True,
|
|
f"{gate}.{field} requires full-walk oracle equivalence")
|
|
require(evidence.get("published_root_equivalent") is True,
|
|
f"{gate}.{field} requires published-root equivalence")
|
|
if field == "post_stop_convergence_measurement":
|
|
require(evidence.get("writes_stopped") is True,
|
|
f"{gate}.{field} requires writes-stopped evidence")
|
|
require(evidence.get("last_mutation_observed") is True,
|
|
f"{gate}.{field} requires last-mutation observation")
|
|
require(evidence.get("first_complete_publication") is True,
|
|
f"{gate}.{field} requires first complete publication evidence")
|
|
samples = evidence_integer(evidence.get("post_stop_samples"), f"{gate}.{field}.post_stop_samples", 1, 2**31 - 1)
|
|
multiple = release_bundle_number(evidence.get("post_stop_work_multiple"),
|
|
f"{gate}.{field}.post_stop_work_multiple", 0)
|
|
limit = release_bundle_number(evidence.get("post_stop_work_multiple_limit"),
|
|
f"{gate}.{field}.post_stop_work_multiple_limit", 1)
|
|
require(multiple <= limit, f"{gate}.{field} exceeds post-stop work multiple limit")
|
|
raw_multiples = evidence.get("post_stop_work_multiples")
|
|
require(isinstance(raw_multiples, list) and len(raw_multiples) == samples,
|
|
f"{gate}.{field} requires measured post-stop work multiple samples")
|
|
observed = [release_bundle_number(value, f"{gate}.{field}.post_stop_work_multiples", 0)
|
|
for value in raw_multiples]
|
|
require(max(observed) == multiple, f"{gate}.{field} worst post-stop multiple mismatch")
|
|
if gate == "G07":
|
|
case_field = {
|
|
"mrf_responsibility_oracle": "mrf_responsibility_cases",
|
|
"commit_boundary_crash_matrix": "commit_crash_cases",
|
|
}[field]
|
|
cases = evidence_string_list(evidence.get(case_field), f"{gate}.{field}.{case_field}")
|
|
missing_cases = sorted(set(SCANNER_HEAL_RELEASE_G07_REQUIRED_CASES[field]) - set(cases))
|
|
require(not missing_cases, f"{gate}.{field} missing cases: {', '.join(missing_cases)}")
|
|
if gate == "G08":
|
|
case_field = {
|
|
"mrf_capacity_evidence": "capacity_cases",
|
|
"disk_full_matrix": "disk_full_cases",
|
|
"replica_loss_matrix": "replica_loss_cases",
|
|
}[field]
|
|
cases = evidence_string_list(evidence.get(case_field), f"{gate}.{field}.{case_field}")
|
|
missing_cases = sorted(set(SCANNER_HEAL_RELEASE_G08_REQUIRED_CASES[field]) - set(cases))
|
|
require(not missing_cases, f"{gate}.{field} missing cases: {', '.join(missing_cases)}")
|
|
if gate == "G14":
|
|
if field == "same_window_field_evidence":
|
|
required_fields = set(SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS["G14"]) - {field}
|
|
same_window_fields = evidence.get("same_window_fields")
|
|
require(isinstance(same_window_fields, list) and
|
|
len(set(same_window_fields)) == len(same_window_fields) and
|
|
all(isinstance(item, str) and item in required_fields for item in same_window_fields),
|
|
"G14.same_window_field_evidence requires named G14 field coverage")
|
|
missing_same_window_fields = sorted(required_fields - set(same_window_fields))
|
|
require(not missing_same_window_fields,
|
|
"G14.same_window_field_evidence missing fields: " + ", ".join(missing_same_window_fields))
|
|
if field == "ec8_4_evidence":
|
|
topology = evidence.get("topology")
|
|
require(isinstance(topology, dict), "G14.ec8_4_evidence missing topology")
|
|
require(topology.get("erasure") == "EC8+4", "G14.ec8_4_evidence must record EC8+4")
|
|
require(evidence_integer(topology.get("nodes"), "G14 topology nodes", 3, 64) >= 3,
|
|
"G14.ec8_4_evidence requires at least three nodes")
|
|
require(evidence_integer(topology.get("drives_per_node"), "G14 topology drives", 4, 64) >= 4,
|
|
"G14.ec8_4_evidence requires at least four drives per node")
|
|
if field == "multi_set_evidence":
|
|
evidence_integer(evidence.get("sets"), "G14 multi_set_evidence.sets", 2, 1024)
|
|
if field == "multi_pool_evidence":
|
|
evidence_integer(evidence.get("pools"), "G14 multi_pool_evidence.pools", 2, 1024)
|
|
if field == "distributed_segment_invalidation_evidence":
|
|
require(evidence.get("invalidation_domain") == "distributed-ec",
|
|
"G14.distributed_segment_invalidation_evidence requires distributed EC invalidation")
|
|
require(evidence.get("distributed_ec_invalidation") is True,
|
|
"G14.distributed_segment_invalidation_evidence requires peer invalidation proof")
|
|
evidence_integer(evidence.get("peer_count"), "G14 distributed_segment_invalidation_evidence.peer_count", 3, 64)
|
|
require(evidence.get("same_window_remote_proof") is True,
|
|
"G14.distributed_segment_invalidation_evidence requires same-window remote proof")
|
|
require(evidence.get("all_peers_bound_to_generation_window") is True,
|
|
"G14.distributed_segment_invalidation_evidence requires peer generation-window binding")
|
|
if field == "profile_evidence":
|
|
evidence_integer(evidence.get("resolved_samples"), f"{gate}.{field}.resolved_samples", 1, 2**63 - 1)
|
|
profile_artifacts = evidence.get("profile_artifacts")
|
|
require(isinstance(profile_artifacts, dict), f"{gate}.{field} missing profile artifacts")
|
|
missing_artifacts = sorted(set(RELEASE_PROFILE_ARTIFACTS) - set(profile_artifacts))
|
|
require(not missing_artifacts,
|
|
f"{gate}.{field} missing profile artifacts: {', '.join(missing_artifacts)}")
|
|
unknown_artifacts = sorted(set(profile_artifacts) - set(RELEASE_PROFILE_ARTIFACTS))
|
|
require(not unknown_artifacts,
|
|
f"{gate}.{field} unknown profile artifacts: {', '.join(unknown_artifacts)}")
|
|
for artifact_kind in RELEASE_PROFILE_ARTIFACTS:
|
|
item = profile_artifacts[artifact_kind]
|
|
require(isinstance(item, dict), f"{gate}.{field}.{artifact_kind} must be an object")
|
|
artifact_field = f"{field}.{artifact_kind}"
|
|
artifact_path = release_bundle_artifact_path(bundle_path, item.get("artifact"), gate, artifact_field)
|
|
require(sha(item.get("sha256")) and digest(artifact_path) == item["sha256"],
|
|
f"{gate}.{artifact_field} artifact hash mismatch")
|
|
artifact_format = evidence_string(item.get("artifact_format"), f"{gate}.{artifact_field}.artifact_format",
|
|
r"[A-Za-z0-9][A-Za-z0-9._+:-]{1,63}")
|
|
require(item.get("source_revision") == source_revision,
|
|
f"{gate}.{artifact_field} source revision mismatch")
|
|
require(item.get("run_id") == run_id, f"{gate}.{artifact_field} run_id mismatch")
|
|
require(item.get("measurement_window_id") == window_id,
|
|
f"{gate}.{artifact_field} measurement window mismatch")
|
|
if is_json_artifact_format(artifact_format):
|
|
validate_release_bundle_json_artifact_payload(
|
|
bundle_path,
|
|
artifact_path,
|
|
source_revision,
|
|
gate,
|
|
field,
|
|
run_id,
|
|
window_id,
|
|
artifact_kind,
|
|
)
|
|
if "resolved_samples" in item:
|
|
evidence_integer(item.get("resolved_samples"), f"{gate}.{artifact_field}.resolved_samples",
|
|
0, 2**63 - 1)
|
|
validate_release_bundle_domain_evidence(gate, field, evidence)
|
|
return window_id
|
|
|
|
|
|
def validate_release_bundle_gate_windows(gate: str, field_windows: dict[str, str]) -> None:
|
|
if gate == "G14" or gate.startswith("P"):
|
|
windows = sorted(set(field_windows.values()))
|
|
require(len(windows) == 1, f"{gate} evidence fields must share one measurement window")
|
|
|
|
|
|
def validate_scanner_heal_release_bundle_gate(bundle_path: Path, source_revision: str,
|
|
requirements: dict[str, dict[str, object]],
|
|
raw_gates: dict[str, object], gate: str) -> list[str]:
|
|
gate_errors: list[str] = []
|
|
requirement = requirements[gate]
|
|
gate_evidence = raw_gates.get(gate)
|
|
if not isinstance(gate_evidence, dict):
|
|
return ["missing gate evidence"]
|
|
if gate_evidence.get("status") != "pass":
|
|
gate_errors.append("gate status must be pass")
|
|
if gate_evidence.get("lane") != requirement["lane"]:
|
|
gate_errors.append("gate lane mismatch")
|
|
if gate_evidence.get("evidence_type") != "measured":
|
|
gate_errors.append("gate evidence type must be measured")
|
|
fields = gate_evidence.get("evidence_fields")
|
|
if not isinstance(fields, dict):
|
|
gate_errors.append("missing gate evidence fields")
|
|
fields = {}
|
|
required_fields = tuple(SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS[gate])
|
|
missing_fields = [field for field in required_fields if field not in fields]
|
|
if missing_fields:
|
|
gate_errors.append(f"missing required fields: {', '.join(missing_fields)}")
|
|
field_windows: dict[str, str] = {}
|
|
for field in required_fields:
|
|
if field not in fields:
|
|
continue
|
|
evidence = fields[field]
|
|
if not isinstance(evidence, dict):
|
|
gate_errors.append(f"{field} must be an object")
|
|
continue
|
|
for marker in ("fixture_only", "dry_run", "synthetic"):
|
|
if evidence.get(marker) is True:
|
|
gate_errors.append(f"{field} is {marker}")
|
|
continue
|
|
try:
|
|
field_windows[field] = validate_release_bundle_artifact(bundle_path, source_revision, gate, field, evidence)
|
|
except (OSError, KeyError, TypeError, ValueError) as error:
|
|
gate_errors.append(str(error))
|
|
if not gate_errors:
|
|
try:
|
|
validate_release_bundle_gate_windows(gate, field_windows)
|
|
except ValueError as error:
|
|
gate_errors.append(str(error))
|
|
return gate_errors
|
|
|
|
|
|
def scanner_heal_release_bundle_status(root: Path, bundle_path: Path) -> dict[str, object]:
|
|
"""Validate a complete hard-gate evidence bundle without accepting synthetic claims."""
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
requirements, release_schema_capable, pending_lanes = scanner_heal_release_requirements(registry)
|
|
require(release_schema_capable, "scanner/heal release bundle requires schema 2 registry")
|
|
bundle_path = bundle_path.resolve()
|
|
bundle = read_json(bundle_path)
|
|
for marker in ("fixture_only", "dry_run", "synthetic"):
|
|
require(bundle.get(marker) is not True, f"scanner/heal release evidence bundle is {marker}")
|
|
require(bundle.get("schema") == 1, "unsupported scanner/heal release evidence bundle schema")
|
|
require(bundle.get("evidence") == "measured", "scanner/heal release evidence bundle must be measured")
|
|
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=root, text=True).strip()
|
|
source_revision = bundle.get("source_revision")
|
|
require(isinstance(source_revision, str) and re.fullmatch(r"[0-9a-f]{40}", source_revision) is not None and
|
|
source_revision == revision,
|
|
"scanner/heal release evidence source revision mismatch")
|
|
raw_gates = bundle.get("gates")
|
|
require(isinstance(raw_gates, dict), "scanner/heal release evidence bundle missing gates")
|
|
|
|
verified: list[str] = []
|
|
rejected: dict[str, list[str]] = {}
|
|
for gate in requirements:
|
|
gate_errors = validate_scanner_heal_release_bundle_gate(bundle_path, source_revision, requirements, raw_gates, gate)
|
|
if gate_errors:
|
|
rejected[gate] = gate_errors
|
|
else:
|
|
verified.append(gate)
|
|
|
|
unknown = sorted(set(raw_gates) - set(requirements))
|
|
if unknown:
|
|
rejected["unknown"] = [f"unknown gates: {', '.join(unknown)}"]
|
|
approved = not rejected and sorted(verified) == sorted(requirements)
|
|
return {
|
|
"schema": 1,
|
|
"decision": "approved" if approved else "blocked",
|
|
"release_approved": approved,
|
|
"release_schema_capable": release_schema_capable,
|
|
"verified_gates": sorted(verified),
|
|
"rejected_gates": rejected,
|
|
"pending_gates": [] if approved else sorted(gate for gate in requirements if gate not in verified),
|
|
"pending_lanes": [] if approved else pending_lanes,
|
|
}
|
|
|
|
|
|
def scanner_heal_release_bundle_gate_status(root: Path, bundle_path: Path, gate: str) -> dict[str, object]:
|
|
"""Validate one measured release-bundle gate without approving the full release."""
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
requirements, release_schema_capable, pending_lanes = scanner_heal_release_requirements(registry)
|
|
require(release_schema_capable, "scanner/heal release bundle gate requires schema 2 registry")
|
|
require(gate in requirements, f"unknown scanner/heal release gate: {gate}")
|
|
bundle_path = bundle_path.resolve()
|
|
bundle = read_json(bundle_path)
|
|
for marker in ("fixture_only", "dry_run", "synthetic"):
|
|
require(bundle.get(marker) is not True, f"scanner/heal release evidence bundle is {marker}")
|
|
require(bundle.get("schema") == 1, "unsupported scanner/heal release evidence bundle schema")
|
|
require(bundle.get("evidence") == "measured", "scanner/heal release evidence bundle must be measured")
|
|
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=root, text=True).strip()
|
|
source_revision = bundle.get("source_revision")
|
|
require(isinstance(source_revision, str) and re.fullmatch(r"[0-9a-f]{40}", source_revision) is not None and
|
|
source_revision == revision,
|
|
"scanner/heal release evidence source revision mismatch")
|
|
raw_gates = bundle.get("gates")
|
|
require(isinstance(raw_gates, dict), "scanner/heal release evidence bundle missing gates")
|
|
unknown = sorted(set(raw_gates) - set(requirements))
|
|
require(not unknown, f"scanner/heal release evidence bundle has unknown gates: {', '.join(unknown)}")
|
|
|
|
gate_errors = validate_scanner_heal_release_bundle_gate(bundle_path, source_revision, requirements, raw_gates, gate)
|
|
verified = not gate_errors
|
|
return {
|
|
"schema": 1,
|
|
"decision": "verified" if verified else "blocked",
|
|
"release_approved": False,
|
|
"release_schema_capable": release_schema_capable,
|
|
"verified_gate": gate if verified else None,
|
|
"rejected_gate": None if verified else gate,
|
|
"rejected_errors": gate_errors,
|
|
"pending_gates": sorted(item for item in requirements if item != gate),
|
|
"pending_lanes": pending_lanes,
|
|
}
|
|
|
|
|
|
def release_bundle_descriptor_path(descriptor_path: Path, raw_path: object, gate: str, field: str) -> Path:
|
|
require(isinstance(raw_path, str) and raw_path.strip(), f"{gate}.{field} missing source artifact")
|
|
path = Path(raw_path)
|
|
resolved = path if path.is_absolute() else descriptor_path.parent / path
|
|
resolved = resolved.resolve(strict=True)
|
|
require(resolved.is_file(), f"{gate}.{field} source artifact is missing")
|
|
require(resolved.stat().st_size > 0, f"{gate}.{field} source artifact is empty")
|
|
return resolved
|
|
|
|
|
|
def release_bundle_artifact_suffix(path: Path) -> str:
|
|
suffix = path.suffix
|
|
if re.fullmatch(r"\.[A-Za-z0-9._+-]{1,16}", suffix or "") is not None:
|
|
return suffix
|
|
return ".artifact"
|
|
|
|
|
|
def reject_release_bundle_markers(item: dict[str, object], label: str) -> None:
|
|
for marker in ("fixture_only", "dry_run", "synthetic"):
|
|
require(item.get(marker) is not True, f"{label} is {marker}")
|
|
|
|
|
|
def copy_release_bundle_artifact(descriptor_path: Path, bundle_dir: Path, evidence: dict[str, object],
|
|
gate: str, field: str, label: str) -> None:
|
|
source = release_bundle_descriptor_path(descriptor_path, evidence.get("artifact"), gate, field)
|
|
source_sha = digest(source)
|
|
if "sha256" in evidence:
|
|
require(sha(evidence.get("sha256")) and evidence["sha256"] == source_sha,
|
|
f"{gate}.{field} descriptor artifact hash mismatch")
|
|
artifact_dir = bundle_dir / "artifacts"
|
|
safe_label = re.sub(r"[^A-Za-z0-9._-]", "-", label)
|
|
target = artifact_dir / f"{safe_label}{release_bundle_artifact_suffix(source)}"
|
|
require(not target.exists(), f"{gate}.{field} duplicate assembled artifact path")
|
|
shutil.copyfile(source, target)
|
|
require(digest(target) == source_sha, f"{gate}.{field} assembled artifact hash mismatch")
|
|
evidence["artifact"] = target.relative_to(bundle_dir).as_posix()
|
|
evidence["sha256"] = source_sha
|
|
|
|
|
|
def copy_release_bundle_profile_raw_artifact(descriptor_path: Path, bundle_dir: Path, item: dict[str, object],
|
|
gate: str, field: str, artifact_kind: str) -> None:
|
|
wrapper = release_bundle_artifact_path(bundle_dir / "release-evidence.json", item.get("artifact"), gate, field)
|
|
payload = read_json(wrapper)
|
|
raw_source = release_bundle_descriptor_path(
|
|
descriptor_path,
|
|
payload.get("raw_profile_artifact"),
|
|
gate,
|
|
f"{field}.raw_profile",
|
|
)
|
|
raw_sha = digest(raw_source)
|
|
require(sha(payload.get("raw_profile_sha256")) and payload["raw_profile_sha256"] == raw_sha,
|
|
f"{gate}.{field} descriptor raw profile hash mismatch")
|
|
raw_bytes = evidence_integer(payload.get("raw_profile_bytes"), f"{gate}.{field}.raw_profile_bytes", 1, 2**63 - 1)
|
|
require(raw_bytes == raw_source.stat().st_size, f"{gate}.{field} descriptor raw profile bytes mismatch")
|
|
artifact_dir = bundle_dir / "artifacts"
|
|
safe_label = re.sub(r"[^A-Za-z0-9._-]", "-", f"{gate}-{field}-{artifact_kind}-raw")
|
|
raw_target = artifact_dir / f"{safe_label}{release_bundle_artifact_suffix(raw_source)}"
|
|
require(not raw_target.exists(), f"{gate}.{field} duplicate assembled raw profile path")
|
|
shutil.copyfile(raw_source, raw_target)
|
|
require(digest(raw_target) == raw_sha, f"{gate}.{field} assembled raw profile hash mismatch")
|
|
payload["raw_profile_artifact"] = raw_target.relative_to(bundle_dir).as_posix()
|
|
payload["raw_profile_sha256"] = raw_sha
|
|
payload["raw_profile_bytes"] = raw_target.stat().st_size
|
|
write_json(wrapper, payload)
|
|
item["sha256"] = digest(wrapper)
|
|
|
|
|
|
def copy_release_bundle_g14_case_artifacts(descriptor_path: Path, bundle_dir: Path, evidence: dict[str, object],
|
|
gate: str, field: str) -> None:
|
|
wrapper = release_bundle_artifact_path(bundle_dir / "release-evidence.json", evidence.get("artifact"), gate, field)
|
|
payload = read_json(wrapper)
|
|
cases = payload.get("case_evidence")
|
|
require(isinstance(cases, list), f"{gate}.{field} descriptor case_evidence must be a list")
|
|
artifact_dir = bundle_dir / "artifacts"
|
|
for index, item in enumerate(cases):
|
|
require(isinstance(item, dict), f"{gate}.{field} descriptor case_evidence[{index}] must be an object")
|
|
case = evidence_string(
|
|
item.get("case"),
|
|
f"{gate}.{field}.case_evidence[{index}].case",
|
|
r"[A-Za-z0-9][A-Za-z0-9._:-]{1,127}",
|
|
)
|
|
source = release_bundle_descriptor_path(
|
|
descriptor_path,
|
|
item.get("artifact"),
|
|
gate,
|
|
f"{field}.case_evidence.{case}",
|
|
)
|
|
source_sha = digest(source)
|
|
require(sha(item.get("sha256")) and item["sha256"] == source_sha,
|
|
f"{gate}.{field} descriptor case_evidence[{index}] artifact hash mismatch")
|
|
safe_case = re.sub(r"[^A-Za-z0-9._-]", "-", case)
|
|
target = artifact_dir / f"{gate}-{field}-case-{safe_case}{release_bundle_artifact_suffix(source)}"
|
|
require(not target.exists(), f"{gate}.{field} duplicate assembled case evidence path")
|
|
shutil.copyfile(source, target)
|
|
require(digest(target) == source_sha, f"{gate}.{field} assembled case evidence hash mismatch")
|
|
item["artifact"] = target.relative_to(bundle_dir).as_posix()
|
|
item["sha256"] = source_sha
|
|
write_json(wrapper, payload)
|
|
evidence["case_evidence"] = payload["case_evidence"]
|
|
evidence["sha256"] = digest(wrapper)
|
|
|
|
|
|
def assemble_scanner_heal_release_bundle_descriptors(
|
|
root: Path,
|
|
descriptor_paths: list[Path],
|
|
directory: Path,
|
|
) -> tuple[Path, dict[str, object]]:
|
|
"""Copy measured Scanner/Heal evidence descriptors into one validated release bundle."""
|
|
require(not directory.exists(), "scanner/heal release bundle directory must be new")
|
|
require(descriptor_paths, "scanner/heal release bundle assembly requires at least one descriptor")
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
requirements, release_schema_capable, _ = scanner_heal_release_requirements(registry)
|
|
require(release_schema_capable, "scanner/heal release bundle assembly requires schema 2 registry")
|
|
|
|
bundle_dir = directory.resolve()
|
|
artifact_dir = bundle_dir / "artifacts"
|
|
artifact_dir.mkdir(parents=True)
|
|
assembled_gates = {}
|
|
source_revision = None
|
|
for descriptor_index, raw_descriptor_path in enumerate(descriptor_paths):
|
|
descriptor_path = raw_descriptor_path.resolve(strict=True)
|
|
descriptor = read_json(descriptor_path)
|
|
label = f"scanner/heal release evidence descriptor {descriptor_index + 1}"
|
|
reject_release_bundle_markers(descriptor, label)
|
|
require(descriptor.get("schema") == 1, f"unsupported {label} schema")
|
|
require(descriptor.get("evidence") == "measured", f"{label} must be measured")
|
|
descriptor_revision = descriptor.get("source_revision")
|
|
require(isinstance(descriptor_revision, str) and re.fullmatch(r"[0-9a-f]{40}", descriptor_revision) is not None,
|
|
f"{label} source revision is invalid")
|
|
if source_revision is None:
|
|
source_revision = descriptor_revision
|
|
else:
|
|
require(descriptor_revision == source_revision, "scanner/heal release descriptor source revisions differ")
|
|
raw_gates = descriptor.get("gates")
|
|
require(isinstance(raw_gates, dict), f"{label} missing gates")
|
|
unknown_gates = sorted(set(raw_gates) - set(requirements))
|
|
require(not unknown_gates, f"{label} has unknown gates: {', '.join(unknown_gates)}")
|
|
|
|
for gate in sorted(raw_gates):
|
|
require(gate not in assembled_gates, f"scanner/heal release descriptor repeats gate: {gate}")
|
|
raw_gate = raw_gates[gate]
|
|
require(isinstance(raw_gate, dict), f"{gate} descriptor gate must be an object")
|
|
reject_release_bundle_markers(raw_gate, f"{gate} descriptor gate")
|
|
raw_fields = raw_gate.get("evidence_fields", raw_gate)
|
|
require(isinstance(raw_fields, dict), f"{gate} descriptor gate missing evidence fields")
|
|
required_fields = set(SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS[gate])
|
|
unknown_fields = sorted(set(raw_fields) - required_fields)
|
|
require(not unknown_fields, f"{gate} descriptor has unknown fields: {', '.join(unknown_fields)}")
|
|
fields = {}
|
|
for field in sorted(raw_fields):
|
|
raw_evidence = raw_fields[field]
|
|
require(isinstance(raw_evidence, dict), f"{gate}.{field} descriptor evidence must be an object")
|
|
evidence = json.loads(json.dumps(raw_evidence))
|
|
reject_release_bundle_markers(evidence, f"{gate}.{field} descriptor evidence")
|
|
copy_release_bundle_artifact(descriptor_path, bundle_dir, evidence, gate, field, f"{gate}-{field}")
|
|
if gate == "G14":
|
|
copy_release_bundle_g14_case_artifacts(descriptor_path, bundle_dir, evidence, gate, field)
|
|
profile_artifacts = evidence.get("profile_artifacts")
|
|
if profile_artifacts is not None:
|
|
require(isinstance(profile_artifacts, dict), f"{gate}.{field} profile artifacts must be an object")
|
|
for artifact_kind, item in sorted(profile_artifacts.items()):
|
|
require(isinstance(item, dict), f"{gate}.{field}.{artifact_kind} descriptor profile artifact must be an object")
|
|
reject_release_bundle_markers(item, f"{gate}.{field}.{artifact_kind} descriptor profile artifact")
|
|
copy_release_bundle_artifact(
|
|
descriptor_path,
|
|
bundle_dir,
|
|
item,
|
|
gate,
|
|
f"{field}.{artifact_kind}",
|
|
f"{gate}-{field}-{artifact_kind}",
|
|
)
|
|
if field == "profile_evidence":
|
|
copy_release_bundle_profile_raw_artifact(
|
|
descriptor_path,
|
|
bundle_dir,
|
|
item,
|
|
gate,
|
|
f"{field}.{artifact_kind}",
|
|
artifact_kind,
|
|
)
|
|
fields[field] = evidence
|
|
assembled_gates[gate] = {
|
|
"status": "pass",
|
|
"lane": requirements[gate]["lane"],
|
|
"evidence_type": "measured",
|
|
"evidence_fields": fields,
|
|
}
|
|
|
|
bundle = bundle_dir / "release-evidence.json"
|
|
write_json(bundle, {
|
|
"schema": 1,
|
|
"evidence": "measured",
|
|
"source_revision": source_revision,
|
|
"gates": assembled_gates,
|
|
})
|
|
return bundle, scanner_heal_release_bundle_status(root, bundle)
|
|
|
|
|
|
def assemble_scanner_heal_release_bundle(root: Path, descriptor_path: Path, directory: Path) -> tuple[Path, dict[str, object]]:
|
|
"""Copy one measured Scanner/Heal evidence descriptor into a validated release bundle."""
|
|
return assemble_scanner_heal_release_bundle_descriptors(root, [descriptor_path], directory)
|
|
|
|
|
|
def write_scanner_heal_release_bundle_fixture(root: Path, directory: Path) -> Path:
|
|
"""Write a non-approvable release bundle shape fixture for dry-run validation."""
|
|
bundle_dir = directory.resolve()
|
|
bundle_dir.mkdir(parents=True, exist_ok=True)
|
|
artifact_dir = bundle_dir / "artifacts"
|
|
artifact_dir.mkdir(parents=True, exist_ok=True)
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
requirements, release_schema_capable, _ = scanner_heal_release_requirements(registry)
|
|
require(release_schema_capable, "scanner/heal release bundle fixture requires schema 2 registry")
|
|
source_revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=root, text=True).strip()
|
|
baseline_revision = "0" * 39 + "1"
|
|
require(baseline_revision != source_revision, "scanner/heal release bundle fixture needs a distinct baseline")
|
|
started = datetime.now(timezone.utc).replace(microsecond=0)
|
|
gates = {}
|
|
for gate, requirement in requirements.items():
|
|
fields = {}
|
|
for field in SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS[gate]:
|
|
artifact = artifact_dir / f"{gate}-{field}.json"
|
|
write_json(artifact, {
|
|
"schema": 1,
|
|
"fixture_only": True,
|
|
"gate": gate,
|
|
"field": field,
|
|
"source_revision": source_revision,
|
|
})
|
|
duration = 60
|
|
evidence = {
|
|
"fixture_only": True,
|
|
"artifact": artifact.relative_to(bundle_dir).as_posix(),
|
|
"sha256": digest(artifact),
|
|
"evidence_type": "measured",
|
|
"source_revision": source_revision,
|
|
"run_id": f"{gate.lower()}-{field.replace('_', '-')}-fixture-run",
|
|
"measurement_window_id": f"{gate.lower()}-fixture-window",
|
|
"started_at": started.isoformat().replace("+00:00", "Z"),
|
|
"command": ["fixture-only", "scanner-heal-release-bundle", gate, field],
|
|
"artifact_format": "json",
|
|
"summary": f"fixture-only parser sample for {gate}.{field}; not release evidence",
|
|
}
|
|
if gate.startswith("P"):
|
|
duration = 900
|
|
evidence["duration_seconds"] = duration
|
|
if gate == "P3" and field == "two_hour_pressure_measurement":
|
|
duration = 7200
|
|
evidence["duration_seconds"] = duration
|
|
evidence["finished_at"] = (started + timedelta(seconds=duration)).isoformat().replace("+00:00", "Z")
|
|
if gate == "G10" and field == "scheduler_bound_evidence":
|
|
evidence.update({
|
|
"max_deferred_items": 256,
|
|
"max_deferred_bytes": 262144,
|
|
"max_retry_age_seconds": 30,
|
|
"duplicate_task_count": 0,
|
|
})
|
|
if gate == "G10" and field == "pressure_recovery_evidence":
|
|
evidence.update({
|
|
"pressure_pacing_engaged": True,
|
|
"recovery_window_seconds": 5,
|
|
"lock_hold_p95_ms": 0,
|
|
"foreground_latency_p95_ms": 10,
|
|
})
|
|
if gate == "P1" and field == "foreground_latency_throughput_measurement":
|
|
evidence.update({
|
|
"foreground_latency_p95_ms": 10,
|
|
"foreground_latency_p99_ms": 20,
|
|
"throughput_ops_per_second": 100,
|
|
"error_count": 0,
|
|
})
|
|
if gate == "P3" and field == "two_hour_pressure_measurement":
|
|
evidence.update({
|
|
"fixed_offered_load": True,
|
|
"foreground_latency_p99_ms": 20,
|
|
"attempt_cost_samples": 1,
|
|
})
|
|
if gate == "P3" and field == "heal_capacity_measurement":
|
|
evidence.update({"completed_heal_objects": 1, "duplicate_task_count": 0})
|
|
if gate == "P3" and field == "recovery_window_measurement":
|
|
evidence.update({"pressure_recovery_window_seconds": 5, "lock_hold_p95_ms": 0})
|
|
if gate == "G02" and field == "bounded_checkpoint_oracle":
|
|
evidence.update({
|
|
"checkpoint_progress_bounded": True,
|
|
"raw_entry_budget": 8,
|
|
"max_raw_entries_per_round": 8,
|
|
"max_objects_processed_per_round": 8,
|
|
"bounded_work_quantum_observed": True,
|
|
"durable_checkpoint_committed": True,
|
|
"no_unbounded_tail": True,
|
|
})
|
|
if gate == "G02" and field == "independent_version_inventory":
|
|
evidence.update({
|
|
"independent_version_inventory_observed": True,
|
|
"objects_expected": 16,
|
|
"objects_retained": 16,
|
|
"versions_retained": 16,
|
|
"bytes_retained": 16,
|
|
})
|
|
if gate == "R-E" and field == "fixed_budget_restart_evidence":
|
|
evidence.update({
|
|
"fixed_budget_restart_converged": True,
|
|
"restart_rounds": 3,
|
|
"raw_entry_budget": 8,
|
|
"no_unbudgeted_final_sweep": True,
|
|
})
|
|
if gate == "R-E" and field == "enumeration_evidence":
|
|
evidence.update({
|
|
"raw_enumeration_observed": True,
|
|
"durable_raw_page_commit_observed": True,
|
|
"raw_page_index_complete": True,
|
|
"enumeration_frontier_retained": True,
|
|
})
|
|
if gate == "R-E" and field == "classification_evidence":
|
|
evidence.update({
|
|
"classification_observed": True,
|
|
"objects_processed": 16,
|
|
"objects_retained": 16,
|
|
"versions_retained": 16,
|
|
"bytes_retained": 16,
|
|
})
|
|
if gate in ("G03", "G09", "R-L"):
|
|
evidence["versions"] = [baseline_revision, source_revision]
|
|
evidence["mixed_version_role"] = SCANNER_HEAL_RELEASE_MIXED_VERSION_ROLES[(gate, field)]
|
|
if gate == "G03":
|
|
evidence["scoped_ack_cases"] = list(SCANNER_HEAL_RELEASE_SCOPED_ACK_CASES[field])
|
|
for required in SCANNER_HEAL_RELEASE_G03_REQUIRED_TRUE_FIELDS[field]:
|
|
evidence[required] = True
|
|
if gate in ("G04", "G07", "R-E", "R-L"):
|
|
evidence["crash_points"] = ["fixture-before-commit"]
|
|
if gate == "G12":
|
|
evidence["quota_path_cases"] = list(SCANNER_HEAL_RELEASE_G12_QUOTA_PATH_CASES[field])
|
|
if (gate, field) in SCANNER_HEAL_RELEASE_MRF_DURABLE_REPLAY_FIELDS:
|
|
evidence["replayed_records"] = 1
|
|
evidence["responsibility_anchor_retained"] = True
|
|
evidence["successor_snapshot_published"] = True
|
|
if gate == "G14" and field == "ec8_4_evidence":
|
|
evidence["topology"] = {"erasure": "EC8+4", "nodes": 3, "drives_per_node": 4}
|
|
if gate == "G14" and field == "same_window_field_evidence":
|
|
evidence["same_window_fields"] = [
|
|
item
|
|
for item in SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS["G14"]
|
|
if item != "same_window_field_evidence"
|
|
]
|
|
if gate == "G14" and field == "multi_set_evidence":
|
|
evidence["sets"] = 2
|
|
if gate == "G14" and field == "multi_pool_evidence":
|
|
evidence["pools"] = 2
|
|
if field == "profile_evidence":
|
|
evidence["resolved_samples"] = 1
|
|
artifacts = {}
|
|
for artifact_kind in RELEASE_PROFILE_ARTIFACTS:
|
|
profile_artifact = artifact_dir / f"{gate}-{field}-{artifact_kind}.json"
|
|
write_json(profile_artifact, {
|
|
"schema": 1,
|
|
"fixture_only": True,
|
|
"gate": gate,
|
|
"field": field,
|
|
"artifact": artifact_kind,
|
|
"source_revision": source_revision,
|
|
})
|
|
artifacts[artifact_kind] = {
|
|
"artifact": profile_artifact.relative_to(bundle_dir).as_posix(),
|
|
"sha256": digest(profile_artifact),
|
|
"artifact_format": "json",
|
|
"source_revision": source_revision,
|
|
"run_id": evidence["run_id"],
|
|
"measurement_window_id": evidence["measurement_window_id"],
|
|
}
|
|
evidence["profile_artifacts"] = artifacts
|
|
fields[field] = evidence
|
|
gates[gate] = {
|
|
"status": "pass",
|
|
"lane": requirement["lane"],
|
|
"evidence_type": "measured",
|
|
"evidence_fields": fields,
|
|
}
|
|
bundle = bundle_dir / "release-evidence.fixture.json"
|
|
write_json(bundle, {
|
|
"schema": 1,
|
|
"fixture_only": True,
|
|
"evidence": "measured",
|
|
"source_revision": source_revision,
|
|
"gates": gates,
|
|
})
|
|
return bundle
|
|
|
|
|
|
def validate(root: Path) -> list[str]:
|
|
errors: list[str] = []
|
|
errors.extend(check_core_fixtures(root))
|
|
errors.extend(check_e2e_modules(root))
|
|
errors.extend(check_vault_test_groups(root))
|
|
errors.extend(check_ilm_build_budget(root))
|
|
errors.extend(check_fuzz_targets(root))
|
|
errors.extend(check_runner_selection(root))
|
|
errors.extend(check_s3_tests_runner(root))
|
|
errors.extend(check_workflow_readiness(root))
|
|
errors.extend(check_profile_definitions(root))
|
|
errors.extend(check_scheduled_alerts(root))
|
|
errors.extend(check_quick_checks(root))
|
|
return errors
|
|
|
|
|
|
class SelfTests(unittest.TestCase):
|
|
def test_quick_checks_rejects_caller_and_execution_bypasses(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
caller = (
|
|
"jobs:\n quick-checks:\n steps:\n"
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
" with:\n persist-credentials: false\n"
|
|
" - uses: ./.github/actions/quick-checks\n"
|
|
)
|
|
action = (
|
|
"runs:\n using: composite\n steps:\n"
|
|
" - uses: taiki-e/install-action@pinned\n"
|
|
" with:\n tool: actionlint@1.7.12\n"
|
|
" - name: Lint workflows\n shell: bash\n run: shellcheck --version && actionlint\n"
|
|
" - name: Error format ratchet\n shell: bash\n"
|
|
" run: ./scripts/check_error_other_format_ratchet.sh\n"
|
|
" - name: Script tests\n shell: bash\n run: make script-tests\n"
|
|
)
|
|
sources = {
|
|
".github/workflows/ci.yml": caller.replace(
|
|
" steps:", " if: github.event_name != 'pull_request' || github.event.action != 'closed'\n steps:"
|
|
),
|
|
".github/actions/quick-checks/action.yml": action,
|
|
}
|
|
for relative, source in sources.items():
|
|
path = root / relative
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
path.write_text(source)
|
|
self.assertEqual(check_quick_checks(root), [])
|
|
for relative in (".github/workflows/ci.yml",):
|
|
source = sources[relative]
|
|
mutations = {
|
|
"different action": source.replace("./.github/actions/quick-checks", "./.github/actions/other"),
|
|
"conditional call": source + " if: false\n",
|
|
"ignored call failure": source + " continue-on-error: true\n",
|
|
"conditional checkout": source.replace(" with:", " if: false\n with:"),
|
|
"ignored job failure": source.replace(" steps:", " continue-on-error: true\n steps:"),
|
|
"changed job condition": (
|
|
source.replace("github.event_name != 'pull_request' || github.event.action != 'closed'", "false")
|
|
if relative.endswith("/ci.yml") else source.replace(" steps:", " if: false\n steps:")
|
|
),
|
|
"persisted credentials": source.replace("persist-credentials: false", "persist-credentials: true"),
|
|
"late checkout": source.replace(" - uses: ./.github/actions/quick-checks\n", "").replace(
|
|
" steps:\n", " steps:\n - uses: ./.github/actions/quick-checks\n"
|
|
),
|
|
"missing job": source.replace(" quick-checks:", " other-checks:"),
|
|
}
|
|
for key in ("'if' : false", '"if": false', "'continue-on-error': true", '"continue-on-error" : true'):
|
|
mutations[f"quoted call {key}"] = source + f" {key}\n"
|
|
mutations[f"quoted checkout {key}"] = source.replace(" with:", f" {key}\n with:")
|
|
job_source = source.replace(
|
|
" if: github.event_name != 'pull_request' || github.event.action != 'closed'\n", ""
|
|
) if "if" in key else source
|
|
mutations[f"quoted job {key}"] = job_source.replace(" steps:", f" {key}\n steps:")
|
|
for dependency in ("needs: prerequisite", "needs: [prerequisite]", "needs:\n - prerequisite", "'needs' : [prerequisite]", '"needs": [prerequisite]'):
|
|
for condition in ("false", "true"):
|
|
prerequisite = f"\n prerequisite:\n if: {condition}\n runs-on: ubuntu-latest\n steps:\n - run: exit 1\n"
|
|
mutations[f"job dependency {dependency} if {condition}"] = source.replace(" steps:", f" {dependency}\n steps:") + prerequisite
|
|
if relative.endswith("/ci.yml"):
|
|
for separator in ("", "\n", " # continued condition\n"):
|
|
mutations[f"continued job condition {separator!r}"] = source.replace(
|
|
" steps:", f"{separator} && false\n steps:"
|
|
)
|
|
for case, mutated in mutations.items():
|
|
with self.subTest(path=relative, case=case):
|
|
(root / relative).write_text(mutated)
|
|
self.assertTrue(check_quick_checks(root))
|
|
(root / relative).write_text(source)
|
|
relative = ".github/actions/quick-checks/action.yml"
|
|
mutations = {
|
|
"not composite": action.replace("using: composite", "using: node24"),
|
|
"only installed actionlint": action.replace("run: shellcheck --version && actionlint", "run: echo actionlint"),
|
|
"missing shellcheck preflight": action.replace("shellcheck --version && ", ""),
|
|
"missing ratchet": action.replace("run: ./scripts/check_error_other_format_ratchet.sh", "run: echo skipped"),
|
|
"missing script tests": action.replace("run: make script-tests", "run: echo skipped"),
|
|
"swallowed script failure": action.replace("run: make script-tests", "run: make script-tests || true"),
|
|
"swallowed lint failure": action.replace("&& actionlint", "&& actionlint || true"),
|
|
"swallowed ratchet failure": action.replace("ratchet.sh", "ratchet.sh || true"),
|
|
"conditional lint": action.replace("run: shellcheck", "if: false\n run: shellcheck"),
|
|
"ignored ratchet failure": action.replace("run: ./scripts/", "continue-on-error: true\n run: ./scripts/"),
|
|
"non-failing shell": action.replace("shell: bash", "shell: bash {0}"),
|
|
"run text in step name": action.replace(
|
|
"name: Lint workflows", "name: |\n run: shellcheck --version && actionlint"
|
|
).replace("\n run: shellcheck --version && actionlint\n", "\n run: shellcheck --version && actionlint\n || true\n"),
|
|
}
|
|
for command in ("shellcheck --version && actionlint", "./scripts/check_error_other_format_ratchet.sh", "make script-tests"):
|
|
for key in ("'if' : false", '"if": false', "'continue-on-error': true", '"continue-on-error" : true'):
|
|
mutations[f"quoted {command} {key}"] = action.replace(f"run: {command}", f"{key}\n run: {command}")
|
|
for separator in ("", "\n", " # continued command\n"):
|
|
mutations[f"continued {command} {separator!r}"] = action.replace(
|
|
f"run: {command}\n", f"run: {command}\n{separator} || true\n"
|
|
)
|
|
for case, mutated in mutations.items():
|
|
with self.subTest(case=case):
|
|
(root / relative).write_text(mutated)
|
|
self.assertTrue(check_quick_checks(root))
|
|
(root / relative).unlink()
|
|
self.assertTrue(check_quick_checks(root))
|
|
|
|
def test_quick_checks_commands_propagate_failure(self) -> None:
|
|
runs = yaml_block((ROOT / ".github/actions/quick-checks/action.yml").read_text().splitlines(), "runs", 0)
|
|
steps = yaml_block(runs or [], "steps", 2) or []
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
(root / "scripts").mkdir()
|
|
commands = ("shellcheck", "actionlint", "./scripts/check_error_other_format_ratchet.sh")
|
|
(root / "Makefile").write_text(".PHONY: script-tests\nscript-tests:\n\texit 17\n")
|
|
for failing in (*commands, "make script-tests"):
|
|
with self.subTest(command=failing):
|
|
run = "shellcheck --version && actionlint" if failing in ("shellcheck", "actionlint") else failing
|
|
step = workflow_step_block(steps, run, key="run", indent=4)
|
|
self.assertIsNotNone(step)
|
|
run_index = next(index for index, line in enumerate(step[1]) if line.startswith(" run:"))
|
|
self.assertFalse(yaml_scalar_continues(step[1], run_index, 6))
|
|
body = step[1][run_index].removeprefix(" run: ")
|
|
for command in commands:
|
|
shim = root / command
|
|
shim.write_text(f"#!/bin/sh\nexit {17 if command == failing else 0}\n")
|
|
shim.chmod(0o755)
|
|
result = subprocess.run(
|
|
["bash", "--noprofile", "--norc", "-e", "-o", "pipefail", "-c", body],
|
|
cwd=root, env=dict(os.environ, PATH=f"{root}{os.pathsep}{os.environ['PATH']}"),
|
|
capture_output=True, text=True,
|
|
)
|
|
self.assertEqual(result.returncode, 2 if failing == "make script-tests" else 17, result.stderr)
|
|
|
|
def test_validate_includes_quick_checks(self) -> None:
|
|
error = "Quick Checks wiring regression"
|
|
with mock.patch(__name__ + ".check_quick_checks", return_value=[error]):
|
|
self.assertIn(error, validate(ROOT))
|
|
|
|
def scanner_heal_fixture(self, directory: Path) -> tuple[Path, Path]:
|
|
"""Parser fixtures only; these files are never runtime evidence."""
|
|
root, run_dir = directory / "repo", directory / "run"
|
|
(root / ".config").mkdir(parents=True)
|
|
run_dir.mkdir()
|
|
registry = read_json(ROOT / ".config/scanner-heal-required-tests.json")
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
requirements = registry["cases"]
|
|
binary = directory / "fake-binary"
|
|
binary.write_bytes(b"parser fixture, not a real build")
|
|
binary.chmod(0o700)
|
|
build = {"path": str(binary), "sha256": digest(binary)}
|
|
write_json(run_dir / "run.json", {"schema": 1, "run_id": "a" * 32, "source_revision": "b" * 40,
|
|
"binary_source_revision": "b" * 40,
|
|
"test_build": {"source_revision": "b" * 40, "dirty": False,
|
|
"lock_blob": "c" * 40, "features": "default"},
|
|
"started_at": datetime.now(timezone.utc).timestamp() - 1,
|
|
"binary": build, "test_binary": build})
|
|
suite = "e2e_test"
|
|
write_json(run_dir / "listing.json", {"rust-suites": {suite: {
|
|
"binary-id": suite, "binary-path": str(binary), "package-name": "e2e_test", "build-platform": "target",
|
|
"testcases": {
|
|
requirement["name"]: {"ignored": False, "filter-match": {"status": "matches"}}
|
|
for requirement in requirements.values()
|
|
}
|
|
}}})
|
|
(run_dir / "junit.xml").write_text(
|
|
"<testsuites><testsuite>"
|
|
+ "".join(
|
|
f'<testcase name="{requirement["name"]}" classname="{requirement["suite"]}" '
|
|
f'timestamp="{datetime.now(timezone.utc).isoformat(timespec="milliseconds")}"/>'
|
|
for requirement in requirements.values()
|
|
)
|
|
+ "</testsuite></testsuites>"
|
|
)
|
|
def oracle_objects(requirement: dict[str, object]) -> list[dict[str, object]]:
|
|
topology = requirement["topology"]
|
|
total_blocks = topology["nodes"] * topology["drives_per_node"]
|
|
erasure_set_drive_count = requirement.get("erasure_set_drive_count", total_blocks)
|
|
erasure = requirement.get("erasure")
|
|
if erasure is None:
|
|
parity_blocks = 4 if erasure_set_drive_count == 12 else erasure_set_drive_count // 2
|
|
data_blocks = erasure_set_drive_count - parity_blocks
|
|
else:
|
|
data_blocks = erasure["data_blocks"]
|
|
parity_blocks = erasure["parity_blocks"]
|
|
physical = {"has_xl_meta": True, "version_id": None, "data_dir": "data-generation",
|
|
"erasure_index": 1, "data_blocks": data_blocks, "parity_blocks": parity_blocks,
|
|
"expected_part_numbers": [1],
|
|
"present_part_fingerprints": {"1": {"size": 12, "sha256": "c" * 64}},
|
|
"inline_data_fingerprint": None}
|
|
obj = {"key": "object", "version_id": None, "expected_bytes": 16, "actual_bytes": 16,
|
|
"expected_sha256": "d" * 64, "actual_sha256": "d" * 64,
|
|
"expected_physical": physical, "physical": physical}
|
|
count = requirement.get("min_objects", 9)
|
|
objects = [dict(obj, key=f"object-{index}") for index in range(count)]
|
|
objects[-1] = dict(objects[-1], expected_physical=None)
|
|
return objects
|
|
|
|
for case_id, requirement in requirements.items():
|
|
objects = oracle_objects(requirement)
|
|
write_json(run_dir / requirement["oracle"], {
|
|
"schema": 1, "evidence": requirement["evidence"], "case": case_id,
|
|
"run_id": "a" * 32, "source_revision": "b" * 40,
|
|
"test_build": {"source_revision": "b" * 40, "dirty": False, "lock_blob": "c" * 40,
|
|
"features": "default", "target": "aarch64-apple-darwin", "profile": "debug", "rustflags_hex": ""},
|
|
"binary_sha256": build["sha256"], "test_binary_sha256": build["sha256"],
|
|
"topology": requirement["topology"],
|
|
"erasure_set_drive_count": requirement.get(
|
|
"erasure_set_drive_count",
|
|
requirement["topology"]["nodes"] * requirement["topology"]["drives_per_node"],
|
|
),
|
|
"sets": requirement.get("sets", 1),
|
|
"pools": requirement.get("pools", 1),
|
|
"outage_target_manifest_required": requirement.get("outage_target_manifest_required", True),
|
|
"distributed_ec_invalidation": True,
|
|
"peer_count": requirement["topology"]["nodes"],
|
|
"same_window_remote_proof": True,
|
|
"all_peers_bound_to_generation_window": True,
|
|
"pid_before": 10, "pid_after": 11,
|
|
"unclean_shutdown_marker": requirement["unclean_shutdown_marker"],
|
|
"objects": objects, "node_listings": [[item["key"] for item in objects]] * requirement["topology"]["nodes"],
|
|
})
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
return root, run_dir
|
|
|
|
def test_scanner_heal_case_lane_is_required_for_runner_profile(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, _ = self.scanner_heal_fixture(Path(tmp))
|
|
registry_path = root / ".config/scanner-heal-required-tests.json"
|
|
registry = read_json(registry_path)
|
|
del registry["cases"]["ec84-target-drive-restart"]["lane"]
|
|
write_json(registry_path, registry)
|
|
with self.assertRaisesRegex(ValueError, "invalid nextest profile lane for ec84-target-drive-restart"):
|
|
scanner_heal_oracle_names(root)
|
|
|
|
def scanner_heal_release_bundle_fixture(self, directory: Path) -> tuple[Path, Path]:
|
|
"""Parser fixtures only; the bundle is not runtime evidence."""
|
|
root, _ = self.scanner_heal_fixture(directory)
|
|
bundle_dir = directory / "bundle"
|
|
artifact_dir = bundle_dir / "artifacts"
|
|
artifact_dir.mkdir(parents=True)
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
requirements, _, _ = scanner_heal_release_requirements(registry)
|
|
gates = {}
|
|
source_revision = "b" * 40
|
|
started = datetime(2026, 9, 8, 0, 0, tzinfo=timezone.utc)
|
|
for gate, requirement in requirements.items():
|
|
fields = {}
|
|
for field in SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS[gate]:
|
|
artifact = artifact_dir / f"{gate}-{field}.json"
|
|
run_id = f"{gate.lower()}-{field.replace('_', '-')}-run"
|
|
window_id = f"{gate.lower()}-window"
|
|
duration = 60
|
|
evidence = {
|
|
"artifact": artifact.relative_to(bundle_dir).as_posix(),
|
|
"evidence_type": "measured",
|
|
"source_revision": source_revision,
|
|
"run_id": run_id,
|
|
"measurement_window_id": window_id,
|
|
"started_at": started.isoformat().replace("+00:00", "Z"),
|
|
"command": ["cargo", "nextest", "run", requirement["description"]],
|
|
"artifact_format": "json",
|
|
"summary": f"parser fixture for {gate}.{field}",
|
|
}
|
|
if gate.startswith("P"):
|
|
duration = 900
|
|
evidence["duration_seconds"] = duration
|
|
if gate == "P3" and field == "two_hour_pressure_measurement":
|
|
duration = 7200
|
|
evidence["duration_seconds"] = duration
|
|
if gate == "P4" and field == "mrf_cleanup_gc_soak_evidence":
|
|
duration = 7200
|
|
evidence["duration_seconds"] = duration
|
|
evidence["finished_at"] = (started + timedelta(seconds=duration)).isoformat().replace("+00:00", "Z")
|
|
if gate == "G10" and field == "scheduler_bound_evidence":
|
|
evidence.update({
|
|
"max_deferred_items": 256,
|
|
"max_deferred_bytes": 262144,
|
|
"max_retry_age_seconds": 30,
|
|
"duplicate_task_count": 0,
|
|
})
|
|
if gate == "G10" and field == "pressure_recovery_evidence":
|
|
evidence.update({
|
|
"pressure_pacing_engaged": True,
|
|
"recovery_window_seconds": 5,
|
|
"lock_hold_p95_ms": 0,
|
|
"foreground_latency_p95_ms": 10,
|
|
})
|
|
if gate == "P1" and field == "foreground_latency_throughput_measurement":
|
|
evidence.update({
|
|
"foreground_latency_p95_ms": 10,
|
|
"foreground_latency_p99_ms": 20,
|
|
"throughput_ops_per_second": 100,
|
|
"error_count": 0,
|
|
})
|
|
if gate == "P3" and field == "two_hour_pressure_measurement":
|
|
evidence.update({
|
|
"fixed_offered_load": True,
|
|
"foreground_latency_p99_ms": 20,
|
|
"attempt_cost_samples": 1,
|
|
})
|
|
if gate == "P3" and field == "heal_capacity_measurement":
|
|
evidence.update({"completed_heal_objects": 1, "duplicate_task_count": 0})
|
|
if gate == "P3" and field == "recovery_window_measurement":
|
|
evidence.update({"pressure_recovery_window_seconds": 5, "lock_hold_p95_ms": 0})
|
|
if gate == "G02" and field == "bounded_checkpoint_oracle":
|
|
evidence.update({
|
|
"checkpoint_progress_bounded": True,
|
|
"raw_entry_budget": 8,
|
|
"max_raw_entries_per_round": 8,
|
|
"max_objects_processed_per_round": 8,
|
|
"bounded_work_quantum_observed": True,
|
|
"durable_checkpoint_committed": True,
|
|
"no_unbounded_tail": True,
|
|
})
|
|
if gate == "G02" and field == "independent_version_inventory":
|
|
evidence.update({
|
|
"independent_version_inventory_observed": True,
|
|
"objects_expected": 16,
|
|
"objects_retained": 16,
|
|
"versions_retained": 16,
|
|
"bytes_retained": 16,
|
|
})
|
|
if gate == "G01" and field == "root_authority_evidence":
|
|
evidence.update({
|
|
"root_authority_cases": list(SCANNER_HEAL_RELEASE_G01_ROOT_AUTHORITY_CASES),
|
|
"root_cas_observed": True,
|
|
"root_readback_observed": True,
|
|
"incomplete_root_rejected": True,
|
|
"stale_root_rejected": True,
|
|
})
|
|
if gate == "G01" and field == "quota_authority_evidence":
|
|
evidence.update({
|
|
"quota_authority_cases": list(SCANNER_HEAL_RELEASE_G01_QUOTA_AUTHORITY_CASES),
|
|
"quota_floor_readback_observed": True,
|
|
"over_limit_put_rejected": True,
|
|
"rejected_object_invisible": True,
|
|
"quota_fails_closed_without_authority": True,
|
|
})
|
|
if gate in ("G03", "G09", "R-L"):
|
|
evidence["versions"] = ["a" * 40, source_revision]
|
|
evidence["mixed_version_role"] = SCANNER_HEAL_RELEASE_MIXED_VERSION_ROLES[(gate, field)]
|
|
if gate in ("G04", "G07", "R-E", "R-L"):
|
|
evidence["crash_points"] = ["before-commit"]
|
|
if gate == "G03":
|
|
evidence["scoped_ack_cases"] = list(SCANNER_HEAL_RELEASE_SCOPED_ACK_CASES[field])
|
|
for required in SCANNER_HEAL_RELEASE_G03_REQUIRED_TRUE_FIELDS[field]:
|
|
evidence[required] = True
|
|
if gate == "G04" and field == "root_floor_intent_crash_evidence":
|
|
evidence["durable_intent_cases"] = list(SCANNER_HEAL_RELEASE_CRASH_BOUNDARY_FIELDS[(gate, field)])
|
|
evidence["persist_failure_blocks_acceptance"] = True
|
|
if gate == "G12":
|
|
evidence["quota_path_cases"] = list(SCANNER_HEAL_RELEASE_G12_QUOTA_PATH_CASES[field])
|
|
if gate == "G05" and field == "per_object_outcome_oracle":
|
|
evidence["per_object_outcome_cases"] = list(SCANNER_HEAL_RELEASE_G05_PER_OBJECT_OUTCOME_CASES)
|
|
evidence["outcome_counts"] = {"repaired": 4, "healthy": 3, "skipped": 2, "failed": 1}
|
|
evidence["status_matches_object_oracle"] = True
|
|
if gate == "G05" and field == "terminal_retention_bounds":
|
|
evidence["terminal_retention_cases"] = list(SCANNER_HEAL_RELEASE_G05_TERMINAL_RETENTION_CASES)
|
|
evidence["terminal_retention_window_seconds"] = 3600
|
|
evidence["max_terminal_record_age_seconds"] = 3599
|
|
evidence["terminal_records_pruned_after_window"] = 2
|
|
if gate == "G06" and field == "concurrent_status_evidence":
|
|
evidence["concurrent_status_cases"] = list(SCANNER_HEAL_RELEASE_G06_CONCURRENT_STATUS_CASES)
|
|
evidence["status_samples"] = 4
|
|
evidence["all_status_responses_http_success"] = True
|
|
evidence["partial_status_reports_degraded"] = True
|
|
if gate == "G06" and field == "legacy_client_compatibility":
|
|
evidence["legacy_client_cases"] = list(SCANNER_HEAL_RELEASE_G06_LEGACY_CLIENT_CASES)
|
|
evidence["rustfs_and_minio_paths_compatible"] = True
|
|
evidence["empty_body_status_requests_accepted"] = True
|
|
if gate == "G06" and field == "truncation_behavior":
|
|
evidence["truncation_cases"] = list(SCANNER_HEAL_RELEASE_G06_TRUNCATION_CASES)
|
|
evidence["truncated_payloads_rejected"] = True
|
|
evidence["max_status_payload_bytes"] = 4096
|
|
if gate == "R-E" and field == "fixed_budget_restart_evidence":
|
|
evidence.update({
|
|
"fixed_budget_restart_converged": True,
|
|
"restart_rounds": 3,
|
|
"raw_entry_budget": 8,
|
|
"no_unbudgeted_final_sweep": True,
|
|
})
|
|
if gate == "R-E" and field == "enumeration_evidence":
|
|
evidence.update({
|
|
"raw_enumeration_observed": True,
|
|
"durable_raw_page_commit_observed": True,
|
|
"raw_page_index_complete": True,
|
|
"enumeration_frontier_retained": True,
|
|
})
|
|
if gate == "R-E" and field == "classification_evidence":
|
|
evidence.update({
|
|
"classification_observed": True,
|
|
"objects_processed": 16,
|
|
"objects_retained": 16,
|
|
"versions_retained": 16,
|
|
"bytes_retained": 16,
|
|
})
|
|
if gate == "G09":
|
|
evidence["mixed_version_cases"] = list(SCANNER_HEAL_RELEASE_MIXED_VERSION_CASES[field])
|
|
if field == "rollback_payload_evidence":
|
|
evidence["rollback_payload_replayed"] = True
|
|
if (gate, field) in SCANNER_HEAL_RELEASE_MRF_DURABLE_REPLAY_FIELDS:
|
|
evidence["replayed_records"] = 2
|
|
evidence["responsibility_anchor_retained"] = True
|
|
evidence["successor_snapshot_published"] = True
|
|
if gate == "P4" and field == "mrf_cleanup_gc_soak_evidence":
|
|
evidence["cleanup_gc_cases"] = list(SCANNER_HEAL_RELEASE_MRF_CLEANUP_GC_SOAK_CASES)
|
|
evidence["verified_idle_gc_observed"] = True
|
|
evidence["pending_responsibilities_after_gc"] = 0
|
|
evidence["stale_journals_after_gc"] = 0
|
|
if gate == "G07":
|
|
case_field = {
|
|
"mrf_responsibility_oracle": "mrf_responsibility_cases",
|
|
"commit_boundary_crash_matrix": "commit_crash_cases",
|
|
}[field]
|
|
evidence[case_field] = list(SCANNER_HEAL_RELEASE_G07_REQUIRED_CASES[field])
|
|
if gate == "G08":
|
|
case_field = {
|
|
"mrf_capacity_evidence": "capacity_cases",
|
|
"disk_full_matrix": "disk_full_cases",
|
|
"replica_loss_matrix": "replica_loss_cases",
|
|
}[field]
|
|
evidence[case_field] = list(SCANNER_HEAL_RELEASE_G08_REQUIRED_CASES[field])
|
|
if gate == "G10" and field == "scheduler_bound_evidence":
|
|
evidence["scheduler_bounds"] = list(SCANNER_HEAL_RELEASE_SCHEDULER_BOUNDS)
|
|
evidence["duplicate_task_bound_observed"] = True
|
|
if gate == "G10" and field == "pressure_recovery_evidence":
|
|
evidence["pressure_metrics"] = {
|
|
"foreground_p95_ms": 40.0,
|
|
"foreground_p99_ms": 90.0,
|
|
"throughput_ops": 250.0,
|
|
"error_rate": 0.0,
|
|
"heal_lock_wait_p99_ms": 25.0,
|
|
"attempt_cost_samples": 8.0,
|
|
"foreground_pressure_samples": 120,
|
|
"foreground_pressure_high_samples": 12,
|
|
}
|
|
if gate == "G11" and field == "maintenance_producer_matrix":
|
|
evidence["producer_identities"] = list(SCANNER_HEAL_REQUIRED_PRODUCER_IDENTITIES)
|
|
evidence["producer_families"] = list(SCANNER_HEAL_REQUIRED_PRODUCER_FAMILIES)
|
|
evidence["matrix_cases"] = list(SCANNER_HEAL_RELEASE_G11_REQUIRED_CASES[field])
|
|
evidence["durable_identity_observed"] = True
|
|
evidence["durable_journal_replay_observed"] = True
|
|
evidence["durable_journal_hydration_bound"] = True
|
|
evidence["generation_window_bound"] = True
|
|
evidence["restart_gap_absent"] = True
|
|
evidence["overflow_absent"] = True
|
|
evidence["ack_clear_generation_observed"] = True
|
|
evidence["invalid_journal_inputs_rejected"] = list(SCANNER_HEAL_DURABLE_JOURNAL_FAIL_CLOSED_CASES)
|
|
if gate == "G11" and field == "complete_producer_inventory":
|
|
evidence["required_producer_identities"] = list(SCANNER_HEAL_REQUIRED_PRODUCER_IDENTITIES)
|
|
evidence["observed_producer_identities"] = list(SCANNER_HEAL_REQUIRED_PRODUCER_IDENTITIES)
|
|
evidence["required_producer_families"] = list(SCANNER_HEAL_REQUIRED_PRODUCER_FAMILIES)
|
|
evidence["observed_producer_families"] = list(SCANNER_HEAL_REQUIRED_PRODUCER_FAMILIES)
|
|
evidence["missing_producer_identities"] = []
|
|
evidence["unknown_producer_excluded"] = True
|
|
if gate == "G13" and field == "quorum_minus_one_matrix":
|
|
evidence["quorum_cases"] = list(SCANNER_HEAL_RELEASE_G13_REQUIRED_CASES[field])
|
|
evidence["no_success_at_quorum_minus_one"] = True
|
|
evidence["exact_quorum_restored"] = True
|
|
if gate == "G13" and field == "unknown_disk_remount_matrix":
|
|
evidence["remount_cases"] = list(SCANNER_HEAL_RELEASE_G13_REQUIRED_CASES[field])
|
|
evidence["unknown_disks_excluded"] = True
|
|
evidence["remounted_disks_revalidated"] = True
|
|
evidence["stale_incarnation_rejected"] = True
|
|
if gate == "G13" and field == "object_lock_dry_run_grace_evidence":
|
|
evidence["grace_cases"] = list(SCANNER_HEAL_RELEASE_G13_REQUIRED_CASES[field])
|
|
evidence["object_lock_denials_preserved"] = True
|
|
evidence["dry_run_mutation_count"] = 0
|
|
evidence["grace_outcomes_retained"] = True
|
|
if gate == "G14" and field == "ec8_4_evidence":
|
|
evidence["topology"] = {"erasure": "EC8+4", "nodes": 3, "drives_per_node": 4}
|
|
if gate == "G14" and field == "same_window_field_evidence":
|
|
evidence["same_window_fields"] = [
|
|
item
|
|
for item in SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS["G14"]
|
|
if item != "same_window_field_evidence"
|
|
]
|
|
if gate == "G14" and field == "multi_set_evidence":
|
|
evidence["sets"] = 2
|
|
if gate == "G14" and field == "multi_pool_evidence":
|
|
evidence["pools"] = 2
|
|
if gate == "G14" and field == "distributed_segment_invalidation_evidence":
|
|
evidence["invalidation_domain"] = "distributed-ec"
|
|
evidence["distributed_ec_invalidation"] = True
|
|
evidence["peer_count"] = 3
|
|
evidence["same_window_remote_proof"] = True
|
|
evidence["all_peers_bound_to_generation_window"] = True
|
|
if gate == "G14":
|
|
case_id = "ec84-multiset-multipool"
|
|
case_artifact = artifact_dir / "cases" / f"{gate}-{field}-{case_id}.json"
|
|
case_artifact.parent.mkdir(parents=True, exist_ok=True)
|
|
write_json(case_artifact, {
|
|
"schema": 1,
|
|
"case": case_id,
|
|
"evidence_type": "measured",
|
|
"source_revision": source_revision,
|
|
"measurement_window_id": window_id,
|
|
"topology": {"erasure": "EC8+4", "nodes": 3, "drives_per_node": 4},
|
|
"sets": 2,
|
|
"pools": 2,
|
|
"invalidation_domain": "distributed-ec",
|
|
"distributed_ec_invalidation": True,
|
|
"peer_count": 3,
|
|
"same_window_remote_proof": True,
|
|
"all_peers_bound_to_generation_window": True,
|
|
})
|
|
evidence["case_evidence"] = [
|
|
{
|
|
"case": case_id,
|
|
"artifact": case_artifact.relative_to(bundle_dir).as_posix(),
|
|
"sha256": digest(case_artifact),
|
|
"source_revision": source_revision,
|
|
"measurement_window_id": window_id,
|
|
},
|
|
]
|
|
if field == "segment_activation_preflight":
|
|
evidence["production_activation"] = True
|
|
evidence["scanner_segment_reuse_activated"] = True
|
|
evidence["proof_inputs"] = list(SCANNER_HEAL_SEGMENT_ACTIVATION_PROOF_INPUTS)
|
|
evidence["fail_closed_checks"] = list(SCANNER_HEAL_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS)
|
|
if field == "cold_segment_reuse_measurement":
|
|
evidence["hot_walked_segments"] = 2
|
|
evidence["cold_walked_segments"] = 0
|
|
evidence["full_walk_oracle_equivalent"] = True
|
|
evidence["published_root_equivalent"] = True
|
|
if field == "post_stop_convergence_measurement":
|
|
evidence["writes_stopped"] = True
|
|
evidence["last_mutation_observed"] = True
|
|
evidence["first_complete_publication"] = True
|
|
evidence["post_stop_samples"] = 2
|
|
evidence["post_stop_work_multiple"] = 1.1
|
|
evidence["post_stop_work_multiple_limit"] = 1.2
|
|
evidence["post_stop_work_multiples"] = [1.0, 1.1]
|
|
if gate == "P4" and field == "retained_responsibility_evidence":
|
|
evidence["duration_seconds"] = 7200
|
|
evidence["finished_at"] = (started + timedelta(seconds=7200)).isoformat().replace("+00:00", "Z")
|
|
evidence["retained_responsibility_cases"] = list(
|
|
SCANNER_HEAL_RELEASE_P4_RETAINED_RESPONSIBILITY_CASES
|
|
)
|
|
evidence["retention_window_seconds"] = 7200
|
|
evidence["idle_cleanup_observed"] = True
|
|
evidence["verified_proof_discharge_observed"] = True
|
|
if gate == "R-D" and field == "manager_disposition_evidence":
|
|
evidence["manager_disposition_cases"] = list(SCANNER_HEAL_RELEASE_RD_MANAGER_CASES)
|
|
evidence["manager_dispositions_are_terminal"] = True
|
|
if gate == "R-D" and field == "event_disposition_evidence":
|
|
evidence["event_disposition_cases"] = list(SCANNER_HEAL_RELEASE_RD_EVENT_CASES)
|
|
evidence["events_correlate_to_manager_dispositions"] = True
|
|
if gate == "R-D" and field == "ledger_disposition_evidence":
|
|
evidence["ledger_disposition_cases"] = list(SCANNER_HEAL_RELEASE_RD_LEDGER_CASES)
|
|
evidence["ledger_correlates_to_events"] = True
|
|
evidence["ledger_replay_preserves_terminal_disposition"] = True
|
|
if gate == "R-D" and field == "grace_handling":
|
|
evidence["grace_cases"] = list(SCANNER_HEAL_RELEASE_RD_GRACE_CASES)
|
|
evidence["grace_window_seconds"] = 300
|
|
evidence["grace_retention_observed"] = True
|
|
evidence["grace_expiry_pruned_terminal_records"] = True
|
|
if gate == "R-L" and field == "legacy_source_conflict_evidence":
|
|
evidence["legacy_source_conflict_cases"] = list(SCANNER_HEAL_RELEASE_RL_REQUIRED_CASES[field])
|
|
evidence["source_conflicts_rejected"] = True
|
|
evidence["takeover_identity_bound"] = True
|
|
evidence["legacy_checksum_gap_rejected"] = True
|
|
if gate == "R-L" and field == "migration_gap_evidence":
|
|
evidence["migration_gap_cases"] = list(SCANNER_HEAL_RELEASE_RL_REQUIRED_CASES[field])
|
|
evidence["migration_gap_closed"] = True
|
|
evidence["legacy_sources_fail_closed"] = True
|
|
evidence["prior_responsibilities_inherited"] = True
|
|
if gate == "R-L" and field == "crash_safe_source_retirement_evidence":
|
|
evidence["source_retirement_cases"] = list(SCANNER_HEAL_RELEASE_RL_REQUIRED_CASES[field])
|
|
evidence["source_retirement_is_crash_safe"] = True
|
|
evidence["old_source_retained_until_successor"] = True
|
|
evidence["recovered_pending_migration"] = True
|
|
if field == "profile_evidence":
|
|
profile_metrics = {
|
|
"resolved_samples": 1,
|
|
"allocation_bytes": 1024,
|
|
"rss_peak_bytes": 4096,
|
|
"save_operations": 2,
|
|
"saved_bytes": 2048,
|
|
}
|
|
evidence.update(profile_metrics)
|
|
artifacts = {}
|
|
for artifact_kind in RELEASE_PROFILE_ARTIFACTS:
|
|
raw_artifact = artifact_dir / f"{gate}-{field}-{artifact_kind}.raw"
|
|
raw_artifact.write_text(f"{artifact_kind} measured profile sample\n", encoding="utf-8")
|
|
profile_artifact = artifact_dir / f"{gate}-{field}-{artifact_kind}.json"
|
|
write_json(profile_artifact, {
|
|
"schema": 1,
|
|
"evidence_type": "measured",
|
|
"source_revision": source_revision,
|
|
"run_id": run_id,
|
|
"measurement_window_id": window_id,
|
|
"gate": gate,
|
|
"field": field,
|
|
"artifact_kind": artifact_kind,
|
|
"raw_profile_artifact": raw_artifact.relative_to(bundle_dir).as_posix(),
|
|
"raw_profile_sha256": digest(raw_artifact),
|
|
"raw_profile_bytes": raw_artifact.stat().st_size,
|
|
"raw_profile_format": "raw",
|
|
**{
|
|
metric: profile_metrics[metric]
|
|
for metric in PROFILE_ARTIFACT_REQUIRED_METRICS[artifact_kind]
|
|
},
|
|
})
|
|
artifacts[artifact_kind] = {
|
|
"artifact": profile_artifact.relative_to(bundle_dir).as_posix(),
|
|
"sha256": digest(profile_artifact),
|
|
"artifact_format": "json",
|
|
"source_revision": source_revision,
|
|
"run_id": run_id,
|
|
"measurement_window_id": window_id,
|
|
}
|
|
evidence["profile_artifacts"] = artifacts
|
|
if gate == "P1" and field == "cold_walk_share_measurement":
|
|
evidence["cold_walk_share"] = 0.5
|
|
evidence["walk_objects"] = 100
|
|
evidence["cold_walk_objects"] = 50
|
|
if gate == "P1" and field == "foreground_latency_throughput_measurement":
|
|
evidence["foreground_p95_ms"] = 50.0
|
|
evidence["foreground_p99_ms"] = 100.0
|
|
evidence["throughput_ops"] = 200.0
|
|
evidence["error_rate"] = 0.0
|
|
if gate == "P3" and field == "two_hour_pressure_measurement":
|
|
evidence["abba_legs"] = ["A1", "B1", "B2", "A2"]
|
|
evidence["scenarios"] = list(SCENARIOS)
|
|
evidence["foreground_p95_ms"] = 50.0
|
|
evidence["foreground_p99_ms"] = 100.0
|
|
evidence["throughput_ops"] = 200.0
|
|
if gate == "P3" and field == "heal_capacity_measurement":
|
|
evidence["heal_capacity"] = {
|
|
"objects": 100,
|
|
"versions": 100,
|
|
"bytes": 1048576,
|
|
"completed_objects": 100,
|
|
}
|
|
if gate == "P3" and field == "recovery_window_measurement":
|
|
evidence["fault_modes"] = ["process-restart", "process-crash-restart"]
|
|
evidence["recovery_p95_ms"] = 500.0
|
|
evidence["recovery_p99_ms"] = 1000.0
|
|
artifact_payload = {
|
|
"schema": 1,
|
|
"evidence_type": "measured",
|
|
"source_revision": source_revision,
|
|
"run_id": run_id,
|
|
"measurement_window_id": window_id,
|
|
"gate": gate,
|
|
"field": field,
|
|
}
|
|
mrf_artifact_kind = SCANNER_HEAL_RELEASE_MRF_ARTIFACT_KINDS.get((gate, field))
|
|
if mrf_artifact_kind is not None:
|
|
artifact_payload["artifact_kind"] = mrf_artifact_kind
|
|
for mirror_field in release_bundle_json_artifact_mirrored_fields(gate, field):
|
|
artifact_payload[mirror_field] = evidence[mirror_field]
|
|
write_json(artifact, artifact_payload)
|
|
evidence["sha256"] = digest(artifact)
|
|
fields[field] = evidence
|
|
gates[gate] = {
|
|
"status": "pass",
|
|
"lane": requirement["lane"],
|
|
"evidence_type": "measured",
|
|
"evidence_fields": fields,
|
|
}
|
|
bundle = bundle_dir / "release-evidence.json"
|
|
write_json(bundle, {"schema": 1, "evidence": "measured", "source_revision": source_revision, "gates": gates})
|
|
return root, bundle
|
|
|
|
def test_scanner_heal_release_bundle_accepts_complete_measured_evidence(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_status(root, bundle)
|
|
self.assertEqual(status["decision"], "approved")
|
|
self.assertTrue(status["release_approved"])
|
|
self.assertEqual(len(status["verified_gates"]), len(SCANNER_HEAL_RELEASE_REQUIRED_GATES))
|
|
self.assertEqual(status["pending_gates"], [])
|
|
self.assertEqual(status["pending_lanes"], [])
|
|
|
|
def test_scanner_heal_release_bundle_assembler_copies_measured_artifacts(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, descriptor = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
bundle, status = assemble_scanner_heal_release_bundle(root, descriptor, Path(tmp) / "assembled")
|
|
self.assertEqual(status["decision"], "approved")
|
|
self.assertTrue(status["release_approved"])
|
|
assembled = read_json(bundle)
|
|
evidence = assembled["gates"]["G01"]["evidence_fields"]["root_authority_evidence"]
|
|
self.assertEqual(evidence["artifact"], "artifacts/G01-root_authority_evidence.json")
|
|
self.assertEqual(evidence["sha256"], digest(bundle.parent / evidence["artifact"]))
|
|
self.assertTrue((bundle.parent / evidence["artifact"]).is_file())
|
|
profile = assembled["gates"]["P1"]["evidence_fields"]["profile_evidence"]
|
|
profile_artifact = profile["profile_artifacts"]["save-frequency"]
|
|
profile_wrapper = bundle.parent / profile_artifact["artifact"]
|
|
profile_payload = read_json(profile_wrapper)
|
|
raw_profile = bundle.parent / profile_payload["raw_profile_artifact"]
|
|
self.assertTrue(raw_profile.is_file())
|
|
self.assertEqual(profile_payload["raw_profile_sha256"], digest(raw_profile))
|
|
self.assertEqual(profile_artifact["sha256"], digest(profile_wrapper))
|
|
|
|
def test_scanner_heal_release_bundle_assembler_merges_measured_descriptors(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, descriptor = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
source = read_json(descriptor)
|
|
descriptor_paths = []
|
|
for gate in ("G01", "G09"):
|
|
partial = descriptor.parent / f"release-bundle-{gate}.json"
|
|
write_json(partial, {
|
|
"schema": 1,
|
|
"evidence": "measured",
|
|
"source_revision": source["source_revision"],
|
|
"gates": {gate: source["gates"][gate]},
|
|
})
|
|
descriptor_paths.append(partial)
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
bundle, status = assemble_scanner_heal_release_bundle_descriptors(
|
|
root,
|
|
descriptor_paths,
|
|
Path(tmp) / "assembled",
|
|
)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertIn("G01", status["verified_gates"])
|
|
self.assertIn("G09", status["verified_gates"])
|
|
self.assertIn("G02", status["pending_gates"])
|
|
assembled = read_json(bundle)
|
|
self.assertEqual(sorted(assembled["gates"]), ["G01", "G09"])
|
|
g09 = assembled["gates"]["G09"]["evidence_fields"]["mixed_version_reader_evidence"]
|
|
self.assertTrue((bundle.parent / g09["artifact"]).is_file())
|
|
|
|
def test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, descriptor = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
with self.assertRaisesRegex(ValueError, "repeats gate: G01"):
|
|
assemble_scanner_heal_release_bundle_descriptors(
|
|
root,
|
|
[descriptor, descriptor],
|
|
Path(tmp) / "assembled",
|
|
)
|
|
|
|
def test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, descriptor = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
source = read_json(descriptor)
|
|
g01 = descriptor.parent / "release-bundle-g01.json"
|
|
g09 = descriptor.parent / "release-bundle-g09.json"
|
|
write_json(g01, {
|
|
"schema": 1,
|
|
"evidence": "measured",
|
|
"source_revision": source["source_revision"],
|
|
"gates": {"G01": source["gates"]["G01"]},
|
|
})
|
|
write_json(g09, {
|
|
"schema": 1,
|
|
"evidence": "measured",
|
|
"source_revision": "c" * 40,
|
|
"gates": {"G09": source["gates"]["G09"]},
|
|
})
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
with self.assertRaisesRegex(ValueError, "source revisions differ"):
|
|
assemble_scanner_heal_release_bundle_descriptors(root, [g01, g09], Path(tmp) / "assembled")
|
|
|
|
def test_scanner_heal_release_bundle_gate_accepts_single_measured_gate(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
data["gates"] = {"G09": data["gates"]["G09"]}
|
|
write_json(bundle, data)
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_gate_status(root, bundle, "G09")
|
|
full_status = scanner_heal_release_bundle_status(root, bundle)
|
|
self.assertEqual(status["decision"], "verified")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertEqual(status["verified_gate"], "G09")
|
|
self.assertIn("G07", status["pending_gates"])
|
|
self.assertEqual(full_status["decision"], "blocked")
|
|
self.assertFalse(full_status["release_approved"])
|
|
|
|
def test_scanner_heal_release_bundle_gate_rejects_missing_field(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
del data["gates"]["G09"]["evidence_fields"]["rollback_payload_evidence"]
|
|
write_json(bundle, data)
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_gate_status(root, bundle, "G09")
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertIn("missing required fields: rollback_payload_evidence", status["rejected_errors"])
|
|
|
|
def test_scanner_heal_release_bundle_gate_rejects_unknown_gate(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
data["gates"] = {"G09": data["gates"]["G09"], "Z99": {"status": "pass"}}
|
|
write_json(bundle, data)
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
with self.assertRaisesRegex(ValueError, "unknown gates: Z99"):
|
|
scanner_heal_release_bundle_gate_status(root, bundle, "G09")
|
|
|
|
def test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, _ = self.scanner_heal_fixture(Path(tmp))
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
descriptor = write_scanner_heal_release_bundle_fixture(root, Path(tmp) / "bundle")
|
|
with self.assertRaisesRegex(ValueError, "fixture_only"):
|
|
assemble_scanner_heal_release_bundle(root, descriptor, Path(tmp) / "assembled")
|
|
|
|
def test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, descriptor = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(descriptor)
|
|
data["gates"]["G01"]["evidence_fields"]["root_authority_evidence"]["sha256"] = "0" * 64
|
|
write_json(descriptor, data)
|
|
|
|
with self.assertRaisesRegex(ValueError, "descriptor artifact hash mismatch"):
|
|
assemble_scanner_heal_release_bundle(root, descriptor, Path(tmp) / "assembled")
|
|
|
|
def test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, _ = self.scanner_heal_fixture(Path(tmp))
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
bundle = write_scanner_heal_release_bundle_fixture(root, Path(tmp) / "bundle")
|
|
with self.assertRaisesRegex(ValueError, "fixture_only"):
|
|
scanner_heal_release_bundle_status(root, bundle)
|
|
data = read_json(bundle)
|
|
self.assertTrue(data["fixture_only"])
|
|
self.assertEqual(data["source_revision"], "b" * 40)
|
|
|
|
def test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields(self) -> None:
|
|
for fault in ("synthetic", "missing-field", "hash", "empty-artifact"):
|
|
with self.subTest(fault=fault), tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
if fault == "synthetic":
|
|
data["evidence"] = "synthetic"
|
|
elif fault == "missing-field":
|
|
del data["gates"]["G09"]["evidence_fields"]["rollback_payload_evidence"]
|
|
elif fault == "hash":
|
|
artifact = bundle.parent / data["gates"]["G01"]["evidence_fields"]["root_authority_evidence"]["artifact"]
|
|
artifact.write_text(artifact.read_text(encoding="utf-8") + "\n", encoding="utf-8")
|
|
else:
|
|
artifact = bundle.parent / data["gates"]["G01"]["evidence_fields"]["root_authority_evidence"]["artifact"]
|
|
artifact.write_text("", encoding="utf-8")
|
|
write_json(bundle, data)
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
if fault == "synthetic":
|
|
with self.assertRaisesRegex(ValueError, "must be measured"):
|
|
scanner_heal_release_bundle_status(root, bundle)
|
|
else:
|
|
status = scanner_heal_release_bundle_status(root, bundle)
|
|
if fault != "synthetic":
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertTrue(status["rejected_gates"])
|
|
|
|
def test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions(self) -> None:
|
|
for fault, gate, field, mutation, expected in (
|
|
("topology", "G14", "ec8_4_evidence", lambda item: item.update({"topology": {"erasure": "EC4+2", "nodes": 2, "drives_per_node": 3}}), "JSON artifact topology mismatch"),
|
|
("missing-duration", "P1", "cold_walk_share_measurement", lambda item: item.pop("duration_seconds"), "duration_seconds"),
|
|
("duration", "P3", "two_hour_pressure_measurement", lambda item: item.update({"duration_seconds": 7199}), "two hours"),
|
|
("mrf-cleanup-soak-duration", "P4", "mrf_cleanup_gc_soak_evidence", lambda item: item.update({"duration_seconds": 7199}), "two hours"),
|
|
("profile", "P1", "profile_evidence", lambda item: item.pop("resolved_samples"), "resolved_samples"),
|
|
(
|
|
"profile-artifact",
|
|
"P1",
|
|
"profile_evidence",
|
|
lambda item: item["profile_artifacts"].pop("flamegraph"),
|
|
"missing profile artifacts",
|
|
),
|
|
(
|
|
"profile-artifact-hash",
|
|
"P1",
|
|
"profile_evidence",
|
|
lambda item: item["profile_artifacts"]["rss-samples"].update({"sha256": "0" * 64}),
|
|
"artifact hash mismatch",
|
|
),
|
|
(
|
|
"profile-artifact-source",
|
|
"P1",
|
|
"profile_evidence",
|
|
lambda item: item["profile_artifacts"]["allocation-profile"].update({"source_revision": "c" * 40}),
|
|
"source revision mismatch",
|
|
),
|
|
(
|
|
"profile-artifact-run",
|
|
"P1",
|
|
"profile_evidence",
|
|
lambda item: item["profile_artifacts"]["flamegraph"].update({"run_id": "p1-different-profile-run"}),
|
|
"run_id mismatch",
|
|
),
|
|
(
|
|
"profile-artifact-window",
|
|
"P1",
|
|
"profile_evidence",
|
|
lambda item: item["profile_artifacts"]["save-frequency"].update(
|
|
{"measurement_window_id": "p1-different-window"}
|
|
),
|
|
"measurement window mismatch",
|
|
),
|
|
("g05-outcome-cases", "G05", "per_object_outcome_oracle", lambda item: item["per_object_outcome_cases"].remove("object-failed-and-retained"), "missing cases"),
|
|
("g05-outcome-counts", "G05", "per_object_outcome_oracle", lambda item: item["outcome_counts"].pop("failed"), "outcome_counts.failed"),
|
|
(
|
|
"g05-retention-window",
|
|
"G05",
|
|
"terminal_retention_bounds",
|
|
lambda item: item.update({"max_terminal_record_age_seconds": item["terminal_retention_window_seconds"] + 1}),
|
|
"exceeds retention window",
|
|
),
|
|
(
|
|
"g06-concurrent-status",
|
|
"G06",
|
|
"concurrent_status_evidence",
|
|
lambda item: item["concurrent_status_cases"].remove("status-while-peer-down"),
|
|
"missing cases",
|
|
),
|
|
(
|
|
"g06-legacy-client",
|
|
"G06",
|
|
"legacy_client_compatibility",
|
|
lambda item: item.update({"empty_body_status_requests_accepted": False}),
|
|
"empty_body_status_requests_accepted",
|
|
),
|
|
(
|
|
"g06-truncation",
|
|
"G06",
|
|
"truncation_behavior",
|
|
lambda item: item["truncation_cases"].remove("truncated-node-status-reject"),
|
|
"missing cases",
|
|
),
|
|
("versions", "G09", "mixed_version_reader_evidence", lambda item: item.update({"versions": [1, 2]}), "mixed-version"),
|
|
("stale-versions", "G09", "mixed_version_writer_evidence", lambda item: item.update({"versions": ["a" * 40, "c" * 40]}), "tested source revision"),
|
|
("g08-capacity-cases", "G08", "mrf_capacity_evidence", lambda item: item.update({"capacity_cases": ["queue-count-limit"]}), "missing cases"),
|
|
(
|
|
"g08-disk-full-cases",
|
|
"G08",
|
|
"disk_full_matrix",
|
|
lambda item: item["disk_full_cases"].remove("cleanup-delete-enospc"),
|
|
"missing cases",
|
|
),
|
|
("g08-replica-loss-cases", "G08", "replica_loss_matrix", lambda item: item.update({"replica_loss_cases": ["single-replica-loss"]}), "missing cases"),
|
|
("g07-responsibility-cases", "G07", "mrf_responsibility_oracle", lambda item: item.update({"mrf_responsibility_cases": ["legacy-journal-replay"]}), "missing cases"),
|
|
("g07-crash-cases", "G07", "commit_boundary_crash_matrix", lambda item: item.pop("commit_crash_cases"), "non-empty string list"),
|
|
("mrf-records", "G07", "mrf_responsibility_oracle", lambda item: item.pop("replayed_records"), "replayed_records"),
|
|
("mrf-anchor", "G07", "commit_boundary_crash_matrix", lambda item: item.update({"responsibility_anchor_retained": False}), "retained MRF responsibility anchors"),
|
|
("mrf-successor", "P4", "retained_responsibility_evidence", lambda item: item.pop("successor_snapshot_published"), "successor snapshot"),
|
|
(
|
|
"g02-raw-budget",
|
|
"G02",
|
|
"bounded_checkpoint_oracle",
|
|
lambda item: item.update({"max_raw_entries_per_round": item["raw_entry_budget"] + 1}),
|
|
"raw entries exceed fixed budget",
|
|
),
|
|
(
|
|
"g02-durable-checkpoint",
|
|
"G02",
|
|
"bounded_checkpoint_oracle",
|
|
lambda item: item.update({"durable_checkpoint_committed": False}),
|
|
"durable_checkpoint_committed",
|
|
),
|
|
(
|
|
"g02-version-inventory",
|
|
"G02",
|
|
"independent_version_inventory",
|
|
lambda item: item.update({"versions_retained": item["objects_retained"] - 1}),
|
|
"version inventory",
|
|
),
|
|
(
|
|
"re-converged",
|
|
"R-E",
|
|
"fixed_budget_restart_evidence",
|
|
lambda item: item.update({"fixed_budget_restart_converged": False}),
|
|
"fixed_budget_restart_converged",
|
|
),
|
|
(
|
|
"re-enumeration-commit",
|
|
"R-E",
|
|
"enumeration_evidence",
|
|
lambda item: item.update({"durable_raw_page_commit_observed": False}),
|
|
"durable_raw_page_commit_observed",
|
|
),
|
|
(
|
|
"re-classification-inventory",
|
|
"R-E",
|
|
"classification_evidence",
|
|
lambda item: item.update({"bytes_retained": item["objects_retained"] - 1}),
|
|
"byte inventory",
|
|
),
|
|
(
|
|
"mrf-cleanup-gc-cases",
|
|
"P4",
|
|
"mrf_cleanup_gc_soak_evidence",
|
|
lambda item: item["cleanup_gc_cases"].remove("repeated-replay-no-resurrection"),
|
|
"cleanup_gc_cases missing cases",
|
|
),
|
|
(
|
|
"mrf-cleanup-gc-idle",
|
|
"P4",
|
|
"mrf_cleanup_gc_soak_evidence",
|
|
lambda item: item.update({"verified_idle_gc_observed": False}),
|
|
"verified idle GC",
|
|
),
|
|
(
|
|
"mrf-cleanup-gc-pending",
|
|
"P4",
|
|
"mrf_cleanup_gc_soak_evidence",
|
|
lambda item: item.update({"pending_responsibilities_after_gc": 1}),
|
|
"zero pending responsibilities",
|
|
),
|
|
(
|
|
"mrf-cleanup-gc-stale",
|
|
"P4",
|
|
"mrf_cleanup_gc_soak_evidence",
|
|
lambda item: item.update({"stale_journals_after_gc": 1}),
|
|
"zero stale journals",
|
|
),
|
|
("same-window-fields", "G14", "same_window_field_evidence", lambda item: item.update({"same_window_fields": ["ec8_4_evidence", "multi_set_evidence"]}), "JSON artifact same_window_fields mismatch"),
|
|
("activation-disabled", "G11", "segment_activation_preflight", lambda item: item.update({"production_activation": False}), "production activation is enabled"),
|
|
(
|
|
"activation-runtime-disabled",
|
|
"G11",
|
|
"segment_activation_preflight",
|
|
lambda item: item.update({"scanner_segment_reuse_activated": False}),
|
|
"runtime activation gate is enabled",
|
|
),
|
|
(
|
|
"activation-missing-fail-closed",
|
|
"G11",
|
|
"segment_activation_preflight",
|
|
lambda item: item["fail_closed_checks"].remove("overflow"),
|
|
"missing G11.segment_activation_preflight.fail_closed_checks",
|
|
),
|
|
(
|
|
"activation-missing-proof-input",
|
|
"G11",
|
|
"segment_activation_preflight",
|
|
lambda item: item["proof_inputs"].remove("process_epoch"),
|
|
"missing G11.segment_activation_preflight.proof_inputs",
|
|
),
|
|
(
|
|
"activation-missing-durable-journal-input",
|
|
"G11",
|
|
"segment_activation_preflight",
|
|
lambda item: item["proof_inputs"].remove("durable_dirty_producer_journal"),
|
|
"missing G11.segment_activation_preflight.proof_inputs",
|
|
),
|
|
(
|
|
"activation-missing-ack-guard",
|
|
"G11",
|
|
"segment_activation_preflight",
|
|
lambda item: item["fail_closed_checks"].remove("stale_ack_generation"),
|
|
"missing G11.segment_activation_preflight.fail_closed_checks",
|
|
),
|
|
(
|
|
"producer-matrix-missing",
|
|
"G11",
|
|
"maintenance_producer_matrix",
|
|
lambda item: item["producer_identities"].remove("tier_expiration"),
|
|
"producer_identities missing cases",
|
|
),
|
|
(
|
|
"producer-matrix-no-journal-replay",
|
|
"G11",
|
|
"maintenance_producer_matrix",
|
|
lambda item: item.update({"durable_journal_replay_observed": False}),
|
|
"durable_journal_replay_observed",
|
|
),
|
|
(
|
|
"producer-matrix-no-hydration-bound",
|
|
"G11",
|
|
"maintenance_producer_matrix",
|
|
lambda item: item.update({"durable_journal_hydration_bound": False}),
|
|
"durable_journal_hydration_bound",
|
|
),
|
|
(
|
|
"producer-matrix-no-ack-clear-generation",
|
|
"G11",
|
|
"maintenance_producer_matrix",
|
|
lambda item: item.update({"ack_clear_generation_observed": False}),
|
|
"ack_clear_generation_observed",
|
|
),
|
|
(
|
|
"producer-matrix-missing-invalid-journal-case",
|
|
"G11",
|
|
"maintenance_producer_matrix",
|
|
lambda item: item["invalid_journal_inputs_rejected"].remove("mixed-version-producer"),
|
|
"invalid_journal_inputs_rejected missing cases",
|
|
),
|
|
(
|
|
"producer-inventory-missing",
|
|
"G11",
|
|
"complete_producer_inventory",
|
|
lambda item: item.update({"missing_producer_identities": ["tier_expiration"]}),
|
|
"zero missing producer identities",
|
|
),
|
|
(
|
|
"producer-inventory-unknown",
|
|
"G11",
|
|
"complete_producer_inventory",
|
|
lambda item: item.update({"unknown_producer_excluded": False}),
|
|
"unknown_producer_excluded",
|
|
),
|
|
(
|
|
"quorum-minus-one-success",
|
|
"G13",
|
|
"quorum_minus_one_matrix",
|
|
lambda item: item.update({"no_success_at_quorum_minus_one": False}),
|
|
"no_success_at_quorum_minus_one",
|
|
),
|
|
(
|
|
"remount-revalidation",
|
|
"G13",
|
|
"unknown_disk_remount_matrix",
|
|
lambda item: item.update({"remounted_disks_revalidated": False}),
|
|
"remounted_disks_revalidated",
|
|
),
|
|
(
|
|
"dry-run-mutates",
|
|
"G13",
|
|
"object_lock_dry_run_grace_evidence",
|
|
lambda item: item.update({"dry_run_mutation_count": 1}),
|
|
"dry_run_mutation_count",
|
|
),
|
|
(
|
|
"distributed-invalidation",
|
|
"G14",
|
|
"distributed_segment_invalidation_evidence",
|
|
lambda item: item.update({"distributed_ec_invalidation": False}),
|
|
"JSON artifact distributed_ec_invalidation mismatch",
|
|
),
|
|
(
|
|
"distributed-peer-window",
|
|
"G14",
|
|
"distributed_segment_invalidation_evidence",
|
|
lambda item: item.update({"all_peers_bound_to_generation_window": False}),
|
|
"JSON artifact all_peers_bound_to_generation_window mismatch",
|
|
),
|
|
(
|
|
"post-stop-work-multiple",
|
|
"P2",
|
|
"post_stop_convergence_measurement",
|
|
lambda item: item.update({"post_stop_work_multiple": 1.3}),
|
|
"post-stop work multiple limit",
|
|
),
|
|
(
|
|
"post-stop-publication",
|
|
"P2",
|
|
"post_stop_convergence_measurement",
|
|
lambda item: item.update({"first_complete_publication": False}),
|
|
"first complete publication",
|
|
),
|
|
(
|
|
"cold-segment-walk",
|
|
"P2",
|
|
"cold_segment_reuse_measurement",
|
|
lambda item: item.update({"cold_walked_segments": 1}),
|
|
"cold_walked_segments",
|
|
),
|
|
(
|
|
"cold-segment-oracle",
|
|
"P2",
|
|
"cold_segment_reuse_measurement",
|
|
lambda item: item.update({"full_walk_oracle_equivalent": False}),
|
|
"full-walk oracle equivalence",
|
|
),
|
|
(
|
|
"p4-retained-responsibility-cases",
|
|
"P4",
|
|
"retained_responsibility_evidence",
|
|
lambda item: item["retained_responsibility_cases"].remove("idle-cleanup-reclaims-replay-source"),
|
|
"retained_responsibility_cases missing cases",
|
|
),
|
|
(
|
|
"p4-retention-window",
|
|
"P4",
|
|
"retained_responsibility_evidence",
|
|
lambda item: item.update({"retention_window_seconds": 7199}),
|
|
"retention_window_seconds",
|
|
),
|
|
(
|
|
"p4-idle-cleanup",
|
|
"P4",
|
|
"retained_responsibility_evidence",
|
|
lambda item: item.update({"idle_cleanup_observed": False}),
|
|
"idle_cleanup_observed",
|
|
),
|
|
(
|
|
"p4-proof-discharge",
|
|
"P4",
|
|
"retained_responsibility_evidence",
|
|
lambda item: item.pop("verified_proof_discharge_observed"),
|
|
"verified_proof_discharge_observed",
|
|
),
|
|
("scheduler-duplicates", "G10", "scheduler_bound_evidence", lambda item: item.pop("duplicate_task_count"), "duplicate_task_count"),
|
|
("p1-throughput", "P1", "foreground_latency_throughput_measurement", lambda item: item.pop("throughput_ops_per_second"), "throughput_ops_per_second"),
|
|
("p3-fixed-load", "P3", "two_hour_pressure_measurement", lambda item: item.update({"fixed_offered_load": False}), "fixed offered load"),
|
|
(
|
|
"g01-root-cases",
|
|
"G01",
|
|
"root_authority_evidence",
|
|
lambda item: item["root_authority_cases"].remove("stale-root-rejected"),
|
|
"root_authority_cases missing cases",
|
|
),
|
|
(
|
|
"g01-quota-fail-closed",
|
|
"G01",
|
|
"quota_authority_evidence",
|
|
lambda item: item.update({"quota_fails_closed_without_authority": False}),
|
|
"quota_fails_closed_without_authority",
|
|
),
|
|
(
|
|
"rd-manager",
|
|
"R-D",
|
|
"manager_disposition_evidence",
|
|
lambda item: item["manager_disposition_cases"].remove("rejected-policy"),
|
|
"missing cases",
|
|
),
|
|
(
|
|
"rd-event",
|
|
"R-D",
|
|
"event_disposition_evidence",
|
|
lambda item: item.update({"events_correlate_to_manager_dispositions": False}),
|
|
"events_correlate_to_manager_dispositions",
|
|
),
|
|
(
|
|
"rd-ledger",
|
|
"R-D",
|
|
"ledger_disposition_evidence",
|
|
lambda item: item.update({"ledger_replay_preserves_terminal_disposition": False}),
|
|
"ledger_replay_preserves_terminal_disposition",
|
|
),
|
|
(
|
|
"rd-grace",
|
|
"R-D",
|
|
"grace_handling",
|
|
lambda item: item["grace_cases"].remove("restart-preserves-grace-clock"),
|
|
"missing cases",
|
|
),
|
|
):
|
|
with self.subTest(fault=fault), tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
mutation(data["gates"][gate]["evidence_fields"][field])
|
|
write_json(bundle, data)
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_status(root, bundle)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertTrue(any(expected in error for error in status["rejected_gates"][gate]))
|
|
|
|
def test_scanner_heal_release_bundle_requires_mixed_version_field_roles(self) -> None:
|
|
for gate, field, wrong_role in (
|
|
("G03", "mixed_peer_ack_fallback_oracle", "mixed-version-reader"),
|
|
("G09", "mixed_version_reader_evidence", "mixed-version-writer"),
|
|
("G09", "mixed_version_writer_evidence", "mixed-version-reader"),
|
|
("G09", "rollback_payload_evidence", "mixed-version-reader"),
|
|
("R-L", "crash_safe_source_retirement_evidence", "migration-gap"),
|
|
):
|
|
with self.subTest(gate=gate, field=field), tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
data["gates"][gate]["evidence_fields"][field]["mixed_version_role"] = wrong_role
|
|
write_json(bundle, data)
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_status(root, bundle)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertTrue(any("mixed-version role" in error for error in status["rejected_gates"][gate]))
|
|
|
|
def test_scanner_heal_release_bundle_requires_field_provenance(self) -> None:
|
|
for fault, mutation, expected in (
|
|
("source", lambda item: item.update({"source_revision": "c" * 40}), "source revision mismatch"),
|
|
("run-id", lambda item: item.pop("run_id"), "run_id"),
|
|
("window-id", lambda item: item.update({"measurement_window_id": item["run_id"]}), "separate run"),
|
|
("started-at", lambda item: item.update({"started_at": "not-a-time"}), "timestamp"),
|
|
("finished-at", lambda item: item.update({"finished_at": "2026-09-07T00:00:00Z"}), "timestamps are inverted"),
|
|
("command", lambda item: item.update({"command": []}), "command provenance"),
|
|
("artifact-format", lambda item: item.pop("artifact_format"), "artifact_format"),
|
|
):
|
|
with self.subTest(fault=fault), tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
mutation(data["gates"]["G01"]["evidence_fields"]["root_authority_evidence"])
|
|
write_json(bundle, data)
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_status(root, bundle)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertTrue(any(expected in error for error in status["rejected_gates"]["G01"]), fault)
|
|
|
|
def test_scanner_heal_release_bundle_requires_json_artifact_provenance(self) -> None:
|
|
for fault, mutation, evidence_path, expected in (
|
|
(
|
|
"summary-window",
|
|
lambda payload: payload.update({"measurement_window_id": "p1-stale-window"}),
|
|
("P1", "profile_evidence"),
|
|
"JSON artifact measurement window mismatch",
|
|
),
|
|
(
|
|
"summary-source",
|
|
lambda payload: payload.update({"source_revision": "c" * 40}),
|
|
("G09", "mixed_version_reader_evidence"),
|
|
"JSON artifact source revision mismatch",
|
|
),
|
|
(
|
|
"profile-kind",
|
|
lambda payload: payload.update({"artifact_kind": "flamegraph"}),
|
|
("P1", "profile_evidence", "allocation-profile"),
|
|
"JSON artifact kind mismatch",
|
|
),
|
|
(
|
|
"profile-run",
|
|
lambda payload: payload.update({"run_id": "p1-different-profile-run"}),
|
|
("P1", "profile_evidence", "rss-samples"),
|
|
"JSON artifact run_id mismatch",
|
|
),
|
|
(
|
|
"profile-raw-hash",
|
|
lambda payload: payload.update({"raw_profile_sha256": "0" * 64}),
|
|
("P1", "profile_evidence", "flamegraph"),
|
|
"raw profile hash mismatch",
|
|
),
|
|
(
|
|
"profile-save-cost",
|
|
lambda payload: payload.pop("saved_bytes"),
|
|
("P1", "profile_evidence", "save-frequency"),
|
|
"save-frequency.saved_bytes",
|
|
),
|
|
(
|
|
"fixture-marker",
|
|
lambda payload: payload.update({"fixture": True}),
|
|
("G08", "disk_full_matrix"),
|
|
"JSON artifact is fixture",
|
|
),
|
|
(
|
|
"g05-outcome-mirror",
|
|
lambda payload: payload["per_object_outcome_cases"].remove("object-skipped-by-policy"),
|
|
("G05", "per_object_outcome_oracle"),
|
|
"per_object_outcome_cases missing cases",
|
|
),
|
|
(
|
|
"g01-authority-mirror",
|
|
lambda payload: payload["root_authority_cases"].remove("incomplete-root-rejected"),
|
|
("G01", "root_authority_evidence"),
|
|
"root_authority_cases missing cases",
|
|
),
|
|
(
|
|
"g06-truncation-mirror",
|
|
lambda payload: payload["truncation_cases"].remove("trailing-data-node-status-reject"),
|
|
("G06", "truncation_behavior"),
|
|
"truncation_cases missing cases",
|
|
),
|
|
(
|
|
"mrf-artifact-kind",
|
|
lambda payload: payload.update({"artifact_kind": "generic-json"}),
|
|
("G08", "disk_full_matrix"),
|
|
"JSON artifact kind must be mrf-disk-full-enospc-matrix",
|
|
),
|
|
(
|
|
"g08-case-mirror",
|
|
lambda payload: payload["disk_full_cases"].remove("manifest-write-enospc"),
|
|
("G08", "disk_full_matrix"),
|
|
"JSON artifact missing cases",
|
|
),
|
|
(
|
|
"p4-gc-mirror",
|
|
lambda payload: payload.update({"pending_responsibilities_after_gc": 1}),
|
|
("P4", "mrf_cleanup_gc_soak_evidence"),
|
|
"JSON artifact requires zero pending responsibilities",
|
|
),
|
|
(
|
|
"rd-grace-mirror",
|
|
lambda payload: payload["grace_cases"].remove("grace-expired-prunes-terminal"),
|
|
("R-D", "grace_handling"),
|
|
"grace_cases missing cases",
|
|
),
|
|
(
|
|
"rl-migration-gap-mirror",
|
|
lambda payload: payload.update({"migration_gap_closed": False}),
|
|
("R-L", "migration_gap_evidence"),
|
|
"migration_gap_closed",
|
|
),
|
|
(
|
|
"g14-topology-mirror",
|
|
lambda payload: payload.update({"topology": {"erasure": "EC8+4", "nodes": 4, "drives_per_node": 4}}),
|
|
("G14", "ec8_4_evidence"),
|
|
"JSON artifact topology mismatch",
|
|
),
|
|
(
|
|
"g14-case-source",
|
|
lambda payload: payload["case_evidence"][0].update({"source_revision": "c" * 40}),
|
|
("G14", "multi_pool_evidence"),
|
|
"JSON artifact case_evidence mismatch",
|
|
),
|
|
):
|
|
with self.subTest(fault=fault), tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
gate, field, *artifact_kind = evidence_path
|
|
evidence = data["gates"][gate]["evidence_fields"][field]
|
|
if artifact_kind:
|
|
item = evidence["profile_artifacts"][artifact_kind[0]]
|
|
else:
|
|
item = evidence
|
|
artifact = bundle.parent / item["artifact"]
|
|
payload = read_json(artifact)
|
|
mutation(payload)
|
|
write_json(artifact, payload)
|
|
item["sha256"] = digest(artifact)
|
|
write_json(bundle, data)
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_status(root, bundle)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertTrue(any(expected in error for error in status["rejected_gates"][gate]), fault)
|
|
|
|
def test_scanner_heal_release_bundle_requires_same_gate_measurement_window(self) -> None:
|
|
for gate, field in (("G14", "multi_pool_evidence"), ("P1", "profile_evidence"), ("P3", "heal_capacity_measurement")):
|
|
with self.subTest(gate=gate), tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
evidence = data["gates"][gate]["evidence_fields"][field]
|
|
evidence["measurement_window_id"] = f"{gate.lower()}-different-window"
|
|
artifacts = [evidence]
|
|
if "profile_artifacts" in evidence:
|
|
artifacts.extend(evidence["profile_artifacts"].values())
|
|
for item in artifacts:
|
|
artifact = bundle.parent / item["artifact"]
|
|
payload = read_json(artifact)
|
|
payload["measurement_window_id"] = evidence["measurement_window_id"]
|
|
write_json(artifact, payload)
|
|
if "measurement_window_id" in item:
|
|
item["measurement_window_id"] = evidence["measurement_window_id"]
|
|
item["sha256"] = digest(artifact)
|
|
write_json(bundle, data)
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_status(root, bundle)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
expected = (
|
|
"case_evidence[0].measurement_window_id mismatch"
|
|
if gate == "G14"
|
|
else "must share one measurement window"
|
|
)
|
|
self.assertTrue(any(expected in error
|
|
for error in status["rejected_gates"][gate]))
|
|
|
|
def test_scanner_heal_release_bundle_requires_domain_evidence(self) -> None:
|
|
for fault, gate, field, mutation, expected in (
|
|
(
|
|
"scoped-ack-cases",
|
|
"G03",
|
|
"scoped_ack_request_identity",
|
|
lambda item: item["scoped_ack_cases"].remove("exact-generation"),
|
|
"missing cases",
|
|
),
|
|
(
|
|
"scoped-ack-fallback",
|
|
"G03",
|
|
"scoped_ack_request_identity",
|
|
lambda item: item.update({"whole_cycle_fallback_observed": False}),
|
|
"whole_cycle_fallback_observed",
|
|
),
|
|
(
|
|
"scoped-ack-capability-snapshot",
|
|
"G03",
|
|
"participating_peer_capability_snapshot",
|
|
lambda item: item.pop("capability_probe_observed"),
|
|
"capability_probe_observed",
|
|
),
|
|
(
|
|
"scoped-ack-mixed-peer-restart",
|
|
"G03",
|
|
"mixed_peer_ack_fallback_oracle",
|
|
lambda item: item.update({"restarted_peer_rejected": False}),
|
|
"restarted_peer_rejected",
|
|
),
|
|
(
|
|
"durable-intent-cases",
|
|
"G04",
|
|
"root_floor_intent_crash_evidence",
|
|
lambda item: item["durable_intent_cases"].remove("persist-failure-no-202"),
|
|
"durable_intent_cases missing cases",
|
|
),
|
|
(
|
|
"durable-intent-persist",
|
|
"G04",
|
|
"root_floor_intent_crash_evidence",
|
|
lambda item: item.pop("persist_failure_blocks_acceptance"),
|
|
"persist_failure_blocks_acceptance",
|
|
),
|
|
(
|
|
"g12-reset-quota-cases",
|
|
"G12",
|
|
"reset_quota_path_evidence",
|
|
lambda item: item["quota_path_cases"].remove("future-reservation-protocol-fail-closed"),
|
|
"quota_path_cases missing cases",
|
|
),
|
|
(
|
|
"g12-settlement-quota-cases",
|
|
"G12",
|
|
"settlement_quota_path_evidence",
|
|
lambda item: item["quota_path_cases"].remove("rejected-object-not-visible"),
|
|
"quota_path_cases missing cases",
|
|
),
|
|
(
|
|
"mixed-version-cases",
|
|
"G09",
|
|
"mixed_version_writer_evidence",
|
|
lambda item: item["mixed_version_cases"].remove("old-reader-new-writer"),
|
|
"mixed_version_cases missing cases",
|
|
),
|
|
(
|
|
"rollback-payload",
|
|
"G09",
|
|
"rollback_payload_evidence",
|
|
lambda item: item.update({"rollback_payload_replayed": False}),
|
|
"rollback_payload_replayed",
|
|
),
|
|
(
|
|
"legacy-source-conflict-cases",
|
|
"R-L",
|
|
"legacy_source_conflict_evidence",
|
|
lambda item: item["legacy_source_conflict_cases"].remove("newer-target-conflict-rejected"),
|
|
"legacy_source_conflict_cases missing cases",
|
|
),
|
|
(
|
|
"migration-gap-closed",
|
|
"R-L",
|
|
"migration_gap_evidence",
|
|
lambda item: item.update({"migration_gap_closed": False}),
|
|
"migration_gap_closed",
|
|
),
|
|
(
|
|
"source-retirement-crash-safe",
|
|
"R-L",
|
|
"crash_safe_source_retirement_evidence",
|
|
lambda item: item.update({"source_retirement_is_crash_safe": False}),
|
|
"source_retirement_is_crash_safe",
|
|
),
|
|
(
|
|
"scheduler-bounds",
|
|
"G10",
|
|
"scheduler_bound_evidence",
|
|
lambda item: item["scheduler_bounds"].remove("lock-hold-bound"),
|
|
"scheduler_bounds missing cases",
|
|
),
|
|
(
|
|
"pressure-metric",
|
|
"G10",
|
|
"pressure_recovery_evidence",
|
|
lambda item: item["pressure_metrics"].pop("heal_lock_wait_p99_ms"),
|
|
"heal_lock_wait_p99_ms",
|
|
),
|
|
(
|
|
"cold-walk-share",
|
|
"P1",
|
|
"cold_walk_share_measurement",
|
|
lambda item: item.update({"cold_walk_share": 1.2}),
|
|
"cold_walk_share exceeds one",
|
|
),
|
|
(
|
|
"foreground-throughput",
|
|
"P1",
|
|
"foreground_latency_throughput_measurement",
|
|
lambda item: item.pop("throughput_ops"),
|
|
"throughput_ops",
|
|
),
|
|
(
|
|
"profile-cost",
|
|
"P1",
|
|
"profile_evidence",
|
|
lambda item: item.pop("save_operations"),
|
|
"save_operations",
|
|
),
|
|
(
|
|
"abba-leg",
|
|
"P3",
|
|
"two_hour_pressure_measurement",
|
|
lambda item: item["abba_legs"].remove("B2"),
|
|
"abba_legs missing cases",
|
|
),
|
|
(
|
|
"pressure-scenario",
|
|
"P3",
|
|
"two_hour_pressure_measurement",
|
|
lambda item: item["scenarios"].remove("running-heal"),
|
|
"scenarios missing cases",
|
|
),
|
|
(
|
|
"heal-capacity",
|
|
"P3",
|
|
"heal_capacity_measurement",
|
|
lambda item: item["heal_capacity"].pop("completed_objects"),
|
|
"completed_objects",
|
|
),
|
|
(
|
|
"recovery-window",
|
|
"P3",
|
|
"recovery_window_measurement",
|
|
lambda item: item["fault_modes"].remove("process-crash-restart"),
|
|
"fault_modes missing cases",
|
|
),
|
|
):
|
|
with self.subTest(fault=fault), tempfile.TemporaryDirectory() as tmp:
|
|
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
|
data = read_json(bundle)
|
|
mutation(data["gates"][gate]["evidence_fields"][field])
|
|
write_json(bundle, data)
|
|
|
|
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
|
status = scanner_heal_release_bundle_status(root, bundle)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertTrue(any(expected in error for error in status["rejected_gates"][gate]), fault)
|
|
|
|
def test_scanner_heal_case_does_not_approve_pending_release(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
self.assertEqual(check_scanner_heal_evidence(root, run_dir, "background-target-restart"), [])
|
|
errors = check_scanner_heal_evidence(root, run_dir, "release")
|
|
self.assertEqual(len(errors), 21)
|
|
self.assertTrue(any(error.startswith("pending R-E:") for error in errors))
|
|
self.assertTrue(any(error.startswith("pending R-D:") for error in errors))
|
|
self.assertTrue(any(error.startswith("pending R-L:") for error in errors))
|
|
|
|
status = scanner_heal_release_status(root, run_dir)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertTrue(status["release_schema_capable"])
|
|
self.assertEqual(status["rejected_cases"], [])
|
|
self.assertEqual(len(status["pending_gates"]), 21)
|
|
self.assertIn("mixed-version-rollback", status["pending_lanes"])
|
|
self.assertIn("ec8-4-multiset", status["pending_lanes"])
|
|
self.assertIn("scheduler-pressure", status["pending_lanes"])
|
|
|
|
def test_scanner_heal_case_only_schema_cannot_approve_release(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
registry["schema"] = 1
|
|
registry["release_pending"] = {}
|
|
registry.pop("release_lanes")
|
|
registry.pop("release_requirements")
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
|
|
status = scanner_heal_release_status(root, run_dir)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertFalse(status["release_schema_capable"])
|
|
self.assertEqual(status["rejected_cases"], [])
|
|
self.assertEqual(status["pending_gates"], [])
|
|
|
|
def test_scanner_heal_release_requirements_cannot_be_cleared(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
registry["release_requirements"] = []
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
|
|
with self.assertRaisesRegex(ValueError, "invalid scanner/heal release requirements"):
|
|
scanner_heal_release_status(root, run_dir)
|
|
|
|
def test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
|
|
errors = check_scanner_heal_evidence(root, run_dir, "release")
|
|
for gate, text in (
|
|
("G03", "Exact scoped ACK"),
|
|
("G09", "mixed-version reader/writer"),
|
|
("G14", "3x4 EC8+4"),
|
|
("P3", "two-hour pressure/heal capacity"),
|
|
("R-L", "Legacy source conflicts"),
|
|
):
|
|
self.assertTrue(any(error.startswith(f"pending {gate}:") and text in error for error in errors), gate)
|
|
|
|
status = scanner_heal_release_status(root, run_dir)
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertIn("mixed-version-rollback", status["pending_lanes"])
|
|
self.assertIn("ec8-4-multiset", status["pending_lanes"])
|
|
self.assertIn("scheduler-pressure", status["pending_lanes"])
|
|
requirements, _, _ = scanner_heal_release_requirements(read_json(root / ".config/scanner-heal-required-tests.json"))
|
|
for gate, expected_fields in SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS.items():
|
|
self.assertEqual(tuple(requirements[gate]["evidence_fields"]), expected_fields, gate)
|
|
|
|
def test_scanner_heal_required_evidence_fields_cannot_be_removed(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
for gate, expected_fields in SCANNER_HEAL_RELEASE_BUNDLE_REQUIRED_EVIDENCE_FIELDS.items():
|
|
for requirement in registry["release_requirements"]:
|
|
if requirement["gate"] == gate:
|
|
requirement["evidence_fields"] = []
|
|
break
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
|
|
with self.subTest(gate=gate):
|
|
with self.assertRaisesRegex(ValueError, f"release gate {gate} missing required evidence fields"):
|
|
scanner_heal_release_status(root, run_dir)
|
|
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
for requirement in registry["release_requirements"]:
|
|
if requirement["gate"] == gate:
|
|
requirement["evidence_fields"] = list(expected_fields)
|
|
break
|
|
|
|
def test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
self.assertEqual(check_scanner_heal_evidence(root, run_dir, "background-target-crash-ec8-4"), [])
|
|
|
|
path = run_dir / "background-target-crash-ec8-4.json"
|
|
oracle = read_json(path)
|
|
oracle["objects"][0]["physical"]["data_blocks"] = 10
|
|
oracle["objects"][0]["physical"]["parity_blocks"] = 2
|
|
oracle["objects"][0]["expected_physical"]["data_blocks"] = 10
|
|
oracle["objects"][0]["expected_physical"]["parity_blocks"] = 2
|
|
write_json(path, oracle)
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
|
|
errors = check_scanner_heal_evidence(root, run_dir, "background-target-crash-ec8-4")
|
|
self.assertTrue(
|
|
any("EC data/parity geometry differs from the required case" in error for error in errors),
|
|
errors,
|
|
)
|
|
|
|
def test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
path = run_dir / "background-target-crash-ec8-4-multi-pool.json"
|
|
oracle = read_json(path)
|
|
oracle["outage_write_deferred_until_rejoin"] = True
|
|
oracle["outage_write"] = {
|
|
"attempted": True,
|
|
"required": False,
|
|
"accepted": True,
|
|
"attempts": 36,
|
|
"service_unavailable": 36,
|
|
}
|
|
write_json(path, oracle)
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
|
|
self.assertEqual(check_scanner_heal_evidence(root, run_dir, "background-target-crash-ec8-4-multi-pool"), [])
|
|
|
|
oracle["outage_write"]["accepted"] = False
|
|
write_json(path, oracle)
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
errors = check_scanner_heal_evidence(root, run_dir, "background-target-crash-ec8-4-multi-pool")
|
|
self.assertTrue(
|
|
any("optional outage-write final S3 body was not accepted" in error for error in errors),
|
|
errors,
|
|
)
|
|
|
|
def test_scanner_heal_pending_gate_cannot_map_to_implemented_lane(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
registry["release_requirements"][0]["lane"] = "single-set-restart"
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
|
|
with self.assertRaisesRegex(ValueError, "mapped to non-pending lane"):
|
|
scanner_heal_release_status(root, run_dir)
|
|
|
|
def test_scanner_heal_release_status_rejects_synthetic_case(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
registry["schema"] = 1
|
|
registry["release_pending"] = {}
|
|
registry.pop("release_lanes")
|
|
registry.pop("release_requirements")
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
path = run_dir / "background-target-crash.json"
|
|
oracle = read_json(path)
|
|
oracle["evidence"] = "synthetic"
|
|
write_json(path, oracle)
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
|
|
status = scanner_heal_release_status(root, run_dir)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertFalse(status["release_schema_capable"])
|
|
self.assertEqual(status["rejected_cases"], ["background-target-crash"])
|
|
self.assertEqual(status["pending_gates"], [])
|
|
|
|
def test_scanner_heal_release_status_rejects_focused_case_run(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
registry["schema"] = 1
|
|
registry["release_pending"] = {}
|
|
registry.pop("release_lanes")
|
|
registry.pop("release_requirements")
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
for case_id, requirement in registry["cases"].items():
|
|
if case_id != "background-target-restart":
|
|
(run_dir / requirement["oracle"]).unlink()
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
|
|
status = scanner_heal_release_status(root, run_dir)
|
|
self.assertEqual(status["decision"], "blocked")
|
|
self.assertFalse(status["release_approved"])
|
|
self.assertEqual(status["verified_cases"], ["background-target-restart"])
|
|
self.assertEqual(status["rejected_cases"], sorted(set(registry["cases"]) - {"background-target-restart"}))
|
|
|
|
def test_scanner_heal_finish_collects_oracles_from_registry(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
alternate = dict(registry["cases"]["background-target-restart"])
|
|
alternate["oracle"] = "alternate-target-restart.json"
|
|
registry["cases"]["alternate-target-restart"] = alternate
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
oracle = read_json(run_dir / "background-target-restart.json")
|
|
oracle["case"] = "alternate-target-restart"
|
|
write_json(run_dir / "alternate-target-restart.json", oracle)
|
|
(run_dir / "background-target-restart.json").unlink()
|
|
(run_dir / "execution.json").unlink()
|
|
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
|
|
self.assertIn("alternate-target-restart.json", read_json(run_dir / "execution.json")["artifacts"])
|
|
self.assertEqual(check_scanner_heal_evidence(root, run_dir, "alternate-target-restart"), [])
|
|
|
|
def test_scanner_heal_release_consumes_multiple_registry_oracles(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
registry = read_json(root / ".config/scanner-heal-required-tests.json")
|
|
alternate = dict(registry["cases"]["background-target-restart"])
|
|
alternate["oracle"] = "alternate-target-restart.json"
|
|
registry["cases"]["alternate-target-restart"] = alternate
|
|
write_json(root / ".config/scanner-heal-required-tests.json", registry)
|
|
oracle = read_json(run_dir / "background-target-restart.json")
|
|
oracle["case"] = "alternate-target-restart"
|
|
write_json(run_dir / "alternate-target-restart.json", oracle)
|
|
(run_dir / "execution.json").unlink()
|
|
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
|
|
artifacts = read_json(run_dir / "execution.json")["artifacts"]
|
|
self.assertIn("background-target-restart.json", artifacts)
|
|
self.assertIn("alternate-target-restart.json", artifacts)
|
|
self.assertEqual(check_scanner_heal_evidence(root, run_dir, "alternate-target-restart"), [])
|
|
errors = check_scanner_heal_evidence(root, run_dir, "release")
|
|
self.assertEqual(len(errors), 21)
|
|
self.assertTrue(all(error.startswith("pending ") for error in errors))
|
|
|
|
def test_scanner_heal_crash_case_rejects_restart_oracle(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
path = run_dir / "background-target-crash.json"
|
|
oracle = read_json(path)
|
|
oracle["evidence"] = "process-restart"
|
|
oracle["unclean_shutdown_marker"] = False
|
|
write_json(path, oracle)
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0, root)
|
|
|
|
self.assertTrue(check_scanner_heal_evidence(root, run_dir, "background-target-crash"))
|
|
|
|
def test_scanner_heal_rejects_broken_execution_and_artifacts(self) -> None:
|
|
for fault in ("exit", "missing", "zero", "skipped", "failed", "retry", "filtered", "ignored", "stale",
|
|
"hash", "binary", "synthetic", "wrong-run", "same-pid", "body", "parts", "listing", "topology"):
|
|
with self.subTest(fault=fault), tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
path = run_dir / "background-target-restart.json"
|
|
oracle = read_json(path)
|
|
if fault == "exit":
|
|
receipt = read_json(run_dir / "execution.json")
|
|
receipt["exit_code"] = 42
|
|
write_json(run_dir / "execution.json", receipt)
|
|
elif fault == "missing":
|
|
path.unlink()
|
|
elif fault == "zero":
|
|
(run_dir / "junit.xml").write_text("<testsuites/>")
|
|
elif fault in ("skipped", "failed", "retry"):
|
|
junit = run_dir / "junit.xml"
|
|
tag = {"skipped": "skipped", "failed": "failure", "retry": "rerunFailure"}[fault]
|
|
junit.write_text(junit.read_text().replace("/></testsuite>", f"><{tag}/></testcase></testsuite>"))
|
|
elif fault in ("filtered", "ignored"):
|
|
listing = read_json(run_dir / "listing.json")
|
|
case = next(iter(listing["rust-suites"]["e2e_test"]["testcases"].values()))
|
|
case["ignored"] = fault == "ignored"
|
|
case["filter-match"]["status"] = "mismatch" if fault == "filtered" else "matches"
|
|
write_json(run_dir / "listing.json", listing)
|
|
elif fault == "stale":
|
|
os.utime(path, (1, 1))
|
|
elif fault == "hash":
|
|
path.write_text(path.read_text() + " ")
|
|
elif fault == "binary":
|
|
Path(read_json(run_dir / "run.json")["binary"]["path"]).write_bytes(b"another build")
|
|
else:
|
|
if fault == "synthetic":
|
|
oracle["evidence"] = "synthetic"
|
|
elif fault == "wrong-run":
|
|
oracle["run_id"] = "f" * 32
|
|
elif fault == "same-pid":
|
|
oracle["pid_after"] = oracle["pid_before"]
|
|
elif fault == "body":
|
|
oracle["objects"][0]["actual_sha256"] = "e" * 64
|
|
elif fault == "parts":
|
|
oracle["objects"][0]["physical"]["present_part_fingerprints"] = {}
|
|
elif fault == "listing":
|
|
oracle["node_listings"][0] = []
|
|
elif fault == "topology":
|
|
oracle["topology"] = {"nodes": 3, "drives_per_node": 4}
|
|
write_json(path, oracle)
|
|
if fault not in ("exit", "missing", "stale", "hash", "binary"):
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0)
|
|
self.assertTrue(check_scanner_heal_evidence(root, run_dir, "background-target-restart"), fault)
|
|
|
|
def test_scanner_heal_receipts_reject_reuse_and_missing_builds(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
with self.assertRaisesRegex(ValueError, "already exists"):
|
|
finish_scanner_heal_receipt(run_dir, 0)
|
|
with self.assertRaisesRegex(ValueError, "must be new"):
|
|
begin_scanner_heal_receipt(root, run_dir, Path("missing"), Path("missing"))
|
|
with mock.patch("subprocess.check_output", side_effect=["", "b" * 40]):
|
|
with self.assertRaises(FileNotFoundError):
|
|
begin_scanner_heal_receipt(root, Path(tmp) / "new-run", Path(tmp) / "missing", Path(tmp) / "missing")
|
|
|
|
def test_scanner_heal_begin_requires_embedded_source_provenance(self) -> None:
|
|
for kind in ("current", "stale", "dirty", "unknown"):
|
|
with self.subTest(kind=kind), tempfile.TemporaryDirectory() as tmp:
|
|
root, _ = self.scanner_heal_fixture(Path(tmp))
|
|
sources = root / "crates/e2e_test/src"
|
|
sources.mkdir(parents=True)
|
|
(sources / "heal_erasure_disk_rebuild_test.rs").write_bytes(b"oracle source")
|
|
(sources / "chaos.rs").write_bytes(b"census source")
|
|
revision = "c" * 40 if kind == "stale" else "b" * 40
|
|
version = f"rustfs\ngit commit : {revision}\ngit status :\n"
|
|
if kind == "dirty":
|
|
version += "modified source\n"
|
|
if kind == "unknown":
|
|
version = "rustfs without build provenance"
|
|
with mock.patch("subprocess.check_output", side_effect=["", "b" * 40, version, "c" * 40]), \
|
|
mock.patch.dict(os.environ, {"RUSTFS_E2E_EXPECTED_FEATURES": "default"}):
|
|
directory = Path(tmp) / "fresh"
|
|
binary = Path(tmp) / "fake-binary"
|
|
if kind == "current":
|
|
begin_scanner_heal_receipt(root, directory, binary, binary)
|
|
self.assertEqual(read_json(directory / "run.json")["binary_source_revision"], "b" * 40)
|
|
else:
|
|
with self.assertRaisesRegex(ValueError, "server binary"):
|
|
begin_scanner_heal_receipt(root, directory, binary, binary)
|
|
self.assertFalse(directory.exists())
|
|
|
|
def test_scanner_heal_rejects_copied_junit_and_wrong_suite_build(self) -> None:
|
|
for fault in ("old-junit", "missing-time", "wrong-binary", "common-source", "lockfile", "features", "dirty-build"):
|
|
with self.subTest(fault=fault), tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
if fault in ("old-junit", "missing-time"):
|
|
path = run_dir / "junit.xml"
|
|
xml = ET.fromstring(path.read_bytes())
|
|
testcase = next(xml.iter("testcase"))
|
|
if fault == "old-junit":
|
|
testcase.set("timestamp", "2000-01-01T00:00:00.000Z")
|
|
else:
|
|
del testcase.attrib["timestamp"]
|
|
# Rewriting/copying gives an old execution a fresh mtime.
|
|
path.write_bytes(ET.tostring(xml))
|
|
elif fault == "wrong-binary":
|
|
path = run_dir / "listing.json"
|
|
listing = read_json(path)
|
|
another = Path(tmp) / "another-binary"
|
|
another.write_bytes(Path(tmp, "fake-binary").read_bytes())
|
|
listing["rust-suites"]["e2e_test"]["binary-path"] = str(another)
|
|
write_json(path, listing)
|
|
else:
|
|
path = run_dir / "background-target-restart.json"
|
|
oracle = read_json(path)
|
|
key, value = {"common-source": ("source_revision", "f" * 40), "lockfile": ("lock_blob", "f" * 40),
|
|
"features": ("features", "default,sftp"), "dirty-build": ("dirty", True)}[fault]
|
|
oracle["test_build"][key] = value
|
|
write_json(path, oracle)
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0)
|
|
self.assertTrue(check_scanner_heal_evidence(root, run_dir, "background-target-restart"), fault)
|
|
|
|
def test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers(self) -> None:
|
|
valid = {"schema": 1, "nodes": 4, "drives_per_node": 1, "pid_before": 10, "pid_after": 11,
|
|
"erasure_index": 1, "data_blocks": 2, "parity_blocks": 2}
|
|
cases = [(field, value) for field, correct in valid.items() for value in (True, float(correct))]
|
|
cases += [("erasure_index", 5), ("erasure_index", 0), ("pid_after", -1)]
|
|
for field, value in cases:
|
|
with self.subTest(field=field, value=value), tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
path = run_dir / "background-target-restart.json"
|
|
oracle = read_json(path)
|
|
if field in ("nodes", "drives_per_node"):
|
|
oracle["topology"][field] = value
|
|
elif field in ("erasure_index", "data_blocks", "parity_blocks"):
|
|
oracle["objects"][-1]["physical"][field] = value
|
|
else:
|
|
oracle[field] = value
|
|
write_json(path, oracle)
|
|
(run_dir / "execution.json").unlink()
|
|
finish_scanner_heal_receipt(run_dir, 0)
|
|
self.assertTrue(check_scanner_heal_evidence(root, run_dir, "background-target-restart"))
|
|
for filename in ("run.json", ".config/scanner-heal-required-tests.json"):
|
|
with self.subTest(filename=filename), tempfile.TemporaryDirectory() as tmp:
|
|
root, run_dir = self.scanner_heal_fixture(Path(tmp))
|
|
path = (root if filename.startswith(".config") else run_dir) / filename
|
|
content = read_json(path)
|
|
content["schema"] = True
|
|
write_json(path, content)
|
|
self.assertTrue(check_scanner_heal_evidence(root, run_dir, "background-target-restart"))
|
|
|
|
def test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
(root / ".config").mkdir()
|
|
fixture = root / "fixture.hex"
|
|
fixture.write_text("4142")
|
|
requirements = {
|
|
"tests": [{"invariant": "commit", "suite": "store", "name": "commit_test"}],
|
|
"fixtures": [{"path": "fixture.hex", "sha256": hashlib.sha256(fixture.read_bytes()).hexdigest()}],
|
|
}
|
|
(root / ".config/ecstore-required-tests.json").write_text(json.dumps(requirements))
|
|
listing = root / "listing.json"
|
|
good = {"ignored": False, "filter-match": {"status": "matches"}}
|
|
for case, testcase in (("selected", good), ("missing", {}), ("ignored", dict(good, ignored=True)),
|
|
("filtered", dict(good, **{"filter-match": {"status": "mismatch"}}))):
|
|
with self.subTest(case=case):
|
|
listing.write_text(json.dumps({"rust-suites": {"store": {"testcases": {"commit_test": testcase}}}}))
|
|
self.assertEqual(bool(check_core_listing(root, listing)), case != "selected")
|
|
listing.write_text(json.dumps({"rust-suites": {"store": {"testcases": {"commit_test": good}}}}))
|
|
fixture.write_text("4143")
|
|
self.assertIn("sha256 mismatch", check_core_listing(root, listing)[0])
|
|
fixture.unlink()
|
|
self.assertIn("fixture missing", check_core_listing(root, listing)[0])
|
|
listing.write_text("not json")
|
|
self.assertIn("cannot read", check_core_listing(root, listing)[0])
|
|
|
|
def test_ilm_lane_keeps_the_measured_cargo_build_budget(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
workflow = root / ".github/workflows/ci.yml"
|
|
workflow.parent.mkdir(parents=True)
|
|
valid = (
|
|
"jobs:\n"
|
|
" test-ilm-integration-serial:\n"
|
|
" steps:\n"
|
|
" - name: Run ignored ILM integration tests serially\n"
|
|
" env:\n"
|
|
" CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}\n"
|
|
" run: cargo nextest run\n"
|
|
" next-job:\n"
|
|
" steps: []\n"
|
|
)
|
|
workflow.write_text(valid)
|
|
self.assertEqual(check_ilm_build_budget(root), [])
|
|
|
|
workflow.write_text(valid.replace(" CARGO_BUILD_JOBS:", " BUILD_JOBS:"))
|
|
self.assertEqual(len(check_ilm_build_budget(root)), 1)
|
|
|
|
def test_vault_tests_require_cross_process_serialization(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
src = root / "crates/e2e_test/src/kms"
|
|
src.mkdir(parents=True)
|
|
(src / "vault_test.rs").write_text("fn test(env: VaultTestEnvironment) {}\n")
|
|
config = root / ".config/nextest.toml"
|
|
config.parent.mkdir()
|
|
config.write_text(
|
|
"[profile.default]\n"
|
|
"[[profile.default.overrides]]\n"
|
|
"filter = 'test(/^kms::vault_test::/)'\n"
|
|
"test-group = 'e2e-vault'\n"
|
|
"[profile.e2e-full]\n"
|
|
)
|
|
self.assertEqual(len(check_vault_test_groups(root)), 1)
|
|
config.write_text(
|
|
config.read_text()
|
|
+ "[[profile.e2e-full.overrides]]\n"
|
|
+ "filter = 'test(/^kms::vault_test::/)'\n"
|
|
+ "test-group = 'e2e-vault'\n"
|
|
)
|
|
self.assertEqual(check_vault_test_groups(root), [])
|
|
|
|
def test_runner_readiness_fails_closed(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
script = root / "scripts/run_e2e_tests.sh"
|
|
script.parent.mkdir(parents=True)
|
|
valid_runner = (
|
|
"start_rustfs() {\n"
|
|
' curl --fail "http://localhost:9000/health/ready" && return 0\n'
|
|
" return 1\n"
|
|
"}\n\n# Function to run tests\n"
|
|
"--include-ignored --test-threads=1\n"
|
|
"if ! start_rustfs; then\n"
|
|
' print_error "Failed to start RustFS properly"\n'
|
|
" exit 1\n"
|
|
"fi\n"
|
|
)
|
|
script.write_text(valid_runner)
|
|
self.assertEqual(check_runner_selection(root), [])
|
|
|
|
script.write_text(valid_runner.replace("/health/ready", "/health"))
|
|
self.assertEqual(len(check_runner_selection(root)), 1)
|
|
|
|
script.write_text(valid_runner.replace("return 1", "nc -z localhost 9000 && return 0"))
|
|
self.assertEqual(len(check_runner_selection(root)), 1)
|
|
|
|
script.write_text(
|
|
valid_runner.replace(
|
|
' print_error "Failed to start RustFS properly"\n exit 1',
|
|
' if [ -n "$RUSTFS_PID" ]; then\n echo continuing\n else\n exit 1\n fi',
|
|
)
|
|
)
|
|
self.assertEqual(len(check_runner_selection(root)), 1)
|
|
|
|
def test_e2e_requires_registration(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
src = root / "crates/e2e_test/src"
|
|
src.mkdir(parents=True)
|
|
(src / "lib.rs").write_text("")
|
|
test_file = src / "boundary_test.rs"
|
|
test_file.write_text("#[test]\nfn boundary() {}\n")
|
|
self.assertEqual(len(check_e2e_modules(root)), 1)
|
|
(src / "lib.rs").write_text("mod boundary_test;\n")
|
|
self.assertEqual(check_e2e_modules(root), [])
|
|
|
|
nested = src / "protocols"
|
|
nested.mkdir()
|
|
(nested / "mod.rs").write_text("mod fixed_port_test;\n")
|
|
(nested / "fixed_port_test.rs").write_text("#[test]\nfn fixed_port() {}\n")
|
|
self.assertEqual(len(check_e2e_modules(root)), 1)
|
|
(src / "lib.rs").write_text("mod boundary_test;\nmod protocols;\n")
|
|
self.assertEqual(check_e2e_modules(root), [])
|
|
|
|
(src / "lib.rs").write_text("#[cfg(any())]\nmod boundary_test;\nmod protocols;\n")
|
|
self.assertEqual(len(check_e2e_modules(root)), 1)
|
|
|
|
(src / "lib.rs").write_text(
|
|
"#[cfg(any())]\n/// hidden module\nmod boundary_test;\n#[cfg_attr(test, cfg(any()))]\nmod protocols;\n"
|
|
)
|
|
self.assertEqual(len(check_e2e_modules(root)), 2)
|
|
|
|
(src / "lib.rs").write_text(
|
|
'const PHANTOM: &str = r#"{\nmod boundary_test;\n"#;\ndiscard! { mod protocols; }\n'
|
|
)
|
|
self.assertEqual(len(check_e2e_modules(root)), 2)
|
|
|
|
(src / "lib.rs").write_text(
|
|
'#[cfg(all(test, target_os = r"windows" /* target_os = "linux" */))]\n'
|
|
"mod boundary_test;\nmod protocols;\n"
|
|
)
|
|
self.assertEqual(len(check_e2e_modules(root)), 1)
|
|
|
|
(src / "lib.rs").write_text(
|
|
'#[cfg(all(test, target_os = r"windows"))] // #[cfg(all(test, target_os = "linux"))]\n'
|
|
"mod boundary_test;\nmod protocols;\n"
|
|
)
|
|
self.assertEqual(len(check_e2e_modules(root)), 1)
|
|
|
|
def test_fuzz_runtime_uses_matrix_target(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
(root / "fuzz").mkdir()
|
|
(root / "scripts/fuzz").mkdir(parents=True)
|
|
(root / ".github/workflows").mkdir(parents=True)
|
|
(root / "fuzz/Cargo.toml").write_text(
|
|
'dep = { path = "../crates/dep" }\n[[bin]]\nname = "one"\n'
|
|
)
|
|
(root / "scripts/fuzz/run.sh").write_text('targets="one"\n')
|
|
(root / ".github/workflows/fuzz.yml").write_text(
|
|
'paths:\n - "crates/dep/**"\n'
|
|
"target: [one]\nFUZZ_TARGET: fixed\n"
|
|
"target: [one]\nFUZZ_TARGET: ${{ matrix.target }}\n"
|
|
"for target in one; do\n"
|
|
" fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/one\n"
|
|
)
|
|
self.assertEqual(len(check_fuzz_targets(root)), 1)
|
|
|
|
def test_s3_runner_rejects_unbounded_failure_locals(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
runner = root / "scripts/s3-tests/run.sh"
|
|
runner.parent.mkdir(parents=True)
|
|
valid_runner = (
|
|
"test_s3_api_ready() {\n"
|
|
' READY_CODE=$(curl "http://${S3_HOST}:${S3_PORT}/health/ready")\n'
|
|
' if [ "${READY_CODE}" != "200" ]; then\n'
|
|
" return 1\n"
|
|
" fi\n"
|
|
" if command -v awscurl; then\n"
|
|
" RESPONSE=$(awscurl --service s3)\n"
|
|
' if echo "${RESPONSE}" | grep -q "<ListAllMyBucketsResult"; then\n'
|
|
" return 0\n"
|
|
" fi\n"
|
|
" return 1\n"
|
|
" fi\n"
|
|
" return 0\n"
|
|
"}\n\n# First, wait\n"
|
|
"tox -- -vv -ra --tb=long\n"
|
|
)
|
|
runner.write_text(valid_runner)
|
|
self.assertEqual(check_s3_tests_runner(root), [])
|
|
runner.write_text(valid_runner.replace("--tb=long", "--showlocals --tb=long"))
|
|
self.assertEqual(len(check_s3_tests_runner(root)), 1)
|
|
runner.write_text(valid_runner.replace("/health/ready", "/health"))
|
|
self.assertEqual(len(check_s3_tests_runner(root)), 1)
|
|
runner.write_text(valid_runner.replace(" return 0\n}\n", " return 0\n return 0\n}\n"))
|
|
self.assertEqual(len(check_s3_tests_runner(root)), 1)
|
|
runner.write_text(
|
|
valid_runner.replace(
|
|
" return 1\n fi\n return 0\n",
|
|
" return 0\n fi\n return 1\n",
|
|
)
|
|
)
|
|
self.assertEqual(len(check_s3_tests_runner(root)), 1)
|
|
runner.write_text(valid_runner.replace(" return 1\n fi", " false || return 0\n return 1\n fi"))
|
|
self.assertEqual(len(check_s3_tests_runner(root)), 1)
|
|
runner.write_text(valid_runner.replace('echo "${RESPONSE}"', 'echo "<ListAllMyBucketsResult"'))
|
|
self.assertEqual(len(check_s3_tests_runner(root)), 1)
|
|
runner.write_text(
|
|
valid_runner.replace(
|
|
"RESPONSE=$(awscurl --service s3)",
|
|
'RESPONSE=$(awscurl --service s3 || echo "<ListAllMyBucketsResult")',
|
|
)
|
|
)
|
|
self.assertEqual(len(check_s3_tests_runner(root)), 1)
|
|
runner.write_text(
|
|
valid_runner.replace(
|
|
"RESPONSE=$(awscurl --service s3)",
|
|
'RESPONSE=$(awscurl --service s3; echo "<ListAllMyBucketsResult")',
|
|
)
|
|
)
|
|
self.assertEqual(len(check_s3_tests_runner(root)), 1)
|
|
with (
|
|
mock.patch(__name__ + ".check_e2e_modules", return_value=[]),
|
|
mock.patch(__name__ + ".check_vault_test_groups", return_value=[]),
|
|
mock.patch(__name__ + ".check_fuzz_targets", return_value=[]),
|
|
mock.patch(__name__ + ".check_core_fixtures", return_value=[]),
|
|
mock.patch(__name__ + ".check_runner_selection", return_value=[]),
|
|
mock.patch(__name__ + ".check_workflow_readiness", return_value=[]),
|
|
mock.patch(__name__ + ".check_profile_definitions", return_value=[]),
|
|
mock.patch(__name__ + ".check_ilm_build_budget", return_value=[]),
|
|
mock.patch(__name__ + ".check_scheduled_alerts", return_value=[]),
|
|
mock.patch(__name__ + ".check_quick_checks", return_value=[]),
|
|
):
|
|
self.assertEqual(len(validate(root)), 1)
|
|
|
|
def test_workflow_readiness_requires_dependency_probe(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
workflows = root / ".github/workflows"
|
|
workflows.mkdir(parents=True)
|
|
valid_workflow = (
|
|
"jobs:\n"
|
|
" test:\n"
|
|
" steps:\n"
|
|
" - name: Wait for RustFS ready\n"
|
|
" run: |\n"
|
|
" for _ in {1..60}; do\n"
|
|
" if curl -sf http://127.0.0.1:9000/health/ready; then\n"
|
|
" exit 0\n"
|
|
" fi\n"
|
|
" done\n"
|
|
" exit 1\n"
|
|
" - name: Run tests\n"
|
|
" run: true\n"
|
|
)
|
|
for name in ("e2e-s3tests.yml", "mint.yml"):
|
|
(workflows / name).write_text(valid_workflow)
|
|
self.assertEqual(check_workflow_readiness(root), [])
|
|
|
|
(workflows / "mint.yml").write_text(valid_workflow.replace("/health/ready", "/health"))
|
|
self.assertEqual(len(check_workflow_readiness(root)), 1)
|
|
(workflows / "mint.yml").write_text(
|
|
valid_workflow.replace(
|
|
"if curl -sf http://127.0.0.1:9000/health/ready; then",
|
|
"if curl -sf http://127.0.0.1:9000/health/ready || true; then",
|
|
)
|
|
)
|
|
self.assertEqual(len(check_workflow_readiness(root)), 1)
|
|
(workflows / "mint.yml").write_text(
|
|
valid_workflow.replace(
|
|
"if curl -sf http://127.0.0.1:9000/health/ready; then",
|
|
"if curl -sf http://127.0.0.1:9000/health/ready || :; then",
|
|
)
|
|
)
|
|
self.assertEqual(len(check_workflow_readiness(root)), 1)
|
|
(workflows / "mint.yml").write_text(
|
|
valid_workflow.replace(
|
|
"if curl -sf http://127.0.0.1:9000/health/ready; then\n exit 0\n fi",
|
|
"curl -sf http://127.0.0.1:9000/health/ready || true\n exit 0",
|
|
)
|
|
)
|
|
self.assertEqual(len(check_workflow_readiness(root)), 1)
|
|
|
|
def test_profile_listing_enforces_selection(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
(root / ".config").mkdir()
|
|
digest = hashlib.sha256(b"suite::two\n").hexdigest()
|
|
(root / ".config/e2e-smoke-selection.txt").write_text(f"sha256={digest}\n")
|
|
listing = root / "listing.json"
|
|
listing.write_text(
|
|
json.dumps(
|
|
{
|
|
"rust-suites": {
|
|
"suite": {
|
|
"testcases": {
|
|
"one": {"filter-match": {"status": "matches"}},
|
|
"two": {"filter-match": {"status": "mismatch"}},
|
|
}
|
|
}
|
|
}
|
|
}
|
|
)
|
|
)
|
|
self.assertEqual(len(check_profile_listing(root, "e2e-smoke", listing)), 1)
|
|
|
|
def test_profile_listing_binds_platform_digest(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
(root / ".config").mkdir()
|
|
darwin_digest = hashlib.sha256(b"suite::darwin\n").hexdigest()
|
|
linux_digest = hashlib.sha256(b"suite::linux\n").hexdigest()
|
|
(root / ".config/e2e-full-selection.txt").write_text(
|
|
f"sha256-darwin={darwin_digest}\nsha256-linux={linux_digest}\n"
|
|
)
|
|
listing = root / "listing.json"
|
|
listing.write_text(
|
|
json.dumps(
|
|
{
|
|
"rust-suites": {
|
|
"suite": {
|
|
"testcases": {"darwin": {"filter-match": {"status": "matches"}}}
|
|
}
|
|
}
|
|
}
|
|
)
|
|
)
|
|
with mock.patch.object(sys, "platform", "linux"):
|
|
self.assertEqual(len(check_profile_listing(root, "e2e-full", listing)), 1)
|
|
|
|
def test_update_profile_selection_changes_only_requested_platform(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
(root / ".config").mkdir()
|
|
darwin_digest = "a" * 64
|
|
(root / ".config/e2e-full-selection.txt").write_text(
|
|
f"sha256-darwin={darwin_digest}\nsha256-linux={'b' * 64}\n"
|
|
)
|
|
listing = root / "listing.json"
|
|
listing.write_text(
|
|
json.dumps(
|
|
{
|
|
"rust-suites": {
|
|
"suite": {
|
|
"testcases": {"linux": {"filter-match": {"status": "matches"}}}
|
|
}
|
|
}
|
|
}
|
|
)
|
|
)
|
|
|
|
count, digest, key = update_profile_selection(root, "e2e-full", listing, "linux")
|
|
|
|
self.assertEqual(count, 1)
|
|
self.assertEqual(key, "sha256-linux")
|
|
self.assertEqual(
|
|
(root / ".config/e2e-full-selection.txt").read_text(),
|
|
f"sha256-darwin={darwin_digest}\nsha256-linux={digest}\n",
|
|
)
|
|
with mock.patch.object(sys, "platform", "linux"):
|
|
self.assertEqual(check_profile_listing(root, "e2e-full", listing), [])
|
|
selection = root / ".config/e2e-smoke-selection.txt"
|
|
selection.write_text(f"sha256={'a' * 64}\n")
|
|
|
|
count, digest, key = update_profile_selection(root, "e2e-smoke", listing, "linux")
|
|
|
|
self.assertEqual((count, key), (1, "sha256"))
|
|
self.assertEqual(selection.read_text(), f"sha256={digest}\n")
|
|
|
|
def test_scheduled_alerts_require_completion_watchdog(self) -> None:
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
alert = (
|
|
" alert-on-failure:\n"
|
|
" if: always() && github.event_name == 'schedule' && "
|
|
"contains(needs.*.result, 'failure')\n"
|
|
" permissions:\n"
|
|
" issues: write\n"
|
|
" steps:\n"
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
" - uses: ./.github/actions/schedule-failure-issue\n"
|
|
" with:\n"
|
|
" github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}\n"
|
|
)
|
|
names: list[str] = []
|
|
for index, relative in enumerate(SCHEDULED_ALERT_WORKFLOWS, start=1):
|
|
path = root / relative
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
names.append(path.stem)
|
|
path.write_text(
|
|
f'name: "{path.stem}"\n'
|
|
f'on:\n schedule:\n - cron: "{index} {index} * * *"\n'
|
|
f'jobs:\n{alert}'
|
|
)
|
|
watchdog = root / ".github/workflows/scheduled-validation-watchdog.yml"
|
|
watchdog.write_text(
|
|
"on:\n workflow_run:\n workflows:\n"
|
|
+ "\n".join(f' - "{name}"' for name in names)
|
|
+ "\njobs:\n"
|
|
+ " alert-on-incomplete-run:\n"
|
|
+ " github.event.workflow_run.event == 'schedule'\n"
|
|
+ " github.event.workflow_run.conclusion != 'success'\n"
|
|
+ " github.event.workflow_run.conclusion != 'failure'\n"
|
|
+ " permissions:\n"
|
|
+ " actions: read\n"
|
|
+ " issues: write\n"
|
|
+ " steps:\n"
|
|
+ " - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
+ " - uses: ./.github/actions/schedule-failure-issue\n"
|
|
+ " with:\n"
|
|
+ " github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}\n"
|
|
+ " workflow-name: ${{ github.event.workflow_run.name }}\n"
|
|
+ " source-run-id: ${{ github.event.workflow_run.id }}\n"
|
|
+ " source-run-attempt: ${{ github.event.workflow_run.run_attempt }}\n"
|
|
+ " source-event: ${{ github.event.workflow_run.event }}\n"
|
|
+ " source-ref-name: ${{ github.event.workflow_run.head_branch }}\n"
|
|
+ " source-sha: ${{ github.event.workflow_run.head_sha }}\n"
|
|
)
|
|
freshness = root / ".github/workflows/scheduled-validation-freshness.yml"
|
|
freshness.write_text(
|
|
"jobs:\n"
|
|
" check-freshness:\n"
|
|
" permissions:\n"
|
|
" actions: read\n"
|
|
" issues: write\n"
|
|
" steps:\n"
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
" - run: python3 scripts/check_scheduled_validation_freshness.py\n"
|
|
" - uses: ./.github/actions/schedule-failure-issue\n"
|
|
" if: failure()\n"
|
|
" with:\n"
|
|
" github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}\n"
|
|
" details-file: ${{ runner.temp }}/scheduled-validation-freshness.md\n"
|
|
)
|
|
checker = root / "scripts/check_scheduled_validation_freshness.py"
|
|
checker.parent.mkdir()
|
|
checker.write_text("")
|
|
self.assertEqual(check_scheduled_alerts(root), [])
|
|
|
|
first = root / SCHEDULED_ALERT_WORKFLOWS[0]
|
|
mutations = (
|
|
("contains(needs.*.result, 'failure')", "false"),
|
|
("issues: write", "issues: read"),
|
|
(
|
|
"uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0",
|
|
"uses: actions/checkout@missing",
|
|
),
|
|
(
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n",
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
" if: github.event_name == 'workflow_dispatch'\n",
|
|
),
|
|
(
|
|
" - uses: ./.github/actions/schedule-failure-issue\n",
|
|
" - uses: ./.github/actions/schedule-failure-issue\n"
|
|
" if: github.event_name == 'workflow_dispatch'\n",
|
|
),
|
|
("uses: ./.github/actions/schedule-failure-issue", "uses: actions/checkout@v7"),
|
|
("github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}", "github-token: missing"),
|
|
)
|
|
for required, replacement in mutations:
|
|
original = first.read_text()
|
|
first.write_text(original.replace(required, replacement))
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
first.write_text(original)
|
|
|
|
first_original = first.read_text()
|
|
real_steps = (
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
" - uses: ./.github/actions/schedule-failure-issue\n"
|
|
" with:\n"
|
|
" github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}\n"
|
|
)
|
|
first.write_text(
|
|
first_original.replace(
|
|
real_steps,
|
|
" - run: |\n"
|
|
" : <<'MARKER'\n"
|
|
" uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
" MARKER\n"
|
|
" - run: |\n"
|
|
" : <<'MARKER'\n"
|
|
" uses: ./.github/actions/schedule-failure-issue\n"
|
|
" github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}\n"
|
|
" MARKER\n",
|
|
)
|
|
)
|
|
self.assertTrue(check_scheduled_alerts(root))
|
|
first.write_text(
|
|
first_original.replace(
|
|
real_steps,
|
|
" - uses: ./.github/actions/schedule-failure-issue\n"
|
|
" with:\n"
|
|
" github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}\n"
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n",
|
|
)
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
first.write_text(first_original)
|
|
|
|
watchdog_mutations = (
|
|
("actions: read", "actions: none"),
|
|
("issues: write", "issues: read"),
|
|
(
|
|
"uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0",
|
|
"uses: actions/checkout@missing",
|
|
),
|
|
(
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n",
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
" if: github.event_name == 'workflow_dispatch'\n",
|
|
),
|
|
(
|
|
" - uses: ./.github/actions/schedule-failure-issue\n",
|
|
" - uses: ./.github/actions/schedule-failure-issue\n"
|
|
" if: github.event_name == 'workflow_dispatch'\n",
|
|
),
|
|
("uses: ./.github/actions/schedule-failure-issue", "uses: actions/checkout@v7"),
|
|
("github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}", "github-token: missing"),
|
|
("source-event: ${{ github.event.workflow_run.event }}", "source-event: watchdog"),
|
|
(
|
|
"source-ref-name: ${{ github.event.workflow_run.head_branch }}",
|
|
"source-ref-name: main",
|
|
),
|
|
("source-sha: ${{ github.event.workflow_run.head_sha }}", "source-sha: missing"),
|
|
)
|
|
for required, replacement in watchdog_mutations:
|
|
original = watchdog.read_text()
|
|
watchdog.write_text(original.replace(required, replacement))
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
watchdog.write_text(original)
|
|
|
|
watchdog_original = watchdog.read_text()
|
|
watchdog.write_text(
|
|
watchdog_original.replace("issues: write", "issues: read")
|
|
+ " decoy:\n permissions:\n issues: write\n"
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
watchdog.write_text(watchdog_original)
|
|
|
|
first_original = first.read_text()
|
|
first.write_text(
|
|
first_original.replace(' schedule:\n - cron: "1 1 * * *"\n', "")
|
|
+ ' decoy:\n strategy:\n matrix:\n cron:\n - "1 1 * * *"\n'
|
|
+ ' runs-on: ubuntu-latest\n steps:\n - run: true\n'
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
first.write_text(first_original)
|
|
|
|
first.write_text(
|
|
first_original.replace(
|
|
' - cron: "1 1 * * *"\n',
|
|
' - cron: "1 1 * * *"\n - cron: "0 5 * * *"\n',
|
|
)
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
first.write_text(
|
|
first_original.replace(
|
|
' - cron: "1 1 * * *"\n',
|
|
' - cron: "1 1 * * *"\n - cron: "2 2 * * *"\n',
|
|
)
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
first.write_text(first_original)
|
|
|
|
watchdog.write_text(
|
|
watchdog_original.replace(f' - "{names[0]}"\n', "")
|
|
+ f' decoy:\n strategy:\n matrix:\n workflow:\n - "{names[0]}"\n'
|
|
+ ' runs-on: ubuntu-latest\n steps:\n - run: true\n'
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
watchdog.write_text(watchdog_original)
|
|
|
|
watchdog.write_text(watchdog_original.replace(f' - "{names[0]}"\n', ""))
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
watchdog.write_text(watchdog_original)
|
|
original = first.read_text()
|
|
first.write_text(re.sub(r'- cron: "\d+ \d+', '- cron: "0 0', original, count=1))
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
first.write_text(original)
|
|
|
|
second = root / SCHEDULED_ALERT_WORKFLOWS[1]
|
|
second_original = second.read_text()
|
|
second.write_text(re.sub(r'- cron: "\d+ \d+', '- cron: "1 1', second_original, count=1))
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
second.write_text(second_original)
|
|
|
|
first.write_text(
|
|
first_original.replace(
|
|
' - cron: "1 1 * * *"\n',
|
|
' - cron: "7 0 * * *"\n timezone: "Asia/Shanghai"\n',
|
|
)
|
|
)
|
|
second.write_text(
|
|
second_original.replace(
|
|
' - cron: "2 2 * * *"\n',
|
|
' - cron: "2 2 * * *"\n - cron: "7 16 * * *"\n',
|
|
)
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
first.write_text(first_original)
|
|
second.write_text(second_original)
|
|
|
|
freshness_original = freshness.read_text()
|
|
freshness.write_text(freshness_original.replace("details-file:", "report-file:"))
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
freshness.write_text(
|
|
freshness_original.replace("github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}", "github-token: missing")
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
freshness.write_text(
|
|
freshness_original.replace(
|
|
"uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0",
|
|
"uses: actions/checkout@missing",
|
|
)
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
freshness.write_text(
|
|
freshness_original.replace(
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n",
|
|
" - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0\n"
|
|
" if: github.event_name == 'workflow_dispatch'\n",
|
|
)
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
freshness.write_text(
|
|
freshness_original.replace("if: failure()", "if: github.event_name == 'workflow_dispatch'")
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
freshness.write_text(
|
|
freshness_original.replace("issues: write", "issues: read")
|
|
+ " decoy:\n permissions:\n issues: write\n"
|
|
)
|
|
self.assertEqual(len(check_scheduled_alerts(root)), 1)
|
|
|
|
def main() -> int:
|
|
if sys.argv[1:] == ["--self-test"]:
|
|
suite = unittest.defaultTestLoader.loadTestsFromTestCase(SelfTests)
|
|
return 0 if unittest.TextTestRunner(verbosity=2).run(suite).wasSuccessful() else 1
|
|
if sys.argv[1:2] in (["--begin-scanner-heal"], ["--finish-scanner-heal"], ["--check-scanner-heal"],
|
|
["--check-scanner-heal-release"], ["--check-scanner-heal-release-bundle"],
|
|
["--check-scanner-heal-release-bundle-gate"],
|
|
["--assemble-scanner-heal-release-bundle"],
|
|
["--write-scanner-heal-release-bundle-fixture"]):
|
|
try:
|
|
if len(sys.argv) == 5 and sys.argv[1] == "--begin-scanner-heal":
|
|
begin_scanner_heal_receipt(ROOT, Path(sys.argv[2]), Path(sys.argv[3]), Path(sys.argv[4]))
|
|
return 0
|
|
if len(sys.argv) == 4 and sys.argv[1] == "--finish-scanner-heal":
|
|
finish_scanner_heal_receipt(Path(sys.argv[2]), int(sys.argv[3]))
|
|
return 0
|
|
if len(sys.argv) == 4 and sys.argv[1] == "--check-scanner-heal":
|
|
errors = check_scanner_heal_evidence(ROOT, Path(sys.argv[2]), sys.argv[3])
|
|
for error in errors:
|
|
print(f"ERROR: {error}", file=sys.stderr)
|
|
if not errors:
|
|
print(f"Case evidence verified: {sys.argv[3]}; this does not approve release")
|
|
return 1 if errors else 0
|
|
if len(sys.argv) == 3 and sys.argv[1] == "--check-scanner-heal-release":
|
|
try:
|
|
status = scanner_heal_release_status(ROOT, Path(sys.argv[2]))
|
|
except (OSError, KeyError, TypeError, ValueError, ET.ParseError) as error:
|
|
print(json.dumps({"schema": 1, "decision": "invalid", "release_approved": False,
|
|
"error": str(error)}, sort_keys=True, separators=(",", ":")))
|
|
return 2
|
|
print(json.dumps(status, sort_keys=True, separators=(",", ":")))
|
|
return 0 if status["release_approved"] else 1
|
|
if len(sys.argv) == 3 and sys.argv[1] == "--check-scanner-heal-release-bundle":
|
|
try:
|
|
status = scanner_heal_release_bundle_status(ROOT, Path(sys.argv[2]))
|
|
except (OSError, KeyError, TypeError, ValueError, ET.ParseError) as error:
|
|
print(json.dumps({"schema": 1, "decision": "invalid", "release_approved": False,
|
|
"error": str(error)}, sort_keys=True, separators=(",", ":")))
|
|
return 2
|
|
print(json.dumps(status, sort_keys=True, separators=(",", ":")))
|
|
return 0 if status["release_approved"] else 1
|
|
if len(sys.argv) == 4 and sys.argv[1] == "--check-scanner-heal-release-bundle-gate":
|
|
try:
|
|
status = scanner_heal_release_bundle_gate_status(ROOT, Path(sys.argv[2]), sys.argv[3])
|
|
except (OSError, KeyError, TypeError, ValueError, ET.ParseError) as error:
|
|
print(json.dumps({"schema": 1, "decision": "invalid", "release_approved": False,
|
|
"error": str(error)}, sort_keys=True, separators=(",", ":")))
|
|
return 2
|
|
print(json.dumps(status, sort_keys=True, separators=(",", ":")))
|
|
return 0 if status["decision"] == "verified" else 1
|
|
if len(sys.argv) >= 4 and sys.argv[1] == "--assemble-scanner-heal-release-bundle":
|
|
try:
|
|
bundle, status = assemble_scanner_heal_release_bundle_descriptors(
|
|
ROOT,
|
|
[Path(item) for item in sys.argv[2:-1]],
|
|
Path(sys.argv[-1]),
|
|
)
|
|
except (OSError, KeyError, TypeError, ValueError, ET.ParseError) as error:
|
|
print(json.dumps({"schema": 1, "decision": "invalid", "release_approved": False,
|
|
"error": str(error)}, sort_keys=True, separators=(",", ":")))
|
|
return 2
|
|
print(bundle)
|
|
print(json.dumps(status, sort_keys=True, separators=(",", ":")))
|
|
return 0 if status["release_approved"] else 1
|
|
if len(sys.argv) == 3 and sys.argv[1] == "--write-scanner-heal-release-bundle-fixture":
|
|
bundle = write_scanner_heal_release_bundle_fixture(ROOT, Path(sys.argv[2]))
|
|
print(bundle)
|
|
return 0
|
|
raise ValueError("expected --begin-scanner-heal DIR BINARY TEST_BINARY, --finish-scanner-heal DIR EXIT, --check-scanner-heal DIR CASE|release, --check-scanner-heal-release DIR, --check-scanner-heal-release-bundle FILE, --check-scanner-heal-release-bundle-gate FILE GATE, --assemble-scanner-heal-release-bundle DESCRIPTOR [DESCRIPTOR ...] DIR, or --write-scanner-heal-release-bundle-fixture DIR")
|
|
except (OSError, KeyError, TypeError, ValueError, subprocess.SubprocessError) as error:
|
|
print(f"ERROR: {error}", file=sys.stderr)
|
|
return 1
|
|
if len(sys.argv) == 3 and sys.argv[1] == "--check-core":
|
|
errors = check_core_listing(ROOT, Path(sys.argv[2]))
|
|
for error in errors:
|
|
print(f"ERROR: {error}", file=sys.stderr)
|
|
return 1 if errors else 0
|
|
if len(sys.argv) == 4 and sys.argv[1] == "--check-profile":
|
|
errors = check_profile_listing(ROOT, sys.argv[2], Path(sys.argv[3]))
|
|
if errors:
|
|
for error in errors:
|
|
print(f"ERROR: {error}", file=sys.stderr)
|
|
return 1
|
|
return 0
|
|
if len(sys.argv) == 5 and sys.argv[1] == "--update-profile":
|
|
try:
|
|
count, digest, key = update_profile_selection(ROOT, sys.argv[2], Path(sys.argv[3]), sys.argv[4])
|
|
except (FileNotFoundError, KeyError, TypeError, ValueError, json.JSONDecodeError) as error:
|
|
print(f"ERROR: cannot update {sys.argv[2]} selection: {error}", file=sys.stderr)
|
|
return 1
|
|
print(f"Updated .config/{sys.argv[2]}-selection.txt: count={count} {key}={digest}")
|
|
return 0
|
|
if sys.argv[1:]:
|
|
print(
|
|
"usage: check_test_wiring.py [--self-test | --check-core LISTING | --check-profile PROFILE LISTING | "
|
|
"--update-profile PROFILE LISTING PLATFORM | --check-scanner-heal-release-bundle FILE | "
|
|
"--check-scanner-heal-release-bundle-gate FILE GATE | "
|
|
"--assemble-scanner-heal-release-bundle DESCRIPTOR [DESCRIPTOR ...] DIR | "
|
|
"--write-scanner-heal-release-bundle-fixture DIR]",
|
|
file=sys.stderr,
|
|
)
|
|
return 2
|
|
|
|
errors = validate(ROOT)
|
|
if errors:
|
|
for error in errors:
|
|
print(f"ERROR: {error}", file=sys.stderr)
|
|
return 1
|
|
print("OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|