mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-29 07:17:52 +00:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 8a48a52632 | |||
| 6feac7b7f7 | |||
| c7fb6d9901 |
@@ -66,7 +66,7 @@ The manifest has the following JSON contract (all fields are required):
|
||||
| `rounds`, `duration_seconds`, `min_free_bytes` | 3..10 groups, 7200..86400 seconds for measured release runs, and the independently estimated free-space reservation in bytes. Synthetic runs may use 1 second. |
|
||||
| `baseline`, `candidate` | Each contains executable `binary`, full 40-character `revision`, and verified `sha256`. The runner rehashes binaries before every leg. |
|
||||
| `fixed` | `config_sha256`, `dataset_sha256`, `release_flags`, `durability`, `disk_type`, `cache_state`, `load_command`, `resource_isolation`, `topology` (`EC8+4`), and positive `offered_load_ops`. Hashes use 64 lowercase hexadecimal characters. |
|
||||
| `release_evidence` | Required for `measured` runs. It binds the 3x4 EC8+4 topology, multi-pool/multi-set coverage, per-node metrics endpoints, same-window distributed sampling, process restart and crash-restart fault modes, mixed-version reader/writer/rollback participation, and allocation/flamegraph/RSS/save-frequency profile artifact requirements. Synthetic runs do not need this field and still cannot approve release evidence. |
|
||||
| `release_evidence` | Required for `measured` runs. It declares the 3x4 EC8+4 topology, multi-pool/multi-set coverage, per-node metrics endpoints, same-window distributed sampling, process restart and crash-restart fault modes, mixed-version reader/writer/rollback participation, and allocation/flamegraph/RSS/save-frequency profile artifact requirements. Synthetic runs do not require it and cannot approve release evidence. |
|
||||
| `oracles` | A map with all five scenario names. Each value contains positive integer `objects`, `versions`, `bytes`, and `sha256` of the independently prepared canonical object/version/content manifest. |
|
||||
| `expected_healed_objects` | A map with all five scenario names and independently seeded repair counts. Running-heal and MRF-replay require a positive count. |
|
||||
|
||||
@@ -79,11 +79,17 @@ the collector reads `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` from its environ
|
||||
The adapter must echo the measured run's `release_evidence` object in every
|
||||
measurement response. A mismatch fails the cell because it means the deployment,
|
||||
mixed-version set, crash mode, or profiler contract no longer matches the
|
||||
operator-reviewed manifest. This echo is provenance binding only; it does not
|
||||
replace the independent correctness oracle, distributed metrics samples, profile
|
||||
artifacts, or ABBA comparison thresholds. The summary tool revalidates the same
|
||||
manifest contract before it can print a measured PASS result, so hand-built or
|
||||
trimmed reports without this provenance fail closed.
|
||||
operator-reviewed manifest. This echo checks consistency with the declared plan;
|
||||
it does not prove that observations share a source window or that the reported
|
||||
client percentiles are exact. Until those original observations are independently
|
||||
verified, measured results are marked unverified and cannot produce a PASS or a
|
||||
release descriptor. The summary command's legacy descriptor option rejects the
|
||||
request until a producer can verify each field against its original window.
|
||||
|
||||
The existing live collector persists the declared load command, scanner/heal
|
||||
configuration snapshots, and adapter/collector output. It is not a verified
|
||||
sanitized capture path; use it only after confirming those inputs and outputs do
|
||||
not contain credentials, private configuration files, or sensitive commands.
|
||||
|
||||
#### Deployment Adapter Contract
|
||||
|
||||
@@ -218,7 +224,7 @@ retain the failure and reconcile the service before continuing.
|
||||
|
||||
The `measure` response echoes the observed `evidence`, `fixed`, `build`,
|
||||
`data_dir`, and `background`, plus `sample_count` (1..3600), `elapsed_seconds`,
|
||||
and `metrics`. All metrics must be finite nonnegative numbers: `p99_ms`,
|
||||
and `metrics`. All metrics must be finite nonnegative numbers: `p95_ms`, `p99_ms`,
|
||||
`throughput_ops`, `rss_bytes`, `cpu_seconds`, `iops`, `rpc_count`,
|
||||
`cache_clone_bytes`, `encode_bytes`, `save_bytes`, `oldest_age_seconds`,
|
||||
`walk_objects`, `cold_walk_objects`, `healed_objects`, `errors`, and `requests`.
|
||||
@@ -235,6 +241,21 @@ in the cell artifact directory; aggregate values alone do not establish their
|
||||
measurement provenance. Missing production instrumentation is a pending gate,
|
||||
not permission to report a fabricated zero.
|
||||
|
||||
For measured cells, retain the original schedule, start, and completion ledgers
|
||||
and their references in `client_request_samples`. The runner reconciles request
|
||||
identities, counts, timestamps, errors, and file hashes before calculating
|
||||
nearest-rank percentiles from the recorded worker-start-to-completion samples.
|
||||
The current adapter timestamps worker start before its ledger flush, so those
|
||||
samples also include local logging and do not establish SDK-call latency.
|
||||
Missing HTTP status does not prove zero 503s. The current ledger format does not
|
||||
prove that the offered load reached the service or that the client and node
|
||||
windows coincide, so a measured release result remains unavailable until those
|
||||
boundaries are verified.
|
||||
The four ABBA legs cover different windows: their per-leg percentiles are
|
||||
reported separately, not renamed as a combined p95 or p99. Ledger reduction
|
||||
uses a size-limited scratch database and samples free space; those checks do not
|
||||
reserve disk space or prevent concurrent writers from crossing the floor.
|
||||
|
||||
For usage-cache saves, `encode_bytes` uses the window delta of
|
||||
`rustfs_scanner_cache_serialized_bytes_total`. `save_bytes` uses the sum of
|
||||
`rustfs_scanner_cache_save_payload_bytes_total` deltas for `cache="main"` and
|
||||
|
||||
@@ -89,7 +89,12 @@ their issue closes.
|
||||
| `test_scanner_heal_w16_recovery_evidence.sh` | dev-tool | Shell self-test for the Scanner/Heal W16 recovery evidence runner | — |
|
||||
| `test_scanner_heal_w13_mrf_evidence.sh` | dev-tool | Shell self-test for the Scanner/Heal W13 MRF evidence runner | — |
|
||||
| `scanner_abba.py` | dev-tool | Scanner/heal ABBA orchestration and evidence gates via `run_scanner_validation_harness.sh --abba` | `docs/operations/scanner-benchmark-runbook.md` |
|
||||
| `scanner_client_trace.py` | dev-tool | Reconcile client request ledgers and calculate per-window exact sample percentiles | `scanner_abba.py` |
|
||||
| `test_scanner_abba.py` | dev-tool | Synthetic ABBA adapter and failure-path tests | `test_scanner_validation_harness.sh` |
|
||||
| `test_scanner_client_trace.py` | dev-tool | Client ledger reconciliation and fail-closed tests | `test_scanner_validation_harness.sh` |
|
||||
| `test_summarize_scanner_heal_perf.py` | dev-tool | Scanner/heal summary and provenance tests | `test_scanner_validation_harness.sh` |
|
||||
| `test_scanner_heal_scheduler_pressure_evidence.py` | dev-tool | Scheduler-pressure evidence fail-closed tests | `test_scanner_heal_scheduler_pressure_evidence.sh` |
|
||||
| `test_scanner_heal_linux_evidence_plan.py` | dev-tool | Linux evidence plan coverage tests | `test_scanner_heal_linux_evidence_plan.sh` |
|
||||
| `test_build_rustfs_options.sh` | dev-tool | Shell test for rustfs build-option wiring | `make test` (script-tests) |
|
||||
| `test_entrypoint_credentials.sh` | dev-tool | Container entrypoint credential-handling test | `make test` (script-tests) |
|
||||
| `test_helm_chart_version.sh` | dev-tool | Test for `helm_chart_version.sh` | — |
|
||||
|
||||
@@ -29,6 +29,7 @@ from scanner_abba import (
|
||||
RELEASE_PROFILE_ARTIFACTS,
|
||||
SCENARIOS,
|
||||
digest,
|
||||
measured_provenance_verified,
|
||||
number,
|
||||
read_json,
|
||||
require,
|
||||
@@ -2737,6 +2738,16 @@ def validate_release_bundle_domain_evidence(gate: str, field: str, evidence: dic
|
||||
f"{gate}.{field}.recovered_pending_migration")
|
||||
|
||||
|
||||
def segment_oracle_provenance_verified(_bundle_dir: Path, _field: str) -> bool:
|
||||
# Original post-stop and segment/oracle evidence is not verified yet.
|
||||
return False
|
||||
|
||||
|
||||
def distributed_window_provenance_verified(_bundle_dir: Path, _field: str) -> bool:
|
||||
# Original multi-node generation and invalidation evidence is not verified yet.
|
||||
return False
|
||||
|
||||
|
||||
def validate_release_bundle_artifact(bundle_path: Path, source_revision: str, gate: str, field: str,
|
||||
evidence: dict[str, object]) -> str:
|
||||
require(evidence.get("evidence_type") == "measured", f"{gate}.{field} must be measured evidence")
|
||||
@@ -2976,6 +2987,18 @@ def validate_release_bundle_artifact(bundle_path: Path, source_revision: str, ga
|
||||
evidence_integer(item.get("resolved_samples"), f"{gate}.{artifact_field}.resolved_samples",
|
||||
0, 2**63 - 1)
|
||||
validate_release_bundle_domain_evidence(gate, field, evidence)
|
||||
if (gate, field) in (("G10", "pressure_recovery_evidence"),
|
||||
("P1", "cold_walk_share_measurement"),
|
||||
("P1", "foreground_latency_throughput_measurement"),
|
||||
("P3", "two_hour_pressure_measurement")):
|
||||
require(measured_provenance_verified(bundle_path.parent),
|
||||
f"{gate}.{field} source windows and client percentiles are not independently verified")
|
||||
if gate == "P2" and field in ("post_stop_convergence_measurement", "cold_segment_reuse_measurement"):
|
||||
require(segment_oracle_provenance_verified(bundle_path.parent, field),
|
||||
f"{gate}.{field} post-stop and segment/oracle provenance is not independently verified")
|
||||
if gate == "G14" and field in ("same_window_field_evidence", "distributed_segment_invalidation_evidence"):
|
||||
require(distributed_window_provenance_verified(bundle_path.parent, field),
|
||||
f"{gate}.{field} multi-node generation and window provenance is not independently verified")
|
||||
return window_id
|
||||
|
||||
|
||||
@@ -3516,6 +3539,19 @@ def validate(root: Path) -> list[str]:
|
||||
|
||||
|
||||
class SelfTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
# Structural fixtures exercise the other validators with a simulated
|
||||
# source verifier. The real verifier is checked separately below.
|
||||
self.provenance_verifier = mock.patch.object(sys.modules[__name__], "measured_provenance_verified", return_value=True)
|
||||
self.provenance_verifier.start()
|
||||
self.addCleanup(self.provenance_verifier.stop)
|
||||
self.segment_verifier = mock.patch.object(sys.modules[__name__], "segment_oracle_provenance_verified", return_value=True)
|
||||
self.segment_verifier.start()
|
||||
self.addCleanup(self.segment_verifier.stop)
|
||||
self.distributed_verifier = mock.patch.object(sys.modules[__name__], "distributed_window_provenance_verified", return_value=True)
|
||||
self.distributed_verifier.start()
|
||||
self.addCleanup(self.distributed_verifier.stop)
|
||||
|
||||
def test_quick_checks_rejects_caller_and_execution_bypasses(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
@@ -4189,6 +4225,28 @@ class SelfTests(unittest.TestCase):
|
||||
write_json(bundle, {"schema": 1, "evidence": "measured", "source_revision": source_revision, "gates": gates})
|
||||
return root, bundle
|
||||
|
||||
def test_scanner_heal_release_bundle_rejects_unverified_measurements(self) -> None:
|
||||
self.provenance_verifier.stop()
|
||||
self.segment_verifier.stop()
|
||||
self.distributed_verifier.stop()
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
||||
payload = read_json(bundle)
|
||||
payload["measurement_provenance"] = "verified"
|
||||
write_json(bundle, payload)
|
||||
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
||||
status = scanner_heal_release_bundle_status(root, bundle)
|
||||
unverified = [scanner_heal_release_bundle_gate_status(root, bundle, gate)
|
||||
for gate in ("G10", "G14", "P1", "P2", "P3")]
|
||||
g09 = scanner_heal_release_bundle_gate_status(root, bundle, "G09")
|
||||
self.assertFalse(status["release_approved"])
|
||||
for gate in ("G10", "G14", "P1", "P2", "P3"):
|
||||
self.assertIn(gate, status["rejected_gates"])
|
||||
for gate in unverified:
|
||||
self.assertEqual(gate["decision"], "blocked")
|
||||
self.assertTrue(any("not independently verified" in error for error in gate["rejected_errors"]))
|
||||
self.assertEqual(g09["verified_gate"], "G09")
|
||||
|
||||
def test_scanner_heal_release_bundle_accepts_complete_measured_evidence(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
|
||||
|
||||
@@ -11,6 +11,7 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
@@ -18,7 +19,7 @@ import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from scanner_abba import digest, read_json, require, write_json
|
||||
from scanner_abba import digest, read_json, require, safe_error_message, write_json
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
G14_FIELDS = (
|
||||
@@ -34,10 +35,6 @@ def git_head() -> str:
|
||||
return subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def iso_from_epoch(value: Any, name: str) -> str:
|
||||
require(type(value) in (int, float) and value > 0, f"invalid {name}")
|
||||
return datetime.fromtimestamp(float(value), timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
@@ -272,8 +269,10 @@ def proof_from_case_directories(raw_values: list[str], out_dir: Path, source_rev
|
||||
|
||||
|
||||
def write_field(out_dir: Path, field: str, proof: dict[str, Any], source_revision: str) -> dict[str, Any]:
|
||||
started_at = proof.get("started_at") or utc_now()
|
||||
finished_at = proof.get("finished_at") or started_at
|
||||
started_at = proof.get("started_at")
|
||||
finished_at = proof.get("finished_at")
|
||||
require(isinstance(started_at, str) and started_at, "missing G14 original window start")
|
||||
require(isinstance(finished_at, str) and finished_at, "missing G14 original window end")
|
||||
evidence: dict[str, Any] = {
|
||||
"artifact": "",
|
||||
"evidence_type": "measured",
|
||||
@@ -325,11 +324,18 @@ def write_field(out_dir: Path, field: str, proof: dict[str, Any], source_revisio
|
||||
return evidence
|
||||
|
||||
|
||||
def g14_inputs_verified(_args: argparse.Namespace, _source_revision: str) -> bool:
|
||||
# The original multi-node observation and invalidation window are not yet
|
||||
# independently verified. A self-declared measured proof cannot open G14.
|
||||
return False
|
||||
|
||||
|
||||
def build_descriptor(args: argparse.Namespace) -> Path:
|
||||
out_dir = args.out_dir.resolve()
|
||||
require(not out_dir.exists(), "output directory must be new")
|
||||
source_revision = args.source_revision or git_head()
|
||||
out_dir.mkdir(parents=True)
|
||||
require(g14_inputs_verified(args, source_revision), "G14 source windows are not independently verified")
|
||||
out_dir.mkdir(parents=True, mode=0o700)
|
||||
if args.proof_json is not None:
|
||||
proof_path = args.proof_json.resolve()
|
||||
proof = load_proof(proof_path, source_revision)
|
||||
@@ -344,26 +350,31 @@ def build_descriptor(args: argparse.Namespace) -> Path:
|
||||
proof = proof_from_case_directories(args.case_dir, out_dir, source_revision)
|
||||
fields = {field: write_field(out_dir, field, proof, source_revision) for field in G14_FIELDS}
|
||||
descriptor = out_dir / "release-bundle-g14.json"
|
||||
write_json(descriptor, {
|
||||
"schema": 1,
|
||||
"evidence": "measured",
|
||||
"source_revision": source_revision,
|
||||
"gates": {
|
||||
"G14": {
|
||||
"status": "pass",
|
||||
"lane": "ec8-4-multiset",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": fields,
|
||||
pending = out_dir / ".release-bundle-g14.pending.json"
|
||||
try:
|
||||
write_json(pending, {
|
||||
"schema": 1,
|
||||
"evidence": "measured",
|
||||
"source_revision": source_revision,
|
||||
"gates": {
|
||||
"G14": {
|
||||
"status": "pass",
|
||||
"lane": "ec8-4-multiset",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": fields,
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
subprocess.check_call([
|
||||
sys.executable,
|
||||
str(ROOT / "scripts/check_test_wiring.py"),
|
||||
"--check-scanner-heal-release-bundle-gate",
|
||||
str(descriptor),
|
||||
"G14",
|
||||
], cwd=ROOT)
|
||||
})
|
||||
subprocess.check_call([
|
||||
sys.executable,
|
||||
str(ROOT / "scripts/check_test_wiring.py"),
|
||||
"--check-scanner-heal-release-bundle-gate",
|
||||
str(pending),
|
||||
"G14",
|
||||
], cwd=ROOT)
|
||||
os.link(pending, descriptor)
|
||||
finally:
|
||||
pending.unlink(missing_ok=True)
|
||||
return descriptor
|
||||
|
||||
|
||||
@@ -452,7 +463,7 @@ def write_self_test_case_dir(root: Path, source_revision: str, case: str, sets:
|
||||
return directory
|
||||
|
||||
|
||||
def run_self_test() -> None:
|
||||
def _run_packaging_self_test() -> None:
|
||||
import tempfile
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
@@ -464,6 +475,12 @@ def run_self_test() -> None:
|
||||
"--out-dir", str(root / "out"),
|
||||
]))
|
||||
require(descriptor.is_file(), "self-test descriptor missing")
|
||||
checked = subprocess.run([
|
||||
sys.executable, str(ROOT / "scripts/check_test_wiring.py"),
|
||||
"--check-scanner-heal-release-bundle-gate", str(descriptor), "G14",
|
||||
], cwd=ROOT, capture_output=True, text=True, check=False)
|
||||
require(checked.returncode == 1 and "not independently verified" in checked.stdout,
|
||||
"self-test accepted unverified G14 evidence")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
@@ -534,6 +551,78 @@ def run_self_test() -> None:
|
||||
raise ValueError("self-test accepted fixture-only G14 case oracle")
|
||||
|
||||
|
||||
def run_self_test() -> None:
|
||||
import contextlib
|
||||
import io
|
||||
import tempfile
|
||||
from unittest import mock
|
||||
|
||||
import check_test_wiring as wiring
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
for proof in ({}, {"started_at": "2026-09-09T00:00:00Z"}):
|
||||
try:
|
||||
write_field(root, G14_FIELDS[0], proof, "a" * 40)
|
||||
except ValueError as err:
|
||||
require("missing G14 original window" in str(err), "wrong missing-window failure")
|
||||
else:
|
||||
raise ValueError("self-test synthesized a G14 window")
|
||||
require(not (root / "artifacts").exists(), "missing G14 window left artifacts")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
stderr = io.StringIO()
|
||||
argv = ["g14", "--proof-json", "unused", "--out-dir", str(Path(tmp) / "out")]
|
||||
error = subprocess.CalledProcessError(1, ["sensitive-command-sentinel"])
|
||||
with mock.patch.object(sys, "argv", argv), mock.patch.object(sys.modules[__name__], "git_head", side_effect=error), \
|
||||
contextlib.redirect_stderr(stderr):
|
||||
require(main() == 1, "self-test accepted a failed source revision")
|
||||
require("sensitive-command-sentinel" not in stderr.getvalue(), "G14 error exposed command arguments")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
source_revision = git_head()
|
||||
proof = write_self_test_proof(root, source_revision)
|
||||
out_dir = root / "out"
|
||||
try:
|
||||
build_descriptor(parse_args(["--proof-json", str(proof), "--out-dir", str(out_dir)]))
|
||||
except ValueError as err:
|
||||
require("not independently verified" in str(err), "wrong unverified G14 failure")
|
||||
else:
|
||||
raise ValueError("self-test accepted unverified G14 inputs")
|
||||
require(not out_dir.exists(), "unverified G14 inputs left an output directory")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
source_revision = git_head()
|
||||
proof = write_self_test_proof(root, source_revision)
|
||||
out_dir = root / "out"
|
||||
with mock.patch.object(sys.modules[__name__], "g14_inputs_verified", return_value=True), \
|
||||
mock.patch.object(subprocess, "check_call", side_effect=subprocess.CalledProcessError(1, "checker")):
|
||||
try:
|
||||
build_descriptor(parse_args(["--proof-json", str(proof), "--out-dir", str(out_dir)]))
|
||||
except subprocess.CalledProcessError:
|
||||
pass
|
||||
else:
|
||||
raise ValueError("self-test accepted a rejected G14 descriptor")
|
||||
require(not (out_dir / "release-bundle-g14.json").exists()
|
||||
and not (out_dir / ".release-bundle-g14.pending.json").exists(),
|
||||
"rejected G14 validation left a descriptor")
|
||||
|
||||
def verify_fixture(command: list[str], cwd: Path) -> None:
|
||||
require(cwd == ROOT and command[2] == "--check-scanner-heal-release-bundle-gate"
|
||||
and command[4] == "G14", "unexpected self-test checker invocation")
|
||||
with mock.patch.object(wiring, "distributed_window_provenance_verified", return_value=True):
|
||||
status = wiring.scanner_heal_release_bundle_gate_status(ROOT, Path(command[3]), "G14")
|
||||
require(status["decision"] == "verified", "fixture gate validation failed")
|
||||
|
||||
# Only synthetic packaging tests simulate the future verifier. Real CLI
|
||||
# invocations and the separate checker retain their fail-closed behavior.
|
||||
with mock.patch.object(sys.modules[__name__], "g14_inputs_verified", return_value=True), \
|
||||
mock.patch.object(subprocess, "check_call", side_effect=verify_fixture):
|
||||
_run_packaging_self_test()
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--proof-json", type=Path)
|
||||
@@ -561,7 +650,7 @@ def main() -> int:
|
||||
print(f"G14 release descriptor verified: {descriptor}")
|
||||
return 0
|
||||
except (ValueError, KeyError, OSError, subprocess.SubprocessError) as err:
|
||||
print(f"ERROR: {err}", file=sys.stderr)
|
||||
print(f"ERROR: {safe_error_message(err)}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
@@ -365,7 +365,7 @@ def performance_steps() -> list[dict[str, Any]]:
|
||||
{
|
||||
"id": "scanner-heal-abba",
|
||||
"description": "Run the isolated EC8+4 Scanner/Heal ABBA matrix through the deployment adapter.",
|
||||
"covers": {"gates": ["G10", "P1", "P2", "P3"], "issues": ["2266", "2273", "2274", "2275"]},
|
||||
"covers": {"gates": ["G10", "P1", "P3"], "issues": ["2266", "2273", "2274", "2275"]},
|
||||
"commands": [
|
||||
existing_script_command(
|
||||
"scripts/python_bin.sh",
|
||||
@@ -385,7 +385,7 @@ def performance_steps() -> list[dict[str, Any]]:
|
||||
{
|
||||
"id": "scheduler-pressure",
|
||||
"description": "Assemble scheduler-pressure, profile, RSS, throughput, and latency descriptors.",
|
||||
"covers": {"gates": ["G10", "P1", "P2", "P3"], "issues": ["2266", "2274", "2275"]},
|
||||
"covers": {"gates": ["G10", "P1", "P3"], "issues": ["2266", "2274", "2275"]},
|
||||
"commands": [
|
||||
existing_script_command(
|
||||
"scripts/python_bin.sh",
|
||||
@@ -402,12 +402,28 @@ def performance_steps() -> list[dict[str, Any]]:
|
||||
"rss-samples=$RUN_ROOT/profile/rss-samples.json",
|
||||
"--profile-artifact",
|
||||
"save-frequency=$RUN_ROOT/profile/save-frequency.json",
|
||||
"--started-at",
|
||||
"$ORIGINAL_WINDOW_STARTED_AT",
|
||||
"--finished-at",
|
||||
"$ORIGINAL_WINDOW_FINISHED_AT",
|
||||
"--run-id",
|
||||
"$ORIGINAL_RUN_ID",
|
||||
"--measurement-window-id",
|
||||
"$ORIGINAL_WINDOW_ID",
|
||||
"--out-dir",
|
||||
"$RUN_ROOT/descriptors/scheduler-pressure",
|
||||
)
|
||||
],
|
||||
"expected_outputs": expected_outputs("$RUN_ROOT/descriptors/scheduler-pressure/release-bundle-scheduler-pressure.json"),
|
||||
},
|
||||
{
|
||||
"id": "p2-independent-evidence",
|
||||
"description": "P2 needs original post-stop and segment observations with independent full-walk and root oracles.",
|
||||
"covers": {"gates": ["P2"], "issues": ["2274"]},
|
||||
"commands": [],
|
||||
"expected_outputs": [],
|
||||
"pending_reason": "No producer verifies the required independent P2 evidence yet.",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@@ -561,7 +577,10 @@ def build_status(plan: dict[str, Any], run_root: Path) -> dict[str, Any]:
|
||||
for key in counts:
|
||||
totals[key] += counts[key]
|
||||
totals["expected"] += len(outputs)
|
||||
if not outputs:
|
||||
pending_reason = step.get("pending_reason")
|
||||
if isinstance(pending_reason, str) and pending_reason:
|
||||
status = "pending"
|
||||
elif not outputs:
|
||||
status = "not_tracked"
|
||||
elif counts["invalid"] or counts["empty"]:
|
||||
status = "invalid"
|
||||
@@ -578,12 +597,17 @@ def build_status(plan: dict[str, Any], run_root: Path) -> dict[str, Any]:
|
||||
"status": status,
|
||||
"commands": step["commands"],
|
||||
}
|
||||
step_statuses.append({
|
||||
if pending_reason:
|
||||
next_step["reason"] = pending_reason
|
||||
step_status = {
|
||||
"id": step["id"],
|
||||
"status": status,
|
||||
"covers": step.get("covers", {}),
|
||||
"outputs": outputs,
|
||||
})
|
||||
}
|
||||
if pending_reason:
|
||||
step_status["reason"] = pending_reason
|
||||
step_statuses.append(step_status)
|
||||
tracked = [step for step in step_statuses if step["status"] != "not_tracked"]
|
||||
if not tracked:
|
||||
stage_state = "not_tracked"
|
||||
@@ -598,7 +622,7 @@ def build_status(plan: dict[str, Any], run_root: Path) -> dict[str, Any]:
|
||||
stage_statuses.append({"id": stage["id"], "status": stage_state, "steps": step_statuses})
|
||||
if totals["invalid"] or totals["empty"]:
|
||||
decision = "invalid"
|
||||
elif totals["missing"]:
|
||||
elif totals["missing"] or pending_gates:
|
||||
decision = "blocked"
|
||||
else:
|
||||
decision = "complete"
|
||||
@@ -753,7 +777,8 @@ def self_test() -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("{}\n")
|
||||
complete = build_status(plan, complete_root)
|
||||
assert complete["decision"] == "complete"
|
||||
assert complete["decision"] == "blocked"
|
||||
assert "P2" in complete["pending_gates"]
|
||||
assert complete["release_approved"] is False
|
||||
print("PASS: scanner/heal Linux evidence plan self-test")
|
||||
|
||||
|
||||
@@ -9,21 +9,23 @@ it never turns synthetic harness output into release evidence.
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from scanner_abba import (
|
||||
P2_WORK_MULTIPLE_LIMIT,
|
||||
RELEASE_PROFILE_ARTIFACTS,
|
||||
RELEASE_SCHEDULER_BOUNDS,
|
||||
SCENARIOS,
|
||||
digest,
|
||||
measured_provenance_verified,
|
||||
read_json,
|
||||
require,
|
||||
sha,
|
||||
@@ -33,29 +35,27 @@ from scanner_abba import (
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
G10_FIELDS = ("scheduler_bound_evidence", "pressure_recovery_evidence")
|
||||
P1_FIELDS = ("cold_walk_share_measurement", "foreground_latency_throughput_measurement", "profile_evidence")
|
||||
P2_FIELDS = ("post_stop_convergence_measurement", "cold_segment_reuse_measurement")
|
||||
P3_FIELDS = ("two_hour_pressure_measurement", "heal_capacity_measurement", "recovery_window_measurement")
|
||||
PRESSURE_METRICS = (
|
||||
"foreground_p95_ms",
|
||||
"foreground_p99_ms",
|
||||
"throughput_ops",
|
||||
"error_rate",
|
||||
"heal_lock_wait_p99_ms",
|
||||
"attempt_cost_samples",
|
||||
)
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def git_head() -> str:
|
||||
return subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
|
||||
|
||||
|
||||
def finite_number(value: Any, name: str, minimum: float = 0.0) -> float:
|
||||
require(type(value) in (int, float) and math.isfinite(value) and value >= minimum, f"invalid {name}")
|
||||
return float(value)
|
||||
@dataclass(frozen=True)
|
||||
class VerifiedPressureObservations:
|
||||
source_revision: str
|
||||
window: tuple[str, str, str, str]
|
||||
fields: dict[str, dict[str, dict[str, Any]]]
|
||||
profile_costs: dict[str, int]
|
||||
profile_identities: dict[str, tuple[str, int]]
|
||||
|
||||
|
||||
def pressure_inputs_verified(_abba_dir: Path, _recovery_path: Path, _profiles: dict[str, Path],
|
||||
_window: tuple[str, str, str, str]) -> VerifiedPressureObservations | None:
|
||||
# The verifier must derive each field from original, independently bound
|
||||
# observations. A self-reported flag or a count from a different window is
|
||||
# not a verified observation.
|
||||
return None
|
||||
|
||||
|
||||
def positive_int(value: Any, name: str, minimum: int = 1) -> int:
|
||||
@@ -63,12 +63,6 @@ def positive_int(value: Any, name: str, minimum: int = 1) -> int:
|
||||
return value
|
||||
|
||||
|
||||
def positive_int_from_sources(cli_value: int | None, manifest_values: dict[str, Any], key: str) -> int:
|
||||
if cli_value is not None:
|
||||
return positive_int(cli_value, key)
|
||||
return positive_int(manifest_values.get(key), key)
|
||||
|
||||
|
||||
def parse_artifact_arg(value: str) -> tuple[str, Path, str]:
|
||||
try:
|
||||
kind, raw_path = value.split("=", 1)
|
||||
@@ -90,6 +84,7 @@ def load_measured_abba(abba_dir: Path, source_revision: str) -> tuple[dict[str,
|
||||
require(manifest.get("evidence") == "measured", "ABBA manifest must be measured")
|
||||
require(report.get("evidence") == "measured", "ABBA report must be measured")
|
||||
require(report.get("status") == "pass" and report.get("performance") == "pass", "ABBA report must pass")
|
||||
require(measured_provenance_verified(abba_dir), "ABBA source windows and client percentiles are not independently verified")
|
||||
require(manifest.get("candidate", {}).get("revision") == source_revision, "candidate revision must match checkout")
|
||||
duration = positive_int(manifest.get("duration_seconds"), "duration_seconds", 7200)
|
||||
require(duration >= 7200, "P3 two-hour pressure evidence requires at least 7200 seconds")
|
||||
@@ -121,38 +116,6 @@ def load_measured_abba(abba_dir: Path, source_revision: str) -> tuple[dict[str,
|
||||
return manifest, report, measures
|
||||
|
||||
|
||||
def comparison_rows(report: dict[str, Any], scenario: str, comparison: str) -> list[dict[str, Any]]:
|
||||
rows = [
|
||||
item for item in report.get("comparisons", [])
|
||||
if item.get("scenario") == scenario and item.get("comparison") == comparison
|
||||
]
|
||||
require(rows, f"missing {scenario}/{comparison} comparisons")
|
||||
return rows
|
||||
|
||||
|
||||
def worst_metric(measures: list[dict[str, Any]], key: str) -> float:
|
||||
return max(finite_number(item["metrics"].get(key), key) for item in measures)
|
||||
|
||||
|
||||
def sum_metric(measures: list[dict[str, Any]], key: str) -> float:
|
||||
return sum(finite_number(item["metrics"].get(key), key) for item in measures)
|
||||
|
||||
|
||||
def post_stop_convergence_multiples(report: dict[str, Any]) -> list[float]:
|
||||
values: list[float] = []
|
||||
for index, comparison in enumerate(report.get("comparisons", [])):
|
||||
raw = comparison.get("p2_post_stop_work_multiples")
|
||||
if raw is None:
|
||||
continue
|
||||
require(isinstance(raw, list), f"comparison {index} p2_post_stop_work_multiples must be a list")
|
||||
for value in raw:
|
||||
if value is None:
|
||||
continue
|
||||
values.append(finite_number(value, "p2 post-stop work multiple", 0.0))
|
||||
require(values, "P2 requires measured post-stop convergence rows")
|
||||
return values
|
||||
|
||||
|
||||
PROFILE_ARTIFACT_REQUIRED_METRICS = {
|
||||
"allocation-profile": ("resolved_samples", "allocation_bytes"),
|
||||
"flamegraph": ("resolved_samples",),
|
||||
@@ -162,7 +125,8 @@ PROFILE_ARTIFACT_REQUIRED_METRICS = {
|
||||
|
||||
|
||||
def copy_profile_artifacts(out_dir: Path, artifacts: dict[str, tuple[Path, str]], source_revision: str,
|
||||
run_id: str, window_id: str, profile_costs: dict[str, int]) -> dict[str, Any]:
|
||||
run_id: str, window_id: str, profile_costs: dict[str, int],
|
||||
profile_identities: dict[str, tuple[str, int]]) -> dict[str, Any]:
|
||||
copied: dict[str, Any] = {}
|
||||
profile_dir = out_dir / "artifacts" / "profiles"
|
||||
raw_dir = profile_dir / "raw"
|
||||
@@ -172,6 +136,9 @@ def copy_profile_artifacts(out_dir: Path, artifacts: dict[str, tuple[Path, str]]
|
||||
source, artifact_format = artifacts[kind]
|
||||
raw_target = raw_dir / f"{kind}{source.suffix or '.artifact'}"
|
||||
shutil.copyfile(source, raw_target)
|
||||
expected_hash, expected_size = profile_identities[kind]
|
||||
require(raw_target.stat().st_size == expected_size and digest(raw_target) == expected_hash,
|
||||
f"profile artifact differs from verified observation: {kind}")
|
||||
target = profile_dir / f"P1-profile_evidence-{kind}.json"
|
||||
payload = {
|
||||
"schema": 1,
|
||||
@@ -239,12 +206,8 @@ def build_descriptor(args: argparse.Namespace) -> Path:
|
||||
out_dir = args.out_dir.resolve()
|
||||
require(not out_dir.exists(), "output directory must be new")
|
||||
source_revision = args.source_revision or git_head()
|
||||
manifest, report, measures = load_measured_abba(args.abba_dir.resolve(), source_revision)
|
||||
recovery = recovery_evidence(args.recovery_window_json.resolve(), source_revision)
|
||||
release_evidence = manifest["release_evidence"]
|
||||
scheduler = release_evidence["scheduler"]
|
||||
profile_measurements = release_evidence.get("profile", {}).get("measurements", {})
|
||||
require(isinstance(profile_measurements, dict), "ABBA manifest profile measurements must be an object")
|
||||
manifest, _, _ = load_measured_abba(args.abba_dir.resolve(), source_revision)
|
||||
recovery_evidence(args.recovery_window_json.resolve(), source_revision)
|
||||
|
||||
profile_inputs = {}
|
||||
profile_formats = {}
|
||||
@@ -256,237 +219,99 @@ def build_descriptor(args: argparse.Namespace) -> Path:
|
||||
missing_profiles = sorted(set(RELEASE_PROFILE_ARTIFACTS) - set(profile_inputs))
|
||||
require(not missing_profiles, "missing profile artifacts: " + ", ".join(missing_profiles))
|
||||
|
||||
out_dir.mkdir(parents=True)
|
||||
duration = positive_int(manifest["duration_seconds"], "duration_seconds", 7200)
|
||||
started_at = args.started_at or utc_now()
|
||||
if args.finished_at:
|
||||
finished_at = args.finished_at
|
||||
else:
|
||||
started_at, finished_at = args.started_at, args.finished_at
|
||||
run_id, window_id = args.run_id, args.measurement_window_id
|
||||
require(all(isinstance(value, str) and value.strip() for value in (started_at, finished_at, run_id, window_id)),
|
||||
"original measurement window timestamps and identifiers are required")
|
||||
try:
|
||||
started = datetime.fromisoformat(started_at.replace("Z", "+00:00"))
|
||||
finished_at = (started + timedelta(seconds=duration)).isoformat().replace("+00:00", "Z")
|
||||
run_id = args.run_id or f"scheduler-pressure-{source_revision[:12]}"
|
||||
window_id = args.measurement_window_id or f"scheduler-pressure-window-{source_revision[:12]}"
|
||||
command = [
|
||||
"scripts/run_scanner_heal_scheduler_pressure_evidence.py",
|
||||
"--abba-dir", "<abba-dir>",
|
||||
"--recovery-window-json", "<recovery-window-json>",
|
||||
"--profile-artifact", "<kind=artifact>",
|
||||
]
|
||||
running_heal = comparison_rows(report, "running-heal", "build")
|
||||
require(any(row.get("w10", {}).get("status") == "observed" for row in running_heal),
|
||||
"G10 requires observed running-heal pacing benefit")
|
||||
require(all(row.get("w11", {}).get("status") == "observed" for row in running_heal),
|
||||
"P3 requires observed bounded retry-window rows")
|
||||
cold_hot = comparison_rows(report, "cold-hot", "build")
|
||||
p1_rows = [row.get("p1") for row in cold_hot]
|
||||
require(all(isinstance(row, dict) and row.get("observed_reduction", -1) >= row.get("required_reduction", 1)
|
||||
for row in p1_rows), "P1 cold-hot rows did not meet required reduction")
|
||||
candidate_walked_segments = 0
|
||||
candidate_cold_segments = 0
|
||||
for index, row in enumerate(p1_rows):
|
||||
require(isinstance(row, dict), f"P1 row {index} missing cold-hot measurement")
|
||||
candidate_walked_segments += int(finite_number(row.get("candidate_walk_objects"),
|
||||
"candidate_walk_objects", 1))
|
||||
candidate_cold_segments += int(finite_number(row.get("candidate_cold_walk_objects"),
|
||||
"candidate_cold_walk_objects", 0))
|
||||
require(candidate_cold_segments == 0, "P2 requires zero cold-segment walks in measured cold-hot rows")
|
||||
p2_multiples = post_stop_convergence_multiples(report)
|
||||
p2_limit = float(P2_WORK_MULTIPLE_LIMIT)
|
||||
p2_worst = max(p2_multiples)
|
||||
require(p2_worst <= p2_limit, "P2 post-stop convergence exceeded work multiple limit")
|
||||
profile_costs = {
|
||||
"resolved_samples": positive_int_from_sources(args.resolved_samples, profile_measurements, "resolved_samples"),
|
||||
"allocation_bytes": positive_int_from_sources(args.allocation_bytes, profile_measurements, "allocation_bytes"),
|
||||
"rss_peak_bytes": positive_int_from_sources(args.rss_peak_bytes, profile_measurements, "rss_peak_bytes"),
|
||||
"save_operations": positive_int_from_sources(args.save_operations, profile_measurements, "save_operations"),
|
||||
"saved_bytes": positive_int_from_sources(args.saved_bytes, profile_measurements, "saved_bytes"),
|
||||
}
|
||||
finished = datetime.fromisoformat(finished_at.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
raise ValueError("invalid original measurement window") from None
|
||||
require(started.tzinfo is not None and finished.tzinfo is not None and (finished - started).total_seconds() >= duration,
|
||||
"invalid original measurement window")
|
||||
window = (run_id, window_id, started_at, finished_at)
|
||||
observations = pressure_inputs_verified(args.abba_dir.resolve(), args.recovery_window_json.resolve(), profile_inputs, window)
|
||||
require(isinstance(observations, VerifiedPressureObservations) and observations.window == window
|
||||
and observations.source_revision == source_revision,
|
||||
"independent verified pressure observations are unavailable")
|
||||
expected_fields = {"G10": G10_FIELDS, "P1": P1_FIELDS, "P3": P3_FIELDS}
|
||||
require(type(observations.fields) is dict and set(observations.fields) == set(expected_fields),
|
||||
"verified pressure observations do not cover required gates")
|
||||
reserved = {"artifact", "sha256", "artifact_format", "schema", "source_revision", "run_id",
|
||||
"measurement_window_id", "started_at", "finished_at", "command", "evidence_type",
|
||||
"gate", "field", "profile_artifacts", "resolved_samples", "allocation_bytes",
|
||||
"rss_peak_bytes", "save_operations", "saved_bytes"}
|
||||
for gate, names in expected_fields.items():
|
||||
rows = observations.fields[gate]
|
||||
require(type(rows) is dict and set(rows) == set(names), f"verified {gate} observations are incomplete")
|
||||
for field in names:
|
||||
row = rows[field]
|
||||
require(type(row) is dict and not reserved.intersection(row)
|
||||
and isinstance(row.get("summary"), str) and row["summary"].strip(),
|
||||
f"invalid verified {gate}.{field} observation")
|
||||
|
||||
common = {
|
||||
"evidence_type": "measured",
|
||||
"source_revision": source_revision,
|
||||
"run_id": run_id,
|
||||
"measurement_window_id": window_id,
|
||||
"started_at": started_at,
|
||||
"finished_at": finished_at,
|
||||
"command": command,
|
||||
}
|
||||
total_walk = int(sum_metric(measures, "walk_objects"))
|
||||
total_cold = int(sum_metric(measures, "cold_walk_objects"))
|
||||
total_healed = int(sum_metric(measures, "healed_objects"))
|
||||
total_bytes = sum(item.get("oracle", {}).get("bytes", 0) for item in measures)
|
||||
total_versions = sum(item.get("oracle", {}).get("versions", 0) for item in measures)
|
||||
profile_refs = copy_profile_artifacts(
|
||||
out_dir,
|
||||
{kind: (profile_inputs[kind], profile_formats[kind]) for kind in RELEASE_PROFILE_ARTIFACTS},
|
||||
source_revision,
|
||||
run_id,
|
||||
window_id,
|
||||
profile_costs,
|
||||
)
|
||||
profile_keys = ("resolved_samples", "allocation_bytes", "rss_peak_bytes", "save_operations", "saved_bytes")
|
||||
require(type(observations.profile_costs) is dict and set(observations.profile_costs) == set(profile_keys),
|
||||
"verified profile observations are incomplete")
|
||||
profile_costs = {key: positive_int(observations.profile_costs[key], key) for key in profile_keys}
|
||||
require(type(observations.profile_identities) is dict
|
||||
and set(observations.profile_identities) == set(RELEASE_PROFILE_ARTIFACTS),
|
||||
"verified profile identities are incomplete")
|
||||
for kind, identity in observations.profile_identities.items():
|
||||
require(type(identity) is tuple and len(identity) == 2 and sha(identity[0])
|
||||
and type(identity[1]) is int and identity[1] > 0, f"invalid verified profile identity: {kind}")
|
||||
for key in profile_keys:
|
||||
supplied = getattr(args, key)
|
||||
require(supplied is None or supplied == profile_costs[key], f"profile {key} differs from verified observation")
|
||||
|
||||
gates: dict[str, Any] = {
|
||||
"G10": {
|
||||
"status": "pass",
|
||||
"lane": "scheduler-pressure",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": {
|
||||
"scheduler_bound_evidence": write_field(out_dir, "G10", "scheduler_bound_evidence", {
|
||||
**common,
|
||||
"summary": "Measured ABBA scheduler-pressure run bounded deferred work and rejected duplicate admission.",
|
||||
"max_deferred_items": scheduler["max_deferred_items"],
|
||||
"max_deferred_bytes": scheduler["max_deferred_bytes"],
|
||||
"max_retry_age_seconds": scheduler["max_retry_age_seconds"],
|
||||
"duplicate_task_count": int(worst_metric(measures, "heal_duplicate_task_count")),
|
||||
"scheduler_bounds": list(RELEASE_SCHEDULER_BOUNDS),
|
||||
"duplicate_task_bound_observed": scheduler["duplicate_task_bound_observed"]
|
||||
and int(worst_metric(measures, "heal_duplicate_task_count")) == 0,
|
||||
}),
|
||||
"pressure_recovery_evidence": write_field(out_dir, "G10", "pressure_recovery_evidence", {
|
||||
**common,
|
||||
"summary": "Measured ABBA running-heal rows observed pressure pacing and foreground recovery metrics.",
|
||||
"pressure_pacing_engaged": True,
|
||||
"recovery_window_seconds": positive_int(recovery.get("pressure_recovery_window_seconds"), "pressure recovery window"),
|
||||
"lock_hold_p95_ms": int(worst_metric(measures, "heal_lock_hold_p95_ms")),
|
||||
"foreground_latency_p95_ms": int(worst_metric(measures, "p95_ms")),
|
||||
"pressure_metrics": {
|
||||
"foreground_p95_ms": worst_metric(measures, "p95_ms"),
|
||||
"foreground_p99_ms": worst_metric(measures, "p99_ms"),
|
||||
"throughput_ops": min(finite_number(item["metrics"].get("throughput_ops"), "throughput_ops", 1) for item in measures),
|
||||
"error_rate": 0.0,
|
||||
"heal_lock_wait_p99_ms": worst_metric(measures, "heal_lock_wait_p99_ms"),
|
||||
"attempt_cost_samples": max(1.0, sum_metric(measures, "healed_objects")),
|
||||
"foreground_pressure_samples": int(sum_metric(measures, "foreground_pressure_samples")),
|
||||
"foreground_pressure_high_samples": max(1, int(sum_metric(measures, "foreground_pressure_high_samples"))),
|
||||
},
|
||||
}),
|
||||
},
|
||||
},
|
||||
"P1": {
|
||||
"status": "pass",
|
||||
"lane": "scheduler-pressure",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": {
|
||||
"cold_walk_share_measurement": write_field(out_dir, "P1", "cold_walk_share_measurement", {
|
||||
**common,
|
||||
"duration_seconds": duration,
|
||||
"summary": "Measured ABBA cold-hot rows met the required cold-walk share reduction.",
|
||||
"cold_walk_share": 0.0 if total_walk == 0 else total_cold / total_walk,
|
||||
"walk_objects": total_walk,
|
||||
"cold_walk_objects": total_cold,
|
||||
}),
|
||||
"foreground_latency_throughput_measurement": write_field(out_dir, "P1", "foreground_latency_throughput_measurement", {
|
||||
**common,
|
||||
"duration_seconds": duration,
|
||||
"summary": "Measured ABBA foreground latency and throughput remained within the release thresholds.",
|
||||
"foreground_latency_p95_ms": int(worst_metric(measures, "p95_ms")),
|
||||
"foreground_latency_p99_ms": int(worst_metric(measures, "p99_ms")),
|
||||
"throughput_ops_per_second": int(min(finite_number(item["metrics"].get("throughput_ops"), "throughput_ops", 1) for item in measures)),
|
||||
"error_count": int(sum_metric(measures, "errors")),
|
||||
"foreground_p95_ms": worst_metric(measures, "p95_ms"),
|
||||
"foreground_p99_ms": worst_metric(measures, "p99_ms"),
|
||||
"throughput_ops": min(finite_number(item["metrics"].get("throughput_ops"), "throughput_ops", 1) for item in measures),
|
||||
"error_rate": 0.0,
|
||||
}),
|
||||
"profile_evidence": write_field(out_dir, "P1", "profile_evidence", {
|
||||
**common,
|
||||
"duration_seconds": duration,
|
||||
"summary": "Measured profile artifacts are bound to the scheduler-pressure measurement window.",
|
||||
**profile_costs,
|
||||
"profile_artifacts": profile_refs,
|
||||
}),
|
||||
},
|
||||
},
|
||||
"P2": {
|
||||
"status": "pass",
|
||||
"lane": "scheduler-pressure",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": {
|
||||
"post_stop_convergence_measurement": write_field(out_dir, "P2", "post_stop_convergence_measurement", {
|
||||
**common,
|
||||
"duration_seconds": duration,
|
||||
"summary": "Measured ABBA rows converged after writes stopped within the bounded work multiple.",
|
||||
"writes_stopped": True,
|
||||
"last_mutation_observed": True,
|
||||
"first_complete_publication": True,
|
||||
"post_stop_samples": len(p2_multiples),
|
||||
"post_stop_work_multiple": p2_worst,
|
||||
"post_stop_work_multiple_limit": p2_limit,
|
||||
"post_stop_work_multiples": p2_multiples,
|
||||
}),
|
||||
"cold_segment_reuse_measurement": write_field(out_dir, "P2", "cold_segment_reuse_measurement", {
|
||||
**common,
|
||||
"duration_seconds": duration,
|
||||
"summary": "Measured ABBA cold-hot rows reused cold segments without walking cold objects.",
|
||||
"hot_walked_segments": candidate_walked_segments,
|
||||
"cold_walked_segments": candidate_cold_segments,
|
||||
"full_walk_oracle_equivalent": True,
|
||||
"published_root_equivalent": True,
|
||||
"walk_objects": candidate_walked_segments,
|
||||
"cold_walk_objects": candidate_cold_segments,
|
||||
}),
|
||||
},
|
||||
},
|
||||
"P3": {
|
||||
"status": "pass",
|
||||
"lane": "scheduler-pressure",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": {
|
||||
"two_hour_pressure_measurement": write_field(out_dir, "P3", "two_hour_pressure_measurement", {
|
||||
**common,
|
||||
"duration_seconds": duration,
|
||||
"summary": "Measured full ABBA scheduler-pressure matrix completed a two-hour fixed-load window per cell.",
|
||||
"fixed_offered_load": True,
|
||||
"foreground_latency_p99_ms": int(worst_metric(measures, "p99_ms")),
|
||||
"attempt_cost_samples": max(1, int(sum_metric(measures, "healed_objects"))),
|
||||
"abba_legs": ["A1", "B1", "B2", "A2"],
|
||||
"scenarios": list(SCENARIOS),
|
||||
"foreground_p95_ms": worst_metric(measures, "p95_ms"),
|
||||
"foreground_p99_ms": worst_metric(measures, "p99_ms"),
|
||||
"throughput_ops": min(finite_number(item["metrics"].get("throughput_ops"), "throughput_ops", 1) for item in measures),
|
||||
}),
|
||||
"heal_capacity_measurement": write_field(out_dir, "P3", "heal_capacity_measurement", {
|
||||
**common,
|
||||
"duration_seconds": duration,
|
||||
"summary": "Measured ABBA cells retained heal capacity without duplicate task admission.",
|
||||
"completed_heal_objects": max(1, total_healed),
|
||||
"duplicate_task_count": int(worst_metric(measures, "heal_duplicate_task_count")),
|
||||
"heal_capacity": {
|
||||
"objects": max(1, total_healed),
|
||||
"versions": max(1, int(total_versions)),
|
||||
"bytes": max(1, int(total_bytes)),
|
||||
"completed_objects": max(1, total_healed),
|
||||
},
|
||||
}),
|
||||
"recovery_window_measurement": write_field(out_dir, "P3", "recovery_window_measurement", {
|
||||
**common,
|
||||
"duration_seconds": duration,
|
||||
"summary": "Measured restart/crash recovery window is bound to the same scheduler-pressure release window.",
|
||||
"pressure_recovery_window_seconds": positive_int(recovery.get("pressure_recovery_window_seconds"), "pressure recovery window"),
|
||||
"lock_hold_p95_ms": positive_int(recovery.get("lock_hold_p95_ms"), "lock hold p95", 0),
|
||||
"fault_modes": ["process-restart", "process-crash-restart"],
|
||||
"recovery_p95_ms": finite_number(recovery.get("recovery_p95_ms"), "recovery p95", 1),
|
||||
"recovery_p99_ms": finite_number(recovery.get("recovery_p99_ms"), "recovery p99", 1),
|
||||
}),
|
||||
},
|
||||
},
|
||||
}
|
||||
descriptor = out_dir / "release-bundle-scheduler-pressure.json"
|
||||
write_json(descriptor, {
|
||||
"schema": 1,
|
||||
"evidence": "measured",
|
||||
"source_revision": source_revision,
|
||||
"gates": gates,
|
||||
})
|
||||
for gate in ("G10", "P1", "P2", "P3"):
|
||||
subprocess.check_call([
|
||||
sys.executable,
|
||||
str(ROOT / "scripts/check_test_wiring.py"),
|
||||
"--check-scanner-heal-release-bundle-gate",
|
||||
str(descriptor),
|
||||
gate,
|
||||
], cwd=ROOT)
|
||||
return descriptor
|
||||
command = ["scripts/run_scanner_heal_scheduler_pressure_evidence.py", "--abba-dir", "<abba-dir>",
|
||||
"--recovery-window-json", "<recovery-window-json>", "--profile-artifact", "<kind=artifact>"]
|
||||
common = {"evidence_type": "measured", "source_revision": source_revision, "run_id": run_id,
|
||||
"measurement_window_id": window_id, "started_at": started_at, "finished_at": finished_at,
|
||||
"command": command}
|
||||
out_dir.parent.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix=".scheduler-pressure-", dir=out_dir.parent) as temporary:
|
||||
staging = Path(temporary) / "bundle"
|
||||
staging.mkdir()
|
||||
profile_refs = copy_profile_artifacts(
|
||||
staging, {kind: (profile_inputs[kind], profile_formats[kind]) for kind in RELEASE_PROFILE_ARTIFACTS},
|
||||
source_revision, run_id, window_id, profile_costs, observations.profile_identities,
|
||||
)
|
||||
gates: dict[str, Any] = {}
|
||||
for gate, names in expected_fields.items():
|
||||
fields = {}
|
||||
for field in names:
|
||||
evidence = {**common, **observations.fields[gate][field]}
|
||||
if gate == "P1" and field == "profile_evidence":
|
||||
evidence.update(profile_costs, profile_artifacts=profile_refs)
|
||||
fields[field] = write_field(staging, gate, field, evidence)
|
||||
gates[gate] = {"status": "pass", "lane": "scheduler-pressure", "evidence_type": "measured",
|
||||
"evidence_fields": fields}
|
||||
descriptor = staging / "release-bundle-scheduler-pressure.json"
|
||||
write_json(descriptor, {"schema": 1, "evidence": "measured", "source_revision": source_revision, "gates": gates})
|
||||
out_dir.mkdir(mode=0o700)
|
||||
# Both directories are on the same filesystem. Move the artifacts
|
||||
# without a second copy, then check the actual destination before
|
||||
# publishing the descriptor.
|
||||
os.rename(staging / "artifacts", out_dir / "artifacts")
|
||||
pending = out_dir / ".release-bundle-scheduler-pressure.pending.json"
|
||||
os.link(descriptor, pending)
|
||||
try:
|
||||
for gate in ("G10", "P1", "P3"):
|
||||
subprocess.check_call([
|
||||
sys.executable,
|
||||
str(ROOT / "scripts/check_test_wiring.py"),
|
||||
"--check-scanner-heal-release-bundle-gate",
|
||||
str(pending),
|
||||
gate,
|
||||
], cwd=ROOT)
|
||||
final = out_dir / descriptor.name
|
||||
os.link(pending, final)
|
||||
finally:
|
||||
pending.unlink()
|
||||
return final
|
||||
|
||||
|
||||
def write_self_test_abba(root: Path, source_revision: str) -> tuple[Path, Path, list[Path]]:
|
||||
@@ -601,8 +426,54 @@ def write_self_test_abba(root: Path, source_revision: str) -> tuple[Path, Path,
|
||||
return abba_dir, recovery, profiles
|
||||
|
||||
|
||||
def self_test_observations(source_revision: str, profiles: dict[str, Path]) -> VerifiedPressureObservations:
|
||||
"""Synthetic packaging fixture; it is never produced by the verifier."""
|
||||
pressure = {"foreground_p95_ms": 10.0, "foreground_p99_ms": 20.0, "throughput_ops": 100.0,
|
||||
"error_rate": 0.0, "heal_lock_wait_p99_ms": 10.0, "attempt_cost_samples": 10,
|
||||
"foreground_pressure_samples": 100, "foreground_pressure_high_samples": 10}
|
||||
profile_costs = {"resolved_samples": 4, "allocation_bytes": 1024, "rss_peak_bytes": 2048,
|
||||
"save_operations": 2, "saved_bytes": 4096}
|
||||
fields = {
|
||||
"G10": {
|
||||
"scheduler_bound_evidence": {"summary": "Fixture bounds", "max_deferred_items": 10,
|
||||
"max_deferred_bytes": 1048576, "max_retry_age_seconds": 30, "duplicate_task_count": 0,
|
||||
"scheduler_bounds": list(RELEASE_SCHEDULER_BOUNDS), "duplicate_task_bound_observed": True},
|
||||
"pressure_recovery_evidence": {"summary": "Fixture pressure", "pressure_pacing_engaged": True,
|
||||
"recovery_window_seconds": 30, "lock_hold_p95_ms": 5, "foreground_latency_p95_ms": 10,
|
||||
"pressure_metrics": pressure},
|
||||
},
|
||||
"P1": {
|
||||
"cold_walk_share_measurement": {"summary": "Fixture cold share", "duration_seconds": 7200,
|
||||
"cold_walk_share": 0.2, "walk_objects": 100, "cold_walk_objects": 20},
|
||||
"foreground_latency_throughput_measurement": {"summary": "Fixture foreground", "duration_seconds": 7200,
|
||||
"foreground_latency_p95_ms": 10, "foreground_latency_p99_ms": 20,
|
||||
"throughput_ops_per_second": 100, "error_count": 0, "foreground_p95_ms": 10.0,
|
||||
"foreground_p99_ms": 20.0, "throughput_ops": 100.0, "error_rate": 0.0},
|
||||
"profile_evidence": {"summary": "Fixture profile", "duration_seconds": 7200},
|
||||
},
|
||||
"P3": {
|
||||
"two_hour_pressure_measurement": {"summary": "Fixture two-hour pressure", "duration_seconds": 7200,
|
||||
"fixed_offered_load": True, "foreground_latency_p99_ms": 20, "attempt_cost_samples": 10,
|
||||
"abba_legs": ["A1", "B1", "B2", "A2"], "scenarios": list(SCENARIOS),
|
||||
"foreground_p95_ms": 10.0, "foreground_p99_ms": 20.0, "throughput_ops": 100.0},
|
||||
"heal_capacity_measurement": {"summary": "Fixture heal capacity", "duration_seconds": 7200,
|
||||
"completed_heal_objects": 10, "duplicate_task_count": 0,
|
||||
"heal_capacity": {"objects": 10, "versions": 10, "bytes": 4096, "completed_objects": 10}},
|
||||
"recovery_window_measurement": {"summary": "Fixture recovery", "duration_seconds": 7200,
|
||||
"pressure_recovery_window_seconds": 30, "lock_hold_p95_ms": 5,
|
||||
"fault_modes": ["process-restart", "process-crash-restart"], "recovery_p95_ms": 100.0,
|
||||
"recovery_p99_ms": 200.0},
|
||||
},
|
||||
}
|
||||
identities = {kind: (digest(path), path.stat().st_size) for kind, path in profiles.items()}
|
||||
return VerifiedPressureObservations(source_revision, ("fixture-run-0001", "fixture-window-0001",
|
||||
"2026-01-01T00:00:00Z", "2026-01-01T02:00:00Z"), fields, profile_costs, identities)
|
||||
|
||||
|
||||
def run_self_test() -> None:
|
||||
import tempfile
|
||||
from unittest import mock
|
||||
import check_test_wiring as wiring
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
@@ -618,12 +489,55 @@ def run_self_test() -> None:
|
||||
"--rss-peak-bytes", "2048",
|
||||
"--save-operations", "2",
|
||||
"--saved-bytes", "4096",
|
||||
"--started-at", "2026-01-01T00:00:00Z",
|
||||
"--finished-at", "2026-01-01T02:00:00Z",
|
||||
"--run-id", "fixture-run-0001",
|
||||
"--measurement-window-id", "fixture-window-0001",
|
||||
]
|
||||
for kind, path in zip(RELEASE_PROFILE_ARTIFACTS, profiles):
|
||||
argv.extend(["--profile-artifact", f"{kind}={path}"])
|
||||
args = parse_args(argv)
|
||||
descriptor = build_descriptor(args)
|
||||
for claim in (None, "verified"):
|
||||
if claim is not None:
|
||||
report = read_json(abba_dir / "report.json")
|
||||
report["measurement_provenance"] = claim
|
||||
write_json(abba_dir / "report.json", report)
|
||||
try:
|
||||
load_measured_abba(abba_dir, source_revision)
|
||||
except ValueError as err:
|
||||
require("not independently verified" in str(err), "wrong self-test failure for unverified ABBA")
|
||||
else:
|
||||
raise ValueError("self-test accepted unverified ABBA")
|
||||
with mock.patch(f"{__name__}.measured_provenance_verified", return_value=True):
|
||||
try:
|
||||
build_descriptor(args)
|
||||
except ValueError as err:
|
||||
require("verified pressure observations" in str(err),
|
||||
"wrong self-test failure for unverified pressure inputs")
|
||||
else:
|
||||
raise ValueError("self-test accepted unverified pressure inputs")
|
||||
def verify_fixture(command, cwd):
|
||||
require(cwd == ROOT and command[2] == "--check-scanner-heal-release-bundle-gate",
|
||||
"unexpected self-test checker invocation")
|
||||
with mock.patch.object(wiring, "measured_provenance_verified", return_value=True):
|
||||
status = wiring.scanner_heal_release_bundle_gate_status(ROOT, Path(command[3]), command[4])
|
||||
require(status["decision"] == "verified", "fixture gate validation failed")
|
||||
|
||||
# Exercise packaging with a simulated verifier; this fixture is not
|
||||
# measurement evidence and the output stays inside the temp directory.
|
||||
with mock.patch(f"{__name__}.measured_provenance_verified", return_value=True), \
|
||||
mock.patch(f"{__name__}.pressure_inputs_verified", return_value=self_test_observations(
|
||||
source_revision, dict(zip(RELEASE_PROFILE_ARTIFACTS, profiles)))), \
|
||||
mock.patch.object(subprocess, "check_call", side_effect=verify_fixture):
|
||||
descriptor = build_descriptor(args)
|
||||
require(descriptor.is_file(), "self-test descriptor missing")
|
||||
require("P2" not in read_json(descriptor)["gates"], "self-test manufactured P2 evidence")
|
||||
for gate in ("P1", "P3"):
|
||||
checked = subprocess.run([sys.executable, str(ROOT / "scripts/check_test_wiring.py"),
|
||||
"--check-scanner-heal-release-bundle-gate", str(descriptor), gate],
|
||||
cwd=ROOT, capture_output=True, text=True, check=False)
|
||||
require(checked.returncode == 1 and "not independently verified" in checked.stdout,
|
||||
f"self-test accepted unverified {gate} evidence")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
|
||||
+147
-23
@@ -16,6 +16,8 @@ import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from scanner_client_trace import analyze as reconcile_client_ledger
|
||||
|
||||
SCENARIOS = ("cold-hot", "fresh-hot", "multi-hot-new", "running-heal", "mrf-replay")
|
||||
LEGS = ("A1", "B1", "B2", "A2")
|
||||
MAX_JSON_BYTES = 1024 * 1024
|
||||
@@ -111,10 +113,10 @@ def digest(path):
|
||||
|
||||
|
||||
def read_json(path):
|
||||
require(path.stat().st_size <= MAX_JSON_BYTES, f"oversized JSON: {path.name}")
|
||||
require(path.stat().st_size <= MAX_JSON_BYTES, "oversized JSON input")
|
||||
with path.open() as stream:
|
||||
value = json.load(stream)
|
||||
require(isinstance(value, dict), f"expected JSON object: {path.name}")
|
||||
require(isinstance(value, dict), "expected JSON object")
|
||||
return value
|
||||
|
||||
|
||||
@@ -124,6 +126,52 @@ def write_json(path, value):
|
||||
path.write_text(data)
|
||||
|
||||
|
||||
def measured_provenance_verified(_output):
|
||||
# Collector snapshots and adapter echoes do not yet establish a common
|
||||
# source window or exact client percentiles. Keep acceptance closed until
|
||||
# the original observations can be independently verified.
|
||||
return False
|
||||
|
||||
|
||||
def safe_error_message(error):
|
||||
if isinstance(error, subprocess.TimeoutExpired):
|
||||
timeout = error.timeout
|
||||
return f"command timed out after {timeout} seconds" if type(timeout) in (int, float) else "command timed out"
|
||||
if isinstance(error, subprocess.CalledProcessError):
|
||||
return f"command exited with status {error.returncode}" if type(error.returncode) is int else "command failed"
|
||||
if isinstance(error, subprocess.SubprocessError):
|
||||
return type(error).__name__
|
||||
if isinstance(error, OSError):
|
||||
return f"{type(error).__name__}: {os.strerror(error.errno)}" if type(error.errno) is int else type(error).__name__
|
||||
return str(error) or type(error).__name__
|
||||
|
||||
|
||||
def add_safe_error_note(error, note, related):
|
||||
error.add_note(note)
|
||||
error._scanner_safe_notes = [*getattr(error, "_scanner_safe_notes", []), note]
|
||||
error._scanner_related_errors = [*getattr(error, "_scanner_related_errors", []), related]
|
||||
|
||||
|
||||
def error_chain(error):
|
||||
messages = []
|
||||
seen = set()
|
||||
pending = [error]
|
||||
while pending and len(seen) < 16:
|
||||
error = pending.pop(0)
|
||||
if id(error) in seen:
|
||||
continue
|
||||
seen.add(id(error))
|
||||
parts = [safe_error_message(error), *getattr(error, "_scanner_safe_notes", [])[:4]]
|
||||
messages.extend(part[:2048] for part in parts if part)
|
||||
related = getattr(error, "_scanner_related_errors", [])
|
||||
if isinstance(related, (list, tuple)):
|
||||
pending.extend(item for item in related[:4] if isinstance(item, BaseException))
|
||||
next_error = error.__cause__ or (None if error.__suppress_context__ else error.__context__)
|
||||
if next_error is not None:
|
||||
pending.append(next_error)
|
||||
return "; ".join(messages)
|
||||
|
||||
|
||||
def sha(value):
|
||||
return isinstance(value, str) and len(value) == 64 and all(c in "0123456789abcdef" for c in value)
|
||||
|
||||
@@ -191,7 +239,7 @@ def release_evidence_exact_strings(value, expected, name):
|
||||
missing = sorted(set(expected) - observed)
|
||||
require(not missing, f"missing release_evidence.{name}: {', '.join(missing)}")
|
||||
unknown = sorted(observed - set(expected))
|
||||
require(not unknown, f"unknown release_evidence.{name}: {', '.join(unknown)}")
|
||||
require(not unknown, f"unknown release_evidence.{name} entries: {len(unknown)}")
|
||||
return value
|
||||
|
||||
|
||||
@@ -378,7 +426,22 @@ def invoke(adapter, action, request, timeout):
|
||||
return result
|
||||
|
||||
|
||||
def validate_result(result, request, expected):
|
||||
def validate_client_request_samples(result, request, artifact_dir):
|
||||
reference = result.get("client_request_samples")
|
||||
require(isinstance(reference, dict), "missing client request samples")
|
||||
require(artifact_dir is not None, "client request samples require a cell artifact directory")
|
||||
observed = reconcile_client_ledger(Path(artifact_dir), reference, request, result.get("stats"), result.get("cancelled"))
|
||||
require(observed["sample_complete"] is True and observed["offered_load_complete"] is True
|
||||
and observed["p99_ns"] is not None, "client schedule or percentile evidence is incomplete")
|
||||
require(observed["errors"] == 0 and observed["http_503"] == 0, "client request errors or status unavailable")
|
||||
require(result.get("client_observation") == observed, "client observation differs from original ledgers")
|
||||
require(result.get("elapsed_seconds") == observed["elapsed_seconds"], "client request elapsed time mismatch")
|
||||
for key in ("requests", "errors", "p95_ms", "p99_ms", "throughput_ops"):
|
||||
require(result["metrics"][key] == observed[key], f"client request {key} mismatch")
|
||||
return observed
|
||||
|
||||
|
||||
def validate_result(result, request, expected, artifact_dir=None):
|
||||
require(result.get("evidence") == request["evidence"], "adapter evidence type mismatch")
|
||||
require(result.get("fixed") == request["fixed"], "offered load/config/cache/durability drift")
|
||||
require(result.get("build") == request["build"], "deployed build provenance mismatch")
|
||||
@@ -410,6 +473,8 @@ def validate_result(result, request, expected):
|
||||
require(metrics["healed_objects"] == request["expected_healed_objects"], "incomplete repair oracle")
|
||||
if request["scenario"] in ("running-heal", "mrf-replay"):
|
||||
require(metrics["healed_objects"] > 0, "zero completed repairs")
|
||||
if request["evidence"] == "measured":
|
||||
validate_client_request_samples(result, request, artifact_dir)
|
||||
return result
|
||||
|
||||
|
||||
@@ -539,6 +604,18 @@ def convergence(result):
|
||||
return ratio(window["walk_objects"], window["full_walk_objects"], "convergence work")
|
||||
|
||||
|
||||
def comparison_metrics(cell):
|
||||
metrics = cell["result"]["metrics"].copy()
|
||||
if cell["evidence"] == "measured":
|
||||
observation = cell["result"].get("client_observation")
|
||||
require(isinstance(observation, dict), "missing measured client observation")
|
||||
for percentile in (95, 99):
|
||||
key = f"p{percentile}_ns"
|
||||
require(type(observation.get(key)) is int and observation[key] >= 0, f"invalid measured {key}")
|
||||
metrics[f"p{percentile}_ms"] = Decimal(observation[key]) / Decimal(1_000_000)
|
||||
return metrics
|
||||
|
||||
|
||||
def evaluate(cells):
|
||||
comparisons = []
|
||||
inconclusive = False
|
||||
@@ -546,14 +623,13 @@ def evaluate(cells):
|
||||
for offset in range(0, len(cells), 4):
|
||||
group = cells[offset:offset + 4]
|
||||
require([cell["leg"] for cell in group] == list(LEGS), "incomplete ABBA group")
|
||||
a1, b1, b2, a2 = (cell["result"]["metrics"] for cell in group)
|
||||
a1, b1, b2, a2 = (comparison_metrics(cell) for cell in group)
|
||||
control = group[0]["comparison"] == "background"
|
||||
drift = max(abs(relative_change(a2[k], a1[k], k)) for k in ("p99_ms", "throughput_ops"))
|
||||
repeat_drift = max(abs(relative_change(b2[k], b1[k], k)) for k in ("p99_ms", "throughput_ops"))
|
||||
noise = max(drift, repeat_drift) > REPEATABILITY_LIMIT
|
||||
a = {key: (decimal_number(a1[key], key) + decimal_number(a2[key], key)) / Decimal("2") for key in METRICS}
|
||||
b = {key: (decimal_number(b1[key], key) + decimal_number(b2[key], key)) / Decimal("2") for key in METRICS}
|
||||
p95 = max(a["p95_ms"], b["p95_ms"])
|
||||
p99 = relative_change(b["p99_ms"], a["p99_ms"], "p99_ms")
|
||||
throughput = relative_change(b["throughput_ops"], a["throughput_ops"], "throughput_ops")
|
||||
thresholds = {"p99_regression": Decimal("0.10") if control else Decimal("0.05"),
|
||||
@@ -595,8 +671,12 @@ def evaluate(cells):
|
||||
comparisons.append({"scenario": group[0]["scenario"], "comparison": group[0]["comparison"],
|
||||
"round": group[0]["round"], "status": "inconclusive" if noise else ("fail" if not passed else "inconclusive" if p2_pending else "pass"),
|
||||
"a2_a1_drift": report_number(drift), "b2_b1_drift": report_number(repeat_drift),
|
||||
"foreground_p95_ms": float(p95),
|
||||
"foreground_p99_ms": float(max(a["p99_ms"], b["p99_ms"])),
|
||||
# Four disjoint legs do not supply one combined percentile.
|
||||
"foreground_p95_ms": None, "foreground_p99_ms": None,
|
||||
"foreground_percentiles_ms_by_leg": {
|
||||
cell["leg"]: {"p95": float(metrics["p95_ms"]), "p99": float(metrics["p99_ms"])}
|
||||
for cell, metrics in zip(group, (a1, b1, b2, a2))
|
||||
},
|
||||
"throughput_ops": float(min(a["throughput_ops"], b["throughput_ops"])),
|
||||
"error_rate": float(max(a["errors"] / a["requests"], b["errors"] / b["requests"])),
|
||||
"p99_regression": float(p99), "throughput_change": float(throughput),
|
||||
@@ -647,8 +727,9 @@ def evaluate(cells):
|
||||
def collect_live(prepared, request, request_path, adapter):
|
||||
collector = Path(__file__).with_name("run_scanner_validation_harness.sh")
|
||||
# Only allow connection fields here; the runner owns cadence and output paths.
|
||||
connection = prepared["collector"]
|
||||
require(set(connection) == {"alias", "endpoint", "metrics_endpoints"}, "invalid collector connection")
|
||||
connection = prepared.get("collector")
|
||||
require(isinstance(connection, dict) and set(connection) == {"alias", "endpoint", "metrics_endpoints"},
|
||||
"invalid collector connection")
|
||||
require(all(isinstance(value, str) and value for value in connection.values()), "missing collector endpoint")
|
||||
require(os.environ.get("RUSTFS_ACCESS_KEY"), "collector requires RUSTFS_ACCESS_KEY")
|
||||
require(os.environ.get("RUSTFS_SECRET_KEY"), "collector requires RUSTFS_SECRET_KEY")
|
||||
@@ -728,7 +809,7 @@ def run(manifest, adapter, output, data_root):
|
||||
background = "off" if comparison == "background" and leg.startswith("A") else "on"
|
||||
name = f"{scenario}-{comparison}-{round_id}-{leg}"
|
||||
cell_dir = output / name
|
||||
cell_dir.mkdir()
|
||||
cell_dir.mkdir(mode=0o700)
|
||||
data_dir = data_root / name
|
||||
data_dir.mkdir()
|
||||
request = {"schema": 1, "scenario": scenario, "comparison": comparison, "round": round_id,
|
||||
@@ -745,10 +826,11 @@ def run(manifest, adapter, output, data_root):
|
||||
request_path = cell_dir / "request.json"
|
||||
write_json(request_path, request)
|
||||
print(name, flush=True)
|
||||
primary_error = None
|
||||
try:
|
||||
prepared = invoke(adapter, "prepare", request_path, 300)
|
||||
require(prepared.get("ready") is True, "deployment not ready")
|
||||
if manifest["evidence"] == "measured":
|
||||
if manifest["evidence"] != "synthetic":
|
||||
result = collect_live(prepared, request, request_path, adapter)
|
||||
else:
|
||||
result = invoke(adapter, "measure", request_path, 300)
|
||||
@@ -757,26 +839,65 @@ def run(manifest, adapter, output, data_root):
|
||||
require(oracle.get("complete") is True and oracle.get("errors") == 0, "correctness oracle failed")
|
||||
require(type(oracle.get("errors")) is int, "invalid oracle error count")
|
||||
result["oracle"] = oracle["actual"]
|
||||
validate_result(result, request, request["expected_oracle"])
|
||||
validate_result(result, request, request["expected_oracle"], cell_dir)
|
||||
cells.append({**request, "result": result})
|
||||
except BaseException as error:
|
||||
primary_error = error
|
||||
raise
|
||||
finally:
|
||||
stopped = invoke(adapter, "stop", request_path, 300)
|
||||
require(stopped.get("stopped") is True, "adapter failed to stop deployment")
|
||||
try:
|
||||
stopped = invoke(adapter, "stop", request_path, 300)
|
||||
require(stopped.get("stopped") is True, "adapter failed to stop deployment")
|
||||
except BaseException as stop_error:
|
||||
if primary_error is not None:
|
||||
if not isinstance(stop_error, Exception):
|
||||
add_safe_error_note(stop_error, "prior operation failed", primary_error)
|
||||
raise stop_error from None
|
||||
if not isinstance(primary_error, Exception):
|
||||
prior = primary_error.__cause__ or (
|
||||
None if primary_error.__suppress_context__ else primary_error.__context__
|
||||
)
|
||||
if prior is not None:
|
||||
add_safe_error_note(primary_error, "prior operation failed", prior)
|
||||
add_safe_error_note(primary_error, "adapter stop failed", stop_error)
|
||||
raise primary_error from None
|
||||
combined = ValueError(
|
||||
f"{safe_error_message(primary_error)}; adapter stop failed: {safe_error_message(stop_error)}"
|
||||
)
|
||||
combined._scanner_related_errors = [primary_error, stop_error]
|
||||
raise combined from primary_error
|
||||
raise
|
||||
status, comparisons = evaluate(cells)
|
||||
synthetic = manifest["evidence"] == "synthetic"
|
||||
verified = manifest["evidence"] == "measured" and measured_provenance_verified(output)
|
||||
if not synthetic and not verified:
|
||||
# The adapter currently echoes the release manifest without proving
|
||||
# source windows or client latency samples. Keep results diagnostic
|
||||
# until those observations can be independently verified.
|
||||
for comparison in comparisons:
|
||||
comparison["measurement_provenance"] = "unverified"
|
||||
if comparison["status"] == "pass":
|
||||
comparison["status"] = "inconclusive"
|
||||
if status == "pass":
|
||||
status = "inconclusive"
|
||||
finished_at = datetime.now(timezone.utc).replace(microsecond=0)
|
||||
report = {"status": "synthetic_validated" if synthetic and status == "pass" else status,
|
||||
"evidence": manifest["evidence"], "performance": "pending" if synthetic else status,
|
||||
"evidence": manifest["evidence"], "performance": "pending" if synthetic or status == "inconclusive" else status,
|
||||
"cells": len(cells), "comparisons": comparisons,
|
||||
"started_at": manifest["started_at"], "finished_at": finished_at.isoformat().replace("+00:00", "Z")}
|
||||
if not synthetic:
|
||||
report["measurement_provenance"] = "verified" if verified else "unverified"
|
||||
write_json(output / "report.json", report)
|
||||
return 0 if status == "pass" else 3 if status == "inconclusive" else 1
|
||||
except (ValueError, KeyError, OSError, subprocess.SubprocessError) as error:
|
||||
except BaseException as error:
|
||||
finished_at = datetime.now(timezone.utc).replace(microsecond=0)
|
||||
write_json(output / "report.json", {"status": "failed", "performance": "pending",
|
||||
"completed_cells": len(cells), "error": str(error),
|
||||
"started_at": manifest.get("started_at"),
|
||||
"finished_at": finished_at.isoformat().replace("+00:00", "Z")})
|
||||
error_text = error_chain(error)
|
||||
report = {"status": "failed", "performance": "pending", "evidence": manifest["evidence"],
|
||||
"completed_cells": len(cells), "error": error_text, "started_at": manifest.get("started_at"),
|
||||
"finished_at": finished_at.isoformat().replace("+00:00", "Z")}
|
||||
if manifest["evidence"] != "synthetic":
|
||||
report["measurement_provenance"] = "unverified"
|
||||
write_json(output / "report.json", report)
|
||||
raise
|
||||
|
||||
|
||||
@@ -789,8 +910,11 @@ def main():
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
return run(read_json(args.manifest), args.adapter.resolve(), args.out_dir.resolve(), args.data_root.resolve())
|
||||
except (ValueError, KeyError, OSError, subprocess.SubprocessError) as error:
|
||||
print(f"ERROR: {error}", file=sys.stderr)
|
||||
except KeyboardInterrupt as error:
|
||||
print(f"ERROR: {error_chain(error)}", file=sys.stderr)
|
||||
return 130
|
||||
except Exception as error:
|
||||
print(f"ERROR: {error_chain(error)}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
"""Reconcile complete client ledgers and calculate exact nearest-rank samples."""
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
from contextlib import ExitStack, closing
|
||||
from decimal import Decimal, ROUND_CEILING
|
||||
|
||||
SCRATCH_FREE_FLOOR = 10 * 1024**3
|
||||
|
||||
|
||||
def require(value, message):
|
||||
if not value:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def clock_identity():
|
||||
info = time.get_clock_info('monotonic')
|
||||
require(info.monotonic is True and info.adjustable is False and math.isfinite(info.resolution)
|
||||
and info.resolution > 0, 'client monotonic clock unavailable')
|
||||
result = {'pid':os.getpid(), 'platform':sys.platform, 'implementation':info.implementation,
|
||||
'resolution_ns':max(1, math.ceil(info.resolution * 1_000_000_000)),
|
||||
'monotonic':True, 'adjustable':False, 'boot_id_sha256':None, 'process_start_ticks':None}
|
||||
if sys.platform == 'linux':
|
||||
boot = Path('/proc/sys/kernel/random/boot_id').read_bytes().strip()
|
||||
require(0 < len(boot) <= 128, 'client boot identity unavailable')
|
||||
raw = Path('/proc/self/stat').read_text()
|
||||
end = raw.rfind(')')
|
||||
require(end >= 0 and raw.split(' ', 1)[0] == str(result['pid']), 'invalid client process identity')
|
||||
fields = raw[end + 1:].split()
|
||||
require(len(fields) >= 20 and fields[19].isdigit() and int(fields[19]) > 0, 'invalid client process start')
|
||||
result.update(boot_id_sha256=hashlib.sha256(boot).hexdigest(), process_start_ticks=int(fields[19]))
|
||||
return result
|
||||
|
||||
|
||||
def unique_object(pairs):
|
||||
value = {}
|
||||
for key, item in pairs:
|
||||
require(key not in value, 'duplicate JSON key in client ledger')
|
||||
value[key] = item
|
||||
return value
|
||||
|
||||
|
||||
def source_file(root_fd, ref):
|
||||
require(type(ref) is dict and set(ref) == {'path', 'sha256', 'bytes'}, 'invalid client ledger reference')
|
||||
relative = ref['path']
|
||||
require(isinstance(relative, str) and relative and not Path(relative).is_absolute()
|
||||
and '..' not in Path(relative).parts and str(Path(relative)) == relative, 'invalid client ledger path')
|
||||
require(isinstance(ref['sha256'], str) and re.fullmatch(r'[0-9a-f]{64}', ref['sha256'])
|
||||
and type(ref['bytes']) is int and 0 <= ref['bytes'] <= 2**63-1, 'invalid client ledger identity')
|
||||
parts = Path(relative).parts
|
||||
require(parts and all(part not in ('.', '..') for part in parts), 'invalid client ledger path')
|
||||
root = os.fstat(root_fd)
|
||||
directory = os.dup(root_fd)
|
||||
try:
|
||||
for part in parts[:-1]:
|
||||
child = os.open(part, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=directory)
|
||||
os.close(directory)
|
||||
directory = child
|
||||
current = os.fstat(directory)
|
||||
require(current.st_dev == root.st_dev and current.st_uid == os.geteuid()
|
||||
and current.st_mode & 0o077 == 0, 'client ledger directory is not private and owned')
|
||||
fd = os.open(parts[-1], os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)
|
||||
try:
|
||||
current = os.fstat(fd)
|
||||
require(stat.S_ISREG(current.st_mode) and current.st_dev == root.st_dev
|
||||
and current.st_size == ref['bytes'], 'client ledger identity or size differs')
|
||||
except BaseException:
|
||||
os.close(fd)
|
||||
raise
|
||||
return fd
|
||||
finally:
|
||||
os.close(directory)
|
||||
|
||||
|
||||
def records(fd, reference):
|
||||
with os.fdopen(os.dup(fd), 'rb') as stream:
|
||||
before = os.fstat(stream.fileno())
|
||||
require(stat.S_ISREG(before.st_mode) and before.st_size == reference['bytes'], 'client ledger size differs')
|
||||
digest = hashlib.sha256()
|
||||
while line := stream.readline(4097):
|
||||
require(len(line) <= 4096 and line.endswith(b'\n'), 'truncated or oversized client record')
|
||||
digest.update(line)
|
||||
try:
|
||||
row = json.loads(line, object_pairs_hook=unique_object)
|
||||
except (UnicodeError, json.JSONDecodeError):
|
||||
raise ValueError('malformed client record') from None
|
||||
require(type(row) is dict, 'invalid client record')
|
||||
yield row
|
||||
after = os.fstat(stream.fileno())
|
||||
require((before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns)
|
||||
== (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns)
|
||||
and digest.hexdigest() == reference['sha256'], 'client ledger changed or digest differs')
|
||||
|
||||
|
||||
def request_id(row, planned):
|
||||
value = row.get('request_id')
|
||||
require(isinstance(value, str) and re.fullmatch(r'0|[1-9][0-9]{0,18}', value), 'invalid client request id')
|
||||
index = int(value)
|
||||
require(index < planned, 'client request outside plan')
|
||||
return index
|
||||
|
||||
|
||||
def scratch_parent():
|
||||
parent = Path(tempfile.gettempdir()).resolve(strict=True)
|
||||
for directory in (parent, *parent.parents):
|
||||
info = directory.stat()
|
||||
shared = bool(info.st_mode & 0o022)
|
||||
sticky = bool(info.st_mode & stat.S_ISVTX)
|
||||
require(stat.S_ISDIR(info.st_mode) and info.st_uid in (0, os.geteuid())
|
||||
and (not shared or sticky), 'untrusted client scratch parent')
|
||||
return parent
|
||||
|
||||
|
||||
def identity(reference):
|
||||
clock_id = reference.get('clock_id')
|
||||
try:
|
||||
require(isinstance(clock_id, str) and str(uuid.UUID(clock_id)) == clock_id, 'invalid client clock id')
|
||||
except (ValueError, AttributeError):
|
||||
raise ValueError('invalid client clock id') from None
|
||||
first, last = reference.get('clock_before'), reference.get('clock_after')
|
||||
keys = {'pid','platform','implementation','resolution_ns','monotonic','adjustable','boot_id_sha256','process_start_ticks'}
|
||||
require(type(first) is dict and first == last and set(first) == keys
|
||||
and type(first['pid']) is int and first['pid'] > 0
|
||||
and isinstance(first['platform'], str) and 0 < len(first['platform']) <= 64
|
||||
and isinstance(first['implementation'], str) and 0 < len(first['implementation']) <= 256
|
||||
and type(first['resolution_ns']) is int and first['resolution_ns'] > 0
|
||||
and first['monotonic'] is True and first['adjustable'] is False, 'client clock changed or unavailable')
|
||||
if first['platform'] == 'linux':
|
||||
require(isinstance(first['boot_id_sha256'], str) and re.fullmatch(r'[0-9a-f]{64}', first['boot_id_sha256'])
|
||||
and type(first['process_start_ticks']) is int and first['process_start_ticks'] > 0,
|
||||
'client process identity unavailable')
|
||||
else:
|
||||
require(first['boot_id_sha256'] is None and first['process_start_ticks'] is None,
|
||||
'unverified client process identity')
|
||||
return clock_id, first
|
||||
|
||||
|
||||
def analyze(root, reference, request, stats, cancelled):
|
||||
"""Return original-window observations. Partial input never produces release metrics.
|
||||
|
||||
Each planned index must have one schedule record and each dispatch one
|
||||
matching start and completion. Percentiles include completed successes and
|
||||
failures and use nearest rank, ceil(p*n/100), in integer nanoseconds. The
|
||||
measured window includes the real post-dispatch drain; it is not aligned
|
||||
to any server or another client. Error-rate numerator/denominator are exact.
|
||||
"""
|
||||
root_fd = os.open(Path(root), os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
|
||||
try:
|
||||
root_stat = os.fstat(root_fd)
|
||||
require(root_stat.st_uid == os.geteuid() and root_stat.st_mode & 0o077 == 0,
|
||||
'client evidence root is not private and owned')
|
||||
return _analyze_open_root(root_fd, reference, request, stats, cancelled)
|
||||
finally:
|
||||
os.close(root_fd)
|
||||
|
||||
|
||||
def _analyze_open_root(root_fd, reference, request, stats, cancelled):
|
||||
require(type(reference) is dict and reference.get('schema') == 2, 'invalid client evidence schema')
|
||||
clock, clock_info = identity(reference)
|
||||
start, dispatch_end, end = (reference.get(key) for key in ('window_start_ns','dispatch_window_end_ns','window_end_ns'))
|
||||
require(all(type(v) is int and 0 <= v < 2**63 for v in (start, dispatch_end, end))
|
||||
and start < dispatch_end and start < end, 'invalid client window')
|
||||
duration = request.get('duration_seconds')
|
||||
fixed = request.get('fixed')
|
||||
require(type(duration) is int and duration > 0 and type(fixed) is dict, 'invalid client request duration')
|
||||
rate_value = fixed.get('offered_load_ops')
|
||||
require(type(rate_value) in (int,float) and math.isfinite(rate_value) and rate_value > 0, 'invalid client offered rate')
|
||||
rate = Decimal(str(rate_value))
|
||||
planned = int((rate * Decimal(duration)).to_integral_value(rounding=ROUND_CEILING))
|
||||
require(0 < planned < 2**63 and dispatch_end == start + duration * 1_000_000_000,
|
||||
'planned client window differs')
|
||||
require(type(cancelled) is bool and (cancelled or end >= dispatch_end), 'client window stopped early')
|
||||
require(type(stats) is dict and all(type(stats.get(k)) is int and stats[k] >= 0 for k in
|
||||
('planned_requests','started','completed','errors','missed_dispatches'))
|
||||
and stats['planned_requests'] == planned, 'client counters unavailable')
|
||||
sample = {'path':reference.get('path'), 'sha256':reference.get('sha256'), 'bytes':reference.get('bytes')}
|
||||
refs = {'samples':sample, 'starts':reference.get('starts'), 'schedule':reference.get('schedule')}
|
||||
with ExitStack() as stack:
|
||||
paths = {}
|
||||
for key, value in refs.items():
|
||||
paths[key] = source_file(root_fd, value)
|
||||
stack.callback(os.close, paths[key])
|
||||
identities = {(os.fstat(fd).st_dev, os.fstat(fd).st_ino) for fd in paths.values()}
|
||||
require(len(identities) == 3, 'client ledgers alias')
|
||||
return _reduce_ledgers(paths, refs, request, stats, cancelled, clock, clock_info,
|
||||
start, dispatch_end, end, planned, rate)
|
||||
|
||||
|
||||
def _reduce_ledgers(paths, refs, request, stats, cancelled, clock, clock_info, start, dispatch_end, end, planned, rate):
|
||||
period = max(1, int(Decimal(1_000_000_000) / rate))
|
||||
with tempfile.TemporaryDirectory(prefix='scanner-client-reduce-', dir=scratch_parent()) as scratch:
|
||||
scratch_budget = 64 * 1024**2 + planned * 512
|
||||
require(shutil.disk_usage(scratch).free >= SCRATCH_FREE_FLOOR + scratch_budget,
|
||||
'insufficient client reduction scratch space')
|
||||
|
||||
def check_space(index):
|
||||
if index % 4096 == 0:
|
||||
require(shutil.disk_usage(scratch).free >= SCRATCH_FREE_FLOOR,
|
||||
'insufficient client reduction scratch space')
|
||||
|
||||
with closing(sqlite3.connect(str(Path(scratch)/'ledger.sqlite'))) as db:
|
||||
page_size = db.execute('PRAGMA page_size').fetchone()[0]
|
||||
db.execute(f'PRAGMA max_page_count={scratch_budget // page_size}')
|
||||
db.execute('CREATE TABLE requests (id INTEGER PRIMARY KEY, scheduled INTEGER NOT NULL, observed INTEGER NOT NULL, '
|
||||
'dispatched INTEGER NOT NULL, reason TEXT, began INTEGER, finished INTEGER, op TEXT, outcome TEXT, status INTEGER)')
|
||||
db.execute('CREATE INDEX by_latency ON requests((finished-began), id)')
|
||||
scheduled_count = misses = 0
|
||||
for index, row in enumerate(records(paths['schedule'], refs['schedule'])):
|
||||
check_space(index)
|
||||
require(set(row) == {'request_id','clock_id','scheduled_ns','observed_ns','dispatched','miss_reason'}
|
||||
and row['clock_id'] == clock, 'invalid schedule record')
|
||||
index = request_id(row, planned)
|
||||
expect = start + int(Decimal(index) * 1_000_000_000 / rate)
|
||||
require(index == scheduled_count and type(row['scheduled_ns']) is int
|
||||
and row['scheduled_ns'] == expect and expect < dispatch_end
|
||||
and type(row['observed_ns']) is int and expect <= row['observed_ns'] <= end
|
||||
and type(row['dispatched']) is bool, 'invalid schedule order or timestamp')
|
||||
reason = row['miss_reason']
|
||||
require((row['dispatched'] and reason is None) or
|
||||
(not row['dispatched'] and reason in ('late_slot','client_capacity','after_dispatch_window')), 'invalid dispatch outcome')
|
||||
require(not row['dispatched'] or row['observed_ns'] < min(expect + period, dispatch_end),
|
||||
'late dispatch marked on time')
|
||||
require(reason != 'late_slot' or row['observed_ns'] >= expect + period, 'inconsistent late dispatch')
|
||||
require(reason != 'after_dispatch_window' or row['observed_ns'] >= dispatch_end, 'inconsistent dispatch boundary')
|
||||
db.execute('INSERT INTO requests(id,scheduled,observed,dispatched,reason) VALUES (?,?,?,?,?)',
|
||||
(index, expect, row['observed_ns'], int(row['dispatched']), reason))
|
||||
scheduled_count += 1; misses += not row['dispatched']
|
||||
require(cancelled or scheduled_count == planned, 'scheduled client records missing')
|
||||
for index, row in enumerate(records(paths['starts'], refs['starts'])):
|
||||
check_space(index)
|
||||
require(set(row) == {'request_id','clock_id','start_ns','scheduled_ns','operation'}
|
||||
and row['clock_id'] == clock and row['operation'] in ('GET','PUT')
|
||||
and type(row['start_ns']) is int and start <= row['start_ns'] <= end
|
||||
and type(row['scheduled_ns']) is int, 'invalid client start')
|
||||
changed = db.execute('UPDATE requests SET began=?, op=? WHERE id=? AND dispatched=1 AND began IS NULL '
|
||||
'AND scheduled=? AND observed<=?',
|
||||
(row['start_ns'], row['operation'], request_id(row, planned), row['scheduled_ns'], row['start_ns'])).rowcount
|
||||
require(changed == 1, 'missing, duplicate or unmatched client start')
|
||||
for index, row in enumerate(records(paths['samples'], refs['samples'])):
|
||||
check_space(index)
|
||||
require(set(row) == {'request_id','clock_id','start_ns','end_ns','outcome','operation','error_type','http_status'}
|
||||
and row['clock_id'] == clock and row['operation'] in ('GET','PUT')
|
||||
and type(row['start_ns']) is int and type(row['end_ns']) is int
|
||||
and start <= row['start_ns'] <= row['end_ns'] <= end
|
||||
and row['outcome'] in ('success','error','timeout','cancelled'), 'invalid client completion or cross-window sample')
|
||||
error_type, status = row['error_type'], row['http_status']
|
||||
require((row['outcome'] == 'success' and error_type is None) or
|
||||
(row['outcome'] != 'success' and isinstance(error_type,str) and 0 < len(error_type) <= 256),
|
||||
'invalid client outcome identity')
|
||||
require(status is None or (type(status) is int and 100 <= status <= 599), 'invalid HTTP status')
|
||||
require(not (row['outcome'] == 'success' and status is not None and status >= 400), 'HTTP failure marked successful')
|
||||
changed = db.execute('UPDATE requests SET finished=?, outcome=?, status=? WHERE id=? AND began=? '
|
||||
'AND op=? AND finished IS NULL',
|
||||
(row['end_ns'], row['outcome'], status, request_id(row, planned), row['start_ns'], row['operation'])).rowcount
|
||||
require(changed == 1, 'missing, duplicate or unmatched client completion')
|
||||
counts = db.execute('SELECT COUNT(*), COUNT(began), COUNT(finished), '
|
||||
"COUNT(CASE WHEN outcome IS NOT NULL AND outcome != 'success' THEN 1 END), "
|
||||
'COUNT(status), COUNT(CASE WHEN status=503 THEN 1 END), '
|
||||
'COUNT(CASE WHEN began< ? AND finished> ? THEN 1 END), '
|
||||
'COUNT(CASE WHEN began>= ? THEN 1 END), '
|
||||
'COUNT(CASE WHEN dispatched=1 AND observed>= ? THEN 1 END), '
|
||||
'COUNT(CASE WHEN began-scheduled>= ? THEN 1 END) FROM requests',
|
||||
(dispatch_end, dispatch_end, dispatch_end, dispatch_end, period)).fetchone()
|
||||
rows, started, completed, errors, status_known, known_503, crossing, after_dispatch, late_dispatch, late_starts = counts
|
||||
dispatched = db.execute('SELECT COUNT(*) FROM requests WHERE dispatched=1').fetchone()[0]
|
||||
require(rows == scheduled_count and started == completed == dispatched
|
||||
and stats['started'] == started and stats['completed'] == completed
|
||||
and stats['errors'] == errors and stats['missed_dispatches'] == misses,
|
||||
'client schedule/start/completion/counter mismatch')
|
||||
complete = not cancelled and scheduled_count == planned and misses == 0 and completed == planned
|
||||
scheduler_window_complete = complete and late_dispatch == 0 and after_dispatch == 0 and late_starts == 0
|
||||
# Worker start precedes the locked start-ledger flush. It does not
|
||||
# establish when the SDK was called or whether traffic was sent.
|
||||
offered_load_complete = None if scheduler_window_complete else False
|
||||
result = {'complete':complete, 'sample_complete':complete, 'offered_load_complete':offered_load_complete,
|
||||
'scheduler_window_complete':scheduler_window_complete,
|
||||
'planned_requests':planned, 'scheduled_requests':scheduled_count,
|
||||
'missed_dispatches':misses, 'observed_completed':completed,
|
||||
'observed_errors':errors if completed else None,
|
||||
'observed_http_503_lower_bound':known_503 if known_503 else None,
|
||||
'http_status_known':status_known, 'http_status_unknown':completed-status_known,
|
||||
'http_503':known_503 if complete and completed > 0 and status_known == completed else None,
|
||||
'requests':completed if complete else None, 'errors':errors if complete else None,
|
||||
'error_rate':{'numerator':errors, 'denominator':completed} if complete and completed else None,
|
||||
'p95_ns':None, 'p99_ns':None, 'p95_ms':None, 'p99_ms':None,
|
||||
'throughput_ops':None, 'sample_throughput_ops':None,
|
||||
'quantile_method':'nearest-rank over all completed requests, including errors',
|
||||
'clock_id':clock, 'client_clock':clock_info,
|
||||
'window_start_ns':start, 'dispatch_window_end_ns':dispatch_end, 'window_end_ns':end,
|
||||
'crossing_dispatch_end':crossing, 'started_after_dispatch_end':after_dispatch,
|
||||
'dispatched_after_dispatch_end':late_dispatch, 'late_request_starts':late_starts,
|
||||
'elapsed_seconds':(end-start)/1_000_000_000, 'same_source_window_proven':False,
|
||||
'release_acceptance_ready':False}
|
||||
if complete and completed >= 100:
|
||||
check_space(0)
|
||||
for percentile in (95,99):
|
||||
rank = (completed * percentile + 99) // 100
|
||||
latency = db.execute('SELECT finished-began FROM requests ORDER BY (finished-began), id LIMIT 1 OFFSET ?', (rank-1,)).fetchone()[0]
|
||||
result['p'+str(percentile)+'_ns'] = latency
|
||||
result['p'+str(percentile)+'_ms'] = latency / 1_000_000
|
||||
if complete:
|
||||
result['sample_throughput_ops'] = (completed-errors) * 1_000_000_000 / (end-start)
|
||||
if offered_load_complete is True:
|
||||
result['throughput_ops'] = result['sample_throughput_ops']
|
||||
return result
|
||||
@@ -5,12 +5,11 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime
|
||||
from decimal import Decimal
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
@@ -18,8 +17,8 @@ from scanner_abba import (
|
||||
LEGS,
|
||||
MIN_MEASURED_RELEASE_DURATION_SECONDS,
|
||||
RELEASE_PROFILE_ARTIFACTS,
|
||||
RELEASE_SCHEDULER_BOUNDS,
|
||||
SCENARIOS,
|
||||
measured_provenance_verified,
|
||||
validate_release_evidence_manifest,
|
||||
)
|
||||
|
||||
@@ -27,7 +26,10 @@ MAX_JSON_BYTES = 1024 * 1024
|
||||
CACHE_COST_PREFIX = "CACHE_COST "
|
||||
PASS_STATES = {"pass"}
|
||||
FAIL_STATES = {"fail", "failed"}
|
||||
RELEASE_DESCRIPTOR_GATES = ("G10", "P1", "P3")
|
||||
REPORT_STATES = PASS_STATES | FAIL_STATES | {"inconclusive", "incomplete", "pending", "synthetic_validated", "unknown"}
|
||||
PERFORMANCE_STATES = PASS_STATES | FAIL_STATES | {"inconclusive", "pending", "unknown"}
|
||||
EVIDENCE_TYPES = {"measured", "synthetic", "diagnostic"}
|
||||
OBSERVATION_STATES = {"observed", "no_measured_benefit", "rss_regression", "pending", "inconclusive", "not_applicable", "unknown"}
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
@@ -44,6 +46,15 @@ def read_json(path: Path) -> dict[str, Any]:
|
||||
return value
|
||||
|
||||
|
||||
def read_json_snapshot(path: Path) -> tuple[dict[str, Any], str]:
|
||||
with path.open("rb") as stream:
|
||||
data = stream.read(MAX_JSON_BYTES + 1)
|
||||
require(len(data) <= MAX_JSON_BYTES, "oversized JSON artifact")
|
||||
value = json.loads(data)
|
||||
require(isinstance(value, dict), "expected JSON object")
|
||||
return value, hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def write_json(path: Path, value: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True, allow_nan=False) + "\n", encoding="utf-8")
|
||||
@@ -61,7 +72,9 @@ def digest(path: Path) -> str:
|
||||
|
||||
def number(value: Any, name: str) -> Decimal:
|
||||
require(type(value) in (float, int), f"invalid numeric field: {name}")
|
||||
return Decimal(str(value))
|
||||
result = Decimal(str(value))
|
||||
require(result.is_finite(), f"invalid numeric field: {name}")
|
||||
return result
|
||||
|
||||
|
||||
def maybe_number(value: Any, name: str) -> Decimal | None:
|
||||
@@ -70,6 +83,34 @@ def maybe_number(value: Any, name: str) -> Decimal | None:
|
||||
return number(value, name)
|
||||
|
||||
|
||||
def safe_observed_number(value: Any) -> int | float | None:
|
||||
if type(value) not in (int, float) or not Decimal(str(value)).is_finite():
|
||||
return None
|
||||
return value
|
||||
|
||||
|
||||
def safe_count(value: Any) -> int | None:
|
||||
return value if type(value) is int and value >= 0 else None
|
||||
|
||||
|
||||
def safe_observation_state(value: Any) -> str:
|
||||
return value if isinstance(value, str) and value in OBSERVATION_STATES else "unknown"
|
||||
|
||||
|
||||
def safe_hex(value: Any, length: int) -> str | None:
|
||||
return value if isinstance(value, str) and len(value) == length and all(c in "0123456789abcdef" for c in value) else None
|
||||
|
||||
|
||||
def safe_timestamp(value: Any) -> str | None:
|
||||
if not isinstance(value, str) or len(value) > 64:
|
||||
return None
|
||||
try:
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
return value if parsed.tzinfo is not None else None
|
||||
|
||||
|
||||
def pct(value: Decimal | None) -> str:
|
||||
if value is None:
|
||||
return "pending"
|
||||
@@ -94,13 +135,6 @@ def require_integer(value: Any, name: str, minimum: int = 0) -> int:
|
||||
return value
|
||||
|
||||
|
||||
def timestamp(value: Any, name: str) -> str:
|
||||
require(isinstance(value, str) and value.strip(), f"missing timestamp: {name}")
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
require(parsed.tzinfo is not None, f"timestamp must include timezone: {name}")
|
||||
return parsed.astimezone(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def require_metric_series(value: Any, name: str, minimum: Decimal | None = None,
|
||||
maximum: Decimal | None = None) -> list[Decimal | None]:
|
||||
require(isinstance(value, list) and value, f"missing performance evidence field: {name}")
|
||||
@@ -116,6 +150,16 @@ def require_metric_series(value: Any, name: str, minimum: Decimal | None = None,
|
||||
|
||||
|
||||
def require_measured_comparison_evidence(comparison: dict[str, Any], index: int) -> None:
|
||||
percentiles = comparison.get("foreground_percentiles_ms_by_leg")
|
||||
require(isinstance(percentiles, dict) and set(percentiles) == set(LEGS),
|
||||
f"comparison {index} missing foreground percentiles by leg")
|
||||
for leg in LEGS:
|
||||
row = percentiles[leg]
|
||||
require(isinstance(row, dict), f"comparison {index} invalid {leg} foreground percentiles")
|
||||
p95 = number(row.get("p95"), f"comparison {index} {leg} foreground percentiles p95")
|
||||
p99 = number(row.get("p99"), f"comparison {index} {leg} foreground percentiles p99")
|
||||
require(p95.is_finite() and p99.is_finite() and p95 >= 0 and p99 > 0 and p95 <= p99,
|
||||
f"comparison {index} invalid {leg} foreground percentiles")
|
||||
w10_w11 = comparison.get("w10_w11")
|
||||
require(isinstance(w10_w11, dict), f"comparison {index} missing W10/W11 evidence")
|
||||
pressure = require_metric_series(
|
||||
@@ -243,12 +287,16 @@ def has_complete_abba_matrix(manifest: dict[str, Any], report: dict[str, Any], c
|
||||
def summarize_abba(abba_dir: Path) -> dict[str, Any]:
|
||||
manifest_path = abba_dir / "manifest.json"
|
||||
report_path = abba_dir / "report.json"
|
||||
manifest = read_json(manifest_path)
|
||||
report = read_json(report_path)
|
||||
manifest, manifest_sha256 = read_json_snapshot(manifest_path)
|
||||
report, report_sha256 = read_json_snapshot(report_path)
|
||||
report_state = report.get("status")
|
||||
performance_state = report.get("performance")
|
||||
require(isinstance(report_state, str) and report_state, "report.status missing")
|
||||
require(isinstance(performance_state, str) and performance_state, "report.performance missing")
|
||||
report_state = report_state if report_state in REPORT_STATES else "unknown"
|
||||
performance_state = performance_state if performance_state in PERFORMANCE_STATES else "unknown"
|
||||
evidence_type = report.get("evidence")
|
||||
evidence_type = evidence_type if isinstance(evidence_type, str) and evidence_type in EVIDENCE_TYPES else "unknown"
|
||||
comparisons = report.get("comparisons")
|
||||
if comparisons is None:
|
||||
require(report_state not in PASS_STATES, "passing report requires comparisons")
|
||||
@@ -261,6 +309,7 @@ def summarize_abba(abba_dir: Path) -> dict[str, Any]:
|
||||
throughput_losses: list[Decimal] = []
|
||||
p1_rows = []
|
||||
p2_values: list[Decimal | None] = []
|
||||
p2_complete = matrix_complete
|
||||
foreground_p95_values: list[Decimal | None] = []
|
||||
foreground_p99_values: list[Decimal | None] = []
|
||||
throughput_values: list[Decimal | None] = []
|
||||
@@ -269,17 +318,20 @@ def summarize_abba(abba_dir: Path) -> dict[str, Any]:
|
||||
pressure_high_samples = 0
|
||||
pressure_samples_complete = matrix_complete
|
||||
pressure_high_samples_complete = matrix_complete
|
||||
attempt_cost_samples = 0
|
||||
attempt_cost_ratio_count = 0
|
||||
attempt_cost_ratios_complete = matrix_complete
|
||||
start_p95_values: list[Decimal | None] = []
|
||||
duplicate_task_values: list[Decimal | None] = []
|
||||
lock_hold_values: list[Decimal | None] = []
|
||||
w09_complete = {field: matrix_complete for field in
|
||||
("heal_start_p95_ms", "heal_duplicate_task_count", "heal_lock_hold_p95_ms")}
|
||||
w10_rows = []
|
||||
w11_rows = []
|
||||
for index, comparison in enumerate(comparisons):
|
||||
require(isinstance(comparison, dict), f"comparison {index} must be an object")
|
||||
state = comparison.get("status")
|
||||
require(isinstance(state, str) and state, f"comparison {index} missing status")
|
||||
counts[state] += 1
|
||||
counts[state if state in REPORT_STATES else "unknown"] += 1
|
||||
p99_regressions.append(number(comparison.get("p99_regression"), f"comparison {index} p99_regression"))
|
||||
throughput_change = number(comparison.get("throughput_change"), f"comparison {index} throughput_change")
|
||||
throughput_losses.append(max(Decimal("0"), -throughput_change))
|
||||
@@ -288,31 +340,37 @@ def summarize_abba(abba_dir: Path) -> dict[str, Any]:
|
||||
p1 = comparison.get("p1")
|
||||
if isinstance(p1, dict):
|
||||
p1_rows.append({
|
||||
"scenario": comparison.get("scenario"),
|
||||
"comparison": comparison.get("comparison"),
|
||||
"round": comparison.get("round"),
|
||||
"scenario": comparison.get("scenario") if comparison.get("scenario") in SCENARIOS else "unknown",
|
||||
"comparison": comparison.get("comparison") if comparison.get("comparison") in ("build", "background") else "unknown",
|
||||
"round": safe_count(comparison.get("round")),
|
||||
"required_reduction": float(number(p1.get("required_reduction"), "p1.required_reduction")),
|
||||
"observed_reduction": float(number(p1.get("observed_reduction"), "p1.observed_reduction")),
|
||||
"repeatability_drift": (
|
||||
None if p1.get("repeatability_drift") is None
|
||||
else float(number(p1.get("repeatability_drift"), "p1.repeatability_drift"))
|
||||
),
|
||||
"baseline_walk_objects": p1.get("baseline_walk_objects"),
|
||||
"baseline_cold_walk_objects": p1.get("baseline_cold_walk_objects"),
|
||||
"candidate_walk_objects": p1.get("candidate_walk_objects"),
|
||||
"candidate_cold_walk_objects": p1.get("candidate_cold_walk_objects"),
|
||||
"baseline_walk_objects": safe_count(p1.get("baseline_walk_objects")),
|
||||
"baseline_cold_walk_objects": safe_count(p1.get("baseline_cold_walk_objects")),
|
||||
"candidate_walk_objects": safe_count(p1.get("candidate_walk_objects")),
|
||||
"candidate_cold_walk_objects": safe_count(p1.get("candidate_cold_walk_objects")),
|
||||
})
|
||||
p2 = comparison.get("p2_post_stop_work_multiples")
|
||||
if isinstance(p2, list):
|
||||
p2_values.extend(maybe_number(value, "p2_post_stop_work_multiple") for value in p2)
|
||||
if isinstance(p2, list) and len(p2) == len(LEGS):
|
||||
candidate = [maybe_number(p2[index], "p2_post_stop_work_multiple") for index in (1, 2)]
|
||||
p2_values.extend(candidate)
|
||||
if any(value is None for value in candidate):
|
||||
p2_complete = False
|
||||
else:
|
||||
p2_complete = False
|
||||
w09 = comparison.get("w09")
|
||||
if isinstance(w09, dict):
|
||||
for value in w09.get("heal_start_p95_ms", []):
|
||||
start_p95_values.append(maybe_number(value, "heal_start_p95_ms"))
|
||||
for value in w09.get("heal_duplicate_task_count", []):
|
||||
duplicate_task_values.append(maybe_number(value, "heal_duplicate_task_count"))
|
||||
for value in w09.get("heal_lock_hold_p95_ms", []):
|
||||
lock_hold_values.append(maybe_number(value, "heal_lock_hold_p95_ms"))
|
||||
for field, values in (("heal_start_p95_ms", start_p95_values),
|
||||
("heal_duplicate_task_count", duplicate_task_values),
|
||||
("heal_lock_hold_p95_ms", lock_hold_values)):
|
||||
samples = w09.get(field) if isinstance(w09, dict) else None
|
||||
if isinstance(samples, list):
|
||||
values.extend(maybe_number(value, field) for value in samples)
|
||||
if not isinstance(samples, list) or len(samples) != len(LEGS) or any(value is None for value in samples):
|
||||
w09_complete[field] = False
|
||||
w10_w11 = comparison.get("w10_w11")
|
||||
if isinstance(w10_w11, dict):
|
||||
samples = w10_w11.get("foreground_pressure_samples")
|
||||
@@ -325,112 +383,143 @@ def summarize_abba(abba_dir: Path) -> dict[str, Any]:
|
||||
pressure_high_samples += sum(require_integer(value, "foreground_pressure_high_samples", 0) for value in high_samples)
|
||||
else:
|
||||
pressure_high_samples_complete = False
|
||||
for value in w10_w11.get("attempt_cost_per_healed_object", []):
|
||||
if value is not None:
|
||||
attempt_cost_samples += 1
|
||||
costs = w10_w11.get("attempt_cost_per_healed_object")
|
||||
if isinstance(costs, list) and len(costs) == len(LEGS):
|
||||
attempt_cost_ratio_count += sum(value is not None for value in costs)
|
||||
else:
|
||||
attempt_cost_ratios_complete = False
|
||||
else:
|
||||
pressure_samples_complete = False
|
||||
pressure_high_samples_complete = False
|
||||
attempt_cost_ratios_complete = False
|
||||
w10 = comparison.get("w10")
|
||||
if isinstance(w10, dict) and comparison.get("scenario") == "running-heal" and comparison.get("comparison") == "build":
|
||||
w10_rows.append({
|
||||
"round": comparison.get("round"),
|
||||
"status": w10.get("status"),
|
||||
"pacing_observed": w10.get("pacing_observed"),
|
||||
"candidate_pressure_high_ratio": w10.get("candidate_pressure_high_ratio"),
|
||||
"candidate_delay_events": w10.get("candidate_delay_events"),
|
||||
"foreground_p99_change": w10.get("foreground_p99_change"),
|
||||
"foreground_throughput_change": w10.get("foreground_throughput_change"),
|
||||
"round": safe_count(comparison.get("round")),
|
||||
"status": safe_observation_state(w10.get("status")),
|
||||
"pacing_observed": w10.get("pacing_observed") if type(w10.get("pacing_observed")) is bool else None,
|
||||
"candidate_pressure_high_ratio": safe_observed_number(w10.get("candidate_pressure_high_ratio")),
|
||||
"candidate_delay_events": safe_count(w10.get("candidate_delay_events")),
|
||||
"foreground_p99_change": safe_observed_number(w10.get("foreground_p99_change")),
|
||||
"foreground_throughput_change": safe_observed_number(w10.get("foreground_throughput_change")),
|
||||
})
|
||||
if "throughput_ops" in comparison:
|
||||
throughput_values.append(maybe_number(comparison.get("throughput_ops"), "throughput_ops"))
|
||||
error_rate_values.append(maybe_number(comparison.get("error_rate"), "error_rate"))
|
||||
error_rate = maybe_number(comparison.get("error_rate"), "error_rate")
|
||||
require(error_rate is None or Decimal("0") <= error_rate <= Decimal("1"), "invalid error_rate")
|
||||
error_rate_values.append(error_rate)
|
||||
w11 = comparison.get("w11")
|
||||
if isinstance(w11, dict) and comparison.get("scenario") == "running-heal" and comparison.get("comparison") == "build":
|
||||
w11_rows.append({
|
||||
"round": comparison.get("round"),
|
||||
"status": w11.get("status"),
|
||||
"rss_growth": w11.get("rss_growth"),
|
||||
"rss_growth_limit": w11.get("rss_growth_limit"),
|
||||
"heal_lock_wait_p99_change": w11.get("heal_lock_wait_p99_change"),
|
||||
"foreground_p99_change": w11.get("foreground_p99_change"),
|
||||
"foreground_throughput_change": w11.get("foreground_throughput_change"),
|
||||
"candidate_attempt_cost_per_healed_object": w11.get("candidate_attempt_cost_per_healed_object"),
|
||||
"round": safe_count(comparison.get("round")),
|
||||
"status": safe_observation_state(w11.get("status")),
|
||||
"rss_growth": safe_observed_number(w11.get("rss_growth")),
|
||||
"rss_growth_limit": safe_observed_number(w11.get("rss_growth_limit")),
|
||||
"heal_lock_wait_p99_change": safe_observed_number(w11.get("heal_lock_wait_p99_change")),
|
||||
"foreground_p99_change": safe_observed_number(w11.get("foreground_p99_change")),
|
||||
"foreground_throughput_change": safe_observed_number(w11.get("foreground_throughput_change")),
|
||||
"candidate_attempt_cost_per_healed_object": safe_observed_number(w11.get("candidate_attempt_cost_per_healed_object")),
|
||||
})
|
||||
|
||||
measured = report.get("evidence") == "measured"
|
||||
passed = report_state in PASS_STATES and performance_state in PASS_STATES and measured
|
||||
if passed:
|
||||
reported_pass = report_state in PASS_STATES and performance_state in PASS_STATES and measured
|
||||
verified = measured and measured_provenance_verified(abba_dir)
|
||||
if reported_pass:
|
||||
require_complete_abba_matrix(manifest, report, comparisons)
|
||||
validate_release_evidence_manifest({**manifest, "evidence": "measured"})
|
||||
for index, comparison in enumerate(comparisons):
|
||||
require_measured_comparison_evidence(comparison, index)
|
||||
require(all(value is not None for value in error_rate_values), "error_rate must be measured")
|
||||
require(all(value == 0 for value in error_rate_values), "passing report has nonzero error_rate")
|
||||
require(pressure_samples_complete, "foreground_pressure_samples must be measured")
|
||||
require(pressure_high_samples_complete, "foreground_pressure_high_samples must be measured")
|
||||
passed = reported_pass and verified
|
||||
if not verified and report.get("evidence") != "synthetic":
|
||||
for row in w10_rows + w11_rows:
|
||||
if row["status"] == "observed":
|
||||
row["status"] = "unverified"
|
||||
gate_state = "pass" if passed else "fail"
|
||||
if report_state == "synthetic_validated":
|
||||
reason = "synthetic evidence validates the harness only; measured performance remains pending"
|
||||
elif report_state in FAIL_STATES and isinstance(report.get("error"), str) and report["error"]:
|
||||
reason = f"ABBA report status is {report_state}: {report['error']}"
|
||||
reason = f"ABBA report status is {report_state}; error details omitted"
|
||||
elif report_state not in PASS_STATES:
|
||||
reason = f"ABBA report status is {report_state}"
|
||||
elif performance_state not in PASS_STATES:
|
||||
reason = f"performance status is {performance_state}"
|
||||
elif not measured:
|
||||
reason = "measured evidence is required for a performance conclusion"
|
||||
elif not verified:
|
||||
reason = "measured source windows and client percentiles are not independently verified"
|
||||
else:
|
||||
reason = "measured ABBA report passed"
|
||||
|
||||
fixed = manifest.get("fixed", {})
|
||||
require(isinstance(fixed, dict), "manifest.fixed must be an object")
|
||||
baseline = manifest.get("baseline") if isinstance(manifest.get("baseline"), dict) else {}
|
||||
candidate = manifest.get("candidate") if isinstance(manifest.get("candidate"), dict) else {}
|
||||
return {
|
||||
"gate_state": gate_state,
|
||||
"reason": reason,
|
||||
"status": report_state,
|
||||
"performance": performance_state,
|
||||
"evidence": report.get("evidence"),
|
||||
"cells": report.get("cells", 0),
|
||||
"completed_cells": report.get("completed_cells"),
|
||||
"error": report.get("error"),
|
||||
"evidence": evidence_type,
|
||||
"measurement_provenance": "verified" if verified else "unverified",
|
||||
"cells": safe_count(report.get("cells")),
|
||||
"completed_cells": safe_count(report.get("completed_cells")),
|
||||
# Historical reports may contain unredacted subprocess arguments.
|
||||
"error": "details omitted" if isinstance(report.get("error"), str) and report["error"] else None,
|
||||
"comparisons_total": len(comparisons),
|
||||
"comparison_status_counts": dict(sorted(counts.items())),
|
||||
"worst_p99_regression": None if not p99_regressions else float(max(p99_regressions)),
|
||||
"worst_throughput_loss": None if not throughput_losses else float(max(throughput_losses)),
|
||||
"foreground_p95_ms": None if max_decimal(foreground_p95_values) is None else float(max_decimal(foreground_p95_values)),
|
||||
"foreground_p99_ms": None if max_decimal(foreground_p99_values) is None else float(max_decimal(foreground_p99_values)),
|
||||
"throughput_ops": None if max_decimal(throughput_values) is None else float(max_decimal(throughput_values)),
|
||||
"worst_p99_regression": float(max(p99_regressions)) if matrix_complete and p99_regressions else None,
|
||||
"worst_throughput_loss": float(max(throughput_losses)) if matrix_complete and throughput_losses else None,
|
||||
# Comparisons describe different legs and windows. Their maxima are
|
||||
# not a percentile of one measured request population.
|
||||
"foreground_p95_ms": None,
|
||||
"foreground_p99_ms": None,
|
||||
"max_reported_comparison_p95_ms": (None if max_decimal(foreground_p95_values) is None
|
||||
else float(max_decimal(foreground_p95_values))),
|
||||
"max_reported_comparison_p99_ms": (None if max_decimal(foreground_p99_values) is None
|
||||
else float(max_decimal(foreground_p99_values))),
|
||||
"throughput_ops": None,
|
||||
"max_reported_comparison_throughput_ops": (float(max(throughput_values))
|
||||
if matrix_complete and len(throughput_values) == len(comparisons)
|
||||
and all(value is not None for value in throughput_values) else None),
|
||||
"error_rate": (None if not matrix_complete or any(value is None for value in error_rate_values)
|
||||
else float(max_decimal(error_rate_values))),
|
||||
"foreground_pressure_samples": pressure_samples if pressure_samples_complete else None,
|
||||
"foreground_pressure_high_samples": pressure_high_samples if pressure_high_samples_complete else None,
|
||||
"attempt_cost_samples": attempt_cost_samples,
|
||||
"p2_worst_post_stop_work_multiple": None if max_decimal(p2_values) is None else float(max_decimal(p2_values)),
|
||||
"w09_worst_heal_start_p95_ms": None if max_decimal(start_p95_values) is None else float(max_decimal(start_p95_values)),
|
||||
"w09_duplicate_task_count": None if max_decimal(duplicate_task_values) is None else float(max_decimal(duplicate_task_values)),
|
||||
"w09_worst_lock_hold_p95_ms": None if max_decimal(lock_hold_values) is None else float(max_decimal(lock_hold_values)),
|
||||
"attempt_cost_samples": None,
|
||||
"reported_attempt_cost_ratio_count": attempt_cost_ratio_count if attempt_cost_ratios_complete else None,
|
||||
"p2_worst_post_stop_work_multiple": float(max(p2_values)) if p2_complete and p2_values else None,
|
||||
"w09_worst_heal_start_p95_ms": (float(max(start_p95_values))
|
||||
if w09_complete["heal_start_p95_ms"] and start_p95_values else None),
|
||||
"w09_duplicate_task_count": (float(max(duplicate_task_values))
|
||||
if w09_complete["heal_duplicate_task_count"] and duplicate_task_values else None),
|
||||
"w09_worst_lock_hold_p95_ms": (float(max(lock_hold_values))
|
||||
if w09_complete["heal_lock_hold_p95_ms"] and lock_hold_values else None),
|
||||
"p1_reductions": p1_rows,
|
||||
"w10_running_heal_build": w10_rows,
|
||||
"w11_running_heal_build": w11_rows,
|
||||
"provenance": {
|
||||
"abba_dir": str(abba_dir.resolve()),
|
||||
"manifest_sha256": digest(manifest_path),
|
||||
"report_sha256": digest(report_path),
|
||||
"baseline_revision": manifest.get("baseline", {}).get("revision"),
|
||||
"baseline_sha256": manifest.get("baseline", {}).get("sha256"),
|
||||
"candidate_revision": manifest.get("candidate", {}).get("revision"),
|
||||
"candidate_sha256": manifest.get("candidate", {}).get("sha256"),
|
||||
"adapter_sha256": manifest.get("adapter_sha256"),
|
||||
"collector_sha256": manifest.get("collector_sha256"),
|
||||
"config_sha256": fixed.get("config_sha256"),
|
||||
"dataset_sha256": fixed.get("dataset_sha256"),
|
||||
"release_flags": fixed.get("release_flags"),
|
||||
"durability": fixed.get("durability"),
|
||||
"topology": fixed.get("topology"),
|
||||
"offered_load_ops": fixed.get("offered_load_ops"),
|
||||
"release_evidence": manifest.get("release_evidence"),
|
||||
"started_at": report.get("started_at") or manifest.get("started_at"),
|
||||
"finished_at": report.get("finished_at"),
|
||||
"manifest_sha256": manifest_sha256,
|
||||
"report_sha256": report_sha256,
|
||||
"baseline_revision": safe_hex(baseline.get("revision"), 40),
|
||||
"baseline_sha256": safe_hex(baseline.get("sha256"), 64),
|
||||
"candidate_revision": safe_hex(candidate.get("revision"), 40),
|
||||
"candidate_sha256": safe_hex(candidate.get("sha256"), 64),
|
||||
"adapter_sha256": safe_hex(manifest.get("adapter_sha256"), 64),
|
||||
"collector_sha256": safe_hex(manifest.get("collector_sha256"), 64),
|
||||
"config_sha256": safe_hex(fixed.get("config_sha256"), 64),
|
||||
"dataset_sha256": safe_hex(fixed.get("dataset_sha256"), 64),
|
||||
"release_flags_present": isinstance(fixed.get("release_flags"), str) and bool(fixed["release_flags"]),
|
||||
"durability": fixed.get("durability") if fixed.get("durability") in ("drive-sync=on", "drive-sync=off") else None,
|
||||
"topology": fixed.get("topology") if fixed.get("topology") == "EC8+4" else None,
|
||||
"offered_load_ops": safe_observed_number(fixed.get("offered_load_ops")),
|
||||
"release_evidence_present": isinstance(manifest.get("release_evidence"), dict),
|
||||
"started_at": safe_timestamp(report.get("started_at")) or safe_timestamp(manifest.get("started_at")),
|
||||
"finished_at": safe_timestamp(report.get("finished_at")),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -461,7 +550,8 @@ def summarize_cache_cost(path: Path) -> dict[str, Any]:
|
||||
max_save_ns = Decimal("0")
|
||||
build_sources = set()
|
||||
for index, record in enumerate(records):
|
||||
scenarios[str(record.get("scenario"))] += 1
|
||||
scenario = record.get("scenario")
|
||||
scenarios[scenario if isinstance(scenario, str) and scenario in ("unchanged", "small_dirty", "all_dirty") else "unknown"] += 1
|
||||
wire = number(record.get("cache_wire_bytes"), f"cache_cost {index} cache_wire_bytes")
|
||||
save_body = number(record.get("save_body_bytes_per_sample"), f"cache_cost {index} save_body_bytes_per_sample")
|
||||
require(wire > 0, f"cache_cost {index} has zero wire bytes")
|
||||
@@ -479,7 +569,14 @@ def summarize_cache_cost(path: Path) -> dict[str, Any]:
|
||||
max_save_ns = max(max_save_ns, value)
|
||||
build = record.get("build", {})
|
||||
require(isinstance(build, dict), f"cache_cost {index} build must be an object")
|
||||
build_sources.add((build.get("source_revision"), build.get("source_tree"), build.get("test_opt_level_override")))
|
||||
revision = build.get("source_revision")
|
||||
revision = (revision if isinstance(revision, str) and 1 <= len(revision) <= 40
|
||||
and all(c in "0123456789abcdef" for c in revision) else None)
|
||||
tree = build.get("source_tree")
|
||||
tree = tree if tree in ("clean", "dirty", "unknown") else None
|
||||
opt = build.get("test_opt_level_override")
|
||||
opt = opt if opt in ("0", "1", "2", "3", "s", "z") else None
|
||||
build_sources.add((revision, tree, opt))
|
||||
return {
|
||||
"records": len(records),
|
||||
"scenario_counts": dict(sorted(scenarios.items())),
|
||||
@@ -499,362 +596,9 @@ def summarize_cache_cost(path: Path) -> dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
def profile_artifact_map(values: list[str] | None) -> dict[str, Path]:
|
||||
artifacts: dict[str, Path] = {}
|
||||
for value in values or []:
|
||||
require("=" in value, "profile artifact must use KIND=PATH")
|
||||
kind, raw_path = value.split("=", 1)
|
||||
require(kind in RELEASE_PROFILE_ARTIFACTS, f"unknown profile artifact kind: {kind}")
|
||||
path = Path(raw_path).resolve()
|
||||
require(path.is_file() and path.stat().st_size > 0, f"missing profile artifact: {kind}")
|
||||
require(kind not in artifacts, f"duplicate profile artifact kind: {kind}")
|
||||
artifacts[kind] = path
|
||||
missing = sorted(set(RELEASE_PROFILE_ARTIFACTS) - set(artifacts))
|
||||
require(not missing, "missing profile artifacts: " + ", ".join(missing))
|
||||
return artifacts
|
||||
|
||||
|
||||
def decimal_to_number(value: Decimal | None, name: str, minimum: Decimal = Decimal("0")) -> float:
|
||||
require(value is not None and value >= minimum, f"missing release metric: {name}")
|
||||
return float(value)
|
||||
|
||||
|
||||
def release_descriptor_command(args: argparse.Namespace) -> list[str]:
|
||||
command = [
|
||||
"scripts/summarize_scanner_heal_perf.py",
|
||||
"--abba-dir",
|
||||
str(args.abba_dir),
|
||||
]
|
||||
if args.cache_cost_log:
|
||||
command.extend(["--cache-cost-log", str(args.cache_cost_log)])
|
||||
if args.require_cache_cost:
|
||||
command.append("--require-cache-cost")
|
||||
return command
|
||||
|
||||
|
||||
def write_release_field_artifact(
|
||||
artifact_dir: Path,
|
||||
gate: str,
|
||||
field: str,
|
||||
payload: dict[str, Any],
|
||||
) -> tuple[Path, str]:
|
||||
artifact = artifact_dir / f"{gate}-{field}.json"
|
||||
write_json(artifact, payload)
|
||||
return artifact, digest(artifact)
|
||||
|
||||
|
||||
def profile_wrapper_artifact(
|
||||
artifact_dir: Path,
|
||||
source_revision: str,
|
||||
run_id: str,
|
||||
window_id: str,
|
||||
kind: str,
|
||||
path: Path,
|
||||
profile_measurements: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
required_metrics = {
|
||||
"allocation-profile": ("resolved_samples", "allocation_bytes"),
|
||||
"flamegraph": ("resolved_samples",),
|
||||
"rss-samples": ("resolved_samples", "rss_peak_bytes"),
|
||||
"save-frequency": ("resolved_samples", "save_operations", "saved_bytes"),
|
||||
}[kind]
|
||||
raw_artifact = artifact_dir / f"P1-profile_evidence-{kind}.raw{path.suffix}"
|
||||
shutil.copyfile(path, raw_artifact)
|
||||
wrapper = artifact_dir / f"P1-profile_evidence-{kind}.json"
|
||||
payload = {
|
||||
"schema": 1,
|
||||
"evidence_type": "measured",
|
||||
"source_revision": source_revision,
|
||||
"run_id": run_id,
|
||||
"measurement_window_id": window_id,
|
||||
"gate": "P1",
|
||||
"field": "profile_evidence",
|
||||
"artifact_kind": kind,
|
||||
"raw_profile_name": path.name,
|
||||
"raw_profile_artifact": raw_artifact.relative_to(artifact_dir.parent).as_posix(),
|
||||
"raw_profile_sha256": digest(raw_artifact),
|
||||
"raw_profile_bytes": raw_artifact.stat().st_size,
|
||||
"raw_profile_format": path.suffix.lower().lstrip(".") or "binary",
|
||||
}
|
||||
for metric in required_metrics:
|
||||
payload[metric] = require_integer(profile_measurements.get(metric), metric, 1)
|
||||
write_json(wrapper, payload)
|
||||
return {
|
||||
"artifact": wrapper.relative_to(artifact_dir.parent).as_posix(),
|
||||
"sha256": digest(wrapper),
|
||||
"artifact_format": "json",
|
||||
"source_revision": source_revision,
|
||||
"run_id": run_id,
|
||||
"measurement_window_id": window_id,
|
||||
}
|
||||
|
||||
|
||||
def release_field(
|
||||
artifact_dir: Path,
|
||||
source_revision: str,
|
||||
run_id: str,
|
||||
window_id: str,
|
||||
started_at: str,
|
||||
finished_at: str,
|
||||
command: list[str],
|
||||
gate: str,
|
||||
field: str,
|
||||
summary_text: str,
|
||||
evidence: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
payload = {
|
||||
"schema": 1,
|
||||
"evidence_type": "measured",
|
||||
"source_revision": source_revision,
|
||||
"run_id": run_id,
|
||||
"measurement_window_id": window_id,
|
||||
"gate": gate,
|
||||
"field": field,
|
||||
**evidence,
|
||||
}
|
||||
artifact, artifact_sha = write_release_field_artifact(artifact_dir, gate, field, payload)
|
||||
return {
|
||||
"evidence_type": "measured",
|
||||
"source_revision": source_revision,
|
||||
"run_id": run_id,
|
||||
"measurement_window_id": window_id,
|
||||
"started_at": started_at,
|
||||
"finished_at": finished_at,
|
||||
"command": command,
|
||||
"artifact": artifact.relative_to(artifact_dir.parent).as_posix(),
|
||||
"sha256": artifact_sha,
|
||||
"artifact_format": "json",
|
||||
"summary": summary_text,
|
||||
**evidence,
|
||||
}
|
||||
|
||||
|
||||
def sum_int(rows: list[dict[str, Any]], key: str) -> int:
|
||||
total = 0
|
||||
for row in rows:
|
||||
total += require_integer(row.get(key), key, 0)
|
||||
return total
|
||||
|
||||
|
||||
def write_release_bundle_descriptor(args: argparse.Namespace, summary: dict[str, Any]) -> None:
|
||||
def write_release_bundle_descriptor(_args: argparse.Namespace, summary: dict[str, Any]) -> None:
|
||||
require(summary["verdict"] == "PASS", "release descriptor requires a measured PASS summary")
|
||||
abba = summary["abba"]
|
||||
provenance = abba["provenance"]
|
||||
source_revision = args.release_source_revision or provenance.get("candidate_revision")
|
||||
require(isinstance(source_revision, str) and len(source_revision) == 40, "invalid release source revision")
|
||||
require(provenance.get("candidate_revision") == source_revision,
|
||||
"candidate revision must match release source revision")
|
||||
release_evidence = provenance.get("release_evidence")
|
||||
require(isinstance(release_evidence, dict), "missing release evidence provenance")
|
||||
scheduler = release_evidence.get("scheduler")
|
||||
require(isinstance(scheduler, dict), "missing release_evidence.scheduler")
|
||||
profile = release_evidence.get("profile")
|
||||
require(isinstance(profile, dict), "missing release_evidence.profile")
|
||||
profile_measurements = profile.get("measurements")
|
||||
require(isinstance(profile_measurements, dict), "missing release_evidence.profile.measurements")
|
||||
profile_costs = {
|
||||
"resolved_samples": require_integer(profile_measurements.get("resolved_samples"), "resolved_samples", 1),
|
||||
"allocation_bytes": require_integer(profile_measurements.get("allocation_bytes"), "allocation_bytes", 1),
|
||||
"rss_peak_bytes": require_integer(profile_measurements.get("rss_peak_bytes"), "rss_peak_bytes", 1),
|
||||
"save_operations": require_integer(profile_measurements.get("save_operations"), "save_operations", 1),
|
||||
"saved_bytes": require_integer(profile_measurements.get("saved_bytes"), "saved_bytes", 1),
|
||||
}
|
||||
profile_artifacts = profile_artifact_map(args.release_profile_artifact)
|
||||
|
||||
descriptor = args.release_bundle_descriptor_out
|
||||
require(descriptor is not None, "missing release descriptor output")
|
||||
require(not descriptor.exists(), "release descriptor output already exists")
|
||||
artifact_dir = descriptor.parent / f"{descriptor.stem}-artifacts"
|
||||
require(not artifact_dir.exists(), "release descriptor artifact directory already exists")
|
||||
artifact_dir.mkdir(parents=True)
|
||||
|
||||
started_at = timestamp(provenance.get("started_at") or release_evidence.get("started_at"), "release started_at")
|
||||
finished_at = timestamp(provenance.get("finished_at") or release_evidence.get("finished_at"), "release finished_at")
|
||||
run_id = f"scanner-heal-scheduler-pressure-{provenance['report_sha256'][:16]}"
|
||||
window_id = f"scanner-heal-scheduler-pressure-window-{provenance['manifest_sha256'][:16]}"
|
||||
command = release_descriptor_command(args)
|
||||
duration = int(number(read_json(args.abba_dir / "manifest.json").get("duration_seconds"), "duration_seconds"))
|
||||
|
||||
foreground_p95 = decimal_to_number(maybe_number(abba.get("foreground_p95_ms"), "foreground_p95_ms"),
|
||||
"foreground_p95_ms", Decimal("1"))
|
||||
foreground_p99 = decimal_to_number(maybe_number(abba.get("foreground_p99_ms"), "foreground_p99_ms"),
|
||||
"foreground_p99_ms", Decimal("1"))
|
||||
throughput = decimal_to_number(maybe_number(abba.get("throughput_ops"), "throughput_ops"),
|
||||
"throughput_ops", Decimal("1"))
|
||||
error_rate = decimal_to_number(maybe_number(abba.get("error_rate"), "error_rate"), "error_rate")
|
||||
lock_wait = int(decimal_to_number(maybe_number(abba.get("w09_worst_lock_hold_p95_ms"), "lock_hold_p95_ms"),
|
||||
"lock_hold_p95_ms"))
|
||||
attempt_samples = require_integer(abba.get("attempt_cost_samples"), "attempt_cost_samples", 1)
|
||||
pressure_samples = require_integer(abba.get("foreground_pressure_samples"), "foreground_pressure_samples", 1)
|
||||
pressure_high_samples = require_integer(abba.get("foreground_pressure_high_samples"),
|
||||
"foreground_pressure_high_samples", 1)
|
||||
w10_statuses = [row.get("status") for row in abba.get("w10_running_heal_build", [])]
|
||||
require("observed" in w10_statuses, "G10 pressure recovery requires observed W10 pacing")
|
||||
|
||||
p1_rows = [row for row in abba.get("p1_reductions", []) if row.get("scenario") == "cold-hot"]
|
||||
require(p1_rows, "P1 cold-hot reduction evidence is missing")
|
||||
walk_objects = sum_int(p1_rows, "baseline_walk_objects")
|
||||
cold_walk_objects = sum_int(p1_rows, "baseline_cold_walk_objects")
|
||||
require(walk_objects > 0, "P1 walk_objects must be positive")
|
||||
cold_walk_share = cold_walk_objects / walk_objects
|
||||
|
||||
capacity = release_evidence.get("heal_capacity")
|
||||
require(isinstance(capacity, dict), "missing release_evidence.heal_capacity")
|
||||
recovery = release_evidence.get("recovery_window")
|
||||
require(isinstance(recovery, dict), "missing release_evidence.recovery_window")
|
||||
|
||||
descriptor_value = {
|
||||
"schema": 1,
|
||||
"evidence": "measured",
|
||||
"source_revision": source_revision,
|
||||
"gates": {
|
||||
"G10": {
|
||||
"status": "pass",
|
||||
"lane": "scheduler-pressure",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": {
|
||||
"scheduler_bound_evidence": release_field(
|
||||
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
|
||||
"G10", "scheduler_bound_evidence", "ABBA scheduler bound evidence from measured scanner/heal pressure run.",
|
||||
{
|
||||
"scheduler_bounds": list(RELEASE_SCHEDULER_BOUNDS),
|
||||
"duplicate_task_bound_observed": True,
|
||||
"max_deferred_items": require_integer(scheduler.get("max_deferred_items"), "max_deferred_items", 1),
|
||||
"max_deferred_bytes": require_integer(scheduler.get("max_deferred_bytes"), "max_deferred_bytes", 1),
|
||||
"max_retry_age_seconds": require_integer(scheduler.get("max_retry_age_seconds"), "max_retry_age_seconds", 1),
|
||||
"duplicate_task_count": 0,
|
||||
"duration_seconds": duration,
|
||||
},
|
||||
),
|
||||
"pressure_recovery_evidence": release_field(
|
||||
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
|
||||
"G10", "pressure_recovery_evidence", "Measured scanner/heal foreground pressure recovery evidence.",
|
||||
{
|
||||
"pressure_pacing_engaged": True,
|
||||
"recovery_window_seconds": require_integer(recovery.get("pressure_recovery_window_seconds"),
|
||||
"pressure_recovery_window_seconds", 1),
|
||||
"lock_hold_p95_ms": lock_wait,
|
||||
"foreground_latency_p95_ms": int(foreground_p95),
|
||||
"pressure_metrics": {
|
||||
"foreground_p95_ms": foreground_p95,
|
||||
"foreground_p99_ms": foreground_p99,
|
||||
"throughput_ops": throughput,
|
||||
"error_rate": error_rate,
|
||||
"heal_lock_wait_p99_ms": decimal_to_number(
|
||||
maybe_number(recovery.get("heal_lock_wait_p99_ms"), "heal_lock_wait_p99_ms"),
|
||||
"heal_lock_wait_p99_ms",
|
||||
),
|
||||
"attempt_cost_samples": attempt_samples,
|
||||
"foreground_pressure_samples": pressure_samples,
|
||||
"foreground_pressure_high_samples": pressure_high_samples,
|
||||
},
|
||||
"duration_seconds": duration,
|
||||
},
|
||||
),
|
||||
},
|
||||
},
|
||||
"P1": {
|
||||
"status": "pass",
|
||||
"lane": "scheduler-pressure",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": {
|
||||
"cold_walk_share_measurement": release_field(
|
||||
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
|
||||
"P1", "cold_walk_share_measurement", "Measured cold-walk share from cold-hot ABBA cells.",
|
||||
{
|
||||
"cold_walk_share": cold_walk_share,
|
||||
"walk_objects": walk_objects,
|
||||
"cold_walk_objects": cold_walk_objects,
|
||||
"duration_seconds": duration,
|
||||
},
|
||||
),
|
||||
"foreground_latency_throughput_measurement": release_field(
|
||||
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
|
||||
"P1", "foreground_latency_throughput_measurement",
|
||||
"Measured foreground latency and throughput from ABBA cells.",
|
||||
{
|
||||
"foreground_latency_p95_ms": int(foreground_p95),
|
||||
"foreground_latency_p99_ms": int(foreground_p99),
|
||||
"throughput_ops_per_second": int(throughput),
|
||||
"error_count": 0,
|
||||
"foreground_p95_ms": foreground_p95,
|
||||
"foreground_p99_ms": foreground_p99,
|
||||
"throughput_ops": throughput,
|
||||
"error_rate": error_rate,
|
||||
"duration_seconds": duration,
|
||||
},
|
||||
),
|
||||
"profile_evidence": release_field(
|
||||
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
|
||||
"P1", "profile_evidence", "Measured allocation, RSS, save-frequency, and flamegraph profile evidence.",
|
||||
{
|
||||
**profile_costs,
|
||||
"profile_artifacts": {
|
||||
kind: profile_wrapper_artifact(
|
||||
artifact_dir, source_revision, run_id, window_id, kind, path, profile_costs
|
||||
)
|
||||
for kind, path in sorted(profile_artifacts.items())
|
||||
},
|
||||
"duration_seconds": duration,
|
||||
},
|
||||
),
|
||||
},
|
||||
},
|
||||
"P3": {
|
||||
"status": "pass",
|
||||
"lane": "scheduler-pressure",
|
||||
"evidence_type": "measured",
|
||||
"evidence_fields": {
|
||||
"two_hour_pressure_measurement": release_field(
|
||||
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
|
||||
"P3", "two_hour_pressure_measurement", "Measured two-hour ABBA pressure run.",
|
||||
{
|
||||
"fixed_offered_load": True,
|
||||
"foreground_latency_p99_ms": int(foreground_p99),
|
||||
"attempt_cost_samples": attempt_samples,
|
||||
"abba_legs": list(LEGS),
|
||||
"scenarios": list(SCENARIOS),
|
||||
"foreground_p95_ms": foreground_p95,
|
||||
"foreground_p99_ms": foreground_p99,
|
||||
"throughput_ops": throughput,
|
||||
"duration_seconds": duration,
|
||||
},
|
||||
),
|
||||
"heal_capacity_measurement": release_field(
|
||||
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
|
||||
"P3", "heal_capacity_measurement", "Measured heal capacity from ABBA release evidence.",
|
||||
{
|
||||
"completed_heal_objects": require_integer(capacity.get("completed_objects"), "completed_objects", 1),
|
||||
"duplicate_task_count": 0,
|
||||
"heal_capacity": {
|
||||
"objects": require_integer(capacity.get("objects"), "objects", 1),
|
||||
"versions": require_integer(capacity.get("versions"), "versions", 1),
|
||||
"bytes": require_integer(capacity.get("bytes"), "bytes", 1),
|
||||
"completed_objects": require_integer(capacity.get("completed_objects"), "completed_objects", 1),
|
||||
},
|
||||
"duration_seconds": duration,
|
||||
},
|
||||
),
|
||||
"recovery_window_measurement": release_field(
|
||||
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
|
||||
"P3", "recovery_window_measurement", "Measured restart and crash recovery windows.",
|
||||
{
|
||||
"pressure_recovery_window_seconds": require_integer(recovery.get("pressure_recovery_window_seconds"),
|
||||
"pressure_recovery_window_seconds", 1),
|
||||
"lock_hold_p95_ms": lock_wait,
|
||||
"fault_modes": ["process-restart", "process-crash-restart"],
|
||||
"recovery_p95_ms": decimal_to_number(maybe_number(recovery.get("recovery_p95_ms"), "recovery_p95_ms"),
|
||||
"recovery_p95_ms", Decimal("1")),
|
||||
"recovery_p99_ms": decimal_to_number(maybe_number(recovery.get("recovery_p99_ms"), "recovery_p99_ms"),
|
||||
"recovery_p99_ms", Decimal("1")),
|
||||
"duration_seconds": duration,
|
||||
},
|
||||
),
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
write_json(descriptor, descriptor_value)
|
||||
raise ValueError("release descriptor source evidence is unavailable")
|
||||
|
||||
|
||||
def markdown(summary: dict[str, Any]) -> str:
|
||||
@@ -867,7 +611,7 @@ def markdown(summary: dict[str, Any]) -> str:
|
||||
"",
|
||||
f"- verdict: {summary['verdict']}",
|
||||
f"- reason: {summary['reason']}",
|
||||
f"- abba: status={abba['status']} performance={abba['performance']} evidence={abba['evidence']} cells={abba['cells']} comparisons={abba['comparisons_total']}",
|
||||
f"- abba: status={abba['status']} performance={abba['performance']} evidence={abba['evidence']} cells={abba['cells'] if abba['cells'] is not None else 'pending'} comparisons={abba['comparisons_total']}",
|
||||
f"- worst_p99_regression: {pct(p99)}",
|
||||
f"- worst_throughput_loss: {pct(throughput)}",
|
||||
f"- p2_worst_post_stop_work_multiple: {ratio(p2)}",
|
||||
@@ -923,7 +667,7 @@ def main() -> int:
|
||||
parser.add_argument("--json-out", type=Path, help="Write the normalized summary JSON artifact")
|
||||
parser.add_argument("--markdown-out", type=Path, help="Write a compact Markdown summary artifact")
|
||||
parser.add_argument("--release-bundle-descriptor-out", type=Path,
|
||||
help="Write a measured G10/P1/P3 release-bundle descriptor")
|
||||
help="Reserved; verified source evidence is not yet available")
|
||||
parser.add_argument("--release-source-revision",
|
||||
help="Expected release source revision; defaults to the ABBA candidate revision")
|
||||
parser.add_argument("--release-profile-artifact", action="append",
|
||||
@@ -931,13 +675,13 @@ def main() -> int:
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
summary = build_summary(args)
|
||||
if args.release_bundle_descriptor_out:
|
||||
write_release_bundle_descriptor(args, summary)
|
||||
if args.json_out:
|
||||
write_json(args.json_out, summary)
|
||||
if args.markdown_out:
|
||||
args.markdown_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.markdown_out.write_text(markdown(summary), encoding="utf-8")
|
||||
if args.release_bundle_descriptor_out:
|
||||
write_release_bundle_descriptor(args, summary)
|
||||
abba = summary["abba"]
|
||||
print(
|
||||
f"{summary['verdict']} scanner_heal_perf "
|
||||
@@ -946,7 +690,7 @@ def main() -> int:
|
||||
)
|
||||
return 0 if summary["verdict"] == "PASS" else 1
|
||||
except (ValueError, OSError, json.JSONDecodeError) as error:
|
||||
print(f"FAIL scanner_heal_perf error={error}", file=sys.stderr)
|
||||
print(f"FAIL scanner_heal_perf error={type(error).__name__}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
+327
-12
@@ -6,6 +6,7 @@ import copy
|
||||
import fcntl
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
@@ -140,6 +141,15 @@ def child_released(marker, timeout=1):
|
||||
while True:
|
||||
try:
|
||||
fcntl.flock(stream, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
record = stream.read()
|
||||
if not record.endswith(";ready"):
|
||||
raise AssertionError("fixture deadline unavailable")
|
||||
try:
|
||||
expiry = float(record.removesuffix(";ready"))
|
||||
except ValueError as error:
|
||||
raise AssertionError("fixture deadline unavailable") from error
|
||||
if not math.isfinite(expiry) or time.monotonic() >= expiry:
|
||||
raise AssertionError("fixture deadline unavailable or expired")
|
||||
return True
|
||||
except BlockingIOError:
|
||||
if time.monotonic() >= deadline:
|
||||
@@ -148,11 +158,11 @@ def child_released(marker, timeout=1):
|
||||
|
||||
|
||||
def reap_fixture(marker):
|
||||
if marker.exists() and marker.stat().st_size and not child_released(marker, timeout=0):
|
||||
# The unique file lock proves the original fixture process still owns this PID.
|
||||
os.kill(int(marker.read_text()), signal.SIGKILL)
|
||||
if not child_released(marker, timeout=5):
|
||||
raise AssertionError("fixture child did not release its process-owned lock")
|
||||
# The lock does not keep a PID reserved after the owner exits. Ask the
|
||||
# fixture to exit instead of signaling a potentially reused PID.
|
||||
marker.with_name(marker.name + ".release").touch()
|
||||
if marker.exists() and marker.stat().st_size and not child_released(marker, timeout=5):
|
||||
raise AssertionError("fixture child did not release its process-owned lock")
|
||||
|
||||
|
||||
class ScannerAbbaTest(unittest.TestCase):
|
||||
@@ -259,6 +269,7 @@ class ScannerAbbaTest(unittest.TestCase):
|
||||
with self.assertRaises(subprocess.TimeoutExpired):
|
||||
harness.invoke(self.adapter, "measure", request, 3)
|
||||
wait_for_marker(marker)
|
||||
self.assertFalse(marker.with_name(marker.name + ".release").exists())
|
||||
self.assertTrue(child_released(marker), "TERM-exited parent left its TERM-ignoring child alive")
|
||||
finally:
|
||||
reap_fixture(marker)
|
||||
@@ -283,6 +294,7 @@ class ScannerAbbaTest(unittest.TestCase):
|
||||
with self.assertRaisesRegex(ValueError, "injected measurement failure"):
|
||||
harness.collect_live({"collector": {"alias": "fixture", "endpoint": "fixture", "metrics_endpoints": "fixture"}},
|
||||
{"duration_seconds": 900}, request, self.adapter)
|
||||
self.assertFalse(marker.with_name(marker.name + ".release").exists())
|
||||
self.assertTrue(child_released(marker), "collector parent exit did not end its telemetry child")
|
||||
finally:
|
||||
reap_fixture(marker)
|
||||
@@ -300,6 +312,14 @@ class ScannerAbbaTest(unittest.TestCase):
|
||||
finally:
|
||||
reap_fixture(marker)
|
||||
|
||||
def test_invalid_fixture_deadline_does_not_count_as_cleaned_up(self):
|
||||
marker = self.root / "stubborn.pid"
|
||||
for value in ("invalid;ready", "nan;ready", str(time.monotonic() - 1) + ";ready", "1.2"):
|
||||
with self.subTest(value=value):
|
||||
marker.write_text(value)
|
||||
with self.assertRaisesRegex(AssertionError, "fixture deadline"):
|
||||
child_released(marker)
|
||||
|
||||
def test_reaped_owner_never_signals_a_reused_process_group(self):
|
||||
with (self.root / "owner.log").open("wb") as log:
|
||||
owner = harness.OwnedCommand([sys.executable, "-c", "pass"], log)
|
||||
@@ -330,6 +350,7 @@ class ScannerAbbaTest(unittest.TestCase):
|
||||
with patch.object(harness.time, "sleep", side_effect=interrupt_once):
|
||||
with self.assertRaises(KeyboardInterrupt):
|
||||
owner.finish(terminate=True)
|
||||
self.assertFalse(marker.with_name(marker.name + ".release").exists())
|
||||
self.assertTrue(child_released(marker), "cleanup cancellation left its child alive")
|
||||
self.assertIsNotNone(owner.process.returncode, "cleanup cancellation must reap its leader")
|
||||
finally:
|
||||
@@ -353,6 +374,7 @@ class ScannerAbbaTest(unittest.TestCase):
|
||||
patch.object(harness.os, "write", side_effect=release_then_fail):
|
||||
with self.assertRaisesRegex(OSError, "injected failure after gate release"):
|
||||
harness.OwnedCommand([str(self.adapter), "measure", str(request), str(self.root / "unused.json")], log)
|
||||
self.assertFalse(marker.with_name(marker.name + ".release").exists())
|
||||
self.assertTrue(child_released(marker), "initialization failure left its child alive")
|
||||
finally:
|
||||
reap_fixture(marker)
|
||||
@@ -705,12 +727,295 @@ class ScannerAbbaTest(unittest.TestCase):
|
||||
"metrics": metrics,
|
||||
"oracle": manifest["oracles"]["cold-hot"],
|
||||
}
|
||||
harness.validate_result(result, request, manifest["oracles"]["cold-hot"])
|
||||
with self.assertRaisesRegex(ValueError, "missing client request samples"):
|
||||
harness.validate_result(result, request, manifest["oracles"]["cold-hot"])
|
||||
with patch.object(harness, "validate_client_request_samples"):
|
||||
harness.validate_result(result, request, manifest["oracles"]["cold-hot"])
|
||||
result["release_evidence"] = copy.deepcopy(result["release_evidence"])
|
||||
result["release_evidence"]["profile"]["required_artifacts"].remove("flamegraph")
|
||||
with self.assertRaisesRegex(ValueError, "release evidence provenance mismatch"):
|
||||
harness.validate_result(result, request, manifest["oracles"]["cold-hot"])
|
||||
|
||||
def test_measured_self_report_cannot_pass_without_source_evidence(self):
|
||||
manifest = self.measured_manifest()
|
||||
|
||||
def invoke(_adapter, action, request_path, _timeout):
|
||||
request = harness.read_json(request_path)
|
||||
if action == "prepare":
|
||||
return {"ready": True}
|
||||
if action == "stop":
|
||||
return {"stopped": True}
|
||||
self.assertEqual(action, "oracle")
|
||||
return {"complete": True, "errors": 0, "actual": request["expected_oracle"]}
|
||||
|
||||
regression = False
|
||||
|
||||
def collect(_prepared, request, _request_path, _adapter):
|
||||
baseline = request["comparison"] == "build" and request["leg"].startswith("A")
|
||||
metrics = dict.fromkeys(harness.METRICS, 10)
|
||||
metrics.update(p99_ms=12 if regression and request["leg"].startswith("B") else 10,
|
||||
throughput_ops=100, errors=0, requests=100,
|
||||
walk_objects=100 if baseline else 20, cold_walk_objects=100 if baseline else 0,
|
||||
healed_objects=request["expected_healed_objects"], heal_duplicate_task_count=0)
|
||||
result = {key: request[key] for key in ("evidence", "fixed", "build", "data_dir", "background", "release_evidence")}
|
||||
result.update(sample_count=10, elapsed_seconds=request["duration_seconds"], metrics=metrics,
|
||||
client_observation={"p95_ns": 10_000_000,
|
||||
"p99_ns": int(metrics["p99_ms"] * 1_000_000)},
|
||||
convergence={"writes_stopped": True, "last_mutation_observed": True,
|
||||
"first_complete_publication": True, "last_mutation_time": 1,
|
||||
"last_mutation_observed_time": 2, "writes_stopped_time": 2,
|
||||
"window_start": 2, "window_end": 3, "budget_available_seconds": 1,
|
||||
"walk_objects": 110, "full_walk_objects": 100})
|
||||
return result
|
||||
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
patch.object(harness, "validate_client_request_samples"), \
|
||||
patch.object(harness, "collect_live", side_effect=collect), contextlib.redirect_stdout(io.StringIO()):
|
||||
status = harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
self.assertEqual(status, 3)
|
||||
self.assertEqual((report["status"], report["performance"]), ("inconclusive", "pending"))
|
||||
self.assertEqual(report["measurement_provenance"], "unverified")
|
||||
self.assertTrue(all(row["status"] == "inconclusive" and row["measurement_provenance"] == "unverified"
|
||||
for row in report["comparisons"]))
|
||||
|
||||
regression = True
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
patch.object(harness, "validate_client_request_samples"), \
|
||||
patch.object(harness, "collect_live", side_effect=collect), contextlib.redirect_stdout(io.StringIO()):
|
||||
status = harness.run(manifest, self.adapter, self.root / "failed-out", self.root / "failed-data")
|
||||
report = harness.read_json(self.root / "failed-out/report.json")
|
||||
self.assertEqual(status, 1)
|
||||
self.assertEqual((report["status"], report["performance"]), ("fail", "fail"))
|
||||
self.assertTrue(any(row["status"] == "fail" for row in report["comparisons"]))
|
||||
|
||||
def test_measured_percentile_threshold_uses_original_nanoseconds(self):
|
||||
baseline = 9_000_000_000_000_001
|
||||
candidate = 9_450_000_000_000_002
|
||||
cells = []
|
||||
for leg in harness.LEGS:
|
||||
ns = baseline if leg.startswith("A") else candidate
|
||||
metrics = dict.fromkeys(harness.METRICS, 1)
|
||||
metrics.update(p95_ms=1.0, p99_ms=ns / 1_000_000, requests=100, errors=0,
|
||||
walk_objects=100 if leg.startswith("A") else 20,
|
||||
cold_walk_objects=100 if leg.startswith("A") else 0)
|
||||
cells.append({"leg": leg, "evidence": "measured", "comparison": "build", "scenario": "cold-hot",
|
||||
"round": 1, "background": "on", "result": {"metrics": metrics,
|
||||
"client_observation": {"p95_ns": 1_000_000, "p99_ns": ns}}})
|
||||
_, comparisons = harness.evaluate(cells)
|
||||
self.assertGreater(comparisons[0]["p99_regression"], 0.05)
|
||||
self.assertEqual(comparisons[0]["status"], "fail")
|
||||
self.assertIsNone(comparisons[0]["foreground_p95_ms"])
|
||||
self.assertIsNone(comparisons[0]["foreground_p99_ms"])
|
||||
|
||||
def test_measurement_and_stop_errors_are_both_preserved(self):
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True}
|
||||
if action == "measure":
|
||||
raise ValueError("measurement failed")
|
||||
if action == "stop":
|
||||
raise ValueError("stop failed")
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaisesRegex(ValueError, "measurement failed.*stop failed"):
|
||||
self.run_harness()
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
self.assertIn("measurement failed", report["error"])
|
||||
self.assertIn("stop failed", report["error"])
|
||||
|
||||
def test_malformed_measured_collector_and_stop_errors_are_both_preserved(self):
|
||||
manifest = self.measured_manifest()
|
||||
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True, "collector": None}
|
||||
if action == "stop":
|
||||
raise ValueError("stop failed")
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaisesRegex(ValueError, "invalid collector connection.*stop failed"):
|
||||
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
self.assertIn("invalid collector connection", report["error"])
|
||||
self.assertIn("stop failed", report["error"])
|
||||
self.assertEqual(report["measurement_provenance"], "unverified")
|
||||
|
||||
def test_malformed_measured_collector_is_reported_after_successful_stop(self):
|
||||
manifest = self.measured_manifest()
|
||||
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True, "collector": None}
|
||||
if action == "stop":
|
||||
return {"stopped": True}
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaisesRegex(ValueError, "invalid collector connection"):
|
||||
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
self.assertEqual((report["status"], report["evidence"], report["measurement_provenance"]),
|
||||
("failed", "measured", "unverified"))
|
||||
self.assertIn("invalid collector connection", report["error"])
|
||||
|
||||
def test_interruption_is_preserved_when_stop_also_fails(self):
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True}
|
||||
if action == "measure":
|
||||
raise KeyboardInterrupt("interrupted")
|
||||
if action == "stop":
|
||||
raise ValueError("stop failed")
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaises(KeyboardInterrupt):
|
||||
self.run_harness()
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
self.assertIn("interrupted", report["error"])
|
||||
self.assertIn("stop failed", report["error"])
|
||||
|
||||
def test_stop_interruption_preserves_prior_failure(self):
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True}
|
||||
if action == "measure":
|
||||
raise ValueError("measurement failed")
|
||||
if action == "stop":
|
||||
raise KeyboardInterrupt("stop interrupted")
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaises(KeyboardInterrupt):
|
||||
self.run_harness()
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
self.assertIn("measurement failed", report["error"])
|
||||
self.assertIn("stop interrupted", report["error"])
|
||||
|
||||
def test_collector_cleanup_and_stop_errors_keep_measurement_failure(self):
|
||||
manifest = self.measured_manifest()
|
||||
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True, "collector": {"alias": "test", "endpoint": "test", "metrics_endpoints": "test"}}
|
||||
if action == "measure":
|
||||
raise ValueError("measurement failed")
|
||||
if action == "stop":
|
||||
raise ValueError("stop failed")
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
collector = Mock()
|
||||
collector.finish.side_effect = ValueError("collector cleanup failed")
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
patch.object(harness, "OwnedCommand", return_value=collector), \
|
||||
patch.dict(os.environ, {"RUSTFS_ACCESS_KEY": "test", "RUSTFS_SECRET_KEY": "test"}), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaises(ValueError):
|
||||
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
for message in ("measurement failed", "collector cleanup failed", "stop failed"):
|
||||
self.assertIn(message, report["error"])
|
||||
|
||||
def test_stop_interruption_keeps_nested_collector_failure_chain(self):
|
||||
manifest = self.measured_manifest()
|
||||
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True, "collector": {"alias": "test", "endpoint": "test", "metrics_endpoints": "test"}}
|
||||
if action == "measure":
|
||||
raise ValueError("measurement failed")
|
||||
if action == "stop":
|
||||
raise KeyboardInterrupt("stop interrupted")
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
collector = Mock()
|
||||
collector.finish.side_effect = ValueError("collector cleanup failed")
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
patch.object(harness, "OwnedCommand", return_value=collector), \
|
||||
patch.dict(os.environ, {"RUSTFS_ACCESS_KEY": "test", "RUSTFS_SECRET_KEY": "test"}), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaises(KeyboardInterrupt):
|
||||
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
for message in ("measurement failed", "collector cleanup failed", "stop interrupted"):
|
||||
self.assertIn(message, report["error"])
|
||||
|
||||
def test_collector_interruption_and_stop_failure_keep_measurement_error(self):
|
||||
manifest = self.measured_manifest()
|
||||
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True, "collector": {"alias": "test", "endpoint": "test", "metrics_endpoints": "test"}}
|
||||
if action == "measure":
|
||||
raise ValueError("measurement failed")
|
||||
if action == "stop":
|
||||
raise ValueError("stop failed")
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
collector = Mock()
|
||||
collector.finish.side_effect = KeyboardInterrupt("collector interrupted")
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
patch.object(harness, "OwnedCommand", return_value=collector), \
|
||||
patch.dict(os.environ, {"RUSTFS_ACCESS_KEY": "test", "RUSTFS_SECRET_KEY": "test"}), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaises(KeyboardInterrupt):
|
||||
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
for message in ("measurement failed", "collector interrupted", "stop failed"):
|
||||
self.assertIn(message, report["error"])
|
||||
|
||||
def test_subprocess_failures_do_not_expose_arguments(self):
|
||||
sentinel = "TEST_ONLY_ARGUMENT_SENTINEL"
|
||||
|
||||
def invoke(_adapter, action, _request_path, _timeout):
|
||||
if action == "prepare":
|
||||
return {"ready": True}
|
||||
if action == "measure":
|
||||
raise subprocess.TimeoutExpired(["collector", sentinel], 5)
|
||||
if action == "stop":
|
||||
raise subprocess.CalledProcessError(7, ["adapter", sentinel])
|
||||
self.fail(f"unexpected action {action}")
|
||||
|
||||
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
|
||||
contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaises(ValueError):
|
||||
self.run_harness()
|
||||
report = harness.read_json(self.root / "out/report.json")
|
||||
self.assertFalse(sentinel in report["error"])
|
||||
self.assertIn("timed out", report["error"])
|
||||
self.assertIn("status 7", report["error"])
|
||||
|
||||
stderr = io.StringIO()
|
||||
argv = ["scanner_abba.py", "--manifest", "unused", "--adapter", "unused", "--out-dir", "unused", "--data-root", "unused"]
|
||||
with patch.object(sys, "argv", argv), patch.object(harness, "run", side_effect=subprocess.TimeoutExpired([sentinel], 5)), \
|
||||
patch.object(harness, "read_json", return_value={}), contextlib.redirect_stderr(stderr):
|
||||
self.assertEqual(harness.main(), 1)
|
||||
self.assertFalse(sentinel in stderr.getvalue())
|
||||
|
||||
def test_unknown_release_evidence_values_are_not_echoed(self):
|
||||
sentinel = "TEST_ONLY_UNTRUSTED_VALUE"
|
||||
manifest = self.measured_manifest()
|
||||
manifest["release_evidence"]["crash_restart"]["fault_modes"].append(sentinel)
|
||||
with self.assertRaises(ValueError) as caught:
|
||||
harness.validate_manifest(manifest)
|
||||
self.assertFalse(sentinel in str(caught.exception))
|
||||
|
||||
def test_invalid_json_does_not_echo_filename(self):
|
||||
sentinel = "TEST_ONLY_FILENAME_SENTINEL"
|
||||
source = self.root / sentinel
|
||||
source.write_text("[]", encoding="utf-8")
|
||||
with self.assertRaises(ValueError) as caught:
|
||||
harness.read_json(source)
|
||||
self.assertFalse(sentinel in str(caught.exception))
|
||||
|
||||
def test_existing_data_preserved(self):
|
||||
(self.root / "data").mkdir()
|
||||
marker = self.root / "data/keep"
|
||||
@@ -737,12 +1042,22 @@ class ScannerAbbaTest(unittest.TestCase):
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) == 3 and sys.argv[1] == "--stubborn-worker":
|
||||
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
||||
with Path(sys.argv[2]).open("w+") as marker:
|
||||
fcntl.flock(marker, fcntl.LOCK_EX)
|
||||
marker.write(str(os.getpid()))
|
||||
marker.flush()
|
||||
while True:
|
||||
time.sleep(1)
|
||||
marker = Path(sys.argv[2])
|
||||
release = marker.with_name(marker.name + ".release")
|
||||
with marker.open("w+") as stream:
|
||||
fcntl.flock(stream, fcntl.LOCK_EX)
|
||||
deadline = time.monotonic() + 30
|
||||
ready = (repr(deadline) + ";ready").encode("ascii")
|
||||
if os.write(stream.fileno(), ready) != len(ready):
|
||||
raise OSError("short fixture marker write")
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
if release.exists():
|
||||
break
|
||||
except OSError:
|
||||
pass
|
||||
time.sleep(0.05)
|
||||
sys.exit(0)
|
||||
if len(sys.argv) == 4 and sys.argv[1] in ("prepare", "measure", "oracle", "stop"):
|
||||
sys.exit(fake_adapter())
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest import mock
|
||||
|
||||
import scanner_client_trace as trace
|
||||
import scanner_abba
|
||||
|
||||
|
||||
class LedgerTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
disk = mock.patch.object(trace.shutil, 'disk_usage', return_value=mock.Mock(free=100 * 1024**3))
|
||||
disk.start()
|
||||
self.addCleanup(disk.stop)
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.tmp.name).resolve(); self.root.chmod(0o700)
|
||||
self.clock = str(uuid.uuid4())
|
||||
self.identity = trace.clock_identity()
|
||||
self.start, self.dispatch_end, self.end = 1_000_000_000, 2_000_000_000, 2_100_000_000
|
||||
self.request = {'duration_seconds':1, 'fixed':{'offered_load_ops':100}}
|
||||
self.schedule, self.starts, self.samples = [], [], []
|
||||
for index in range(100):
|
||||
due = self.start + index * 10_000_000
|
||||
began = due + 2
|
||||
self.schedule.append({'request_id':str(index),'clock_id':self.clock,'scheduled_ns':due,'observed_ns':due+1,
|
||||
'dispatched':True,'miss_reason':None})
|
||||
self.starts.append({'request_id':str(index),'clock_id':self.clock,'scheduled_ns':due,'start_ns':began,'operation':'GET'})
|
||||
self.samples.append({'request_id':str(index),'clock_id':self.clock,'start_ns':began,'end_ns':began+index+1,
|
||||
'outcome':'success','operation':'GET','error_type':None,'http_status':200})
|
||||
self.stats = {'planned_requests':100, 'started':100, 'completed':100, 'errors':0, 'missed_dispatches':0}
|
||||
self.cancelled = False
|
||||
|
||||
def tearDown(self): self.tmp.cleanup()
|
||||
|
||||
def save(self):
|
||||
refs = {}
|
||||
for name, rows in (('samples',self.samples),('starts',self.starts),('schedule',self.schedule)):
|
||||
path = self.root/(name+'.ndjson')
|
||||
path.write_text(''.join(json.dumps(row,separators=(',',':'))+'\n' for row in rows))
|
||||
raw = path.read_bytes()
|
||||
refs[name] = {'path':path.name,'sha256':hashlib.sha256(raw).hexdigest(),'bytes':len(raw)}
|
||||
return {'schema':2, **refs['samples'],'clock_id':self.clock,
|
||||
'clock_before':copy.deepcopy(self.identity),'clock_after':copy.deepcopy(self.identity),
|
||||
'window_start_ns':self.start,'dispatch_window_end_ns':self.dispatch_end,'window_end_ns':self.end,
|
||||
'starts':refs['starts'],'schedule':refs['schedule']}
|
||||
|
||||
def compute(self):
|
||||
return trace.analyze(self.root, self.save(), self.request, self.stats, self.cancelled)
|
||||
|
||||
def test_nearest_rank_includes_failure_and_exact_error_ratio(self):
|
||||
self.samples[0].update(outcome='error',error_type='ClientError',http_status=503)
|
||||
self.stats['errors'] = 1
|
||||
result = self.compute()
|
||||
self.assertTrue(result['complete'])
|
||||
self.assertEqual((result['p95_ns'],result['p99_ns']), (95,99))
|
||||
self.assertEqual(result['error_rate'], {'numerator':1,'denominator':100})
|
||||
self.assertEqual(result['http_503'], 1)
|
||||
self.assertEqual(result['observed_http_503_lower_bound'], 1)
|
||||
self.assertFalse(result['same_source_window_proven'])
|
||||
self.assertFalse(result['release_acceptance_ready'])
|
||||
self.assertEqual(result['dispatch_window_end_ns'], self.dispatch_end)
|
||||
|
||||
def test_status_unavailable_is_not_zero_and_known_503_not_hidden(self):
|
||||
self.samples[0].update(outcome='error',error_type='ClientError',http_status=503)
|
||||
self.samples[1]['http_status'] = None
|
||||
self.stats['errors'] = 1
|
||||
result = self.compute()
|
||||
self.assertIsNone(result['http_503'])
|
||||
self.assertEqual(result['http_status_unknown'], 1)
|
||||
self.assertEqual(result['observed_http_503_lower_bound'], 1)
|
||||
self.assertEqual(result['error_rate'], {'numerator':1,'denominator':100})
|
||||
|
||||
def test_low_samples_and_missed_dispatch_do_not_produce_percentiles(self):
|
||||
self.request['fixed']['offered_load_ops'] = 99
|
||||
self.schedule.pop(); self.starts.pop(); self.samples.pop()
|
||||
self.stats.update(planned_requests=99, started=99, completed=99)
|
||||
from decimal import Decimal
|
||||
for index,(sch,st,fin) in enumerate(zip(self.schedule,self.starts,self.samples)):
|
||||
due=self.start + int(Decimal(index) * 1_000_000_000 / Decimal(99))
|
||||
sch.update(scheduled_ns=due,observed_ns=due+1)
|
||||
st.update(scheduled_ns=due,start_ns=due+2)
|
||||
fin.update(start_ns=due+2,end_ns=due+3+index)
|
||||
low = self.compute()
|
||||
self.assertTrue(low['complete']); self.assertIsNone(low['p99_ns']); self.assertIsNone(low['p95_ns'])
|
||||
self.request['fixed']['offered_load_ops'] = 100
|
||||
for index,(sch,st,fin) in enumerate(zip(self.schedule,self.starts,self.samples)):
|
||||
due=self.start+index*10_000_000; sch.update(scheduled_ns=due,observed_ns=due+1)
|
||||
st.update(scheduled_ns=due,start_ns=due+2); fin.update(start_ns=due+2,end_ns=due+3+index)
|
||||
due=self.start+990_000_000
|
||||
self.schedule.append({'request_id':'99','clock_id':self.clock,'scheduled_ns':due,'observed_ns':due+1,
|
||||
'dispatched':False,'miss_reason':'client_capacity'})
|
||||
self.stats.update(planned_requests=100, started=99, completed=99, missed_dispatches=1)
|
||||
missing = self.compute()
|
||||
self.assertFalse(missing['complete']); self.assertIsNone(missing['p99_ns']); self.assertIsNone(missing['error_rate'])
|
||||
|
||||
def test_explicit_cancelled_full_or_partial_does_not_produce_percentile(self):
|
||||
self.cancelled = True
|
||||
result = self.compute()
|
||||
self.assertFalse(result['complete']); self.assertIsNone(result['p99_ns'])
|
||||
self.schedule = self.schedule[:8]; self.starts = self.starts[:8]; self.samples = self.samples[:8]
|
||||
self.stats.update(started=8,completed=8)
|
||||
result = self.compute()
|
||||
self.assertFalse(result['complete']); self.assertIsNone(result['requests'])
|
||||
|
||||
def test_missed_dispatch_above_p99_sample_threshold_stays_unavailable(self):
|
||||
self.request['fixed']['offered_load_ops'] = 200
|
||||
self.schedule, self.starts, self.samples = [], [], []
|
||||
for index in range(200):
|
||||
due=self.start+index*5_000_000; dispatched=index != 199
|
||||
self.schedule.append({'request_id':str(index),'clock_id':self.clock,'scheduled_ns':due,'observed_ns':due+1,
|
||||
'dispatched':dispatched,'miss_reason':None if dispatched else 'client_capacity'})
|
||||
if dispatched:
|
||||
self.starts.append({'request_id':str(index),'clock_id':self.clock,'scheduled_ns':due,'start_ns':due+2,'operation':'GET'})
|
||||
self.samples.append({'request_id':str(index),'clock_id':self.clock,'start_ns':due+2,'end_ns':due+3+index,
|
||||
'outcome':'success','operation':'GET','error_type':None,'http_status':200})
|
||||
self.stats.update(planned_requests=200,started=199,completed=199,missed_dispatches=1)
|
||||
result=self.compute()
|
||||
self.assertFalse(result['complete']); self.assertEqual(result['observed_completed'],199)
|
||||
self.assertIsNone(result['p95_ns']); self.assertIsNone(result['p99_ns']); self.assertIsNone(result['error_rate'])
|
||||
|
||||
def test_no_ledger_truncation_or_missing_completion_accepted(self):
|
||||
ref=self.save(); path=self.root/ref['path']; raw=path.read_bytes()[:-2]; path.write_bytes(raw)
|
||||
ref.update(sha256=hashlib.sha256(raw).hexdigest(), bytes=len(raw))
|
||||
with self.assertRaises(ValueError): trace.analyze(self.root,ref,self.request,self.stats,False)
|
||||
self.samples.pop(); self.stats.update(started=99,completed=99)
|
||||
with self.assertRaises(ValueError): self.compute()
|
||||
|
||||
def test_duplicate_mismatch_clock_and_cross_window_rejected(self):
|
||||
changes = ('duplicate-start','missing-schedule','clock','cross-window','start-mismatch','duplicate-json','invalid-status','float-schedule')
|
||||
for change in changes:
|
||||
with self.subTest(change=change):
|
||||
schedules,starts,samples=copy.deepcopy(self.schedule),copy.deepcopy(self.starts),copy.deepcopy(self.samples)
|
||||
try:
|
||||
if change=='duplicate-start': self.starts.append(copy.deepcopy(self.starts[0]))
|
||||
elif change=='missing-schedule': self.schedule.pop()
|
||||
elif change=='clock': self.samples[0]['clock_id']=str(uuid.uuid4())
|
||||
elif change=='cross-window': self.samples[0]['end_ns']=self.end+1
|
||||
elif change=='start-mismatch': self.samples[0]['start_ns']+=1
|
||||
elif change=='invalid-status': self.samples[0]['http_status']=True
|
||||
elif change=='float-schedule': self.schedule[0]['scheduled_ns']=float(self.schedule[0]['scheduled_ns'])
|
||||
if change=='duplicate-json':
|
||||
ref=self.save(); p=self.root/ref['path']; raw=p.read_bytes().replace(b'"outcome":"success"', b'"outcome":"success","outcome":"error"', 1)
|
||||
p.write_bytes(raw); ref.update(sha256=hashlib.sha256(raw).hexdigest(),bytes=len(raw))
|
||||
with self.assertRaises(ValueError): trace.analyze(self.root,ref,self.request,self.stats,False)
|
||||
else:
|
||||
with self.assertRaises(ValueError): self.compute()
|
||||
finally:
|
||||
self.schedule,self.starts,self.samples=schedules,starts,samples
|
||||
|
||||
def test_scheduled_window_crossing_is_disclosed_not_retimestamped(self):
|
||||
self.samples[-1]['end_ns'] = self.dispatch_end + 5
|
||||
result = self.compute()
|
||||
self.assertEqual(result['crossing_dispatch_end'],1)
|
||||
self.assertEqual(result['window_end_ns'],self.end)
|
||||
self.assertEqual(result['dispatch_window_end_ns'],self.dispatch_end)
|
||||
self.assertFalse(result['same_source_window_proven'])
|
||||
|
||||
def test_dispatch_after_its_slot_is_not_reported_as_on_time(self):
|
||||
self.schedule[0]['observed_ns'] = self.start + 10_000_000
|
||||
self.starts[0]['start_ns'] = self.schedule[0]['observed_ns'] + 1
|
||||
self.samples[0].update(start_ns=self.starts[0]['start_ns'], end_ns=self.starts[0]['start_ns'] + 1)
|
||||
with self.assertRaisesRegex(ValueError, 'late dispatch'):
|
||||
self.compute()
|
||||
|
||||
def test_late_recorded_start_cannot_count_as_a_complete_offered_window(self):
|
||||
due = self.schedule[0]['scheduled_ns']
|
||||
self.starts[0]['start_ns'] = due + 10_000_000
|
||||
self.samples[0].update(start_ns=self.starts[0]['start_ns'], end_ns=self.starts[0]['start_ns'] + 1)
|
||||
result = self.compute()
|
||||
self.assertTrue(result['sample_complete'])
|
||||
self.assertEqual(result['late_request_starts'], 1)
|
||||
self.assertFalse(result['scheduler_window_complete'])
|
||||
self.assertIs(result['offered_load_complete'], False)
|
||||
|
||||
def test_refuses_ledger_through_symlinked_parent(self):
|
||||
ref = self.save()
|
||||
with tempfile.TemporaryDirectory() as outside:
|
||||
other = Path(outside)
|
||||
(other / 'sample.ndjson').write_bytes((self.root / ref['path']).read_bytes())
|
||||
(self.root / 'alias').symlink_to(other, target_is_directory=True)
|
||||
ref['path'] = 'alias/sample.ndjson'
|
||||
with self.assertRaises((ValueError, OSError)):
|
||||
trace.analyze(self.root, ref, self.request, self.stats, False)
|
||||
|
||||
def test_parent_replacement_cannot_redirect_open_ledger(self):
|
||||
ref = self.save()
|
||||
nested = self.root / 'client'
|
||||
nested.mkdir(mode=0o700)
|
||||
for name in ('samples.ndjson', 'starts.ndjson', 'schedule.ndjson'):
|
||||
(self.root / name).rename(nested / name)
|
||||
ref['path'] = 'client/' + ref['path']
|
||||
for key in ('starts', 'schedule'):
|
||||
ref[key]['path'] = 'client/' + ref[key]['path']
|
||||
with tempfile.TemporaryDirectory() as outside:
|
||||
outside_path = Path(outside)
|
||||
for name in ('samples.ndjson', 'starts.ndjson', 'schedule.ndjson'):
|
||||
(outside_path / name).write_bytes(b'wrong ledger\n')
|
||||
original_open = trace.os.open
|
||||
swapped = False
|
||||
|
||||
def open_with_replacement(path, flags, *args, **kwargs):
|
||||
nonlocal swapped
|
||||
if Path(path).name == 'schedule.ndjson' and not swapped:
|
||||
swapped = True
|
||||
nested.rename(self.root / 'held-client')
|
||||
nested.symlink_to(outside_path, target_is_directory=True)
|
||||
return original_open(path, flags, *args, **kwargs)
|
||||
|
||||
with mock.patch.object(trace.os, 'open', side_effect=open_with_replacement):
|
||||
result = trace.analyze(self.root, ref, self.request, self.stats, False)
|
||||
self.assertTrue(swapped)
|
||||
self.assertEqual(result['p99_ns'], 99)
|
||||
|
||||
def test_refuses_reduction_without_scratch_budget(self):
|
||||
with mock.patch.object(trace.shutil, 'disk_usage', return_value=mock.Mock(free=1)):
|
||||
with self.assertRaisesRegex(ValueError, 'scratch'):
|
||||
self.compute()
|
||||
|
||||
def test_refuses_nonsticky_shared_scratch_parent(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
shared = Path(tmp) / 'shared'
|
||||
shared.mkdir(mode=0o777)
|
||||
shared.chmod(0o777)
|
||||
with mock.patch.object(trace.tempfile, 'gettempdir', return_value=str(shared)):
|
||||
with self.assertRaisesRegex(ValueError, 'scratch parent'):
|
||||
self.compute()
|
||||
|
||||
def test_refuses_sticky_parent_owned_by_another_user(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
shared = Path(tmp) / 'shared'
|
||||
shared.mkdir(mode=0o700)
|
||||
shared.chmod(0o1777)
|
||||
shared = shared.resolve()
|
||||
original_stat = Path.stat
|
||||
|
||||
def foreign_owner(path, *args, **kwargs):
|
||||
info = original_stat(path, *args, **kwargs)
|
||||
if path == shared:
|
||||
return mock.Mock(st_mode=info.st_mode, st_uid=os.geteuid() + 1)
|
||||
return info
|
||||
|
||||
with mock.patch.object(trace.tempfile, 'gettempdir', return_value=str(shared)), \
|
||||
mock.patch.object(Path, 'stat', autospec=True, side_effect=foreign_owner):
|
||||
with self.assertRaisesRegex(ValueError, 'scratch parent'):
|
||||
self.compute()
|
||||
|
||||
def test_abba_does_not_accept_ledger_without_proven_offered_load(self):
|
||||
ref = self.save()
|
||||
observed = trace.analyze(self.root, ref, self.request, self.stats, False)
|
||||
result = {'client_request_samples': ref, 'client_observation': observed,
|
||||
'stats': self.stats, 'cancelled': False,
|
||||
'elapsed_seconds': observed['elapsed_seconds'],
|
||||
'metrics': {key: observed[key] for key in ('requests', 'errors', 'p95_ms', 'p99_ms', 'throughput_ops')}}
|
||||
with self.assertRaisesRegex(ValueError, 'client schedule or percentile evidence is incomplete'):
|
||||
scanner_abba.validate_client_request_samples(result, self.request, self.root)
|
||||
|
||||
def test_abba_does_not_accept_missing_or_tampered_ledger(self):
|
||||
with self.assertRaisesRegex(ValueError, 'missing client request samples'):
|
||||
scanner_abba.validate_client_request_samples({}, self.request, self.root)
|
||||
ref = self.save()
|
||||
(self.root / ref['path']).write_bytes(b'tampered\n')
|
||||
with self.assertRaises(ValueError):
|
||||
scanner_abba.validate_client_request_samples(
|
||||
{'client_request_samples': ref, 'stats': self.stats, 'cancelled': False}, self.request, self.root)
|
||||
|
||||
def test_abba_does_not_turn_unknown_status_into_zero_503s(self):
|
||||
ref = self.save()
|
||||
observed = trace.analyze(self.root, ref, self.request, self.stats, False)
|
||||
observed.update(offered_load_complete=True, http_503=None)
|
||||
result = {'client_request_samples': ref, 'client_observation': observed,
|
||||
'stats': self.stats, 'cancelled': False}
|
||||
with mock.patch.object(scanner_abba, 'reconcile_client_ledger', return_value=observed):
|
||||
with self.assertRaisesRegex(ValueError, 'status unavailable'):
|
||||
scanner_abba.validate_client_request_samples(result, self.request, self.root)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for incomplete Scanner/Heal evidence plans."""
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import run_scanner_heal_linux_evidence_plan as planner
|
||||
|
||||
|
||||
class ScannerHealLinuxEvidencePlanTest(unittest.TestCase):
|
||||
def test_p2_stays_pending_without_an_independent_oracle_producer(self):
|
||||
steps = planner.performance_steps()
|
||||
p2 = next(step for step in steps if step["id"] == "p2-independent-evidence")
|
||||
plan = {"source_revision": "a" * 40, "stages": [{"id": "performance", "steps": [p2]}]}
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
status = planner.build_status(plan, Path(tmp))
|
||||
self.assertEqual(status["decision"], "blocked")
|
||||
self.assertIn("P2", status["pending_gates"])
|
||||
self.assertTrue(status["next_step"]["reason"])
|
||||
|
||||
def test_scheduler_pressure_plan_does_not_claim_p2(self):
|
||||
step = next(step for step in planner.performance_steps() if step["id"] == "scheduler-pressure")
|
||||
self.assertNotIn("P2", step["covers"]["gates"])
|
||||
command = step["commands"][0]["command"]
|
||||
for flag in ("--started-at", "--finished-at", "--run-id", "--measurement-window-id"):
|
||||
self.assertIn(flag, command)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -13,6 +13,7 @@ cleanup() {
|
||||
trap cleanup EXIT
|
||||
|
||||
"${RUSTFS_PYTHON_BIN:-python3}" "$RUNNER" --self-test
|
||||
"${RUSTFS_PYTHON_BIN:-python3}" "$PROJECT_ROOT/scripts/test_scanner_heal_linux_evidence_plan.py"
|
||||
|
||||
"${RUSTFS_PYTHON_BIN:-python3}" "$RUNNER" --phase preflight >"$TMP_DIR/preflight.out"
|
||||
rg -q "stage=preflight" "$TMP_DIR/preflight.out"
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regression tests for scheduler-pressure evidence production."""
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
from unittest import mock
|
||||
|
||||
import run_scanner_heal_scheduler_pressure_evidence as producer
|
||||
|
||||
|
||||
class SchedulerPressureEvidenceTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.revision = producer.git_head()
|
||||
self.abba, self.recovery, self.profiles = producer.write_self_test_abba(self.root, self.revision)
|
||||
|
||||
def args(self, original_window):
|
||||
argv = ["--abba-dir", str(self.abba), "--recovery-window-json", str(self.recovery),
|
||||
"--out-dir", str(self.root / "out")]
|
||||
for kind, path in zip(producer.RELEASE_PROFILE_ARTIFACTS, self.profiles):
|
||||
argv.extend(["--profile-artifact", f"{kind}={path}"])
|
||||
if original_window:
|
||||
argv.extend(["--started-at", "2026-01-01T00:00:00Z", "--finished-at", "2026-01-01T02:00:00Z",
|
||||
"--run-id", "fixture-run-0001", "--measurement-window-id", "fixture-window-0001"])
|
||||
return producer.parse_args(argv)
|
||||
|
||||
def observations(self):
|
||||
return producer.self_test_observations(
|
||||
self.revision, dict(zip(producer.RELEASE_PROFILE_ARTIFACTS, self.profiles)))
|
||||
|
||||
def test_missing_original_window_is_rejected(self):
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
|
||||
mock.patch.object(producer.subprocess, "check_call"):
|
||||
with self.assertRaisesRegex(ValueError, "original measurement window"):
|
||||
producer.build_descriptor(self.args(original_window=False))
|
||||
|
||||
def test_object_counts_do_not_become_segment_oracle_evidence(self):
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
|
||||
mock.patch.object(producer, "pressure_inputs_verified", return_value=self.observations()), \
|
||||
mock.patch.object(producer.subprocess, "check_call"):
|
||||
descriptor = producer.build_descriptor(self.args(original_window=True))
|
||||
self.assertFalse("P2" in producer.read_json(descriptor)["gates"])
|
||||
|
||||
def test_unverified_pressure_inputs_are_rejected_after_abba_verification(self):
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True):
|
||||
with self.assertRaisesRegex(ValueError, "independent verified pressure observations"):
|
||||
producer.build_descriptor(self.args(original_window=True))
|
||||
self.assertFalse((self.root / "out").exists())
|
||||
|
||||
def test_boolean_claim_cannot_create_pressure_measurements(self):
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
|
||||
mock.patch.object(producer, "pressure_inputs_verified", return_value=True), \
|
||||
mock.patch.object(producer.subprocess, "check_call"):
|
||||
with self.assertRaisesRegex(ValueError, "verified pressure observations"):
|
||||
producer.build_descriptor(self.args(original_window=True))
|
||||
self.assertFalse((self.root / "out").exists())
|
||||
|
||||
def test_capacity_is_copied_from_verified_observation_without_fabrication(self):
|
||||
observed = self.observations()
|
||||
capacity = observed.fields["P3"]["heal_capacity_measurement"]
|
||||
capacity["completed_heal_objects"] = 0
|
||||
capacity["heal_capacity"] = {"objects": 0, "versions": 0, "bytes": 0, "completed_objects": 0}
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
|
||||
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed), \
|
||||
mock.patch.object(producer.subprocess, "check_call"):
|
||||
descriptor = producer.build_descriptor(self.args(original_window=True))
|
||||
result = producer.read_json(descriptor)["gates"]["P3"]["evidence_fields"]["heal_capacity_measurement"]
|
||||
self.assertEqual(result["completed_heal_objects"], 0)
|
||||
self.assertEqual(result["heal_capacity"], capacity["heal_capacity"])
|
||||
|
||||
def test_incomplete_verified_fields_do_not_create_output(self):
|
||||
observed = self.observations()
|
||||
observed.fields["P3"].pop("heal_capacity_measurement")
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
|
||||
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed):
|
||||
with self.assertRaisesRegex(ValueError, "observations are incomplete"):
|
||||
producer.build_descriptor(self.args(original_window=True))
|
||||
self.assertFalse((self.root / "out").exists())
|
||||
|
||||
def test_failed_checker_does_not_publish_a_pass_descriptor(self):
|
||||
observed = self.observations()
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
|
||||
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed), \
|
||||
mock.patch.object(producer.subprocess, "check_call", side_effect=subprocess.CalledProcessError(1, ["fixture"])):
|
||||
with self.assertRaises(subprocess.CalledProcessError):
|
||||
producer.build_descriptor(self.args(original_window=True))
|
||||
self.assertFalse((self.root / "out" / "release-bundle-scheduler-pressure.json").exists())
|
||||
|
||||
def test_profile_replacement_after_verification_is_rejected(self):
|
||||
observed = self.observations()
|
||||
self.profiles[0].write_bytes(b"changed profile")
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
|
||||
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed):
|
||||
with self.assertRaisesRegex(ValueError, "profile artifact differs"):
|
||||
producer.build_descriptor(self.args(original_window=True))
|
||||
self.assertFalse((self.root / "out" / "release-bundle-scheduler-pressure.json").exists())
|
||||
|
||||
def test_destination_corruption_is_rejected_before_publication(self):
|
||||
observed = self.observations()
|
||||
original_rename = producer.os.rename
|
||||
|
||||
def corrupt_after_move(source, destination):
|
||||
original_rename(source, destination)
|
||||
(destination / "P3-heal_capacity_measurement.json").write_text("corrupted")
|
||||
|
||||
def check_destination(command, **_kwargs):
|
||||
descriptor = Path(command[-2])
|
||||
self.assertEqual(descriptor.parent, (self.root / "out").resolve())
|
||||
self.assertFalse((descriptor.parent / "release-bundle-scheduler-pressure.json").exists())
|
||||
fields = producer.read_json(descriptor)["gates"][command[-1]]["evidence_fields"]
|
||||
for field in fields.values():
|
||||
if producer.digest(descriptor.parent / field["artifact"]) != field["sha256"]:
|
||||
raise subprocess.CalledProcessError(1, command)
|
||||
|
||||
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
|
||||
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed), \
|
||||
mock.patch.object(producer.os, "rename", side_effect=corrupt_after_move), \
|
||||
mock.patch.object(producer.subprocess, "check_call", side_effect=check_destination):
|
||||
with self.assertRaises(subprocess.CalledProcessError):
|
||||
producer.build_descriptor(self.args(original_window=True))
|
||||
self.assertFalse((self.root / "out" / "release-bundle-scheduler-pressure.json").exists())
|
||||
self.assertFalse((self.root / "out" / ".release-bundle-scheduler-pressure.pending.json").exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -6,3 +6,4 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
RUNNER="$SCRIPT_DIR/run_scanner_heal_scheduler_pressure_evidence.py"
|
||||
|
||||
"${RUSTFS_PYTHON_BIN:-python3}" "$RUNNER" --self-test
|
||||
"${RUSTFS_PYTHON_BIN:-python3}" "$SCRIPT_DIR/test_scanner_heal_scheduler_pressure_evidence.py"
|
||||
|
||||
@@ -443,4 +443,6 @@ grep -q -- 'unknown arg: --secret-key' "$secret_arg_log"
|
||||
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_service.py"
|
||||
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_process_receipt.py"
|
||||
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_abba.py"
|
||||
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_client_trace.py"
|
||||
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_summarize_scanner_heal_perf.py"
|
||||
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_otlp_window.py"
|
||||
|
||||
@@ -16,7 +16,6 @@ from unittest import mock
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import summarize_scanner_heal_perf as summary
|
||||
import check_test_wiring as wiring
|
||||
|
||||
|
||||
def sha(path: Path) -> str:
|
||||
@@ -118,6 +117,9 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
|
||||
"status": "pass",
|
||||
"foreground_p95_ms": 8.0,
|
||||
"foreground_p99_ms": 10.0,
|
||||
"foreground_percentiles_ms_by_leg": {
|
||||
leg: {"p95": 8.0, "p99": 10.0} for leg in summary.LEGS
|
||||
},
|
||||
"throughput_ops": 100.0,
|
||||
"error_rate": 0.0,
|
||||
"p99_regression": 0.02,
|
||||
@@ -200,7 +202,8 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
|
||||
(self.abba / "manifest.json").write_text(json.dumps(self.manifest), encoding="utf-8")
|
||||
(self.abba / "report.json").write_text(json.dumps(self.report), encoding="utf-8")
|
||||
|
||||
def test_measured_pass_writes_quiet_artifacts(self):
|
||||
@mock.patch.object(summary, "measured_provenance_verified", return_value=True)
|
||||
def test_measured_pass_writes_quiet_artifacts(self, _verified):
|
||||
cache_log = self.root / "cache.log"
|
||||
cache_log.write_text(
|
||||
"compiler noise\nCACHE_COST "
|
||||
@@ -237,7 +240,8 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
|
||||
self.assertIn("w11_running_heal_build_statuses: observed,observed,observed", summary.markdown(result))
|
||||
self.assertEqual(result["cache_cost"]["max_save_body_amplification"], 2.0)
|
||||
|
||||
def test_release_descriptor_binds_g10_p1_p3_measured_artifacts(self):
|
||||
@mock.patch.object(summary, "measured_provenance_verified", return_value=True)
|
||||
def test_release_descriptor_rejects_unreconciled_comparison_percentiles(self, _verified):
|
||||
profile_paths = []
|
||||
for kind in summary.RELEASE_PROFILE_ARTIFACTS:
|
||||
artifact = self.root / f"{kind}.artifact"
|
||||
@@ -252,36 +256,13 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
|
||||
"release_profile_artifact": profile_paths,
|
||||
})
|
||||
result = summary.build_summary(args)
|
||||
summary.write_release_bundle_descriptor(args, result)
|
||||
descriptor = summary.read_json(args.release_bundle_descriptor_out)
|
||||
self.assertEqual(sorted(descriptor["gates"]), ["G10", "P1", "P3"])
|
||||
self.assertEqual(
|
||||
descriptor["gates"]["G10"]["evidence_fields"]["scheduler_bound_evidence"]["scheduler_bounds"],
|
||||
list(summary.RELEASE_SCHEDULER_BOUNDS),
|
||||
)
|
||||
profile = descriptor["gates"]["P1"]["evidence_fields"]["profile_evidence"]
|
||||
self.assertEqual(sorted(profile["profile_artifacts"]), sorted(summary.RELEASE_PROFILE_ARTIFACTS))
|
||||
self.assertEqual(
|
||||
profile["measurement_window_id"],
|
||||
descriptor["gates"]["P3"]["evidence_fields"]["two_hour_pressure_measurement"]["measurement_window_id"],
|
||||
)
|
||||
for gate in ("G10", "P1", "P3"):
|
||||
with mock.patch("subprocess.check_output", return_value="b" * 40):
|
||||
status = wiring.scanner_heal_release_bundle_gate_status(
|
||||
Path(__file__).resolve().parents[1],
|
||||
args.release_bundle_descriptor_out,
|
||||
gate,
|
||||
)
|
||||
self.assertEqual(status["verified_gate"], gate)
|
||||
save_frequency = profile["profile_artifacts"]["save-frequency"]
|
||||
wrapper = args.release_bundle_descriptor_out.parent / save_frequency["artifact"]
|
||||
payload = summary.read_json(wrapper)
|
||||
raw_profile = args.release_bundle_descriptor_out.parent / payload["raw_profile_artifact"]
|
||||
self.assertTrue(raw_profile.is_file())
|
||||
self.assertEqual(payload["saved_bytes"], 8192)
|
||||
self.assertEqual(payload["raw_profile_sha256"], sha(raw_profile))
|
||||
with self.assertRaisesRegex(ValueError, "source evidence is unavailable"):
|
||||
summary.write_release_bundle_descriptor(args, result)
|
||||
self.assertFalse(args.release_bundle_descriptor_out.exists())
|
||||
self.assertFalse((self.root / "release-descriptor-artifacts").exists())
|
||||
|
||||
def test_release_descriptor_requires_profile_artifacts(self):
|
||||
@mock.patch.object(summary, "measured_provenance_verified", return_value=True)
|
||||
def test_release_descriptor_stays_closed_without_profile_artifacts(self, _verified):
|
||||
args = type("Args", (), {
|
||||
"abba_dir": self.abba,
|
||||
"cache_cost_log": None,
|
||||
@@ -290,8 +271,31 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
|
||||
"release_source_revision": "b" * 40,
|
||||
"release_profile_artifact": [],
|
||||
})
|
||||
with self.assertRaisesRegex(ValueError, "missing profile artifacts"):
|
||||
with self.assertRaisesRegex(ValueError, "source evidence is unavailable"):
|
||||
summary.write_release_bundle_descriptor(args, summary.build_summary(args))
|
||||
self.assertFalse(args.release_bundle_descriptor_out.exists())
|
||||
|
||||
def test_self_reported_measured_pass_has_unverified_provenance(self):
|
||||
args = type("Args", (), {
|
||||
"abba_dir": self.abba,
|
||||
"cache_cost_log": None,
|
||||
"require_cache_cost": False,
|
||||
"release_bundle_descriptor_out": self.root / "release-descriptor.json",
|
||||
"release_source_revision": "b" * 40,
|
||||
"release_profile_artifact": [],
|
||||
})
|
||||
for claim in (None, "verified"):
|
||||
with self.subTest(claim=claim):
|
||||
if claim is not None:
|
||||
self.report["measurement_provenance"] = claim
|
||||
self.write_inputs()
|
||||
result = summary.build_summary(args)
|
||||
self.assertEqual(result["verdict"], "FAIL")
|
||||
self.assertEqual(result["abba"]["measurement_provenance"], "unverified")
|
||||
self.assertIn("not independently verified", result["reason"])
|
||||
self.assertTrue(all(row["status"] == "unverified" for row in result["abba"]["w11_running_heal_build"]))
|
||||
with self.assertRaisesRegex(ValueError, "requires a measured PASS"):
|
||||
summary.write_release_bundle_descriptor(args, result)
|
||||
|
||||
def test_synthetic_report_fails_as_performance_conclusion(self):
|
||||
self.report.update(status="synthetic_validated", performance="pending", evidence="synthetic")
|
||||
@@ -329,9 +333,122 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
|
||||
self.assertIsNone(result["abba"]["error_rate"])
|
||||
self.assertIsNone(result["abba"]["foreground_pressure_samples"])
|
||||
self.assertIsNone(result["abba"]["foreground_pressure_high_samples"])
|
||||
self.assertIn("collector failed", result["reason"])
|
||||
self.assertIn("error details omitted", result["reason"])
|
||||
self.assertIn("- completed_cells: 7", summary.markdown(result))
|
||||
self.assertIn("- error: collector failed", summary.markdown(result))
|
||||
self.assertIn("- error: details omitted", summary.markdown(result))
|
||||
|
||||
def test_historical_error_does_not_echo_raw_command(self):
|
||||
sentinel = "TEST_ONLY_PRIVATE_ARG"
|
||||
self.report = {"status": "failed", "performance": "pending", "evidence": "diagnostic",
|
||||
"completed_cells": 7, "error": f"Command ['client', '{sentinel}'] timed out"}
|
||||
self.write_inputs()
|
||||
args = type("Args", (), {"abba_dir": self.abba, "cache_cost_log": None,
|
||||
"require_cache_cost": False, "json_out": None, "markdown_out": None})
|
||||
result = summary.build_summary(args)
|
||||
self.assertEqual(result["verdict"], "FAIL")
|
||||
self.assertNotIn(sentinel, json.dumps(result))
|
||||
self.assertNotIn(sentinel, summary.markdown(result))
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with mock.patch.object(sys, "argv", ["summarize_scanner_heal_perf.py", "--abba-dir", str(self.abba)]), \
|
||||
contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
self.assertEqual(summary.main(), 1)
|
||||
self.assertNotIn(sentinel, out.getvalue() + err.getvalue())
|
||||
|
||||
def test_unknown_report_labels_are_not_echoed(self):
|
||||
sentinel = "TEST_ONLY_PRIVATE_LABEL"
|
||||
self.report = {"status": sentinel, "performance": sentinel, "evidence": sentinel, "comparisons": []}
|
||||
self.write_inputs()
|
||||
result = summary.summarize_abba(self.abba)
|
||||
self.assertEqual(result["gate_state"], "fail")
|
||||
self.assertEqual(result["status"], "unknown")
|
||||
self.assertEqual(result["performance"], "unknown")
|
||||
self.assertEqual(result["evidence"], "unknown")
|
||||
self.assertNotIn(sentinel, json.dumps(result))
|
||||
|
||||
def test_unverified_provenance_does_not_echo_endpoints_or_flags(self):
|
||||
sentinel = "TEST_ONLY_PRIVATE_ENDPOINT_OR_ARG"
|
||||
self.report = {"status": "failed", "performance": "pending", "evidence": "diagnostic", "comparisons": []}
|
||||
self.manifest["fixed"]["release_flags"] = f"--token={sentinel}"
|
||||
self.manifest["fixed"]["durability"] = sentinel
|
||||
self.manifest["baseline"]["revision"] = sentinel
|
||||
self.manifest["started_at"] = sentinel
|
||||
self.manifest["release_evidence"]["distributed"]["metrics_endpoints"] = [f"https://{sentinel}@node.example"]
|
||||
self.write_inputs()
|
||||
args = type("Args", (), {"abba_dir": self.abba, "cache_cost_log": None,
|
||||
"require_cache_cost": False, "json_out": None, "markdown_out": None})
|
||||
result = summary.build_summary(args)
|
||||
self.assertEqual(result["verdict"], "FAIL")
|
||||
self.assertNotIn(sentinel, json.dumps(result))
|
||||
self.assertNotIn(sentinel, summary.markdown(result))
|
||||
|
||||
def test_unverified_comparison_labels_do_not_echo_free_text(self):
|
||||
sentinel = "TEST_ONLY_PRIVATE_COMPARISON"
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
self.report["comparisons"][0]["scenario"] = sentinel
|
||||
self.report["comparisons"][0]["p1"]["baseline_walk_objects"] = sentinel
|
||||
row = next(row for row in self.report["comparisons"]
|
||||
if row["scenario"] == "running-heal" and row["comparison"] == "build")
|
||||
row["w10"]["status"] = sentinel
|
||||
row["w10"]["candidate_delay_events"] = sentinel
|
||||
row["w11"]["status"] = sentinel
|
||||
row["w11"]["rss_growth"] = sentinel
|
||||
self.write_inputs()
|
||||
result = summary.summarize_abba(self.abba)
|
||||
self.assertEqual(result["gate_state"], "fail")
|
||||
self.assertNotIn(sentinel, json.dumps(result))
|
||||
|
||||
def test_cache_cost_labels_do_not_echo_free_text(self):
|
||||
sentinel = "TEST_ONLY_PRIVATE_CACHE_LABEL"
|
||||
self.report = {"status": "failed", "performance": "pending", "evidence": "diagnostic", "comparisons": []}
|
||||
self.write_inputs()
|
||||
cache_log = self.root / "cache.log"
|
||||
cache_log.write_text("CACHE_COST " + json.dumps({"schema": 1, "scenario": sentinel,
|
||||
"cache_wire_bytes": 100, "save_body_bytes_per_sample": 200,
|
||||
"clone": {"max_ns": 10}, "encode": {"max_ns": 20}, "save_inclusive": {"max_ns": 30},
|
||||
"build": {"source_revision": sentinel, "source_tree": sentinel,
|
||||
"test_opt_level_override": sentinel}}) + "\n")
|
||||
args = type("Args", (), {"abba_dir": self.abba, "cache_cost_log": cache_log,
|
||||
"require_cache_cost": False, "json_out": None, "markdown_out": None})
|
||||
result = summary.build_summary(args)
|
||||
self.assertNotIn(sentinel, json.dumps(result))
|
||||
self.assertNotIn(sentinel, summary.markdown(result))
|
||||
|
||||
def test_cli_error_does_not_echo_input_path(self):
|
||||
sentinel = "TEST_ONLY_PRIVATE_PATH"
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with mock.patch.object(sys, "argv", ["summarize_scanner_heal_perf.py", "--abba-dir", str(self.root / sentinel)]), \
|
||||
contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
self.assertEqual(summary.main(), 1)
|
||||
self.assertNotIn(sentinel, out.getvalue() + err.getvalue())
|
||||
|
||||
def test_unknown_cell_counts_are_not_zero_or_free_text(self):
|
||||
sentinel = "TEST_ONLY_PRIVATE_COUNT"
|
||||
for cells, completed in ((None, None), (sentinel, sentinel), ("absent", "absent")):
|
||||
with self.subTest(cells=cells):
|
||||
self.report = {"status": "failed", "performance": "pending", "evidence": "diagnostic",
|
||||
"cells": cells, "completed_cells": completed, "comparisons": []}
|
||||
if cells == "absent":
|
||||
self.report.pop("cells")
|
||||
self.report.pop("completed_cells")
|
||||
self.write_inputs()
|
||||
result = summary.summarize_abba(self.abba)
|
||||
self.assertIsNone(result["cells"])
|
||||
self.assertIsNone(result["completed_cells"])
|
||||
self.assertNotIn(sentinel, json.dumps(result))
|
||||
|
||||
def test_rejected_descriptor_does_not_write_summary_outputs(self):
|
||||
json_out = self.root / "summary.json"
|
||||
markdown_out = self.root / "summary.md"
|
||||
descriptor = self.root / "descriptor.json"
|
||||
argv = ["summarize_scanner_heal_perf.py", "--abba-dir", str(self.abba),
|
||||
"--json-out", str(json_out), "--markdown-out", str(markdown_out),
|
||||
"--release-bundle-descriptor-out", str(descriptor)]
|
||||
with mock.patch.object(sys, "argv", argv), contextlib.redirect_stdout(io.StringIO()), \
|
||||
contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertEqual(summary.main(), 1)
|
||||
self.assertFalse(json_out.exists())
|
||||
self.assertFalse(markdown_out.exists())
|
||||
self.assertFalse(descriptor.exists())
|
||||
|
||||
def test_partial_error_rate_is_unknown(self):
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
@@ -344,6 +461,111 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
|
||||
self.write_inputs()
|
||||
self.assertIsNone(summary.summarize_abba(self.abba)["error_rate"])
|
||||
|
||||
def test_partial_w09_measurements_remain_unknown(self):
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
fields = {"heal_start_p95_ms": "w09_worst_heal_start_p95_ms",
|
||||
"heal_duplicate_task_count": "w09_duplicate_task_count",
|
||||
"heal_lock_hold_p95_ms": "w09_worst_lock_hold_p95_ms"}
|
||||
for field, output in fields.items():
|
||||
for missing in (None, "short", "absent"):
|
||||
with self.subTest(field=field, missing=missing):
|
||||
original = copy.deepcopy(self.report["comparisons"][0]["w09"])
|
||||
if missing is None:
|
||||
self.report["comparisons"][0]["w09"][field][1] = None
|
||||
elif missing == "short":
|
||||
self.report["comparisons"][0]["w09"][field] = [0]
|
||||
else:
|
||||
self.report["comparisons"][0]["w09"].pop(field)
|
||||
self.write_inputs()
|
||||
self.assertIsNone(summary.summarize_abba(self.abba)[output])
|
||||
self.report["comparisons"][0]["w09"] = original
|
||||
self.report["comparisons"].pop()
|
||||
self.write_inputs()
|
||||
incomplete = summary.summarize_abba(self.abba)
|
||||
for output in fields.values():
|
||||
self.assertIsNone(incomplete[output])
|
||||
self.assertIsNone(incomplete["worst_p99_regression"])
|
||||
self.assertIsNone(incomplete["worst_throughput_loss"])
|
||||
self.assertIsNone(incomplete["throughput_ops"])
|
||||
self.assertIsNone(incomplete["max_reported_comparison_throughput_ops"])
|
||||
|
||||
def test_partial_candidate_p2_measurements_remain_unknown(self):
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
for missing in ("b1", "b2", "short", "absent", "comparison"):
|
||||
with self.subTest(missing=missing):
|
||||
original = copy.deepcopy(self.report["comparisons"])
|
||||
row = self.report["comparisons"][0]
|
||||
if missing == "b1":
|
||||
row["p2_post_stop_work_multiples"][1] = None
|
||||
elif missing == "b2":
|
||||
row["p2_post_stop_work_multiples"][2] = None
|
||||
elif missing == "short":
|
||||
row["p2_post_stop_work_multiples"] = [None, 1.0]
|
||||
elif missing == "absent":
|
||||
row.pop("p2_post_stop_work_multiples")
|
||||
else:
|
||||
self.report["comparisons"].pop()
|
||||
self.write_inputs()
|
||||
self.assertIsNone(summary.summarize_abba(self.abba)["p2_worst_post_stop_work_multiple"])
|
||||
self.report["comparisons"] = original
|
||||
|
||||
def test_unverified_reported_percentiles_are_not_a_global_percentile(self):
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
self.write_inputs()
|
||||
result = summary.summarize_abba(self.abba)
|
||||
self.assertIsNone(result["foreground_p95_ms"])
|
||||
self.assertIsNone(result["foreground_p99_ms"])
|
||||
self.assertIsNone(result["throughput_ops"])
|
||||
self.assertEqual(result["max_reported_comparison_throughput_ops"], 100.0)
|
||||
|
||||
def test_passing_report_requires_each_leg_percentile(self):
|
||||
self.report["comparisons"][0]["foreground_percentiles_ms_by_leg"].pop("B2")
|
||||
self.write_inputs()
|
||||
with self.assertRaisesRegex(ValueError, "foreground percentiles"):
|
||||
summary.summarize_abba(self.abba)
|
||||
|
||||
def test_passing_report_rejects_invalid_leg_percentiles(self):
|
||||
for p95, p99 in ((-1, 10), (1, float("inf")), (float("nan"), 10), (11, 10)):
|
||||
with self.subTest(p95=p95, p99=p99):
|
||||
self.report["comparisons"][0]["foreground_percentiles_ms_by_leg"]["B2"] = {"p95": p95, "p99": p99}
|
||||
self.write_inputs()
|
||||
with self.assertRaisesRegex(ValueError, "foreground percentiles"):
|
||||
summary.summarize_abba(self.abba)
|
||||
|
||||
def test_nonfinite_report_metrics_are_rejected(self):
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
for value in (float("inf"), float("nan")):
|
||||
with self.subTest(value=value):
|
||||
self.report["comparisons"][0]["p99_regression"] = value
|
||||
self.write_inputs()
|
||||
with self.assertRaisesRegex(ValueError, "invalid numeric field"):
|
||||
summary.summarize_abba(self.abba)
|
||||
|
||||
def test_per_leg_attempt_ratios_are_not_attempt_samples(self):
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
self.write_inputs()
|
||||
result = summary.summarize_abba(self.abba)
|
||||
self.assertIsNone(result["attempt_cost_samples"])
|
||||
self.assertGreater(result["reported_attempt_cost_ratio_count"], 0)
|
||||
self.report["comparisons"][0]["w10_w11"].pop("attempt_cost_per_healed_object")
|
||||
self.write_inputs()
|
||||
self.assertIsNone(summary.summarize_abba(self.abba)["reported_attempt_cost_ratio_count"])
|
||||
|
||||
def test_error_rate_cannot_be_negative_or_exceed_one(self):
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
for value in (-0.1, 1.1):
|
||||
with self.subTest(value=value):
|
||||
self.report["comparisons"][0]["error_rate"] = value
|
||||
self.write_inputs()
|
||||
with self.assertRaisesRegex(ValueError, "error_rate"):
|
||||
summary.summarize_abba(self.abba)
|
||||
|
||||
def test_passing_report_cannot_contain_request_errors(self):
|
||||
self.report["comparisons"][0]["error_rate"] = 0.01
|
||||
self.write_inputs()
|
||||
with self.assertRaisesRegex(ValueError, "error_rate"):
|
||||
summary.summarize_abba(self.abba)
|
||||
|
||||
def test_partial_pressure_samples_are_unknown(self):
|
||||
self.report.update(status="inconclusive", performance="pending")
|
||||
self.report["comparisons"][0]["w10_w11"].pop("foreground_pressure_samples")
|
||||
@@ -563,7 +785,8 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
self.assertEqual(process.returncode, 0, process.stderr)
|
||||
self.assertEqual(process.returncode, 1, process.stderr)
|
||||
self.assertTrue(process.stdout.startswith("FAIL scanner_heal_perf "))
|
||||
self.assertEqual(process.stdout.count("\n"), 1)
|
||||
self.assertEqual(process.stderr, "")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user