Compare commits

...

3 Commits

Author SHA1 Message Date
overtrue 8a48a52632 test: release scanner fixtures without signaling stale pids 2026-10-29 05:23:56 +08:00
overtrue 6feac7b7f7 fix: keep unverified G14 descriptors unpublished 2026-10-29 02:59:21 +08:00
overtrue c7fb6d9901 fix: reject unverified scanner performance evidence 2026-10-29 02:37:58 +08:00
17 changed files with 2135 additions and 854 deletions
+28 -7
View File
@@ -66,7 +66,7 @@ The manifest has the following JSON contract (all fields are required):
| `rounds`, `duration_seconds`, `min_free_bytes` | 3..10 groups, 7200..86400 seconds for measured release runs, and the independently estimated free-space reservation in bytes. Synthetic runs may use 1 second. |
| `baseline`, `candidate` | Each contains executable `binary`, full 40-character `revision`, and verified `sha256`. The runner rehashes binaries before every leg. |
| `fixed` | `config_sha256`, `dataset_sha256`, `release_flags`, `durability`, `disk_type`, `cache_state`, `load_command`, `resource_isolation`, `topology` (`EC8+4`), and positive `offered_load_ops`. Hashes use 64 lowercase hexadecimal characters. |
| `release_evidence` | Required for `measured` runs. It binds the 3x4 EC8+4 topology, multi-pool/multi-set coverage, per-node metrics endpoints, same-window distributed sampling, process restart and crash-restart fault modes, mixed-version reader/writer/rollback participation, and allocation/flamegraph/RSS/save-frequency profile artifact requirements. Synthetic runs do not need this field and still cannot approve release evidence. |
| `release_evidence` | Required for `measured` runs. It declares the 3x4 EC8+4 topology, multi-pool/multi-set coverage, per-node metrics endpoints, same-window distributed sampling, process restart and crash-restart fault modes, mixed-version reader/writer/rollback participation, and allocation/flamegraph/RSS/save-frequency profile artifact requirements. Synthetic runs do not require it and cannot approve release evidence. |
| `oracles` | A map with all five scenario names. Each value contains positive integer `objects`, `versions`, `bytes`, and `sha256` of the independently prepared canonical object/version/content manifest. |
| `expected_healed_objects` | A map with all five scenario names and independently seeded repair counts. Running-heal and MRF-replay require a positive count. |
@@ -79,11 +79,17 @@ the collector reads `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` from its environ
The adapter must echo the measured run's `release_evidence` object in every
measurement response. A mismatch fails the cell because it means the deployment,
mixed-version set, crash mode, or profiler contract no longer matches the
operator-reviewed manifest. This echo is provenance binding only; it does not
replace the independent correctness oracle, distributed metrics samples, profile
artifacts, or ABBA comparison thresholds. The summary tool revalidates the same
manifest contract before it can print a measured PASS result, so hand-built or
trimmed reports without this provenance fail closed.
operator-reviewed manifest. This echo checks consistency with the declared plan;
it does not prove that observations share a source window or that the reported
client percentiles are exact. Until those original observations are independently
verified, measured results are marked unverified and cannot produce a PASS or a
release descriptor. The summary command's legacy descriptor option rejects the
request until a producer can verify each field against its original window.
The existing live collector persists the declared load command, scanner/heal
configuration snapshots, and adapter/collector output. It is not a verified
sanitized capture path; use it only after confirming those inputs and outputs do
not contain credentials, private configuration files, or sensitive commands.
#### Deployment Adapter Contract
@@ -218,7 +224,7 @@ retain the failure and reconcile the service before continuing.
The `measure` response echoes the observed `evidence`, `fixed`, `build`,
`data_dir`, and `background`, plus `sample_count` (1..3600), `elapsed_seconds`,
and `metrics`. All metrics must be finite nonnegative numbers: `p99_ms`,
and `metrics`. All metrics must be finite nonnegative numbers: `p95_ms`, `p99_ms`,
`throughput_ops`, `rss_bytes`, `cpu_seconds`, `iops`, `rpc_count`,
`cache_clone_bytes`, `encode_bytes`, `save_bytes`, `oldest_age_seconds`,
`walk_objects`, `cold_walk_objects`, `healed_objects`, `errors`, and `requests`.
@@ -235,6 +241,21 @@ in the cell artifact directory; aggregate values alone do not establish their
measurement provenance. Missing production instrumentation is a pending gate,
not permission to report a fabricated zero.
For measured cells, retain the original schedule, start, and completion ledgers
and their references in `client_request_samples`. The runner reconciles request
identities, counts, timestamps, errors, and file hashes before calculating
nearest-rank percentiles from the recorded worker-start-to-completion samples.
The current adapter timestamps worker start before its ledger flush, so those
samples also include local logging and do not establish SDK-call latency.
Missing HTTP status does not prove zero 503s. The current ledger format does not
prove that the offered load reached the service or that the client and node
windows coincide, so a measured release result remains unavailable until those
boundaries are verified.
The four ABBA legs cover different windows: their per-leg percentiles are
reported separately, not renamed as a combined p95 or p99. Ledger reduction
uses a size-limited scratch database and samples free space; those checks do not
reserve disk space or prevent concurrent writers from crossing the floor.
For usage-cache saves, `encode_bytes` uses the window delta of
`rustfs_scanner_cache_serialized_bytes_total`. `save_bytes` uses the sum of
`rustfs_scanner_cache_save_payload_bytes_total` deltas for `cache="main"` and
+5
View File
@@ -89,7 +89,12 @@ their issue closes.
| `test_scanner_heal_w16_recovery_evidence.sh` | dev-tool | Shell self-test for the Scanner/Heal W16 recovery evidence runner | — |
| `test_scanner_heal_w13_mrf_evidence.sh` | dev-tool | Shell self-test for the Scanner/Heal W13 MRF evidence runner | — |
| `scanner_abba.py` | dev-tool | Scanner/heal ABBA orchestration and evidence gates via `run_scanner_validation_harness.sh --abba` | `docs/operations/scanner-benchmark-runbook.md` |
| `scanner_client_trace.py` | dev-tool | Reconcile client request ledgers and calculate per-window exact sample percentiles | `scanner_abba.py` |
| `test_scanner_abba.py` | dev-tool | Synthetic ABBA adapter and failure-path tests | `test_scanner_validation_harness.sh` |
| `test_scanner_client_trace.py` | dev-tool | Client ledger reconciliation and fail-closed tests | `test_scanner_validation_harness.sh` |
| `test_summarize_scanner_heal_perf.py` | dev-tool | Scanner/heal summary and provenance tests | `test_scanner_validation_harness.sh` |
| `test_scanner_heal_scheduler_pressure_evidence.py` | dev-tool | Scheduler-pressure evidence fail-closed tests | `test_scanner_heal_scheduler_pressure_evidence.sh` |
| `test_scanner_heal_linux_evidence_plan.py` | dev-tool | Linux evidence plan coverage tests | `test_scanner_heal_linux_evidence_plan.sh` |
| `test_build_rustfs_options.sh` | dev-tool | Shell test for rustfs build-option wiring | `make test` (script-tests) |
| `test_entrypoint_credentials.sh` | dev-tool | Container entrypoint credential-handling test | `make test` (script-tests) |
| `test_helm_chart_version.sh` | dev-tool | Test for `helm_chart_version.sh` | — |
+58
View File
@@ -29,6 +29,7 @@ from scanner_abba import (
RELEASE_PROFILE_ARTIFACTS,
SCENARIOS,
digest,
measured_provenance_verified,
number,
read_json,
require,
@@ -2737,6 +2738,16 @@ def validate_release_bundle_domain_evidence(gate: str, field: str, evidence: dic
f"{gate}.{field}.recovered_pending_migration")
def segment_oracle_provenance_verified(_bundle_dir: Path, _field: str) -> bool:
# Original post-stop and segment/oracle evidence is not verified yet.
return False
def distributed_window_provenance_verified(_bundle_dir: Path, _field: str) -> bool:
# Original multi-node generation and invalidation evidence is not verified yet.
return False
def validate_release_bundle_artifact(bundle_path: Path, source_revision: str, gate: str, field: str,
evidence: dict[str, object]) -> str:
require(evidence.get("evidence_type") == "measured", f"{gate}.{field} must be measured evidence")
@@ -2976,6 +2987,18 @@ def validate_release_bundle_artifact(bundle_path: Path, source_revision: str, ga
evidence_integer(item.get("resolved_samples"), f"{gate}.{artifact_field}.resolved_samples",
0, 2**63 - 1)
validate_release_bundle_domain_evidence(gate, field, evidence)
if (gate, field) in (("G10", "pressure_recovery_evidence"),
("P1", "cold_walk_share_measurement"),
("P1", "foreground_latency_throughput_measurement"),
("P3", "two_hour_pressure_measurement")):
require(measured_provenance_verified(bundle_path.parent),
f"{gate}.{field} source windows and client percentiles are not independently verified")
if gate == "P2" and field in ("post_stop_convergence_measurement", "cold_segment_reuse_measurement"):
require(segment_oracle_provenance_verified(bundle_path.parent, field),
f"{gate}.{field} post-stop and segment/oracle provenance is not independently verified")
if gate == "G14" and field in ("same_window_field_evidence", "distributed_segment_invalidation_evidence"):
require(distributed_window_provenance_verified(bundle_path.parent, field),
f"{gate}.{field} multi-node generation and window provenance is not independently verified")
return window_id
@@ -3516,6 +3539,19 @@ def validate(root: Path) -> list[str]:
class SelfTests(unittest.TestCase):
def setUp(self) -> None:
# Structural fixtures exercise the other validators with a simulated
# source verifier. The real verifier is checked separately below.
self.provenance_verifier = mock.patch.object(sys.modules[__name__], "measured_provenance_verified", return_value=True)
self.provenance_verifier.start()
self.addCleanup(self.provenance_verifier.stop)
self.segment_verifier = mock.patch.object(sys.modules[__name__], "segment_oracle_provenance_verified", return_value=True)
self.segment_verifier.start()
self.addCleanup(self.segment_verifier.stop)
self.distributed_verifier = mock.patch.object(sys.modules[__name__], "distributed_window_provenance_verified", return_value=True)
self.distributed_verifier.start()
self.addCleanup(self.distributed_verifier.stop)
def test_quick_checks_rejects_caller_and_execution_bypasses(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
@@ -4189,6 +4225,28 @@ class SelfTests(unittest.TestCase):
write_json(bundle, {"schema": 1, "evidence": "measured", "source_revision": source_revision, "gates": gates})
return root, bundle
def test_scanner_heal_release_bundle_rejects_unverified_measurements(self) -> None:
self.provenance_verifier.stop()
self.segment_verifier.stop()
self.distributed_verifier.stop()
with tempfile.TemporaryDirectory() as tmp:
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
payload = read_json(bundle)
payload["measurement_provenance"] = "verified"
write_json(bundle, payload)
with mock.patch("subprocess.check_output", return_value="b" * 40):
status = scanner_heal_release_bundle_status(root, bundle)
unverified = [scanner_heal_release_bundle_gate_status(root, bundle, gate)
for gate in ("G10", "G14", "P1", "P2", "P3")]
g09 = scanner_heal_release_bundle_gate_status(root, bundle, "G09")
self.assertFalse(status["release_approved"])
for gate in ("G10", "G14", "P1", "P2", "P3"):
self.assertIn(gate, status["rejected_gates"])
for gate in unverified:
self.assertEqual(gate["decision"], "blocked")
self.assertTrue(any("not independently verified" in error for error in gate["rejected_errors"]))
self.assertEqual(g09["verified_gate"], "G09")
def test_scanner_heal_release_bundle_accepts_complete_measured_evidence(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
root, bundle = self.scanner_heal_release_bundle_fixture(Path(tmp))
+118 -29
View File
@@ -11,6 +11,7 @@ from __future__ import annotations
import argparse
from datetime import datetime, timezone
import os
import re
import shutil
import subprocess
@@ -18,7 +19,7 @@ import sys
from pathlib import Path
from typing import Any
from scanner_abba import digest, read_json, require, write_json
from scanner_abba import digest, read_json, require, safe_error_message, write_json
ROOT = Path(__file__).resolve().parents[1]
G14_FIELDS = (
@@ -34,10 +35,6 @@ def git_head() -> str:
return subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
def iso_from_epoch(value: Any, name: str) -> str:
require(type(value) in (int, float) and value > 0, f"invalid {name}")
return datetime.fromtimestamp(float(value), timezone.utc).isoformat().replace("+00:00", "Z")
@@ -272,8 +269,10 @@ def proof_from_case_directories(raw_values: list[str], out_dir: Path, source_rev
def write_field(out_dir: Path, field: str, proof: dict[str, Any], source_revision: str) -> dict[str, Any]:
started_at = proof.get("started_at") or utc_now()
finished_at = proof.get("finished_at") or started_at
started_at = proof.get("started_at")
finished_at = proof.get("finished_at")
require(isinstance(started_at, str) and started_at, "missing G14 original window start")
require(isinstance(finished_at, str) and finished_at, "missing G14 original window end")
evidence: dict[str, Any] = {
"artifact": "",
"evidence_type": "measured",
@@ -325,11 +324,18 @@ def write_field(out_dir: Path, field: str, proof: dict[str, Any], source_revisio
return evidence
def g14_inputs_verified(_args: argparse.Namespace, _source_revision: str) -> bool:
# The original multi-node observation and invalidation window are not yet
# independently verified. A self-declared measured proof cannot open G14.
return False
def build_descriptor(args: argparse.Namespace) -> Path:
out_dir = args.out_dir.resolve()
require(not out_dir.exists(), "output directory must be new")
source_revision = args.source_revision or git_head()
out_dir.mkdir(parents=True)
require(g14_inputs_verified(args, source_revision), "G14 source windows are not independently verified")
out_dir.mkdir(parents=True, mode=0o700)
if args.proof_json is not None:
proof_path = args.proof_json.resolve()
proof = load_proof(proof_path, source_revision)
@@ -344,26 +350,31 @@ def build_descriptor(args: argparse.Namespace) -> Path:
proof = proof_from_case_directories(args.case_dir, out_dir, source_revision)
fields = {field: write_field(out_dir, field, proof, source_revision) for field in G14_FIELDS}
descriptor = out_dir / "release-bundle-g14.json"
write_json(descriptor, {
"schema": 1,
"evidence": "measured",
"source_revision": source_revision,
"gates": {
"G14": {
"status": "pass",
"lane": "ec8-4-multiset",
"evidence_type": "measured",
"evidence_fields": fields,
pending = out_dir / ".release-bundle-g14.pending.json"
try:
write_json(pending, {
"schema": 1,
"evidence": "measured",
"source_revision": source_revision,
"gates": {
"G14": {
"status": "pass",
"lane": "ec8-4-multiset",
"evidence_type": "measured",
"evidence_fields": fields,
},
},
},
})
subprocess.check_call([
sys.executable,
str(ROOT / "scripts/check_test_wiring.py"),
"--check-scanner-heal-release-bundle-gate",
str(descriptor),
"G14",
], cwd=ROOT)
})
subprocess.check_call([
sys.executable,
str(ROOT / "scripts/check_test_wiring.py"),
"--check-scanner-heal-release-bundle-gate",
str(pending),
"G14",
], cwd=ROOT)
os.link(pending, descriptor)
finally:
pending.unlink(missing_ok=True)
return descriptor
@@ -452,7 +463,7 @@ def write_self_test_case_dir(root: Path, source_revision: str, case: str, sets:
return directory
def run_self_test() -> None:
def _run_packaging_self_test() -> None:
import tempfile
with tempfile.TemporaryDirectory() as tmp:
@@ -464,6 +475,12 @@ def run_self_test() -> None:
"--out-dir", str(root / "out"),
]))
require(descriptor.is_file(), "self-test descriptor missing")
checked = subprocess.run([
sys.executable, str(ROOT / "scripts/check_test_wiring.py"),
"--check-scanner-heal-release-bundle-gate", str(descriptor), "G14",
], cwd=ROOT, capture_output=True, text=True, check=False)
require(checked.returncode == 1 and "not independently verified" in checked.stdout,
"self-test accepted unverified G14 evidence")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
@@ -534,6 +551,78 @@ def run_self_test() -> None:
raise ValueError("self-test accepted fixture-only G14 case oracle")
def run_self_test() -> None:
import contextlib
import io
import tempfile
from unittest import mock
import check_test_wiring as wiring
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
for proof in ({}, {"started_at": "2026-09-09T00:00:00Z"}):
try:
write_field(root, G14_FIELDS[0], proof, "a" * 40)
except ValueError as err:
require("missing G14 original window" in str(err), "wrong missing-window failure")
else:
raise ValueError("self-test synthesized a G14 window")
require(not (root / "artifacts").exists(), "missing G14 window left artifacts")
with tempfile.TemporaryDirectory() as tmp:
stderr = io.StringIO()
argv = ["g14", "--proof-json", "unused", "--out-dir", str(Path(tmp) / "out")]
error = subprocess.CalledProcessError(1, ["sensitive-command-sentinel"])
with mock.patch.object(sys, "argv", argv), mock.patch.object(sys.modules[__name__], "git_head", side_effect=error), \
contextlib.redirect_stderr(stderr):
require(main() == 1, "self-test accepted a failed source revision")
require("sensitive-command-sentinel" not in stderr.getvalue(), "G14 error exposed command arguments")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
source_revision = git_head()
proof = write_self_test_proof(root, source_revision)
out_dir = root / "out"
try:
build_descriptor(parse_args(["--proof-json", str(proof), "--out-dir", str(out_dir)]))
except ValueError as err:
require("not independently verified" in str(err), "wrong unverified G14 failure")
else:
raise ValueError("self-test accepted unverified G14 inputs")
require(not out_dir.exists(), "unverified G14 inputs left an output directory")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
source_revision = git_head()
proof = write_self_test_proof(root, source_revision)
out_dir = root / "out"
with mock.patch.object(sys.modules[__name__], "g14_inputs_verified", return_value=True), \
mock.patch.object(subprocess, "check_call", side_effect=subprocess.CalledProcessError(1, "checker")):
try:
build_descriptor(parse_args(["--proof-json", str(proof), "--out-dir", str(out_dir)]))
except subprocess.CalledProcessError:
pass
else:
raise ValueError("self-test accepted a rejected G14 descriptor")
require(not (out_dir / "release-bundle-g14.json").exists()
and not (out_dir / ".release-bundle-g14.pending.json").exists(),
"rejected G14 validation left a descriptor")
def verify_fixture(command: list[str], cwd: Path) -> None:
require(cwd == ROOT and command[2] == "--check-scanner-heal-release-bundle-gate"
and command[4] == "G14", "unexpected self-test checker invocation")
with mock.patch.object(wiring, "distributed_window_provenance_verified", return_value=True):
status = wiring.scanner_heal_release_bundle_gate_status(ROOT, Path(command[3]), "G14")
require(status["decision"] == "verified", "fixture gate validation failed")
# Only synthetic packaging tests simulate the future verifier. Real CLI
# invocations and the separate checker retain their fail-closed behavior.
with mock.patch.object(sys.modules[__name__], "g14_inputs_verified", return_value=True), \
mock.patch.object(subprocess, "check_call", side_effect=verify_fixture):
_run_packaging_self_test()
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--proof-json", type=Path)
@@ -561,7 +650,7 @@ def main() -> int:
print(f"G14 release descriptor verified: {descriptor}")
return 0
except (ValueError, KeyError, OSError, subprocess.SubprocessError) as err:
print(f"ERROR: {err}", file=sys.stderr)
print(f"ERROR: {safe_error_message(err)}", file=sys.stderr)
return 1
@@ -365,7 +365,7 @@ def performance_steps() -> list[dict[str, Any]]:
{
"id": "scanner-heal-abba",
"description": "Run the isolated EC8+4 Scanner/Heal ABBA matrix through the deployment adapter.",
"covers": {"gates": ["G10", "P1", "P2", "P3"], "issues": ["2266", "2273", "2274", "2275"]},
"covers": {"gates": ["G10", "P1", "P3"], "issues": ["2266", "2273", "2274", "2275"]},
"commands": [
existing_script_command(
"scripts/python_bin.sh",
@@ -385,7 +385,7 @@ def performance_steps() -> list[dict[str, Any]]:
{
"id": "scheduler-pressure",
"description": "Assemble scheduler-pressure, profile, RSS, throughput, and latency descriptors.",
"covers": {"gates": ["G10", "P1", "P2", "P3"], "issues": ["2266", "2274", "2275"]},
"covers": {"gates": ["G10", "P1", "P3"], "issues": ["2266", "2274", "2275"]},
"commands": [
existing_script_command(
"scripts/python_bin.sh",
@@ -402,12 +402,28 @@ def performance_steps() -> list[dict[str, Any]]:
"rss-samples=$RUN_ROOT/profile/rss-samples.json",
"--profile-artifact",
"save-frequency=$RUN_ROOT/profile/save-frequency.json",
"--started-at",
"$ORIGINAL_WINDOW_STARTED_AT",
"--finished-at",
"$ORIGINAL_WINDOW_FINISHED_AT",
"--run-id",
"$ORIGINAL_RUN_ID",
"--measurement-window-id",
"$ORIGINAL_WINDOW_ID",
"--out-dir",
"$RUN_ROOT/descriptors/scheduler-pressure",
)
],
"expected_outputs": expected_outputs("$RUN_ROOT/descriptors/scheduler-pressure/release-bundle-scheduler-pressure.json"),
},
{
"id": "p2-independent-evidence",
"description": "P2 needs original post-stop and segment observations with independent full-walk and root oracles.",
"covers": {"gates": ["P2"], "issues": ["2274"]},
"commands": [],
"expected_outputs": [],
"pending_reason": "No producer verifies the required independent P2 evidence yet.",
},
]
@@ -561,7 +577,10 @@ def build_status(plan: dict[str, Any], run_root: Path) -> dict[str, Any]:
for key in counts:
totals[key] += counts[key]
totals["expected"] += len(outputs)
if not outputs:
pending_reason = step.get("pending_reason")
if isinstance(pending_reason, str) and pending_reason:
status = "pending"
elif not outputs:
status = "not_tracked"
elif counts["invalid"] or counts["empty"]:
status = "invalid"
@@ -578,12 +597,17 @@ def build_status(plan: dict[str, Any], run_root: Path) -> dict[str, Any]:
"status": status,
"commands": step["commands"],
}
step_statuses.append({
if pending_reason:
next_step["reason"] = pending_reason
step_status = {
"id": step["id"],
"status": status,
"covers": step.get("covers", {}),
"outputs": outputs,
})
}
if pending_reason:
step_status["reason"] = pending_reason
step_statuses.append(step_status)
tracked = [step for step in step_statuses if step["status"] != "not_tracked"]
if not tracked:
stage_state = "not_tracked"
@@ -598,7 +622,7 @@ def build_status(plan: dict[str, Any], run_root: Path) -> dict[str, Any]:
stage_statuses.append({"id": stage["id"], "status": stage_state, "steps": step_statuses})
if totals["invalid"] or totals["empty"]:
decision = "invalid"
elif totals["missing"]:
elif totals["missing"] or pending_gates:
decision = "blocked"
else:
decision = "complete"
@@ -753,7 +777,8 @@ def self_test() -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("{}\n")
complete = build_status(plan, complete_root)
assert complete["decision"] == "complete"
assert complete["decision"] == "blocked"
assert "P2" in complete["pending_gates"]
assert complete["release_approved"] is False
print("PASS: scanner/heal Linux evidence plan self-test")
@@ -9,21 +9,23 @@ it never turns synthetic harness output into release evidence.
from __future__ import annotations
import argparse
from datetime import datetime, timedelta, timezone
from dataclasses import dataclass
from datetime import datetime
import json
import math
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any
from scanner_abba import (
P2_WORK_MULTIPLE_LIMIT,
RELEASE_PROFILE_ARTIFACTS,
RELEASE_SCHEDULER_BOUNDS,
SCENARIOS,
digest,
measured_provenance_verified,
read_json,
require,
sha,
@@ -33,29 +35,27 @@ from scanner_abba import (
ROOT = Path(__file__).resolve().parents[1]
G10_FIELDS = ("scheduler_bound_evidence", "pressure_recovery_evidence")
P1_FIELDS = ("cold_walk_share_measurement", "foreground_latency_throughput_measurement", "profile_evidence")
P2_FIELDS = ("post_stop_convergence_measurement", "cold_segment_reuse_measurement")
P3_FIELDS = ("two_hour_pressure_measurement", "heal_capacity_measurement", "recovery_window_measurement")
PRESSURE_METRICS = (
"foreground_p95_ms",
"foreground_p99_ms",
"throughput_ops",
"error_rate",
"heal_lock_wait_p99_ms",
"attempt_cost_samples",
)
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
def git_head() -> str:
return subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
def finite_number(value: Any, name: str, minimum: float = 0.0) -> float:
require(type(value) in (int, float) and math.isfinite(value) and value >= minimum, f"invalid {name}")
return float(value)
@dataclass(frozen=True)
class VerifiedPressureObservations:
source_revision: str
window: tuple[str, str, str, str]
fields: dict[str, dict[str, dict[str, Any]]]
profile_costs: dict[str, int]
profile_identities: dict[str, tuple[str, int]]
def pressure_inputs_verified(_abba_dir: Path, _recovery_path: Path, _profiles: dict[str, Path],
_window: tuple[str, str, str, str]) -> VerifiedPressureObservations | None:
# The verifier must derive each field from original, independently bound
# observations. A self-reported flag or a count from a different window is
# not a verified observation.
return None
def positive_int(value: Any, name: str, minimum: int = 1) -> int:
@@ -63,12 +63,6 @@ def positive_int(value: Any, name: str, minimum: int = 1) -> int:
return value
def positive_int_from_sources(cli_value: int | None, manifest_values: dict[str, Any], key: str) -> int:
if cli_value is not None:
return positive_int(cli_value, key)
return positive_int(manifest_values.get(key), key)
def parse_artifact_arg(value: str) -> tuple[str, Path, str]:
try:
kind, raw_path = value.split("=", 1)
@@ -90,6 +84,7 @@ def load_measured_abba(abba_dir: Path, source_revision: str) -> tuple[dict[str,
require(manifest.get("evidence") == "measured", "ABBA manifest must be measured")
require(report.get("evidence") == "measured", "ABBA report must be measured")
require(report.get("status") == "pass" and report.get("performance") == "pass", "ABBA report must pass")
require(measured_provenance_verified(abba_dir), "ABBA source windows and client percentiles are not independently verified")
require(manifest.get("candidate", {}).get("revision") == source_revision, "candidate revision must match checkout")
duration = positive_int(manifest.get("duration_seconds"), "duration_seconds", 7200)
require(duration >= 7200, "P3 two-hour pressure evidence requires at least 7200 seconds")
@@ -121,38 +116,6 @@ def load_measured_abba(abba_dir: Path, source_revision: str) -> tuple[dict[str,
return manifest, report, measures
def comparison_rows(report: dict[str, Any], scenario: str, comparison: str) -> list[dict[str, Any]]:
rows = [
item for item in report.get("comparisons", [])
if item.get("scenario") == scenario and item.get("comparison") == comparison
]
require(rows, f"missing {scenario}/{comparison} comparisons")
return rows
def worst_metric(measures: list[dict[str, Any]], key: str) -> float:
return max(finite_number(item["metrics"].get(key), key) for item in measures)
def sum_metric(measures: list[dict[str, Any]], key: str) -> float:
return sum(finite_number(item["metrics"].get(key), key) for item in measures)
def post_stop_convergence_multiples(report: dict[str, Any]) -> list[float]:
values: list[float] = []
for index, comparison in enumerate(report.get("comparisons", [])):
raw = comparison.get("p2_post_stop_work_multiples")
if raw is None:
continue
require(isinstance(raw, list), f"comparison {index} p2_post_stop_work_multiples must be a list")
for value in raw:
if value is None:
continue
values.append(finite_number(value, "p2 post-stop work multiple", 0.0))
require(values, "P2 requires measured post-stop convergence rows")
return values
PROFILE_ARTIFACT_REQUIRED_METRICS = {
"allocation-profile": ("resolved_samples", "allocation_bytes"),
"flamegraph": ("resolved_samples",),
@@ -162,7 +125,8 @@ PROFILE_ARTIFACT_REQUIRED_METRICS = {
def copy_profile_artifacts(out_dir: Path, artifacts: dict[str, tuple[Path, str]], source_revision: str,
run_id: str, window_id: str, profile_costs: dict[str, int]) -> dict[str, Any]:
run_id: str, window_id: str, profile_costs: dict[str, int],
profile_identities: dict[str, tuple[str, int]]) -> dict[str, Any]:
copied: dict[str, Any] = {}
profile_dir = out_dir / "artifacts" / "profiles"
raw_dir = profile_dir / "raw"
@@ -172,6 +136,9 @@ def copy_profile_artifacts(out_dir: Path, artifacts: dict[str, tuple[Path, str]]
source, artifact_format = artifacts[kind]
raw_target = raw_dir / f"{kind}{source.suffix or '.artifact'}"
shutil.copyfile(source, raw_target)
expected_hash, expected_size = profile_identities[kind]
require(raw_target.stat().st_size == expected_size and digest(raw_target) == expected_hash,
f"profile artifact differs from verified observation: {kind}")
target = profile_dir / f"P1-profile_evidence-{kind}.json"
payload = {
"schema": 1,
@@ -239,12 +206,8 @@ def build_descriptor(args: argparse.Namespace) -> Path:
out_dir = args.out_dir.resolve()
require(not out_dir.exists(), "output directory must be new")
source_revision = args.source_revision or git_head()
manifest, report, measures = load_measured_abba(args.abba_dir.resolve(), source_revision)
recovery = recovery_evidence(args.recovery_window_json.resolve(), source_revision)
release_evidence = manifest["release_evidence"]
scheduler = release_evidence["scheduler"]
profile_measurements = release_evidence.get("profile", {}).get("measurements", {})
require(isinstance(profile_measurements, dict), "ABBA manifest profile measurements must be an object")
manifest, _, _ = load_measured_abba(args.abba_dir.resolve(), source_revision)
recovery_evidence(args.recovery_window_json.resolve(), source_revision)
profile_inputs = {}
profile_formats = {}
@@ -256,237 +219,99 @@ def build_descriptor(args: argparse.Namespace) -> Path:
missing_profiles = sorted(set(RELEASE_PROFILE_ARTIFACTS) - set(profile_inputs))
require(not missing_profiles, "missing profile artifacts: " + ", ".join(missing_profiles))
out_dir.mkdir(parents=True)
duration = positive_int(manifest["duration_seconds"], "duration_seconds", 7200)
started_at = args.started_at or utc_now()
if args.finished_at:
finished_at = args.finished_at
else:
started_at, finished_at = args.started_at, args.finished_at
run_id, window_id = args.run_id, args.measurement_window_id
require(all(isinstance(value, str) and value.strip() for value in (started_at, finished_at, run_id, window_id)),
"original measurement window timestamps and identifiers are required")
try:
started = datetime.fromisoformat(started_at.replace("Z", "+00:00"))
finished_at = (started + timedelta(seconds=duration)).isoformat().replace("+00:00", "Z")
run_id = args.run_id or f"scheduler-pressure-{source_revision[:12]}"
window_id = args.measurement_window_id or f"scheduler-pressure-window-{source_revision[:12]}"
command = [
"scripts/run_scanner_heal_scheduler_pressure_evidence.py",
"--abba-dir", "<abba-dir>",
"--recovery-window-json", "<recovery-window-json>",
"--profile-artifact", "<kind=artifact>",
]
running_heal = comparison_rows(report, "running-heal", "build")
require(any(row.get("w10", {}).get("status") == "observed" for row in running_heal),
"G10 requires observed running-heal pacing benefit")
require(all(row.get("w11", {}).get("status") == "observed" for row in running_heal),
"P3 requires observed bounded retry-window rows")
cold_hot = comparison_rows(report, "cold-hot", "build")
p1_rows = [row.get("p1") for row in cold_hot]
require(all(isinstance(row, dict) and row.get("observed_reduction", -1) >= row.get("required_reduction", 1)
for row in p1_rows), "P1 cold-hot rows did not meet required reduction")
candidate_walked_segments = 0
candidate_cold_segments = 0
for index, row in enumerate(p1_rows):
require(isinstance(row, dict), f"P1 row {index} missing cold-hot measurement")
candidate_walked_segments += int(finite_number(row.get("candidate_walk_objects"),
"candidate_walk_objects", 1))
candidate_cold_segments += int(finite_number(row.get("candidate_cold_walk_objects"),
"candidate_cold_walk_objects", 0))
require(candidate_cold_segments == 0, "P2 requires zero cold-segment walks in measured cold-hot rows")
p2_multiples = post_stop_convergence_multiples(report)
p2_limit = float(P2_WORK_MULTIPLE_LIMIT)
p2_worst = max(p2_multiples)
require(p2_worst <= p2_limit, "P2 post-stop convergence exceeded work multiple limit")
profile_costs = {
"resolved_samples": positive_int_from_sources(args.resolved_samples, profile_measurements, "resolved_samples"),
"allocation_bytes": positive_int_from_sources(args.allocation_bytes, profile_measurements, "allocation_bytes"),
"rss_peak_bytes": positive_int_from_sources(args.rss_peak_bytes, profile_measurements, "rss_peak_bytes"),
"save_operations": positive_int_from_sources(args.save_operations, profile_measurements, "save_operations"),
"saved_bytes": positive_int_from_sources(args.saved_bytes, profile_measurements, "saved_bytes"),
}
finished = datetime.fromisoformat(finished_at.replace("Z", "+00:00"))
except ValueError:
raise ValueError("invalid original measurement window") from None
require(started.tzinfo is not None and finished.tzinfo is not None and (finished - started).total_seconds() >= duration,
"invalid original measurement window")
window = (run_id, window_id, started_at, finished_at)
observations = pressure_inputs_verified(args.abba_dir.resolve(), args.recovery_window_json.resolve(), profile_inputs, window)
require(isinstance(observations, VerifiedPressureObservations) and observations.window == window
and observations.source_revision == source_revision,
"independent verified pressure observations are unavailable")
expected_fields = {"G10": G10_FIELDS, "P1": P1_FIELDS, "P3": P3_FIELDS}
require(type(observations.fields) is dict and set(observations.fields) == set(expected_fields),
"verified pressure observations do not cover required gates")
reserved = {"artifact", "sha256", "artifact_format", "schema", "source_revision", "run_id",
"measurement_window_id", "started_at", "finished_at", "command", "evidence_type",
"gate", "field", "profile_artifacts", "resolved_samples", "allocation_bytes",
"rss_peak_bytes", "save_operations", "saved_bytes"}
for gate, names in expected_fields.items():
rows = observations.fields[gate]
require(type(rows) is dict and set(rows) == set(names), f"verified {gate} observations are incomplete")
for field in names:
row = rows[field]
require(type(row) is dict and not reserved.intersection(row)
and isinstance(row.get("summary"), str) and row["summary"].strip(),
f"invalid verified {gate}.{field} observation")
common = {
"evidence_type": "measured",
"source_revision": source_revision,
"run_id": run_id,
"measurement_window_id": window_id,
"started_at": started_at,
"finished_at": finished_at,
"command": command,
}
total_walk = int(sum_metric(measures, "walk_objects"))
total_cold = int(sum_metric(measures, "cold_walk_objects"))
total_healed = int(sum_metric(measures, "healed_objects"))
total_bytes = sum(item.get("oracle", {}).get("bytes", 0) for item in measures)
total_versions = sum(item.get("oracle", {}).get("versions", 0) for item in measures)
profile_refs = copy_profile_artifacts(
out_dir,
{kind: (profile_inputs[kind], profile_formats[kind]) for kind in RELEASE_PROFILE_ARTIFACTS},
source_revision,
run_id,
window_id,
profile_costs,
)
profile_keys = ("resolved_samples", "allocation_bytes", "rss_peak_bytes", "save_operations", "saved_bytes")
require(type(observations.profile_costs) is dict and set(observations.profile_costs) == set(profile_keys),
"verified profile observations are incomplete")
profile_costs = {key: positive_int(observations.profile_costs[key], key) for key in profile_keys}
require(type(observations.profile_identities) is dict
and set(observations.profile_identities) == set(RELEASE_PROFILE_ARTIFACTS),
"verified profile identities are incomplete")
for kind, identity in observations.profile_identities.items():
require(type(identity) is tuple and len(identity) == 2 and sha(identity[0])
and type(identity[1]) is int and identity[1] > 0, f"invalid verified profile identity: {kind}")
for key in profile_keys:
supplied = getattr(args, key)
require(supplied is None or supplied == profile_costs[key], f"profile {key} differs from verified observation")
gates: dict[str, Any] = {
"G10": {
"status": "pass",
"lane": "scheduler-pressure",
"evidence_type": "measured",
"evidence_fields": {
"scheduler_bound_evidence": write_field(out_dir, "G10", "scheduler_bound_evidence", {
**common,
"summary": "Measured ABBA scheduler-pressure run bounded deferred work and rejected duplicate admission.",
"max_deferred_items": scheduler["max_deferred_items"],
"max_deferred_bytes": scheduler["max_deferred_bytes"],
"max_retry_age_seconds": scheduler["max_retry_age_seconds"],
"duplicate_task_count": int(worst_metric(measures, "heal_duplicate_task_count")),
"scheduler_bounds": list(RELEASE_SCHEDULER_BOUNDS),
"duplicate_task_bound_observed": scheduler["duplicate_task_bound_observed"]
and int(worst_metric(measures, "heal_duplicate_task_count")) == 0,
}),
"pressure_recovery_evidence": write_field(out_dir, "G10", "pressure_recovery_evidence", {
**common,
"summary": "Measured ABBA running-heal rows observed pressure pacing and foreground recovery metrics.",
"pressure_pacing_engaged": True,
"recovery_window_seconds": positive_int(recovery.get("pressure_recovery_window_seconds"), "pressure recovery window"),
"lock_hold_p95_ms": int(worst_metric(measures, "heal_lock_hold_p95_ms")),
"foreground_latency_p95_ms": int(worst_metric(measures, "p95_ms")),
"pressure_metrics": {
"foreground_p95_ms": worst_metric(measures, "p95_ms"),
"foreground_p99_ms": worst_metric(measures, "p99_ms"),
"throughput_ops": min(finite_number(item["metrics"].get("throughput_ops"), "throughput_ops", 1) for item in measures),
"error_rate": 0.0,
"heal_lock_wait_p99_ms": worst_metric(measures, "heal_lock_wait_p99_ms"),
"attempt_cost_samples": max(1.0, sum_metric(measures, "healed_objects")),
"foreground_pressure_samples": int(sum_metric(measures, "foreground_pressure_samples")),
"foreground_pressure_high_samples": max(1, int(sum_metric(measures, "foreground_pressure_high_samples"))),
},
}),
},
},
"P1": {
"status": "pass",
"lane": "scheduler-pressure",
"evidence_type": "measured",
"evidence_fields": {
"cold_walk_share_measurement": write_field(out_dir, "P1", "cold_walk_share_measurement", {
**common,
"duration_seconds": duration,
"summary": "Measured ABBA cold-hot rows met the required cold-walk share reduction.",
"cold_walk_share": 0.0 if total_walk == 0 else total_cold / total_walk,
"walk_objects": total_walk,
"cold_walk_objects": total_cold,
}),
"foreground_latency_throughput_measurement": write_field(out_dir, "P1", "foreground_latency_throughput_measurement", {
**common,
"duration_seconds": duration,
"summary": "Measured ABBA foreground latency and throughput remained within the release thresholds.",
"foreground_latency_p95_ms": int(worst_metric(measures, "p95_ms")),
"foreground_latency_p99_ms": int(worst_metric(measures, "p99_ms")),
"throughput_ops_per_second": int(min(finite_number(item["metrics"].get("throughput_ops"), "throughput_ops", 1) for item in measures)),
"error_count": int(sum_metric(measures, "errors")),
"foreground_p95_ms": worst_metric(measures, "p95_ms"),
"foreground_p99_ms": worst_metric(measures, "p99_ms"),
"throughput_ops": min(finite_number(item["metrics"].get("throughput_ops"), "throughput_ops", 1) for item in measures),
"error_rate": 0.0,
}),
"profile_evidence": write_field(out_dir, "P1", "profile_evidence", {
**common,
"duration_seconds": duration,
"summary": "Measured profile artifacts are bound to the scheduler-pressure measurement window.",
**profile_costs,
"profile_artifacts": profile_refs,
}),
},
},
"P2": {
"status": "pass",
"lane": "scheduler-pressure",
"evidence_type": "measured",
"evidence_fields": {
"post_stop_convergence_measurement": write_field(out_dir, "P2", "post_stop_convergence_measurement", {
**common,
"duration_seconds": duration,
"summary": "Measured ABBA rows converged after writes stopped within the bounded work multiple.",
"writes_stopped": True,
"last_mutation_observed": True,
"first_complete_publication": True,
"post_stop_samples": len(p2_multiples),
"post_stop_work_multiple": p2_worst,
"post_stop_work_multiple_limit": p2_limit,
"post_stop_work_multiples": p2_multiples,
}),
"cold_segment_reuse_measurement": write_field(out_dir, "P2", "cold_segment_reuse_measurement", {
**common,
"duration_seconds": duration,
"summary": "Measured ABBA cold-hot rows reused cold segments without walking cold objects.",
"hot_walked_segments": candidate_walked_segments,
"cold_walked_segments": candidate_cold_segments,
"full_walk_oracle_equivalent": True,
"published_root_equivalent": True,
"walk_objects": candidate_walked_segments,
"cold_walk_objects": candidate_cold_segments,
}),
},
},
"P3": {
"status": "pass",
"lane": "scheduler-pressure",
"evidence_type": "measured",
"evidence_fields": {
"two_hour_pressure_measurement": write_field(out_dir, "P3", "two_hour_pressure_measurement", {
**common,
"duration_seconds": duration,
"summary": "Measured full ABBA scheduler-pressure matrix completed a two-hour fixed-load window per cell.",
"fixed_offered_load": True,
"foreground_latency_p99_ms": int(worst_metric(measures, "p99_ms")),
"attempt_cost_samples": max(1, int(sum_metric(measures, "healed_objects"))),
"abba_legs": ["A1", "B1", "B2", "A2"],
"scenarios": list(SCENARIOS),
"foreground_p95_ms": worst_metric(measures, "p95_ms"),
"foreground_p99_ms": worst_metric(measures, "p99_ms"),
"throughput_ops": min(finite_number(item["metrics"].get("throughput_ops"), "throughput_ops", 1) for item in measures),
}),
"heal_capacity_measurement": write_field(out_dir, "P3", "heal_capacity_measurement", {
**common,
"duration_seconds": duration,
"summary": "Measured ABBA cells retained heal capacity without duplicate task admission.",
"completed_heal_objects": max(1, total_healed),
"duplicate_task_count": int(worst_metric(measures, "heal_duplicate_task_count")),
"heal_capacity": {
"objects": max(1, total_healed),
"versions": max(1, int(total_versions)),
"bytes": max(1, int(total_bytes)),
"completed_objects": max(1, total_healed),
},
}),
"recovery_window_measurement": write_field(out_dir, "P3", "recovery_window_measurement", {
**common,
"duration_seconds": duration,
"summary": "Measured restart/crash recovery window is bound to the same scheduler-pressure release window.",
"pressure_recovery_window_seconds": positive_int(recovery.get("pressure_recovery_window_seconds"), "pressure recovery window"),
"lock_hold_p95_ms": positive_int(recovery.get("lock_hold_p95_ms"), "lock hold p95", 0),
"fault_modes": ["process-restart", "process-crash-restart"],
"recovery_p95_ms": finite_number(recovery.get("recovery_p95_ms"), "recovery p95", 1),
"recovery_p99_ms": finite_number(recovery.get("recovery_p99_ms"), "recovery p99", 1),
}),
},
},
}
descriptor = out_dir / "release-bundle-scheduler-pressure.json"
write_json(descriptor, {
"schema": 1,
"evidence": "measured",
"source_revision": source_revision,
"gates": gates,
})
for gate in ("G10", "P1", "P2", "P3"):
subprocess.check_call([
sys.executable,
str(ROOT / "scripts/check_test_wiring.py"),
"--check-scanner-heal-release-bundle-gate",
str(descriptor),
gate,
], cwd=ROOT)
return descriptor
command = ["scripts/run_scanner_heal_scheduler_pressure_evidence.py", "--abba-dir", "<abba-dir>",
"--recovery-window-json", "<recovery-window-json>", "--profile-artifact", "<kind=artifact>"]
common = {"evidence_type": "measured", "source_revision": source_revision, "run_id": run_id,
"measurement_window_id": window_id, "started_at": started_at, "finished_at": finished_at,
"command": command}
out_dir.parent.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory(prefix=".scheduler-pressure-", dir=out_dir.parent) as temporary:
staging = Path(temporary) / "bundle"
staging.mkdir()
profile_refs = copy_profile_artifacts(
staging, {kind: (profile_inputs[kind], profile_formats[kind]) for kind in RELEASE_PROFILE_ARTIFACTS},
source_revision, run_id, window_id, profile_costs, observations.profile_identities,
)
gates: dict[str, Any] = {}
for gate, names in expected_fields.items():
fields = {}
for field in names:
evidence = {**common, **observations.fields[gate][field]}
if gate == "P1" and field == "profile_evidence":
evidence.update(profile_costs, profile_artifacts=profile_refs)
fields[field] = write_field(staging, gate, field, evidence)
gates[gate] = {"status": "pass", "lane": "scheduler-pressure", "evidence_type": "measured",
"evidence_fields": fields}
descriptor = staging / "release-bundle-scheduler-pressure.json"
write_json(descriptor, {"schema": 1, "evidence": "measured", "source_revision": source_revision, "gates": gates})
out_dir.mkdir(mode=0o700)
# Both directories are on the same filesystem. Move the artifacts
# without a second copy, then check the actual destination before
# publishing the descriptor.
os.rename(staging / "artifacts", out_dir / "artifacts")
pending = out_dir / ".release-bundle-scheduler-pressure.pending.json"
os.link(descriptor, pending)
try:
for gate in ("G10", "P1", "P3"):
subprocess.check_call([
sys.executable,
str(ROOT / "scripts/check_test_wiring.py"),
"--check-scanner-heal-release-bundle-gate",
str(pending),
gate,
], cwd=ROOT)
final = out_dir / descriptor.name
os.link(pending, final)
finally:
pending.unlink()
return final
def write_self_test_abba(root: Path, source_revision: str) -> tuple[Path, Path, list[Path]]:
@@ -601,8 +426,54 @@ def write_self_test_abba(root: Path, source_revision: str) -> tuple[Path, Path,
return abba_dir, recovery, profiles
def self_test_observations(source_revision: str, profiles: dict[str, Path]) -> VerifiedPressureObservations:
"""Synthetic packaging fixture; it is never produced by the verifier."""
pressure = {"foreground_p95_ms": 10.0, "foreground_p99_ms": 20.0, "throughput_ops": 100.0,
"error_rate": 0.0, "heal_lock_wait_p99_ms": 10.0, "attempt_cost_samples": 10,
"foreground_pressure_samples": 100, "foreground_pressure_high_samples": 10}
profile_costs = {"resolved_samples": 4, "allocation_bytes": 1024, "rss_peak_bytes": 2048,
"save_operations": 2, "saved_bytes": 4096}
fields = {
"G10": {
"scheduler_bound_evidence": {"summary": "Fixture bounds", "max_deferred_items": 10,
"max_deferred_bytes": 1048576, "max_retry_age_seconds": 30, "duplicate_task_count": 0,
"scheduler_bounds": list(RELEASE_SCHEDULER_BOUNDS), "duplicate_task_bound_observed": True},
"pressure_recovery_evidence": {"summary": "Fixture pressure", "pressure_pacing_engaged": True,
"recovery_window_seconds": 30, "lock_hold_p95_ms": 5, "foreground_latency_p95_ms": 10,
"pressure_metrics": pressure},
},
"P1": {
"cold_walk_share_measurement": {"summary": "Fixture cold share", "duration_seconds": 7200,
"cold_walk_share": 0.2, "walk_objects": 100, "cold_walk_objects": 20},
"foreground_latency_throughput_measurement": {"summary": "Fixture foreground", "duration_seconds": 7200,
"foreground_latency_p95_ms": 10, "foreground_latency_p99_ms": 20,
"throughput_ops_per_second": 100, "error_count": 0, "foreground_p95_ms": 10.0,
"foreground_p99_ms": 20.0, "throughput_ops": 100.0, "error_rate": 0.0},
"profile_evidence": {"summary": "Fixture profile", "duration_seconds": 7200},
},
"P3": {
"two_hour_pressure_measurement": {"summary": "Fixture two-hour pressure", "duration_seconds": 7200,
"fixed_offered_load": True, "foreground_latency_p99_ms": 20, "attempt_cost_samples": 10,
"abba_legs": ["A1", "B1", "B2", "A2"], "scenarios": list(SCENARIOS),
"foreground_p95_ms": 10.0, "foreground_p99_ms": 20.0, "throughput_ops": 100.0},
"heal_capacity_measurement": {"summary": "Fixture heal capacity", "duration_seconds": 7200,
"completed_heal_objects": 10, "duplicate_task_count": 0,
"heal_capacity": {"objects": 10, "versions": 10, "bytes": 4096, "completed_objects": 10}},
"recovery_window_measurement": {"summary": "Fixture recovery", "duration_seconds": 7200,
"pressure_recovery_window_seconds": 30, "lock_hold_p95_ms": 5,
"fault_modes": ["process-restart", "process-crash-restart"], "recovery_p95_ms": 100.0,
"recovery_p99_ms": 200.0},
},
}
identities = {kind: (digest(path), path.stat().st_size) for kind, path in profiles.items()}
return VerifiedPressureObservations(source_revision, ("fixture-run-0001", "fixture-window-0001",
"2026-01-01T00:00:00Z", "2026-01-01T02:00:00Z"), fields, profile_costs, identities)
def run_self_test() -> None:
import tempfile
from unittest import mock
import check_test_wiring as wiring
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
@@ -618,12 +489,55 @@ def run_self_test() -> None:
"--rss-peak-bytes", "2048",
"--save-operations", "2",
"--saved-bytes", "4096",
"--started-at", "2026-01-01T00:00:00Z",
"--finished-at", "2026-01-01T02:00:00Z",
"--run-id", "fixture-run-0001",
"--measurement-window-id", "fixture-window-0001",
]
for kind, path in zip(RELEASE_PROFILE_ARTIFACTS, profiles):
argv.extend(["--profile-artifact", f"{kind}={path}"])
args = parse_args(argv)
descriptor = build_descriptor(args)
for claim in (None, "verified"):
if claim is not None:
report = read_json(abba_dir / "report.json")
report["measurement_provenance"] = claim
write_json(abba_dir / "report.json", report)
try:
load_measured_abba(abba_dir, source_revision)
except ValueError as err:
require("not independently verified" in str(err), "wrong self-test failure for unverified ABBA")
else:
raise ValueError("self-test accepted unverified ABBA")
with mock.patch(f"{__name__}.measured_provenance_verified", return_value=True):
try:
build_descriptor(args)
except ValueError as err:
require("verified pressure observations" in str(err),
"wrong self-test failure for unverified pressure inputs")
else:
raise ValueError("self-test accepted unverified pressure inputs")
def verify_fixture(command, cwd):
require(cwd == ROOT and command[2] == "--check-scanner-heal-release-bundle-gate",
"unexpected self-test checker invocation")
with mock.patch.object(wiring, "measured_provenance_verified", return_value=True):
status = wiring.scanner_heal_release_bundle_gate_status(ROOT, Path(command[3]), command[4])
require(status["decision"] == "verified", "fixture gate validation failed")
# Exercise packaging with a simulated verifier; this fixture is not
# measurement evidence and the output stays inside the temp directory.
with mock.patch(f"{__name__}.measured_provenance_verified", return_value=True), \
mock.patch(f"{__name__}.pressure_inputs_verified", return_value=self_test_observations(
source_revision, dict(zip(RELEASE_PROFILE_ARTIFACTS, profiles)))), \
mock.patch.object(subprocess, "check_call", side_effect=verify_fixture):
descriptor = build_descriptor(args)
require(descriptor.is_file(), "self-test descriptor missing")
require("P2" not in read_json(descriptor)["gates"], "self-test manufactured P2 evidence")
for gate in ("P1", "P3"):
checked = subprocess.run([sys.executable, str(ROOT / "scripts/check_test_wiring.py"),
"--check-scanner-heal-release-bundle-gate", str(descriptor), gate],
cwd=ROOT, capture_output=True, text=True, check=False)
require(checked.returncode == 1 and "not independently verified" in checked.stdout,
f"self-test accepted unverified {gate} evidence")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
+147 -23
View File
@@ -16,6 +16,8 @@ import sys
import time
from datetime import datetime, timezone
from scanner_client_trace import analyze as reconcile_client_ledger
SCENARIOS = ("cold-hot", "fresh-hot", "multi-hot-new", "running-heal", "mrf-replay")
LEGS = ("A1", "B1", "B2", "A2")
MAX_JSON_BYTES = 1024 * 1024
@@ -111,10 +113,10 @@ def digest(path):
def read_json(path):
require(path.stat().st_size <= MAX_JSON_BYTES, f"oversized JSON: {path.name}")
require(path.stat().st_size <= MAX_JSON_BYTES, "oversized JSON input")
with path.open() as stream:
value = json.load(stream)
require(isinstance(value, dict), f"expected JSON object: {path.name}")
require(isinstance(value, dict), "expected JSON object")
return value
@@ -124,6 +126,52 @@ def write_json(path, value):
path.write_text(data)
def measured_provenance_verified(_output):
# Collector snapshots and adapter echoes do not yet establish a common
# source window or exact client percentiles. Keep acceptance closed until
# the original observations can be independently verified.
return False
def safe_error_message(error):
if isinstance(error, subprocess.TimeoutExpired):
timeout = error.timeout
return f"command timed out after {timeout} seconds" if type(timeout) in (int, float) else "command timed out"
if isinstance(error, subprocess.CalledProcessError):
return f"command exited with status {error.returncode}" if type(error.returncode) is int else "command failed"
if isinstance(error, subprocess.SubprocessError):
return type(error).__name__
if isinstance(error, OSError):
return f"{type(error).__name__}: {os.strerror(error.errno)}" if type(error.errno) is int else type(error).__name__
return str(error) or type(error).__name__
def add_safe_error_note(error, note, related):
error.add_note(note)
error._scanner_safe_notes = [*getattr(error, "_scanner_safe_notes", []), note]
error._scanner_related_errors = [*getattr(error, "_scanner_related_errors", []), related]
def error_chain(error):
messages = []
seen = set()
pending = [error]
while pending and len(seen) < 16:
error = pending.pop(0)
if id(error) in seen:
continue
seen.add(id(error))
parts = [safe_error_message(error), *getattr(error, "_scanner_safe_notes", [])[:4]]
messages.extend(part[:2048] for part in parts if part)
related = getattr(error, "_scanner_related_errors", [])
if isinstance(related, (list, tuple)):
pending.extend(item for item in related[:4] if isinstance(item, BaseException))
next_error = error.__cause__ or (None if error.__suppress_context__ else error.__context__)
if next_error is not None:
pending.append(next_error)
return "; ".join(messages)
def sha(value):
return isinstance(value, str) and len(value) == 64 and all(c in "0123456789abcdef" for c in value)
@@ -191,7 +239,7 @@ def release_evidence_exact_strings(value, expected, name):
missing = sorted(set(expected) - observed)
require(not missing, f"missing release_evidence.{name}: {', '.join(missing)}")
unknown = sorted(observed - set(expected))
require(not unknown, f"unknown release_evidence.{name}: {', '.join(unknown)}")
require(not unknown, f"unknown release_evidence.{name} entries: {len(unknown)}")
return value
@@ -378,7 +426,22 @@ def invoke(adapter, action, request, timeout):
return result
def validate_result(result, request, expected):
def validate_client_request_samples(result, request, artifact_dir):
reference = result.get("client_request_samples")
require(isinstance(reference, dict), "missing client request samples")
require(artifact_dir is not None, "client request samples require a cell artifact directory")
observed = reconcile_client_ledger(Path(artifact_dir), reference, request, result.get("stats"), result.get("cancelled"))
require(observed["sample_complete"] is True and observed["offered_load_complete"] is True
and observed["p99_ns"] is not None, "client schedule or percentile evidence is incomplete")
require(observed["errors"] == 0 and observed["http_503"] == 0, "client request errors or status unavailable")
require(result.get("client_observation") == observed, "client observation differs from original ledgers")
require(result.get("elapsed_seconds") == observed["elapsed_seconds"], "client request elapsed time mismatch")
for key in ("requests", "errors", "p95_ms", "p99_ms", "throughput_ops"):
require(result["metrics"][key] == observed[key], f"client request {key} mismatch")
return observed
def validate_result(result, request, expected, artifact_dir=None):
require(result.get("evidence") == request["evidence"], "adapter evidence type mismatch")
require(result.get("fixed") == request["fixed"], "offered load/config/cache/durability drift")
require(result.get("build") == request["build"], "deployed build provenance mismatch")
@@ -410,6 +473,8 @@ def validate_result(result, request, expected):
require(metrics["healed_objects"] == request["expected_healed_objects"], "incomplete repair oracle")
if request["scenario"] in ("running-heal", "mrf-replay"):
require(metrics["healed_objects"] > 0, "zero completed repairs")
if request["evidence"] == "measured":
validate_client_request_samples(result, request, artifact_dir)
return result
@@ -539,6 +604,18 @@ def convergence(result):
return ratio(window["walk_objects"], window["full_walk_objects"], "convergence work")
def comparison_metrics(cell):
metrics = cell["result"]["metrics"].copy()
if cell["evidence"] == "measured":
observation = cell["result"].get("client_observation")
require(isinstance(observation, dict), "missing measured client observation")
for percentile in (95, 99):
key = f"p{percentile}_ns"
require(type(observation.get(key)) is int and observation[key] >= 0, f"invalid measured {key}")
metrics[f"p{percentile}_ms"] = Decimal(observation[key]) / Decimal(1_000_000)
return metrics
def evaluate(cells):
comparisons = []
inconclusive = False
@@ -546,14 +623,13 @@ def evaluate(cells):
for offset in range(0, len(cells), 4):
group = cells[offset:offset + 4]
require([cell["leg"] for cell in group] == list(LEGS), "incomplete ABBA group")
a1, b1, b2, a2 = (cell["result"]["metrics"] for cell in group)
a1, b1, b2, a2 = (comparison_metrics(cell) for cell in group)
control = group[0]["comparison"] == "background"
drift = max(abs(relative_change(a2[k], a1[k], k)) for k in ("p99_ms", "throughput_ops"))
repeat_drift = max(abs(relative_change(b2[k], b1[k], k)) for k in ("p99_ms", "throughput_ops"))
noise = max(drift, repeat_drift) > REPEATABILITY_LIMIT
a = {key: (decimal_number(a1[key], key) + decimal_number(a2[key], key)) / Decimal("2") for key in METRICS}
b = {key: (decimal_number(b1[key], key) + decimal_number(b2[key], key)) / Decimal("2") for key in METRICS}
p95 = max(a["p95_ms"], b["p95_ms"])
p99 = relative_change(b["p99_ms"], a["p99_ms"], "p99_ms")
throughput = relative_change(b["throughput_ops"], a["throughput_ops"], "throughput_ops")
thresholds = {"p99_regression": Decimal("0.10") if control else Decimal("0.05"),
@@ -595,8 +671,12 @@ def evaluate(cells):
comparisons.append({"scenario": group[0]["scenario"], "comparison": group[0]["comparison"],
"round": group[0]["round"], "status": "inconclusive" if noise else ("fail" if not passed else "inconclusive" if p2_pending else "pass"),
"a2_a1_drift": report_number(drift), "b2_b1_drift": report_number(repeat_drift),
"foreground_p95_ms": float(p95),
"foreground_p99_ms": float(max(a["p99_ms"], b["p99_ms"])),
# Four disjoint legs do not supply one combined percentile.
"foreground_p95_ms": None, "foreground_p99_ms": None,
"foreground_percentiles_ms_by_leg": {
cell["leg"]: {"p95": float(metrics["p95_ms"]), "p99": float(metrics["p99_ms"])}
for cell, metrics in zip(group, (a1, b1, b2, a2))
},
"throughput_ops": float(min(a["throughput_ops"], b["throughput_ops"])),
"error_rate": float(max(a["errors"] / a["requests"], b["errors"] / b["requests"])),
"p99_regression": float(p99), "throughput_change": float(throughput),
@@ -647,8 +727,9 @@ def evaluate(cells):
def collect_live(prepared, request, request_path, adapter):
collector = Path(__file__).with_name("run_scanner_validation_harness.sh")
# Only allow connection fields here; the runner owns cadence and output paths.
connection = prepared["collector"]
require(set(connection) == {"alias", "endpoint", "metrics_endpoints"}, "invalid collector connection")
connection = prepared.get("collector")
require(isinstance(connection, dict) and set(connection) == {"alias", "endpoint", "metrics_endpoints"},
"invalid collector connection")
require(all(isinstance(value, str) and value for value in connection.values()), "missing collector endpoint")
require(os.environ.get("RUSTFS_ACCESS_KEY"), "collector requires RUSTFS_ACCESS_KEY")
require(os.environ.get("RUSTFS_SECRET_KEY"), "collector requires RUSTFS_SECRET_KEY")
@@ -728,7 +809,7 @@ def run(manifest, adapter, output, data_root):
background = "off" if comparison == "background" and leg.startswith("A") else "on"
name = f"{scenario}-{comparison}-{round_id}-{leg}"
cell_dir = output / name
cell_dir.mkdir()
cell_dir.mkdir(mode=0o700)
data_dir = data_root / name
data_dir.mkdir()
request = {"schema": 1, "scenario": scenario, "comparison": comparison, "round": round_id,
@@ -745,10 +826,11 @@ def run(manifest, adapter, output, data_root):
request_path = cell_dir / "request.json"
write_json(request_path, request)
print(name, flush=True)
primary_error = None
try:
prepared = invoke(adapter, "prepare", request_path, 300)
require(prepared.get("ready") is True, "deployment not ready")
if manifest["evidence"] == "measured":
if manifest["evidence"] != "synthetic":
result = collect_live(prepared, request, request_path, adapter)
else:
result = invoke(adapter, "measure", request_path, 300)
@@ -757,26 +839,65 @@ def run(manifest, adapter, output, data_root):
require(oracle.get("complete") is True and oracle.get("errors") == 0, "correctness oracle failed")
require(type(oracle.get("errors")) is int, "invalid oracle error count")
result["oracle"] = oracle["actual"]
validate_result(result, request, request["expected_oracle"])
validate_result(result, request, request["expected_oracle"], cell_dir)
cells.append({**request, "result": result})
except BaseException as error:
primary_error = error
raise
finally:
stopped = invoke(adapter, "stop", request_path, 300)
require(stopped.get("stopped") is True, "adapter failed to stop deployment")
try:
stopped = invoke(adapter, "stop", request_path, 300)
require(stopped.get("stopped") is True, "adapter failed to stop deployment")
except BaseException as stop_error:
if primary_error is not None:
if not isinstance(stop_error, Exception):
add_safe_error_note(stop_error, "prior operation failed", primary_error)
raise stop_error from None
if not isinstance(primary_error, Exception):
prior = primary_error.__cause__ or (
None if primary_error.__suppress_context__ else primary_error.__context__
)
if prior is not None:
add_safe_error_note(primary_error, "prior operation failed", prior)
add_safe_error_note(primary_error, "adapter stop failed", stop_error)
raise primary_error from None
combined = ValueError(
f"{safe_error_message(primary_error)}; adapter stop failed: {safe_error_message(stop_error)}"
)
combined._scanner_related_errors = [primary_error, stop_error]
raise combined from primary_error
raise
status, comparisons = evaluate(cells)
synthetic = manifest["evidence"] == "synthetic"
verified = manifest["evidence"] == "measured" and measured_provenance_verified(output)
if not synthetic and not verified:
# The adapter currently echoes the release manifest without proving
# source windows or client latency samples. Keep results diagnostic
# until those observations can be independently verified.
for comparison in comparisons:
comparison["measurement_provenance"] = "unverified"
if comparison["status"] == "pass":
comparison["status"] = "inconclusive"
if status == "pass":
status = "inconclusive"
finished_at = datetime.now(timezone.utc).replace(microsecond=0)
report = {"status": "synthetic_validated" if synthetic and status == "pass" else status,
"evidence": manifest["evidence"], "performance": "pending" if synthetic else status,
"evidence": manifest["evidence"], "performance": "pending" if synthetic or status == "inconclusive" else status,
"cells": len(cells), "comparisons": comparisons,
"started_at": manifest["started_at"], "finished_at": finished_at.isoformat().replace("+00:00", "Z")}
if not synthetic:
report["measurement_provenance"] = "verified" if verified else "unverified"
write_json(output / "report.json", report)
return 0 if status == "pass" else 3 if status == "inconclusive" else 1
except (ValueError, KeyError, OSError, subprocess.SubprocessError) as error:
except BaseException as error:
finished_at = datetime.now(timezone.utc).replace(microsecond=0)
write_json(output / "report.json", {"status": "failed", "performance": "pending",
"completed_cells": len(cells), "error": str(error),
"started_at": manifest.get("started_at"),
"finished_at": finished_at.isoformat().replace("+00:00", "Z")})
error_text = error_chain(error)
report = {"status": "failed", "performance": "pending", "evidence": manifest["evidence"],
"completed_cells": len(cells), "error": error_text, "started_at": manifest.get("started_at"),
"finished_at": finished_at.isoformat().replace("+00:00", "Z")}
if manifest["evidence"] != "synthetic":
report["measurement_provenance"] = "unverified"
write_json(output / "report.json", report)
raise
@@ -789,8 +910,11 @@ def main():
args = parser.parse_args()
try:
return run(read_json(args.manifest), args.adapter.resolve(), args.out_dir.resolve(), args.data_root.resolve())
except (ValueError, KeyError, OSError, subprocess.SubprocessError) as error:
print(f"ERROR: {error}", file=sys.stderr)
except KeyboardInterrupt as error:
print(f"ERROR: {error_chain(error)}", file=sys.stderr)
return 130
except Exception as error:
print(f"ERROR: {error_chain(error)}", file=sys.stderr)
return 1
+315
View File
@@ -0,0 +1,315 @@
"""Reconcile complete client ledgers and calculate exact nearest-rank samples."""
import hashlib
import json
import math
import os
from pathlib import Path
import re
import shutil
import sqlite3
import stat
import sys
import tempfile
import time
import uuid
from contextlib import ExitStack, closing
from decimal import Decimal, ROUND_CEILING
SCRATCH_FREE_FLOOR = 10 * 1024**3
def require(value, message):
if not value:
raise ValueError(message)
def clock_identity():
info = time.get_clock_info('monotonic')
require(info.monotonic is True and info.adjustable is False and math.isfinite(info.resolution)
and info.resolution > 0, 'client monotonic clock unavailable')
result = {'pid':os.getpid(), 'platform':sys.platform, 'implementation':info.implementation,
'resolution_ns':max(1, math.ceil(info.resolution * 1_000_000_000)),
'monotonic':True, 'adjustable':False, 'boot_id_sha256':None, 'process_start_ticks':None}
if sys.platform == 'linux':
boot = Path('/proc/sys/kernel/random/boot_id').read_bytes().strip()
require(0 < len(boot) <= 128, 'client boot identity unavailable')
raw = Path('/proc/self/stat').read_text()
end = raw.rfind(')')
require(end >= 0 and raw.split(' ', 1)[0] == str(result['pid']), 'invalid client process identity')
fields = raw[end + 1:].split()
require(len(fields) >= 20 and fields[19].isdigit() and int(fields[19]) > 0, 'invalid client process start')
result.update(boot_id_sha256=hashlib.sha256(boot).hexdigest(), process_start_ticks=int(fields[19]))
return result
def unique_object(pairs):
value = {}
for key, item in pairs:
require(key not in value, 'duplicate JSON key in client ledger')
value[key] = item
return value
def source_file(root_fd, ref):
require(type(ref) is dict and set(ref) == {'path', 'sha256', 'bytes'}, 'invalid client ledger reference')
relative = ref['path']
require(isinstance(relative, str) and relative and not Path(relative).is_absolute()
and '..' not in Path(relative).parts and str(Path(relative)) == relative, 'invalid client ledger path')
require(isinstance(ref['sha256'], str) and re.fullmatch(r'[0-9a-f]{64}', ref['sha256'])
and type(ref['bytes']) is int and 0 <= ref['bytes'] <= 2**63-1, 'invalid client ledger identity')
parts = Path(relative).parts
require(parts and all(part not in ('.', '..') for part in parts), 'invalid client ledger path')
root = os.fstat(root_fd)
directory = os.dup(root_fd)
try:
for part in parts[:-1]:
child = os.open(part, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=directory)
os.close(directory)
directory = child
current = os.fstat(directory)
require(current.st_dev == root.st_dev and current.st_uid == os.geteuid()
and current.st_mode & 0o077 == 0, 'client ledger directory is not private and owned')
fd = os.open(parts[-1], os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)
try:
current = os.fstat(fd)
require(stat.S_ISREG(current.st_mode) and current.st_dev == root.st_dev
and current.st_size == ref['bytes'], 'client ledger identity or size differs')
except BaseException:
os.close(fd)
raise
return fd
finally:
os.close(directory)
def records(fd, reference):
with os.fdopen(os.dup(fd), 'rb') as stream:
before = os.fstat(stream.fileno())
require(stat.S_ISREG(before.st_mode) and before.st_size == reference['bytes'], 'client ledger size differs')
digest = hashlib.sha256()
while line := stream.readline(4097):
require(len(line) <= 4096 and line.endswith(b'\n'), 'truncated or oversized client record')
digest.update(line)
try:
row = json.loads(line, object_pairs_hook=unique_object)
except (UnicodeError, json.JSONDecodeError):
raise ValueError('malformed client record') from None
require(type(row) is dict, 'invalid client record')
yield row
after = os.fstat(stream.fileno())
require((before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns)
== (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns)
and digest.hexdigest() == reference['sha256'], 'client ledger changed or digest differs')
def request_id(row, planned):
value = row.get('request_id')
require(isinstance(value, str) and re.fullmatch(r'0|[1-9][0-9]{0,18}', value), 'invalid client request id')
index = int(value)
require(index < planned, 'client request outside plan')
return index
def scratch_parent():
parent = Path(tempfile.gettempdir()).resolve(strict=True)
for directory in (parent, *parent.parents):
info = directory.stat()
shared = bool(info.st_mode & 0o022)
sticky = bool(info.st_mode & stat.S_ISVTX)
require(stat.S_ISDIR(info.st_mode) and info.st_uid in (0, os.geteuid())
and (not shared or sticky), 'untrusted client scratch parent')
return parent
def identity(reference):
clock_id = reference.get('clock_id')
try:
require(isinstance(clock_id, str) and str(uuid.UUID(clock_id)) == clock_id, 'invalid client clock id')
except (ValueError, AttributeError):
raise ValueError('invalid client clock id') from None
first, last = reference.get('clock_before'), reference.get('clock_after')
keys = {'pid','platform','implementation','resolution_ns','monotonic','adjustable','boot_id_sha256','process_start_ticks'}
require(type(first) is dict and first == last and set(first) == keys
and type(first['pid']) is int and first['pid'] > 0
and isinstance(first['platform'], str) and 0 < len(first['platform']) <= 64
and isinstance(first['implementation'], str) and 0 < len(first['implementation']) <= 256
and type(first['resolution_ns']) is int and first['resolution_ns'] > 0
and first['monotonic'] is True and first['adjustable'] is False, 'client clock changed or unavailable')
if first['platform'] == 'linux':
require(isinstance(first['boot_id_sha256'], str) and re.fullmatch(r'[0-9a-f]{64}', first['boot_id_sha256'])
and type(first['process_start_ticks']) is int and first['process_start_ticks'] > 0,
'client process identity unavailable')
else:
require(first['boot_id_sha256'] is None and first['process_start_ticks'] is None,
'unverified client process identity')
return clock_id, first
def analyze(root, reference, request, stats, cancelled):
"""Return original-window observations. Partial input never produces release metrics.
Each planned index must have one schedule record and each dispatch one
matching start and completion. Percentiles include completed successes and
failures and use nearest rank, ceil(p*n/100), in integer nanoseconds. The
measured window includes the real post-dispatch drain; it is not aligned
to any server or another client. Error-rate numerator/denominator are exact.
"""
root_fd = os.open(Path(root), os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
try:
root_stat = os.fstat(root_fd)
require(root_stat.st_uid == os.geteuid() and root_stat.st_mode & 0o077 == 0,
'client evidence root is not private and owned')
return _analyze_open_root(root_fd, reference, request, stats, cancelled)
finally:
os.close(root_fd)
def _analyze_open_root(root_fd, reference, request, stats, cancelled):
require(type(reference) is dict and reference.get('schema') == 2, 'invalid client evidence schema')
clock, clock_info = identity(reference)
start, dispatch_end, end = (reference.get(key) for key in ('window_start_ns','dispatch_window_end_ns','window_end_ns'))
require(all(type(v) is int and 0 <= v < 2**63 for v in (start, dispatch_end, end))
and start < dispatch_end and start < end, 'invalid client window')
duration = request.get('duration_seconds')
fixed = request.get('fixed')
require(type(duration) is int and duration > 0 and type(fixed) is dict, 'invalid client request duration')
rate_value = fixed.get('offered_load_ops')
require(type(rate_value) in (int,float) and math.isfinite(rate_value) and rate_value > 0, 'invalid client offered rate')
rate = Decimal(str(rate_value))
planned = int((rate * Decimal(duration)).to_integral_value(rounding=ROUND_CEILING))
require(0 < planned < 2**63 and dispatch_end == start + duration * 1_000_000_000,
'planned client window differs')
require(type(cancelled) is bool and (cancelled or end >= dispatch_end), 'client window stopped early')
require(type(stats) is dict and all(type(stats.get(k)) is int and stats[k] >= 0 for k in
('planned_requests','started','completed','errors','missed_dispatches'))
and stats['planned_requests'] == planned, 'client counters unavailable')
sample = {'path':reference.get('path'), 'sha256':reference.get('sha256'), 'bytes':reference.get('bytes')}
refs = {'samples':sample, 'starts':reference.get('starts'), 'schedule':reference.get('schedule')}
with ExitStack() as stack:
paths = {}
for key, value in refs.items():
paths[key] = source_file(root_fd, value)
stack.callback(os.close, paths[key])
identities = {(os.fstat(fd).st_dev, os.fstat(fd).st_ino) for fd in paths.values()}
require(len(identities) == 3, 'client ledgers alias')
return _reduce_ledgers(paths, refs, request, stats, cancelled, clock, clock_info,
start, dispatch_end, end, planned, rate)
def _reduce_ledgers(paths, refs, request, stats, cancelled, clock, clock_info, start, dispatch_end, end, planned, rate):
period = max(1, int(Decimal(1_000_000_000) / rate))
with tempfile.TemporaryDirectory(prefix='scanner-client-reduce-', dir=scratch_parent()) as scratch:
scratch_budget = 64 * 1024**2 + planned * 512
require(shutil.disk_usage(scratch).free >= SCRATCH_FREE_FLOOR + scratch_budget,
'insufficient client reduction scratch space')
def check_space(index):
if index % 4096 == 0:
require(shutil.disk_usage(scratch).free >= SCRATCH_FREE_FLOOR,
'insufficient client reduction scratch space')
with closing(sqlite3.connect(str(Path(scratch)/'ledger.sqlite'))) as db:
page_size = db.execute('PRAGMA page_size').fetchone()[0]
db.execute(f'PRAGMA max_page_count={scratch_budget // page_size}')
db.execute('CREATE TABLE requests (id INTEGER PRIMARY KEY, scheduled INTEGER NOT NULL, observed INTEGER NOT NULL, '
'dispatched INTEGER NOT NULL, reason TEXT, began INTEGER, finished INTEGER, op TEXT, outcome TEXT, status INTEGER)')
db.execute('CREATE INDEX by_latency ON requests((finished-began), id)')
scheduled_count = misses = 0
for index, row in enumerate(records(paths['schedule'], refs['schedule'])):
check_space(index)
require(set(row) == {'request_id','clock_id','scheduled_ns','observed_ns','dispatched','miss_reason'}
and row['clock_id'] == clock, 'invalid schedule record')
index = request_id(row, planned)
expect = start + int(Decimal(index) * 1_000_000_000 / rate)
require(index == scheduled_count and type(row['scheduled_ns']) is int
and row['scheduled_ns'] == expect and expect < dispatch_end
and type(row['observed_ns']) is int and expect <= row['observed_ns'] <= end
and type(row['dispatched']) is bool, 'invalid schedule order or timestamp')
reason = row['miss_reason']
require((row['dispatched'] and reason is None) or
(not row['dispatched'] and reason in ('late_slot','client_capacity','after_dispatch_window')), 'invalid dispatch outcome')
require(not row['dispatched'] or row['observed_ns'] < min(expect + period, dispatch_end),
'late dispatch marked on time')
require(reason != 'late_slot' or row['observed_ns'] >= expect + period, 'inconsistent late dispatch')
require(reason != 'after_dispatch_window' or row['observed_ns'] >= dispatch_end, 'inconsistent dispatch boundary')
db.execute('INSERT INTO requests(id,scheduled,observed,dispatched,reason) VALUES (?,?,?,?,?)',
(index, expect, row['observed_ns'], int(row['dispatched']), reason))
scheduled_count += 1; misses += not row['dispatched']
require(cancelled or scheduled_count == planned, 'scheduled client records missing')
for index, row in enumerate(records(paths['starts'], refs['starts'])):
check_space(index)
require(set(row) == {'request_id','clock_id','start_ns','scheduled_ns','operation'}
and row['clock_id'] == clock and row['operation'] in ('GET','PUT')
and type(row['start_ns']) is int and start <= row['start_ns'] <= end
and type(row['scheduled_ns']) is int, 'invalid client start')
changed = db.execute('UPDATE requests SET began=?, op=? WHERE id=? AND dispatched=1 AND began IS NULL '
'AND scheduled=? AND observed<=?',
(row['start_ns'], row['operation'], request_id(row, planned), row['scheduled_ns'], row['start_ns'])).rowcount
require(changed == 1, 'missing, duplicate or unmatched client start')
for index, row in enumerate(records(paths['samples'], refs['samples'])):
check_space(index)
require(set(row) == {'request_id','clock_id','start_ns','end_ns','outcome','operation','error_type','http_status'}
and row['clock_id'] == clock and row['operation'] in ('GET','PUT')
and type(row['start_ns']) is int and type(row['end_ns']) is int
and start <= row['start_ns'] <= row['end_ns'] <= end
and row['outcome'] in ('success','error','timeout','cancelled'), 'invalid client completion or cross-window sample')
error_type, status = row['error_type'], row['http_status']
require((row['outcome'] == 'success' and error_type is None) or
(row['outcome'] != 'success' and isinstance(error_type,str) and 0 < len(error_type) <= 256),
'invalid client outcome identity')
require(status is None or (type(status) is int and 100 <= status <= 599), 'invalid HTTP status')
require(not (row['outcome'] == 'success' and status is not None and status >= 400), 'HTTP failure marked successful')
changed = db.execute('UPDATE requests SET finished=?, outcome=?, status=? WHERE id=? AND began=? '
'AND op=? AND finished IS NULL',
(row['end_ns'], row['outcome'], status, request_id(row, planned), row['start_ns'], row['operation'])).rowcount
require(changed == 1, 'missing, duplicate or unmatched client completion')
counts = db.execute('SELECT COUNT(*), COUNT(began), COUNT(finished), '
"COUNT(CASE WHEN outcome IS NOT NULL AND outcome != 'success' THEN 1 END), "
'COUNT(status), COUNT(CASE WHEN status=503 THEN 1 END), '
'COUNT(CASE WHEN began< ? AND finished> ? THEN 1 END), '
'COUNT(CASE WHEN began>= ? THEN 1 END), '
'COUNT(CASE WHEN dispatched=1 AND observed>= ? THEN 1 END), '
'COUNT(CASE WHEN began-scheduled>= ? THEN 1 END) FROM requests',
(dispatch_end, dispatch_end, dispatch_end, dispatch_end, period)).fetchone()
rows, started, completed, errors, status_known, known_503, crossing, after_dispatch, late_dispatch, late_starts = counts
dispatched = db.execute('SELECT COUNT(*) FROM requests WHERE dispatched=1').fetchone()[0]
require(rows == scheduled_count and started == completed == dispatched
and stats['started'] == started and stats['completed'] == completed
and stats['errors'] == errors and stats['missed_dispatches'] == misses,
'client schedule/start/completion/counter mismatch')
complete = not cancelled and scheduled_count == planned and misses == 0 and completed == planned
scheduler_window_complete = complete and late_dispatch == 0 and after_dispatch == 0 and late_starts == 0
# Worker start precedes the locked start-ledger flush. It does not
# establish when the SDK was called or whether traffic was sent.
offered_load_complete = None if scheduler_window_complete else False
result = {'complete':complete, 'sample_complete':complete, 'offered_load_complete':offered_load_complete,
'scheduler_window_complete':scheduler_window_complete,
'planned_requests':planned, 'scheduled_requests':scheduled_count,
'missed_dispatches':misses, 'observed_completed':completed,
'observed_errors':errors if completed else None,
'observed_http_503_lower_bound':known_503 if known_503 else None,
'http_status_known':status_known, 'http_status_unknown':completed-status_known,
'http_503':known_503 if complete and completed > 0 and status_known == completed else None,
'requests':completed if complete else None, 'errors':errors if complete else None,
'error_rate':{'numerator':errors, 'denominator':completed} if complete and completed else None,
'p95_ns':None, 'p99_ns':None, 'p95_ms':None, 'p99_ms':None,
'throughput_ops':None, 'sample_throughput_ops':None,
'quantile_method':'nearest-rank over all completed requests, including errors',
'clock_id':clock, 'client_clock':clock_info,
'window_start_ns':start, 'dispatch_window_end_ns':dispatch_end, 'window_end_ns':end,
'crossing_dispatch_end':crossing, 'started_after_dispatch_end':after_dispatch,
'dispatched_after_dispatch_end':late_dispatch, 'late_request_starts':late_starts,
'elapsed_seconds':(end-start)/1_000_000_000, 'same_source_window_proven':False,
'release_acceptance_ready':False}
if complete and completed >= 100:
check_space(0)
for percentile in (95,99):
rank = (completed * percentile + 99) // 100
latency = db.execute('SELECT finished-began FROM requests ORDER BY (finished-began), id LIMIT 1 OFFSET ?', (rank-1,)).fetchone()[0]
result['p'+str(percentile)+'_ns'] = latency
result['p'+str(percentile)+'_ms'] = latency / 1_000_000
if complete:
result['sample_throughput_ops'] = (completed-errors) * 1_000_000_000 / (end-start)
if offered_load_complete is True:
result['throughput_ops'] = result['sample_throughput_ops']
return result
+191 -447
View File
@@ -5,12 +5,11 @@ from __future__ import annotations
import argparse
from collections import Counter
from datetime import datetime, timezone
from datetime import datetime
from decimal import Decimal
import hashlib
import json
from pathlib import Path
import shutil
import sys
from typing import Any
@@ -18,8 +17,8 @@ from scanner_abba import (
LEGS,
MIN_MEASURED_RELEASE_DURATION_SECONDS,
RELEASE_PROFILE_ARTIFACTS,
RELEASE_SCHEDULER_BOUNDS,
SCENARIOS,
measured_provenance_verified,
validate_release_evidence_manifest,
)
@@ -27,7 +26,10 @@ MAX_JSON_BYTES = 1024 * 1024
CACHE_COST_PREFIX = "CACHE_COST "
PASS_STATES = {"pass"}
FAIL_STATES = {"fail", "failed"}
RELEASE_DESCRIPTOR_GATES = ("G10", "P1", "P3")
REPORT_STATES = PASS_STATES | FAIL_STATES | {"inconclusive", "incomplete", "pending", "synthetic_validated", "unknown"}
PERFORMANCE_STATES = PASS_STATES | FAIL_STATES | {"inconclusive", "pending", "unknown"}
EVIDENCE_TYPES = {"measured", "synthetic", "diagnostic"}
OBSERVATION_STATES = {"observed", "no_measured_benefit", "rss_regression", "pending", "inconclusive", "not_applicable", "unknown"}
def require(condition: bool, message: str) -> None:
@@ -44,6 +46,15 @@ def read_json(path: Path) -> dict[str, Any]:
return value
def read_json_snapshot(path: Path) -> tuple[dict[str, Any], str]:
with path.open("rb") as stream:
data = stream.read(MAX_JSON_BYTES + 1)
require(len(data) <= MAX_JSON_BYTES, "oversized JSON artifact")
value = json.loads(data)
require(isinstance(value, dict), "expected JSON object")
return value, hashlib.sha256(data).hexdigest()
def write_json(path: Path, value: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(value, indent=2, sort_keys=True, allow_nan=False) + "\n", encoding="utf-8")
@@ -61,7 +72,9 @@ def digest(path: Path) -> str:
def number(value: Any, name: str) -> Decimal:
require(type(value) in (float, int), f"invalid numeric field: {name}")
return Decimal(str(value))
result = Decimal(str(value))
require(result.is_finite(), f"invalid numeric field: {name}")
return result
def maybe_number(value: Any, name: str) -> Decimal | None:
@@ -70,6 +83,34 @@ def maybe_number(value: Any, name: str) -> Decimal | None:
return number(value, name)
def safe_observed_number(value: Any) -> int | float | None:
if type(value) not in (int, float) or not Decimal(str(value)).is_finite():
return None
return value
def safe_count(value: Any) -> int | None:
return value if type(value) is int and value >= 0 else None
def safe_observation_state(value: Any) -> str:
return value if isinstance(value, str) and value in OBSERVATION_STATES else "unknown"
def safe_hex(value: Any, length: int) -> str | None:
return value if isinstance(value, str) and len(value) == length and all(c in "0123456789abcdef" for c in value) else None
def safe_timestamp(value: Any) -> str | None:
if not isinstance(value, str) or len(value) > 64:
return None
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return None
return value if parsed.tzinfo is not None else None
def pct(value: Decimal | None) -> str:
if value is None:
return "pending"
@@ -94,13 +135,6 @@ def require_integer(value: Any, name: str, minimum: int = 0) -> int:
return value
def timestamp(value: Any, name: str) -> str:
require(isinstance(value, str) and value.strip(), f"missing timestamp: {name}")
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
require(parsed.tzinfo is not None, f"timestamp must include timezone: {name}")
return parsed.astimezone(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
def require_metric_series(value: Any, name: str, minimum: Decimal | None = None,
maximum: Decimal | None = None) -> list[Decimal | None]:
require(isinstance(value, list) and value, f"missing performance evidence field: {name}")
@@ -116,6 +150,16 @@ def require_metric_series(value: Any, name: str, minimum: Decimal | None = None,
def require_measured_comparison_evidence(comparison: dict[str, Any], index: int) -> None:
percentiles = comparison.get("foreground_percentiles_ms_by_leg")
require(isinstance(percentiles, dict) and set(percentiles) == set(LEGS),
f"comparison {index} missing foreground percentiles by leg")
for leg in LEGS:
row = percentiles[leg]
require(isinstance(row, dict), f"comparison {index} invalid {leg} foreground percentiles")
p95 = number(row.get("p95"), f"comparison {index} {leg} foreground percentiles p95")
p99 = number(row.get("p99"), f"comparison {index} {leg} foreground percentiles p99")
require(p95.is_finite() and p99.is_finite() and p95 >= 0 and p99 > 0 and p95 <= p99,
f"comparison {index} invalid {leg} foreground percentiles")
w10_w11 = comparison.get("w10_w11")
require(isinstance(w10_w11, dict), f"comparison {index} missing W10/W11 evidence")
pressure = require_metric_series(
@@ -243,12 +287,16 @@ def has_complete_abba_matrix(manifest: dict[str, Any], report: dict[str, Any], c
def summarize_abba(abba_dir: Path) -> dict[str, Any]:
manifest_path = abba_dir / "manifest.json"
report_path = abba_dir / "report.json"
manifest = read_json(manifest_path)
report = read_json(report_path)
manifest, manifest_sha256 = read_json_snapshot(manifest_path)
report, report_sha256 = read_json_snapshot(report_path)
report_state = report.get("status")
performance_state = report.get("performance")
require(isinstance(report_state, str) and report_state, "report.status missing")
require(isinstance(performance_state, str) and performance_state, "report.performance missing")
report_state = report_state if report_state in REPORT_STATES else "unknown"
performance_state = performance_state if performance_state in PERFORMANCE_STATES else "unknown"
evidence_type = report.get("evidence")
evidence_type = evidence_type if isinstance(evidence_type, str) and evidence_type in EVIDENCE_TYPES else "unknown"
comparisons = report.get("comparisons")
if comparisons is None:
require(report_state not in PASS_STATES, "passing report requires comparisons")
@@ -261,6 +309,7 @@ def summarize_abba(abba_dir: Path) -> dict[str, Any]:
throughput_losses: list[Decimal] = []
p1_rows = []
p2_values: list[Decimal | None] = []
p2_complete = matrix_complete
foreground_p95_values: list[Decimal | None] = []
foreground_p99_values: list[Decimal | None] = []
throughput_values: list[Decimal | None] = []
@@ -269,17 +318,20 @@ def summarize_abba(abba_dir: Path) -> dict[str, Any]:
pressure_high_samples = 0
pressure_samples_complete = matrix_complete
pressure_high_samples_complete = matrix_complete
attempt_cost_samples = 0
attempt_cost_ratio_count = 0
attempt_cost_ratios_complete = matrix_complete
start_p95_values: list[Decimal | None] = []
duplicate_task_values: list[Decimal | None] = []
lock_hold_values: list[Decimal | None] = []
w09_complete = {field: matrix_complete for field in
("heal_start_p95_ms", "heal_duplicate_task_count", "heal_lock_hold_p95_ms")}
w10_rows = []
w11_rows = []
for index, comparison in enumerate(comparisons):
require(isinstance(comparison, dict), f"comparison {index} must be an object")
state = comparison.get("status")
require(isinstance(state, str) and state, f"comparison {index} missing status")
counts[state] += 1
counts[state if state in REPORT_STATES else "unknown"] += 1
p99_regressions.append(number(comparison.get("p99_regression"), f"comparison {index} p99_regression"))
throughput_change = number(comparison.get("throughput_change"), f"comparison {index} throughput_change")
throughput_losses.append(max(Decimal("0"), -throughput_change))
@@ -288,31 +340,37 @@ def summarize_abba(abba_dir: Path) -> dict[str, Any]:
p1 = comparison.get("p1")
if isinstance(p1, dict):
p1_rows.append({
"scenario": comparison.get("scenario"),
"comparison": comparison.get("comparison"),
"round": comparison.get("round"),
"scenario": comparison.get("scenario") if comparison.get("scenario") in SCENARIOS else "unknown",
"comparison": comparison.get("comparison") if comparison.get("comparison") in ("build", "background") else "unknown",
"round": safe_count(comparison.get("round")),
"required_reduction": float(number(p1.get("required_reduction"), "p1.required_reduction")),
"observed_reduction": float(number(p1.get("observed_reduction"), "p1.observed_reduction")),
"repeatability_drift": (
None if p1.get("repeatability_drift") is None
else float(number(p1.get("repeatability_drift"), "p1.repeatability_drift"))
),
"baseline_walk_objects": p1.get("baseline_walk_objects"),
"baseline_cold_walk_objects": p1.get("baseline_cold_walk_objects"),
"candidate_walk_objects": p1.get("candidate_walk_objects"),
"candidate_cold_walk_objects": p1.get("candidate_cold_walk_objects"),
"baseline_walk_objects": safe_count(p1.get("baseline_walk_objects")),
"baseline_cold_walk_objects": safe_count(p1.get("baseline_cold_walk_objects")),
"candidate_walk_objects": safe_count(p1.get("candidate_walk_objects")),
"candidate_cold_walk_objects": safe_count(p1.get("candidate_cold_walk_objects")),
})
p2 = comparison.get("p2_post_stop_work_multiples")
if isinstance(p2, list):
p2_values.extend(maybe_number(value, "p2_post_stop_work_multiple") for value in p2)
if isinstance(p2, list) and len(p2) == len(LEGS):
candidate = [maybe_number(p2[index], "p2_post_stop_work_multiple") for index in (1, 2)]
p2_values.extend(candidate)
if any(value is None for value in candidate):
p2_complete = False
else:
p2_complete = False
w09 = comparison.get("w09")
if isinstance(w09, dict):
for value in w09.get("heal_start_p95_ms", []):
start_p95_values.append(maybe_number(value, "heal_start_p95_ms"))
for value in w09.get("heal_duplicate_task_count", []):
duplicate_task_values.append(maybe_number(value, "heal_duplicate_task_count"))
for value in w09.get("heal_lock_hold_p95_ms", []):
lock_hold_values.append(maybe_number(value, "heal_lock_hold_p95_ms"))
for field, values in (("heal_start_p95_ms", start_p95_values),
("heal_duplicate_task_count", duplicate_task_values),
("heal_lock_hold_p95_ms", lock_hold_values)):
samples = w09.get(field) if isinstance(w09, dict) else None
if isinstance(samples, list):
values.extend(maybe_number(value, field) for value in samples)
if not isinstance(samples, list) or len(samples) != len(LEGS) or any(value is None for value in samples):
w09_complete[field] = False
w10_w11 = comparison.get("w10_w11")
if isinstance(w10_w11, dict):
samples = w10_w11.get("foreground_pressure_samples")
@@ -325,112 +383,143 @@ def summarize_abba(abba_dir: Path) -> dict[str, Any]:
pressure_high_samples += sum(require_integer(value, "foreground_pressure_high_samples", 0) for value in high_samples)
else:
pressure_high_samples_complete = False
for value in w10_w11.get("attempt_cost_per_healed_object", []):
if value is not None:
attempt_cost_samples += 1
costs = w10_w11.get("attempt_cost_per_healed_object")
if isinstance(costs, list) and len(costs) == len(LEGS):
attempt_cost_ratio_count += sum(value is not None for value in costs)
else:
attempt_cost_ratios_complete = False
else:
pressure_samples_complete = False
pressure_high_samples_complete = False
attempt_cost_ratios_complete = False
w10 = comparison.get("w10")
if isinstance(w10, dict) and comparison.get("scenario") == "running-heal" and comparison.get("comparison") == "build":
w10_rows.append({
"round": comparison.get("round"),
"status": w10.get("status"),
"pacing_observed": w10.get("pacing_observed"),
"candidate_pressure_high_ratio": w10.get("candidate_pressure_high_ratio"),
"candidate_delay_events": w10.get("candidate_delay_events"),
"foreground_p99_change": w10.get("foreground_p99_change"),
"foreground_throughput_change": w10.get("foreground_throughput_change"),
"round": safe_count(comparison.get("round")),
"status": safe_observation_state(w10.get("status")),
"pacing_observed": w10.get("pacing_observed") if type(w10.get("pacing_observed")) is bool else None,
"candidate_pressure_high_ratio": safe_observed_number(w10.get("candidate_pressure_high_ratio")),
"candidate_delay_events": safe_count(w10.get("candidate_delay_events")),
"foreground_p99_change": safe_observed_number(w10.get("foreground_p99_change")),
"foreground_throughput_change": safe_observed_number(w10.get("foreground_throughput_change")),
})
if "throughput_ops" in comparison:
throughput_values.append(maybe_number(comparison.get("throughput_ops"), "throughput_ops"))
error_rate_values.append(maybe_number(comparison.get("error_rate"), "error_rate"))
error_rate = maybe_number(comparison.get("error_rate"), "error_rate")
require(error_rate is None or Decimal("0") <= error_rate <= Decimal("1"), "invalid error_rate")
error_rate_values.append(error_rate)
w11 = comparison.get("w11")
if isinstance(w11, dict) and comparison.get("scenario") == "running-heal" and comparison.get("comparison") == "build":
w11_rows.append({
"round": comparison.get("round"),
"status": w11.get("status"),
"rss_growth": w11.get("rss_growth"),
"rss_growth_limit": w11.get("rss_growth_limit"),
"heal_lock_wait_p99_change": w11.get("heal_lock_wait_p99_change"),
"foreground_p99_change": w11.get("foreground_p99_change"),
"foreground_throughput_change": w11.get("foreground_throughput_change"),
"candidate_attempt_cost_per_healed_object": w11.get("candidate_attempt_cost_per_healed_object"),
"round": safe_count(comparison.get("round")),
"status": safe_observation_state(w11.get("status")),
"rss_growth": safe_observed_number(w11.get("rss_growth")),
"rss_growth_limit": safe_observed_number(w11.get("rss_growth_limit")),
"heal_lock_wait_p99_change": safe_observed_number(w11.get("heal_lock_wait_p99_change")),
"foreground_p99_change": safe_observed_number(w11.get("foreground_p99_change")),
"foreground_throughput_change": safe_observed_number(w11.get("foreground_throughput_change")),
"candidate_attempt_cost_per_healed_object": safe_observed_number(w11.get("candidate_attempt_cost_per_healed_object")),
})
measured = report.get("evidence") == "measured"
passed = report_state in PASS_STATES and performance_state in PASS_STATES and measured
if passed:
reported_pass = report_state in PASS_STATES and performance_state in PASS_STATES and measured
verified = measured and measured_provenance_verified(abba_dir)
if reported_pass:
require_complete_abba_matrix(manifest, report, comparisons)
validate_release_evidence_manifest({**manifest, "evidence": "measured"})
for index, comparison in enumerate(comparisons):
require_measured_comparison_evidence(comparison, index)
require(all(value is not None for value in error_rate_values), "error_rate must be measured")
require(all(value == 0 for value in error_rate_values), "passing report has nonzero error_rate")
require(pressure_samples_complete, "foreground_pressure_samples must be measured")
require(pressure_high_samples_complete, "foreground_pressure_high_samples must be measured")
passed = reported_pass and verified
if not verified and report.get("evidence") != "synthetic":
for row in w10_rows + w11_rows:
if row["status"] == "observed":
row["status"] = "unverified"
gate_state = "pass" if passed else "fail"
if report_state == "synthetic_validated":
reason = "synthetic evidence validates the harness only; measured performance remains pending"
elif report_state in FAIL_STATES and isinstance(report.get("error"), str) and report["error"]:
reason = f"ABBA report status is {report_state}: {report['error']}"
reason = f"ABBA report status is {report_state}; error details omitted"
elif report_state not in PASS_STATES:
reason = f"ABBA report status is {report_state}"
elif performance_state not in PASS_STATES:
reason = f"performance status is {performance_state}"
elif not measured:
reason = "measured evidence is required for a performance conclusion"
elif not verified:
reason = "measured source windows and client percentiles are not independently verified"
else:
reason = "measured ABBA report passed"
fixed = manifest.get("fixed", {})
require(isinstance(fixed, dict), "manifest.fixed must be an object")
baseline = manifest.get("baseline") if isinstance(manifest.get("baseline"), dict) else {}
candidate = manifest.get("candidate") if isinstance(manifest.get("candidate"), dict) else {}
return {
"gate_state": gate_state,
"reason": reason,
"status": report_state,
"performance": performance_state,
"evidence": report.get("evidence"),
"cells": report.get("cells", 0),
"completed_cells": report.get("completed_cells"),
"error": report.get("error"),
"evidence": evidence_type,
"measurement_provenance": "verified" if verified else "unverified",
"cells": safe_count(report.get("cells")),
"completed_cells": safe_count(report.get("completed_cells")),
# Historical reports may contain unredacted subprocess arguments.
"error": "details omitted" if isinstance(report.get("error"), str) and report["error"] else None,
"comparisons_total": len(comparisons),
"comparison_status_counts": dict(sorted(counts.items())),
"worst_p99_regression": None if not p99_regressions else float(max(p99_regressions)),
"worst_throughput_loss": None if not throughput_losses else float(max(throughput_losses)),
"foreground_p95_ms": None if max_decimal(foreground_p95_values) is None else float(max_decimal(foreground_p95_values)),
"foreground_p99_ms": None if max_decimal(foreground_p99_values) is None else float(max_decimal(foreground_p99_values)),
"throughput_ops": None if max_decimal(throughput_values) is None else float(max_decimal(throughput_values)),
"worst_p99_regression": float(max(p99_regressions)) if matrix_complete and p99_regressions else None,
"worst_throughput_loss": float(max(throughput_losses)) if matrix_complete and throughput_losses else None,
# Comparisons describe different legs and windows. Their maxima are
# not a percentile of one measured request population.
"foreground_p95_ms": None,
"foreground_p99_ms": None,
"max_reported_comparison_p95_ms": (None if max_decimal(foreground_p95_values) is None
else float(max_decimal(foreground_p95_values))),
"max_reported_comparison_p99_ms": (None if max_decimal(foreground_p99_values) is None
else float(max_decimal(foreground_p99_values))),
"throughput_ops": None,
"max_reported_comparison_throughput_ops": (float(max(throughput_values))
if matrix_complete and len(throughput_values) == len(comparisons)
and all(value is not None for value in throughput_values) else None),
"error_rate": (None if not matrix_complete or any(value is None for value in error_rate_values)
else float(max_decimal(error_rate_values))),
"foreground_pressure_samples": pressure_samples if pressure_samples_complete else None,
"foreground_pressure_high_samples": pressure_high_samples if pressure_high_samples_complete else None,
"attempt_cost_samples": attempt_cost_samples,
"p2_worst_post_stop_work_multiple": None if max_decimal(p2_values) is None else float(max_decimal(p2_values)),
"w09_worst_heal_start_p95_ms": None if max_decimal(start_p95_values) is None else float(max_decimal(start_p95_values)),
"w09_duplicate_task_count": None if max_decimal(duplicate_task_values) is None else float(max_decimal(duplicate_task_values)),
"w09_worst_lock_hold_p95_ms": None if max_decimal(lock_hold_values) is None else float(max_decimal(lock_hold_values)),
"attempt_cost_samples": None,
"reported_attempt_cost_ratio_count": attempt_cost_ratio_count if attempt_cost_ratios_complete else None,
"p2_worst_post_stop_work_multiple": float(max(p2_values)) if p2_complete and p2_values else None,
"w09_worst_heal_start_p95_ms": (float(max(start_p95_values))
if w09_complete["heal_start_p95_ms"] and start_p95_values else None),
"w09_duplicate_task_count": (float(max(duplicate_task_values))
if w09_complete["heal_duplicate_task_count"] and duplicate_task_values else None),
"w09_worst_lock_hold_p95_ms": (float(max(lock_hold_values))
if w09_complete["heal_lock_hold_p95_ms"] and lock_hold_values else None),
"p1_reductions": p1_rows,
"w10_running_heal_build": w10_rows,
"w11_running_heal_build": w11_rows,
"provenance": {
"abba_dir": str(abba_dir.resolve()),
"manifest_sha256": digest(manifest_path),
"report_sha256": digest(report_path),
"baseline_revision": manifest.get("baseline", {}).get("revision"),
"baseline_sha256": manifest.get("baseline", {}).get("sha256"),
"candidate_revision": manifest.get("candidate", {}).get("revision"),
"candidate_sha256": manifest.get("candidate", {}).get("sha256"),
"adapter_sha256": manifest.get("adapter_sha256"),
"collector_sha256": manifest.get("collector_sha256"),
"config_sha256": fixed.get("config_sha256"),
"dataset_sha256": fixed.get("dataset_sha256"),
"release_flags": fixed.get("release_flags"),
"durability": fixed.get("durability"),
"topology": fixed.get("topology"),
"offered_load_ops": fixed.get("offered_load_ops"),
"release_evidence": manifest.get("release_evidence"),
"started_at": report.get("started_at") or manifest.get("started_at"),
"finished_at": report.get("finished_at"),
"manifest_sha256": manifest_sha256,
"report_sha256": report_sha256,
"baseline_revision": safe_hex(baseline.get("revision"), 40),
"baseline_sha256": safe_hex(baseline.get("sha256"), 64),
"candidate_revision": safe_hex(candidate.get("revision"), 40),
"candidate_sha256": safe_hex(candidate.get("sha256"), 64),
"adapter_sha256": safe_hex(manifest.get("adapter_sha256"), 64),
"collector_sha256": safe_hex(manifest.get("collector_sha256"), 64),
"config_sha256": safe_hex(fixed.get("config_sha256"), 64),
"dataset_sha256": safe_hex(fixed.get("dataset_sha256"), 64),
"release_flags_present": isinstance(fixed.get("release_flags"), str) and bool(fixed["release_flags"]),
"durability": fixed.get("durability") if fixed.get("durability") in ("drive-sync=on", "drive-sync=off") else None,
"topology": fixed.get("topology") if fixed.get("topology") == "EC8+4" else None,
"offered_load_ops": safe_observed_number(fixed.get("offered_load_ops")),
"release_evidence_present": isinstance(manifest.get("release_evidence"), dict),
"started_at": safe_timestamp(report.get("started_at")) or safe_timestamp(manifest.get("started_at")),
"finished_at": safe_timestamp(report.get("finished_at")),
},
}
@@ -461,7 +550,8 @@ def summarize_cache_cost(path: Path) -> dict[str, Any]:
max_save_ns = Decimal("0")
build_sources = set()
for index, record in enumerate(records):
scenarios[str(record.get("scenario"))] += 1
scenario = record.get("scenario")
scenarios[scenario if isinstance(scenario, str) and scenario in ("unchanged", "small_dirty", "all_dirty") else "unknown"] += 1
wire = number(record.get("cache_wire_bytes"), f"cache_cost {index} cache_wire_bytes")
save_body = number(record.get("save_body_bytes_per_sample"), f"cache_cost {index} save_body_bytes_per_sample")
require(wire > 0, f"cache_cost {index} has zero wire bytes")
@@ -479,7 +569,14 @@ def summarize_cache_cost(path: Path) -> dict[str, Any]:
max_save_ns = max(max_save_ns, value)
build = record.get("build", {})
require(isinstance(build, dict), f"cache_cost {index} build must be an object")
build_sources.add((build.get("source_revision"), build.get("source_tree"), build.get("test_opt_level_override")))
revision = build.get("source_revision")
revision = (revision if isinstance(revision, str) and 1 <= len(revision) <= 40
and all(c in "0123456789abcdef" for c in revision) else None)
tree = build.get("source_tree")
tree = tree if tree in ("clean", "dirty", "unknown") else None
opt = build.get("test_opt_level_override")
opt = opt if opt in ("0", "1", "2", "3", "s", "z") else None
build_sources.add((revision, tree, opt))
return {
"records": len(records),
"scenario_counts": dict(sorted(scenarios.items())),
@@ -499,362 +596,9 @@ def summarize_cache_cost(path: Path) -> dict[str, Any]:
}
def profile_artifact_map(values: list[str] | None) -> dict[str, Path]:
artifacts: dict[str, Path] = {}
for value in values or []:
require("=" in value, "profile artifact must use KIND=PATH")
kind, raw_path = value.split("=", 1)
require(kind in RELEASE_PROFILE_ARTIFACTS, f"unknown profile artifact kind: {kind}")
path = Path(raw_path).resolve()
require(path.is_file() and path.stat().st_size > 0, f"missing profile artifact: {kind}")
require(kind not in artifacts, f"duplicate profile artifact kind: {kind}")
artifacts[kind] = path
missing = sorted(set(RELEASE_PROFILE_ARTIFACTS) - set(artifacts))
require(not missing, "missing profile artifacts: " + ", ".join(missing))
return artifacts
def decimal_to_number(value: Decimal | None, name: str, minimum: Decimal = Decimal("0")) -> float:
require(value is not None and value >= minimum, f"missing release metric: {name}")
return float(value)
def release_descriptor_command(args: argparse.Namespace) -> list[str]:
command = [
"scripts/summarize_scanner_heal_perf.py",
"--abba-dir",
str(args.abba_dir),
]
if args.cache_cost_log:
command.extend(["--cache-cost-log", str(args.cache_cost_log)])
if args.require_cache_cost:
command.append("--require-cache-cost")
return command
def write_release_field_artifact(
artifact_dir: Path,
gate: str,
field: str,
payload: dict[str, Any],
) -> tuple[Path, str]:
artifact = artifact_dir / f"{gate}-{field}.json"
write_json(artifact, payload)
return artifact, digest(artifact)
def profile_wrapper_artifact(
artifact_dir: Path,
source_revision: str,
run_id: str,
window_id: str,
kind: str,
path: Path,
profile_measurements: dict[str, Any],
) -> dict[str, Any]:
required_metrics = {
"allocation-profile": ("resolved_samples", "allocation_bytes"),
"flamegraph": ("resolved_samples",),
"rss-samples": ("resolved_samples", "rss_peak_bytes"),
"save-frequency": ("resolved_samples", "save_operations", "saved_bytes"),
}[kind]
raw_artifact = artifact_dir / f"P1-profile_evidence-{kind}.raw{path.suffix}"
shutil.copyfile(path, raw_artifact)
wrapper = artifact_dir / f"P1-profile_evidence-{kind}.json"
payload = {
"schema": 1,
"evidence_type": "measured",
"source_revision": source_revision,
"run_id": run_id,
"measurement_window_id": window_id,
"gate": "P1",
"field": "profile_evidence",
"artifact_kind": kind,
"raw_profile_name": path.name,
"raw_profile_artifact": raw_artifact.relative_to(artifact_dir.parent).as_posix(),
"raw_profile_sha256": digest(raw_artifact),
"raw_profile_bytes": raw_artifact.stat().st_size,
"raw_profile_format": path.suffix.lower().lstrip(".") or "binary",
}
for metric in required_metrics:
payload[metric] = require_integer(profile_measurements.get(metric), metric, 1)
write_json(wrapper, payload)
return {
"artifact": wrapper.relative_to(artifact_dir.parent).as_posix(),
"sha256": digest(wrapper),
"artifact_format": "json",
"source_revision": source_revision,
"run_id": run_id,
"measurement_window_id": window_id,
}
def release_field(
artifact_dir: Path,
source_revision: str,
run_id: str,
window_id: str,
started_at: str,
finished_at: str,
command: list[str],
gate: str,
field: str,
summary_text: str,
evidence: dict[str, Any],
) -> dict[str, Any]:
payload = {
"schema": 1,
"evidence_type": "measured",
"source_revision": source_revision,
"run_id": run_id,
"measurement_window_id": window_id,
"gate": gate,
"field": field,
**evidence,
}
artifact, artifact_sha = write_release_field_artifact(artifact_dir, gate, field, payload)
return {
"evidence_type": "measured",
"source_revision": source_revision,
"run_id": run_id,
"measurement_window_id": window_id,
"started_at": started_at,
"finished_at": finished_at,
"command": command,
"artifact": artifact.relative_to(artifact_dir.parent).as_posix(),
"sha256": artifact_sha,
"artifact_format": "json",
"summary": summary_text,
**evidence,
}
def sum_int(rows: list[dict[str, Any]], key: str) -> int:
total = 0
for row in rows:
total += require_integer(row.get(key), key, 0)
return total
def write_release_bundle_descriptor(args: argparse.Namespace, summary: dict[str, Any]) -> None:
def write_release_bundle_descriptor(_args: argparse.Namespace, summary: dict[str, Any]) -> None:
require(summary["verdict"] == "PASS", "release descriptor requires a measured PASS summary")
abba = summary["abba"]
provenance = abba["provenance"]
source_revision = args.release_source_revision or provenance.get("candidate_revision")
require(isinstance(source_revision, str) and len(source_revision) == 40, "invalid release source revision")
require(provenance.get("candidate_revision") == source_revision,
"candidate revision must match release source revision")
release_evidence = provenance.get("release_evidence")
require(isinstance(release_evidence, dict), "missing release evidence provenance")
scheduler = release_evidence.get("scheduler")
require(isinstance(scheduler, dict), "missing release_evidence.scheduler")
profile = release_evidence.get("profile")
require(isinstance(profile, dict), "missing release_evidence.profile")
profile_measurements = profile.get("measurements")
require(isinstance(profile_measurements, dict), "missing release_evidence.profile.measurements")
profile_costs = {
"resolved_samples": require_integer(profile_measurements.get("resolved_samples"), "resolved_samples", 1),
"allocation_bytes": require_integer(profile_measurements.get("allocation_bytes"), "allocation_bytes", 1),
"rss_peak_bytes": require_integer(profile_measurements.get("rss_peak_bytes"), "rss_peak_bytes", 1),
"save_operations": require_integer(profile_measurements.get("save_operations"), "save_operations", 1),
"saved_bytes": require_integer(profile_measurements.get("saved_bytes"), "saved_bytes", 1),
}
profile_artifacts = profile_artifact_map(args.release_profile_artifact)
descriptor = args.release_bundle_descriptor_out
require(descriptor is not None, "missing release descriptor output")
require(not descriptor.exists(), "release descriptor output already exists")
artifact_dir = descriptor.parent / f"{descriptor.stem}-artifacts"
require(not artifact_dir.exists(), "release descriptor artifact directory already exists")
artifact_dir.mkdir(parents=True)
started_at = timestamp(provenance.get("started_at") or release_evidence.get("started_at"), "release started_at")
finished_at = timestamp(provenance.get("finished_at") or release_evidence.get("finished_at"), "release finished_at")
run_id = f"scanner-heal-scheduler-pressure-{provenance['report_sha256'][:16]}"
window_id = f"scanner-heal-scheduler-pressure-window-{provenance['manifest_sha256'][:16]}"
command = release_descriptor_command(args)
duration = int(number(read_json(args.abba_dir / "manifest.json").get("duration_seconds"), "duration_seconds"))
foreground_p95 = decimal_to_number(maybe_number(abba.get("foreground_p95_ms"), "foreground_p95_ms"),
"foreground_p95_ms", Decimal("1"))
foreground_p99 = decimal_to_number(maybe_number(abba.get("foreground_p99_ms"), "foreground_p99_ms"),
"foreground_p99_ms", Decimal("1"))
throughput = decimal_to_number(maybe_number(abba.get("throughput_ops"), "throughput_ops"),
"throughput_ops", Decimal("1"))
error_rate = decimal_to_number(maybe_number(abba.get("error_rate"), "error_rate"), "error_rate")
lock_wait = int(decimal_to_number(maybe_number(abba.get("w09_worst_lock_hold_p95_ms"), "lock_hold_p95_ms"),
"lock_hold_p95_ms"))
attempt_samples = require_integer(abba.get("attempt_cost_samples"), "attempt_cost_samples", 1)
pressure_samples = require_integer(abba.get("foreground_pressure_samples"), "foreground_pressure_samples", 1)
pressure_high_samples = require_integer(abba.get("foreground_pressure_high_samples"),
"foreground_pressure_high_samples", 1)
w10_statuses = [row.get("status") for row in abba.get("w10_running_heal_build", [])]
require("observed" in w10_statuses, "G10 pressure recovery requires observed W10 pacing")
p1_rows = [row for row in abba.get("p1_reductions", []) if row.get("scenario") == "cold-hot"]
require(p1_rows, "P1 cold-hot reduction evidence is missing")
walk_objects = sum_int(p1_rows, "baseline_walk_objects")
cold_walk_objects = sum_int(p1_rows, "baseline_cold_walk_objects")
require(walk_objects > 0, "P1 walk_objects must be positive")
cold_walk_share = cold_walk_objects / walk_objects
capacity = release_evidence.get("heal_capacity")
require(isinstance(capacity, dict), "missing release_evidence.heal_capacity")
recovery = release_evidence.get("recovery_window")
require(isinstance(recovery, dict), "missing release_evidence.recovery_window")
descriptor_value = {
"schema": 1,
"evidence": "measured",
"source_revision": source_revision,
"gates": {
"G10": {
"status": "pass",
"lane": "scheduler-pressure",
"evidence_type": "measured",
"evidence_fields": {
"scheduler_bound_evidence": release_field(
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
"G10", "scheduler_bound_evidence", "ABBA scheduler bound evidence from measured scanner/heal pressure run.",
{
"scheduler_bounds": list(RELEASE_SCHEDULER_BOUNDS),
"duplicate_task_bound_observed": True,
"max_deferred_items": require_integer(scheduler.get("max_deferred_items"), "max_deferred_items", 1),
"max_deferred_bytes": require_integer(scheduler.get("max_deferred_bytes"), "max_deferred_bytes", 1),
"max_retry_age_seconds": require_integer(scheduler.get("max_retry_age_seconds"), "max_retry_age_seconds", 1),
"duplicate_task_count": 0,
"duration_seconds": duration,
},
),
"pressure_recovery_evidence": release_field(
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
"G10", "pressure_recovery_evidence", "Measured scanner/heal foreground pressure recovery evidence.",
{
"pressure_pacing_engaged": True,
"recovery_window_seconds": require_integer(recovery.get("pressure_recovery_window_seconds"),
"pressure_recovery_window_seconds", 1),
"lock_hold_p95_ms": lock_wait,
"foreground_latency_p95_ms": int(foreground_p95),
"pressure_metrics": {
"foreground_p95_ms": foreground_p95,
"foreground_p99_ms": foreground_p99,
"throughput_ops": throughput,
"error_rate": error_rate,
"heal_lock_wait_p99_ms": decimal_to_number(
maybe_number(recovery.get("heal_lock_wait_p99_ms"), "heal_lock_wait_p99_ms"),
"heal_lock_wait_p99_ms",
),
"attempt_cost_samples": attempt_samples,
"foreground_pressure_samples": pressure_samples,
"foreground_pressure_high_samples": pressure_high_samples,
},
"duration_seconds": duration,
},
),
},
},
"P1": {
"status": "pass",
"lane": "scheduler-pressure",
"evidence_type": "measured",
"evidence_fields": {
"cold_walk_share_measurement": release_field(
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
"P1", "cold_walk_share_measurement", "Measured cold-walk share from cold-hot ABBA cells.",
{
"cold_walk_share": cold_walk_share,
"walk_objects": walk_objects,
"cold_walk_objects": cold_walk_objects,
"duration_seconds": duration,
},
),
"foreground_latency_throughput_measurement": release_field(
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
"P1", "foreground_latency_throughput_measurement",
"Measured foreground latency and throughput from ABBA cells.",
{
"foreground_latency_p95_ms": int(foreground_p95),
"foreground_latency_p99_ms": int(foreground_p99),
"throughput_ops_per_second": int(throughput),
"error_count": 0,
"foreground_p95_ms": foreground_p95,
"foreground_p99_ms": foreground_p99,
"throughput_ops": throughput,
"error_rate": error_rate,
"duration_seconds": duration,
},
),
"profile_evidence": release_field(
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
"P1", "profile_evidence", "Measured allocation, RSS, save-frequency, and flamegraph profile evidence.",
{
**profile_costs,
"profile_artifacts": {
kind: profile_wrapper_artifact(
artifact_dir, source_revision, run_id, window_id, kind, path, profile_costs
)
for kind, path in sorted(profile_artifacts.items())
},
"duration_seconds": duration,
},
),
},
},
"P3": {
"status": "pass",
"lane": "scheduler-pressure",
"evidence_type": "measured",
"evidence_fields": {
"two_hour_pressure_measurement": release_field(
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
"P3", "two_hour_pressure_measurement", "Measured two-hour ABBA pressure run.",
{
"fixed_offered_load": True,
"foreground_latency_p99_ms": int(foreground_p99),
"attempt_cost_samples": attempt_samples,
"abba_legs": list(LEGS),
"scenarios": list(SCENARIOS),
"foreground_p95_ms": foreground_p95,
"foreground_p99_ms": foreground_p99,
"throughput_ops": throughput,
"duration_seconds": duration,
},
),
"heal_capacity_measurement": release_field(
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
"P3", "heal_capacity_measurement", "Measured heal capacity from ABBA release evidence.",
{
"completed_heal_objects": require_integer(capacity.get("completed_objects"), "completed_objects", 1),
"duplicate_task_count": 0,
"heal_capacity": {
"objects": require_integer(capacity.get("objects"), "objects", 1),
"versions": require_integer(capacity.get("versions"), "versions", 1),
"bytes": require_integer(capacity.get("bytes"), "bytes", 1),
"completed_objects": require_integer(capacity.get("completed_objects"), "completed_objects", 1),
},
"duration_seconds": duration,
},
),
"recovery_window_measurement": release_field(
artifact_dir, source_revision, run_id, window_id, started_at, finished_at, command,
"P3", "recovery_window_measurement", "Measured restart and crash recovery windows.",
{
"pressure_recovery_window_seconds": require_integer(recovery.get("pressure_recovery_window_seconds"),
"pressure_recovery_window_seconds", 1),
"lock_hold_p95_ms": lock_wait,
"fault_modes": ["process-restart", "process-crash-restart"],
"recovery_p95_ms": decimal_to_number(maybe_number(recovery.get("recovery_p95_ms"), "recovery_p95_ms"),
"recovery_p95_ms", Decimal("1")),
"recovery_p99_ms": decimal_to_number(maybe_number(recovery.get("recovery_p99_ms"), "recovery_p99_ms"),
"recovery_p99_ms", Decimal("1")),
"duration_seconds": duration,
},
),
},
},
},
}
write_json(descriptor, descriptor_value)
raise ValueError("release descriptor source evidence is unavailable")
def markdown(summary: dict[str, Any]) -> str:
@@ -867,7 +611,7 @@ def markdown(summary: dict[str, Any]) -> str:
"",
f"- verdict: {summary['verdict']}",
f"- reason: {summary['reason']}",
f"- abba: status={abba['status']} performance={abba['performance']} evidence={abba['evidence']} cells={abba['cells']} comparisons={abba['comparisons_total']}",
f"- abba: status={abba['status']} performance={abba['performance']} evidence={abba['evidence']} cells={abba['cells'] if abba['cells'] is not None else 'pending'} comparisons={abba['comparisons_total']}",
f"- worst_p99_regression: {pct(p99)}",
f"- worst_throughput_loss: {pct(throughput)}",
f"- p2_worst_post_stop_work_multiple: {ratio(p2)}",
@@ -923,7 +667,7 @@ def main() -> int:
parser.add_argument("--json-out", type=Path, help="Write the normalized summary JSON artifact")
parser.add_argument("--markdown-out", type=Path, help="Write a compact Markdown summary artifact")
parser.add_argument("--release-bundle-descriptor-out", type=Path,
help="Write a measured G10/P1/P3 release-bundle descriptor")
help="Reserved; verified source evidence is not yet available")
parser.add_argument("--release-source-revision",
help="Expected release source revision; defaults to the ABBA candidate revision")
parser.add_argument("--release-profile-artifact", action="append",
@@ -931,13 +675,13 @@ def main() -> int:
args = parser.parse_args()
try:
summary = build_summary(args)
if args.release_bundle_descriptor_out:
write_release_bundle_descriptor(args, summary)
if args.json_out:
write_json(args.json_out, summary)
if args.markdown_out:
args.markdown_out.parent.mkdir(parents=True, exist_ok=True)
args.markdown_out.write_text(markdown(summary), encoding="utf-8")
if args.release_bundle_descriptor_out:
write_release_bundle_descriptor(args, summary)
abba = summary["abba"]
print(
f"{summary['verdict']} scanner_heal_perf "
@@ -946,7 +690,7 @@ def main() -> int:
)
return 0 if summary["verdict"] == "PASS" else 1
except (ValueError, OSError, json.JSONDecodeError) as error:
print(f"FAIL scanner_heal_perf error={error}", file=sys.stderr)
print(f"FAIL scanner_heal_perf error={type(error).__name__}", file=sys.stderr)
return 1
+327 -12
View File
@@ -6,6 +6,7 @@ import copy
import fcntl
import io
import json
import math
import os
from pathlib import Path
import shlex
@@ -140,6 +141,15 @@ def child_released(marker, timeout=1):
while True:
try:
fcntl.flock(stream, fcntl.LOCK_EX | fcntl.LOCK_NB)
record = stream.read()
if not record.endswith(";ready"):
raise AssertionError("fixture deadline unavailable")
try:
expiry = float(record.removesuffix(";ready"))
except ValueError as error:
raise AssertionError("fixture deadline unavailable") from error
if not math.isfinite(expiry) or time.monotonic() >= expiry:
raise AssertionError("fixture deadline unavailable or expired")
return True
except BlockingIOError:
if time.monotonic() >= deadline:
@@ -148,11 +158,11 @@ def child_released(marker, timeout=1):
def reap_fixture(marker):
if marker.exists() and marker.stat().st_size and not child_released(marker, timeout=0):
# The unique file lock proves the original fixture process still owns this PID.
os.kill(int(marker.read_text()), signal.SIGKILL)
if not child_released(marker, timeout=5):
raise AssertionError("fixture child did not release its process-owned lock")
# The lock does not keep a PID reserved after the owner exits. Ask the
# fixture to exit instead of signaling a potentially reused PID.
marker.with_name(marker.name + ".release").touch()
if marker.exists() and marker.stat().st_size and not child_released(marker, timeout=5):
raise AssertionError("fixture child did not release its process-owned lock")
class ScannerAbbaTest(unittest.TestCase):
@@ -259,6 +269,7 @@ class ScannerAbbaTest(unittest.TestCase):
with self.assertRaises(subprocess.TimeoutExpired):
harness.invoke(self.adapter, "measure", request, 3)
wait_for_marker(marker)
self.assertFalse(marker.with_name(marker.name + ".release").exists())
self.assertTrue(child_released(marker), "TERM-exited parent left its TERM-ignoring child alive")
finally:
reap_fixture(marker)
@@ -283,6 +294,7 @@ class ScannerAbbaTest(unittest.TestCase):
with self.assertRaisesRegex(ValueError, "injected measurement failure"):
harness.collect_live({"collector": {"alias": "fixture", "endpoint": "fixture", "metrics_endpoints": "fixture"}},
{"duration_seconds": 900}, request, self.adapter)
self.assertFalse(marker.with_name(marker.name + ".release").exists())
self.assertTrue(child_released(marker), "collector parent exit did not end its telemetry child")
finally:
reap_fixture(marker)
@@ -300,6 +312,14 @@ class ScannerAbbaTest(unittest.TestCase):
finally:
reap_fixture(marker)
def test_invalid_fixture_deadline_does_not_count_as_cleaned_up(self):
marker = self.root / "stubborn.pid"
for value in ("invalid;ready", "nan;ready", str(time.monotonic() - 1) + ";ready", "1.2"):
with self.subTest(value=value):
marker.write_text(value)
with self.assertRaisesRegex(AssertionError, "fixture deadline"):
child_released(marker)
def test_reaped_owner_never_signals_a_reused_process_group(self):
with (self.root / "owner.log").open("wb") as log:
owner = harness.OwnedCommand([sys.executable, "-c", "pass"], log)
@@ -330,6 +350,7 @@ class ScannerAbbaTest(unittest.TestCase):
with patch.object(harness.time, "sleep", side_effect=interrupt_once):
with self.assertRaises(KeyboardInterrupt):
owner.finish(terminate=True)
self.assertFalse(marker.with_name(marker.name + ".release").exists())
self.assertTrue(child_released(marker), "cleanup cancellation left its child alive")
self.assertIsNotNone(owner.process.returncode, "cleanup cancellation must reap its leader")
finally:
@@ -353,6 +374,7 @@ class ScannerAbbaTest(unittest.TestCase):
patch.object(harness.os, "write", side_effect=release_then_fail):
with self.assertRaisesRegex(OSError, "injected failure after gate release"):
harness.OwnedCommand([str(self.adapter), "measure", str(request), str(self.root / "unused.json")], log)
self.assertFalse(marker.with_name(marker.name + ".release").exists())
self.assertTrue(child_released(marker), "initialization failure left its child alive")
finally:
reap_fixture(marker)
@@ -705,12 +727,295 @@ class ScannerAbbaTest(unittest.TestCase):
"metrics": metrics,
"oracle": manifest["oracles"]["cold-hot"],
}
harness.validate_result(result, request, manifest["oracles"]["cold-hot"])
with self.assertRaisesRegex(ValueError, "missing client request samples"):
harness.validate_result(result, request, manifest["oracles"]["cold-hot"])
with patch.object(harness, "validate_client_request_samples"):
harness.validate_result(result, request, manifest["oracles"]["cold-hot"])
result["release_evidence"] = copy.deepcopy(result["release_evidence"])
result["release_evidence"]["profile"]["required_artifacts"].remove("flamegraph")
with self.assertRaisesRegex(ValueError, "release evidence provenance mismatch"):
harness.validate_result(result, request, manifest["oracles"]["cold-hot"])
def test_measured_self_report_cannot_pass_without_source_evidence(self):
manifest = self.measured_manifest()
def invoke(_adapter, action, request_path, _timeout):
request = harness.read_json(request_path)
if action == "prepare":
return {"ready": True}
if action == "stop":
return {"stopped": True}
self.assertEqual(action, "oracle")
return {"complete": True, "errors": 0, "actual": request["expected_oracle"]}
regression = False
def collect(_prepared, request, _request_path, _adapter):
baseline = request["comparison"] == "build" and request["leg"].startswith("A")
metrics = dict.fromkeys(harness.METRICS, 10)
metrics.update(p99_ms=12 if regression and request["leg"].startswith("B") else 10,
throughput_ops=100, errors=0, requests=100,
walk_objects=100 if baseline else 20, cold_walk_objects=100 if baseline else 0,
healed_objects=request["expected_healed_objects"], heal_duplicate_task_count=0)
result = {key: request[key] for key in ("evidence", "fixed", "build", "data_dir", "background", "release_evidence")}
result.update(sample_count=10, elapsed_seconds=request["duration_seconds"], metrics=metrics,
client_observation={"p95_ns": 10_000_000,
"p99_ns": int(metrics["p99_ms"] * 1_000_000)},
convergence={"writes_stopped": True, "last_mutation_observed": True,
"first_complete_publication": True, "last_mutation_time": 1,
"last_mutation_observed_time": 2, "writes_stopped_time": 2,
"window_start": 2, "window_end": 3, "budget_available_seconds": 1,
"walk_objects": 110, "full_walk_objects": 100})
return result
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
patch.object(harness, "validate_client_request_samples"), \
patch.object(harness, "collect_live", side_effect=collect), contextlib.redirect_stdout(io.StringIO()):
status = harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
report = harness.read_json(self.root / "out/report.json")
self.assertEqual(status, 3)
self.assertEqual((report["status"], report["performance"]), ("inconclusive", "pending"))
self.assertEqual(report["measurement_provenance"], "unverified")
self.assertTrue(all(row["status"] == "inconclusive" and row["measurement_provenance"] == "unverified"
for row in report["comparisons"]))
regression = True
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
patch.object(harness, "validate_client_request_samples"), \
patch.object(harness, "collect_live", side_effect=collect), contextlib.redirect_stdout(io.StringIO()):
status = harness.run(manifest, self.adapter, self.root / "failed-out", self.root / "failed-data")
report = harness.read_json(self.root / "failed-out/report.json")
self.assertEqual(status, 1)
self.assertEqual((report["status"], report["performance"]), ("fail", "fail"))
self.assertTrue(any(row["status"] == "fail" for row in report["comparisons"]))
def test_measured_percentile_threshold_uses_original_nanoseconds(self):
baseline = 9_000_000_000_000_001
candidate = 9_450_000_000_000_002
cells = []
for leg in harness.LEGS:
ns = baseline if leg.startswith("A") else candidate
metrics = dict.fromkeys(harness.METRICS, 1)
metrics.update(p95_ms=1.0, p99_ms=ns / 1_000_000, requests=100, errors=0,
walk_objects=100 if leg.startswith("A") else 20,
cold_walk_objects=100 if leg.startswith("A") else 0)
cells.append({"leg": leg, "evidence": "measured", "comparison": "build", "scenario": "cold-hot",
"round": 1, "background": "on", "result": {"metrics": metrics,
"client_observation": {"p95_ns": 1_000_000, "p99_ns": ns}}})
_, comparisons = harness.evaluate(cells)
self.assertGreater(comparisons[0]["p99_regression"], 0.05)
self.assertEqual(comparisons[0]["status"], "fail")
self.assertIsNone(comparisons[0]["foreground_p95_ms"])
self.assertIsNone(comparisons[0]["foreground_p99_ms"])
def test_measurement_and_stop_errors_are_both_preserved(self):
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True}
if action == "measure":
raise ValueError("measurement failed")
if action == "stop":
raise ValueError("stop failed")
self.fail(f"unexpected action {action}")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaisesRegex(ValueError, "measurement failed.*stop failed"):
self.run_harness()
report = harness.read_json(self.root / "out/report.json")
self.assertIn("measurement failed", report["error"])
self.assertIn("stop failed", report["error"])
def test_malformed_measured_collector_and_stop_errors_are_both_preserved(self):
manifest = self.measured_manifest()
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True, "collector": None}
if action == "stop":
raise ValueError("stop failed")
self.fail(f"unexpected action {action}")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaisesRegex(ValueError, "invalid collector connection.*stop failed"):
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
report = harness.read_json(self.root / "out/report.json")
self.assertIn("invalid collector connection", report["error"])
self.assertIn("stop failed", report["error"])
self.assertEqual(report["measurement_provenance"], "unverified")
def test_malformed_measured_collector_is_reported_after_successful_stop(self):
manifest = self.measured_manifest()
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True, "collector": None}
if action == "stop":
return {"stopped": True}
self.fail(f"unexpected action {action}")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaisesRegex(ValueError, "invalid collector connection"):
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
report = harness.read_json(self.root / "out/report.json")
self.assertEqual((report["status"], report["evidence"], report["measurement_provenance"]),
("failed", "measured", "unverified"))
self.assertIn("invalid collector connection", report["error"])
def test_interruption_is_preserved_when_stop_also_fails(self):
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True}
if action == "measure":
raise KeyboardInterrupt("interrupted")
if action == "stop":
raise ValueError("stop failed")
self.fail(f"unexpected action {action}")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(KeyboardInterrupt):
self.run_harness()
report = harness.read_json(self.root / "out/report.json")
self.assertIn("interrupted", report["error"])
self.assertIn("stop failed", report["error"])
def test_stop_interruption_preserves_prior_failure(self):
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True}
if action == "measure":
raise ValueError("measurement failed")
if action == "stop":
raise KeyboardInterrupt("stop interrupted")
self.fail(f"unexpected action {action}")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(KeyboardInterrupt):
self.run_harness()
report = harness.read_json(self.root / "out/report.json")
self.assertIn("measurement failed", report["error"])
self.assertIn("stop interrupted", report["error"])
def test_collector_cleanup_and_stop_errors_keep_measurement_failure(self):
manifest = self.measured_manifest()
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True, "collector": {"alias": "test", "endpoint": "test", "metrics_endpoints": "test"}}
if action == "measure":
raise ValueError("measurement failed")
if action == "stop":
raise ValueError("stop failed")
self.fail(f"unexpected action {action}")
collector = Mock()
collector.finish.side_effect = ValueError("collector cleanup failed")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
patch.object(harness, "OwnedCommand", return_value=collector), \
patch.dict(os.environ, {"RUSTFS_ACCESS_KEY": "test", "RUSTFS_SECRET_KEY": "test"}), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(ValueError):
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
report = harness.read_json(self.root / "out/report.json")
for message in ("measurement failed", "collector cleanup failed", "stop failed"):
self.assertIn(message, report["error"])
def test_stop_interruption_keeps_nested_collector_failure_chain(self):
manifest = self.measured_manifest()
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True, "collector": {"alias": "test", "endpoint": "test", "metrics_endpoints": "test"}}
if action == "measure":
raise ValueError("measurement failed")
if action == "stop":
raise KeyboardInterrupt("stop interrupted")
self.fail(f"unexpected action {action}")
collector = Mock()
collector.finish.side_effect = ValueError("collector cleanup failed")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
patch.object(harness, "OwnedCommand", return_value=collector), \
patch.dict(os.environ, {"RUSTFS_ACCESS_KEY": "test", "RUSTFS_SECRET_KEY": "test"}), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(KeyboardInterrupt):
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
report = harness.read_json(self.root / "out/report.json")
for message in ("measurement failed", "collector cleanup failed", "stop interrupted"):
self.assertIn(message, report["error"])
def test_collector_interruption_and_stop_failure_keep_measurement_error(self):
manifest = self.measured_manifest()
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True, "collector": {"alias": "test", "endpoint": "test", "metrics_endpoints": "test"}}
if action == "measure":
raise ValueError("measurement failed")
if action == "stop":
raise ValueError("stop failed")
self.fail(f"unexpected action {action}")
collector = Mock()
collector.finish.side_effect = KeyboardInterrupt("collector interrupted")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
patch.object(harness, "OwnedCommand", return_value=collector), \
patch.dict(os.environ, {"RUSTFS_ACCESS_KEY": "test", "RUSTFS_SECRET_KEY": "test"}), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(KeyboardInterrupt):
harness.run(manifest, self.adapter, self.root / "out", self.root / "data")
report = harness.read_json(self.root / "out/report.json")
for message in ("measurement failed", "collector interrupted", "stop failed"):
self.assertIn(message, report["error"])
def test_subprocess_failures_do_not_expose_arguments(self):
sentinel = "TEST_ONLY_ARGUMENT_SENTINEL"
def invoke(_adapter, action, _request_path, _timeout):
if action == "prepare":
return {"ready": True}
if action == "measure":
raise subprocess.TimeoutExpired(["collector", sentinel], 5)
if action == "stop":
raise subprocess.CalledProcessError(7, ["adapter", sentinel])
self.fail(f"unexpected action {action}")
with patch.object(harness, "SCENARIOS", ("cold-hot",)), patch.object(harness, "invoke", side_effect=invoke), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(ValueError):
self.run_harness()
report = harness.read_json(self.root / "out/report.json")
self.assertFalse(sentinel in report["error"])
self.assertIn("timed out", report["error"])
self.assertIn("status 7", report["error"])
stderr = io.StringIO()
argv = ["scanner_abba.py", "--manifest", "unused", "--adapter", "unused", "--out-dir", "unused", "--data-root", "unused"]
with patch.object(sys, "argv", argv), patch.object(harness, "run", side_effect=subprocess.TimeoutExpired([sentinel], 5)), \
patch.object(harness, "read_json", return_value={}), contextlib.redirect_stderr(stderr):
self.assertEqual(harness.main(), 1)
self.assertFalse(sentinel in stderr.getvalue())
def test_unknown_release_evidence_values_are_not_echoed(self):
sentinel = "TEST_ONLY_UNTRUSTED_VALUE"
manifest = self.measured_manifest()
manifest["release_evidence"]["crash_restart"]["fault_modes"].append(sentinel)
with self.assertRaises(ValueError) as caught:
harness.validate_manifest(manifest)
self.assertFalse(sentinel in str(caught.exception))
def test_invalid_json_does_not_echo_filename(self):
sentinel = "TEST_ONLY_FILENAME_SENTINEL"
source = self.root / sentinel
source.write_text("[]", encoding="utf-8")
with self.assertRaises(ValueError) as caught:
harness.read_json(source)
self.assertFalse(sentinel in str(caught.exception))
def test_existing_data_preserved(self):
(self.root / "data").mkdir()
marker = self.root / "data/keep"
@@ -737,12 +1042,22 @@ class ScannerAbbaTest(unittest.TestCase):
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--stubborn-worker":
signal.signal(signal.SIGTERM, signal.SIG_IGN)
with Path(sys.argv[2]).open("w+") as marker:
fcntl.flock(marker, fcntl.LOCK_EX)
marker.write(str(os.getpid()))
marker.flush()
while True:
time.sleep(1)
marker = Path(sys.argv[2])
release = marker.with_name(marker.name + ".release")
with marker.open("w+") as stream:
fcntl.flock(stream, fcntl.LOCK_EX)
deadline = time.monotonic() + 30
ready = (repr(deadline) + ";ready").encode("ascii")
if os.write(stream.fileno(), ready) != len(ready):
raise OSError("short fixture marker write")
while time.monotonic() < deadline:
try:
if release.exists():
break
except OSError:
pass
time.sleep(0.05)
sys.exit(0)
if len(sys.argv) == 4 and sys.argv[1] in ("prepare", "measure", "oracle", "stop"):
sys.exit(fake_adapter())
unittest.main()
+283
View File
@@ -0,0 +1,283 @@
import copy
import hashlib
import json
import os
from pathlib import Path
import tempfile
import unittest
import uuid
from unittest import mock
import scanner_client_trace as trace
import scanner_abba
class LedgerTests(unittest.TestCase):
def setUp(self):
disk = mock.patch.object(trace.shutil, 'disk_usage', return_value=mock.Mock(free=100 * 1024**3))
disk.start()
self.addCleanup(disk.stop)
self.tmp = tempfile.TemporaryDirectory()
self.root = Path(self.tmp.name).resolve(); self.root.chmod(0o700)
self.clock = str(uuid.uuid4())
self.identity = trace.clock_identity()
self.start, self.dispatch_end, self.end = 1_000_000_000, 2_000_000_000, 2_100_000_000
self.request = {'duration_seconds':1, 'fixed':{'offered_load_ops':100}}
self.schedule, self.starts, self.samples = [], [], []
for index in range(100):
due = self.start + index * 10_000_000
began = due + 2
self.schedule.append({'request_id':str(index),'clock_id':self.clock,'scheduled_ns':due,'observed_ns':due+1,
'dispatched':True,'miss_reason':None})
self.starts.append({'request_id':str(index),'clock_id':self.clock,'scheduled_ns':due,'start_ns':began,'operation':'GET'})
self.samples.append({'request_id':str(index),'clock_id':self.clock,'start_ns':began,'end_ns':began+index+1,
'outcome':'success','operation':'GET','error_type':None,'http_status':200})
self.stats = {'planned_requests':100, 'started':100, 'completed':100, 'errors':0, 'missed_dispatches':0}
self.cancelled = False
def tearDown(self): self.tmp.cleanup()
def save(self):
refs = {}
for name, rows in (('samples',self.samples),('starts',self.starts),('schedule',self.schedule)):
path = self.root/(name+'.ndjson')
path.write_text(''.join(json.dumps(row,separators=(',',':'))+'\n' for row in rows))
raw = path.read_bytes()
refs[name] = {'path':path.name,'sha256':hashlib.sha256(raw).hexdigest(),'bytes':len(raw)}
return {'schema':2, **refs['samples'],'clock_id':self.clock,
'clock_before':copy.deepcopy(self.identity),'clock_after':copy.deepcopy(self.identity),
'window_start_ns':self.start,'dispatch_window_end_ns':self.dispatch_end,'window_end_ns':self.end,
'starts':refs['starts'],'schedule':refs['schedule']}
def compute(self):
return trace.analyze(self.root, self.save(), self.request, self.stats, self.cancelled)
def test_nearest_rank_includes_failure_and_exact_error_ratio(self):
self.samples[0].update(outcome='error',error_type='ClientError',http_status=503)
self.stats['errors'] = 1
result = self.compute()
self.assertTrue(result['complete'])
self.assertEqual((result['p95_ns'],result['p99_ns']), (95,99))
self.assertEqual(result['error_rate'], {'numerator':1,'denominator':100})
self.assertEqual(result['http_503'], 1)
self.assertEqual(result['observed_http_503_lower_bound'], 1)
self.assertFalse(result['same_source_window_proven'])
self.assertFalse(result['release_acceptance_ready'])
self.assertEqual(result['dispatch_window_end_ns'], self.dispatch_end)
def test_status_unavailable_is_not_zero_and_known_503_not_hidden(self):
self.samples[0].update(outcome='error',error_type='ClientError',http_status=503)
self.samples[1]['http_status'] = None
self.stats['errors'] = 1
result = self.compute()
self.assertIsNone(result['http_503'])
self.assertEqual(result['http_status_unknown'], 1)
self.assertEqual(result['observed_http_503_lower_bound'], 1)
self.assertEqual(result['error_rate'], {'numerator':1,'denominator':100})
def test_low_samples_and_missed_dispatch_do_not_produce_percentiles(self):
self.request['fixed']['offered_load_ops'] = 99
self.schedule.pop(); self.starts.pop(); self.samples.pop()
self.stats.update(planned_requests=99, started=99, completed=99)
from decimal import Decimal
for index,(sch,st,fin) in enumerate(zip(self.schedule,self.starts,self.samples)):
due=self.start + int(Decimal(index) * 1_000_000_000 / Decimal(99))
sch.update(scheduled_ns=due,observed_ns=due+1)
st.update(scheduled_ns=due,start_ns=due+2)
fin.update(start_ns=due+2,end_ns=due+3+index)
low = self.compute()
self.assertTrue(low['complete']); self.assertIsNone(low['p99_ns']); self.assertIsNone(low['p95_ns'])
self.request['fixed']['offered_load_ops'] = 100
for index,(sch,st,fin) in enumerate(zip(self.schedule,self.starts,self.samples)):
due=self.start+index*10_000_000; sch.update(scheduled_ns=due,observed_ns=due+1)
st.update(scheduled_ns=due,start_ns=due+2); fin.update(start_ns=due+2,end_ns=due+3+index)
due=self.start+990_000_000
self.schedule.append({'request_id':'99','clock_id':self.clock,'scheduled_ns':due,'observed_ns':due+1,
'dispatched':False,'miss_reason':'client_capacity'})
self.stats.update(planned_requests=100, started=99, completed=99, missed_dispatches=1)
missing = self.compute()
self.assertFalse(missing['complete']); self.assertIsNone(missing['p99_ns']); self.assertIsNone(missing['error_rate'])
def test_explicit_cancelled_full_or_partial_does_not_produce_percentile(self):
self.cancelled = True
result = self.compute()
self.assertFalse(result['complete']); self.assertIsNone(result['p99_ns'])
self.schedule = self.schedule[:8]; self.starts = self.starts[:8]; self.samples = self.samples[:8]
self.stats.update(started=8,completed=8)
result = self.compute()
self.assertFalse(result['complete']); self.assertIsNone(result['requests'])
def test_missed_dispatch_above_p99_sample_threshold_stays_unavailable(self):
self.request['fixed']['offered_load_ops'] = 200
self.schedule, self.starts, self.samples = [], [], []
for index in range(200):
due=self.start+index*5_000_000; dispatched=index != 199
self.schedule.append({'request_id':str(index),'clock_id':self.clock,'scheduled_ns':due,'observed_ns':due+1,
'dispatched':dispatched,'miss_reason':None if dispatched else 'client_capacity'})
if dispatched:
self.starts.append({'request_id':str(index),'clock_id':self.clock,'scheduled_ns':due,'start_ns':due+2,'operation':'GET'})
self.samples.append({'request_id':str(index),'clock_id':self.clock,'start_ns':due+2,'end_ns':due+3+index,
'outcome':'success','operation':'GET','error_type':None,'http_status':200})
self.stats.update(planned_requests=200,started=199,completed=199,missed_dispatches=1)
result=self.compute()
self.assertFalse(result['complete']); self.assertEqual(result['observed_completed'],199)
self.assertIsNone(result['p95_ns']); self.assertIsNone(result['p99_ns']); self.assertIsNone(result['error_rate'])
def test_no_ledger_truncation_or_missing_completion_accepted(self):
ref=self.save(); path=self.root/ref['path']; raw=path.read_bytes()[:-2]; path.write_bytes(raw)
ref.update(sha256=hashlib.sha256(raw).hexdigest(), bytes=len(raw))
with self.assertRaises(ValueError): trace.analyze(self.root,ref,self.request,self.stats,False)
self.samples.pop(); self.stats.update(started=99,completed=99)
with self.assertRaises(ValueError): self.compute()
def test_duplicate_mismatch_clock_and_cross_window_rejected(self):
changes = ('duplicate-start','missing-schedule','clock','cross-window','start-mismatch','duplicate-json','invalid-status','float-schedule')
for change in changes:
with self.subTest(change=change):
schedules,starts,samples=copy.deepcopy(self.schedule),copy.deepcopy(self.starts),copy.deepcopy(self.samples)
try:
if change=='duplicate-start': self.starts.append(copy.deepcopy(self.starts[0]))
elif change=='missing-schedule': self.schedule.pop()
elif change=='clock': self.samples[0]['clock_id']=str(uuid.uuid4())
elif change=='cross-window': self.samples[0]['end_ns']=self.end+1
elif change=='start-mismatch': self.samples[0]['start_ns']+=1
elif change=='invalid-status': self.samples[0]['http_status']=True
elif change=='float-schedule': self.schedule[0]['scheduled_ns']=float(self.schedule[0]['scheduled_ns'])
if change=='duplicate-json':
ref=self.save(); p=self.root/ref['path']; raw=p.read_bytes().replace(b'"outcome":"success"', b'"outcome":"success","outcome":"error"', 1)
p.write_bytes(raw); ref.update(sha256=hashlib.sha256(raw).hexdigest(),bytes=len(raw))
with self.assertRaises(ValueError): trace.analyze(self.root,ref,self.request,self.stats,False)
else:
with self.assertRaises(ValueError): self.compute()
finally:
self.schedule,self.starts,self.samples=schedules,starts,samples
def test_scheduled_window_crossing_is_disclosed_not_retimestamped(self):
self.samples[-1]['end_ns'] = self.dispatch_end + 5
result = self.compute()
self.assertEqual(result['crossing_dispatch_end'],1)
self.assertEqual(result['window_end_ns'],self.end)
self.assertEqual(result['dispatch_window_end_ns'],self.dispatch_end)
self.assertFalse(result['same_source_window_proven'])
def test_dispatch_after_its_slot_is_not_reported_as_on_time(self):
self.schedule[0]['observed_ns'] = self.start + 10_000_000
self.starts[0]['start_ns'] = self.schedule[0]['observed_ns'] + 1
self.samples[0].update(start_ns=self.starts[0]['start_ns'], end_ns=self.starts[0]['start_ns'] + 1)
with self.assertRaisesRegex(ValueError, 'late dispatch'):
self.compute()
def test_late_recorded_start_cannot_count_as_a_complete_offered_window(self):
due = self.schedule[0]['scheduled_ns']
self.starts[0]['start_ns'] = due + 10_000_000
self.samples[0].update(start_ns=self.starts[0]['start_ns'], end_ns=self.starts[0]['start_ns'] + 1)
result = self.compute()
self.assertTrue(result['sample_complete'])
self.assertEqual(result['late_request_starts'], 1)
self.assertFalse(result['scheduler_window_complete'])
self.assertIs(result['offered_load_complete'], False)
def test_refuses_ledger_through_symlinked_parent(self):
ref = self.save()
with tempfile.TemporaryDirectory() as outside:
other = Path(outside)
(other / 'sample.ndjson').write_bytes((self.root / ref['path']).read_bytes())
(self.root / 'alias').symlink_to(other, target_is_directory=True)
ref['path'] = 'alias/sample.ndjson'
with self.assertRaises((ValueError, OSError)):
trace.analyze(self.root, ref, self.request, self.stats, False)
def test_parent_replacement_cannot_redirect_open_ledger(self):
ref = self.save()
nested = self.root / 'client'
nested.mkdir(mode=0o700)
for name in ('samples.ndjson', 'starts.ndjson', 'schedule.ndjson'):
(self.root / name).rename(nested / name)
ref['path'] = 'client/' + ref['path']
for key in ('starts', 'schedule'):
ref[key]['path'] = 'client/' + ref[key]['path']
with tempfile.TemporaryDirectory() as outside:
outside_path = Path(outside)
for name in ('samples.ndjson', 'starts.ndjson', 'schedule.ndjson'):
(outside_path / name).write_bytes(b'wrong ledger\n')
original_open = trace.os.open
swapped = False
def open_with_replacement(path, flags, *args, **kwargs):
nonlocal swapped
if Path(path).name == 'schedule.ndjson' and not swapped:
swapped = True
nested.rename(self.root / 'held-client')
nested.symlink_to(outside_path, target_is_directory=True)
return original_open(path, flags, *args, **kwargs)
with mock.patch.object(trace.os, 'open', side_effect=open_with_replacement):
result = trace.analyze(self.root, ref, self.request, self.stats, False)
self.assertTrue(swapped)
self.assertEqual(result['p99_ns'], 99)
def test_refuses_reduction_without_scratch_budget(self):
with mock.patch.object(trace.shutil, 'disk_usage', return_value=mock.Mock(free=1)):
with self.assertRaisesRegex(ValueError, 'scratch'):
self.compute()
def test_refuses_nonsticky_shared_scratch_parent(self):
with tempfile.TemporaryDirectory() as tmp:
shared = Path(tmp) / 'shared'
shared.mkdir(mode=0o777)
shared.chmod(0o777)
with mock.patch.object(trace.tempfile, 'gettempdir', return_value=str(shared)):
with self.assertRaisesRegex(ValueError, 'scratch parent'):
self.compute()
def test_refuses_sticky_parent_owned_by_another_user(self):
with tempfile.TemporaryDirectory() as tmp:
shared = Path(tmp) / 'shared'
shared.mkdir(mode=0o700)
shared.chmod(0o1777)
shared = shared.resolve()
original_stat = Path.stat
def foreign_owner(path, *args, **kwargs):
info = original_stat(path, *args, **kwargs)
if path == shared:
return mock.Mock(st_mode=info.st_mode, st_uid=os.geteuid() + 1)
return info
with mock.patch.object(trace.tempfile, 'gettempdir', return_value=str(shared)), \
mock.patch.object(Path, 'stat', autospec=True, side_effect=foreign_owner):
with self.assertRaisesRegex(ValueError, 'scratch parent'):
self.compute()
def test_abba_does_not_accept_ledger_without_proven_offered_load(self):
ref = self.save()
observed = trace.analyze(self.root, ref, self.request, self.stats, False)
result = {'client_request_samples': ref, 'client_observation': observed,
'stats': self.stats, 'cancelled': False,
'elapsed_seconds': observed['elapsed_seconds'],
'metrics': {key: observed[key] for key in ('requests', 'errors', 'p95_ms', 'p99_ms', 'throughput_ops')}}
with self.assertRaisesRegex(ValueError, 'client schedule or percentile evidence is incomplete'):
scanner_abba.validate_client_request_samples(result, self.request, self.root)
def test_abba_does_not_accept_missing_or_tampered_ledger(self):
with self.assertRaisesRegex(ValueError, 'missing client request samples'):
scanner_abba.validate_client_request_samples({}, self.request, self.root)
ref = self.save()
(self.root / ref['path']).write_bytes(b'tampered\n')
with self.assertRaises(ValueError):
scanner_abba.validate_client_request_samples(
{'client_request_samples': ref, 'stats': self.stats, 'cancelled': False}, self.request, self.root)
def test_abba_does_not_turn_unknown_status_into_zero_503s(self):
ref = self.save()
observed = trace.analyze(self.root, ref, self.request, self.stats, False)
observed.update(offered_load_complete=True, http_503=None)
result = {'client_request_samples': ref, 'client_observation': observed,
'stats': self.stats, 'cancelled': False}
with mock.patch.object(scanner_abba, 'reconcile_client_ledger', return_value=observed):
with self.assertRaisesRegex(ValueError, 'status unavailable'):
scanner_abba.validate_client_request_samples(result, self.request, self.root)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,31 @@
#!/usr/bin/env python3
"""Tests for incomplete Scanner/Heal evidence plans."""
import tempfile
import unittest
from pathlib import Path
import run_scanner_heal_linux_evidence_plan as planner
class ScannerHealLinuxEvidencePlanTest(unittest.TestCase):
def test_p2_stays_pending_without_an_independent_oracle_producer(self):
steps = planner.performance_steps()
p2 = next(step for step in steps if step["id"] == "p2-independent-evidence")
plan = {"source_revision": "a" * 40, "stages": [{"id": "performance", "steps": [p2]}]}
with tempfile.TemporaryDirectory() as tmp:
status = planner.build_status(plan, Path(tmp))
self.assertEqual(status["decision"], "blocked")
self.assertIn("P2", status["pending_gates"])
self.assertTrue(status["next_step"]["reason"])
def test_scheduler_pressure_plan_does_not_claim_p2(self):
step = next(step for step in planner.performance_steps() if step["id"] == "scheduler-pressure")
self.assertNotIn("P2", step["covers"]["gates"])
command = step["commands"][0]["command"]
for flag in ("--started-at", "--finished-at", "--run-id", "--measurement-window-id"):
self.assertIn(flag, command)
if __name__ == "__main__":
unittest.main()
@@ -13,6 +13,7 @@ cleanup() {
trap cleanup EXIT
"${RUSTFS_PYTHON_BIN:-python3}" "$RUNNER" --self-test
"${RUSTFS_PYTHON_BIN:-python3}" "$PROJECT_ROOT/scripts/test_scanner_heal_linux_evidence_plan.py"
"${RUSTFS_PYTHON_BIN:-python3}" "$RUNNER" --phase preflight >"$TMP_DIR/preflight.out"
rg -q "stage=preflight" "$TMP_DIR/preflight.out"
@@ -0,0 +1,130 @@
#!/usr/bin/env python3
"""Regression tests for scheduler-pressure evidence production."""
import tempfile
import unittest
from pathlib import Path
import subprocess
from unittest import mock
import run_scanner_heal_scheduler_pressure_evidence as producer
class SchedulerPressureEvidenceTest(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.revision = producer.git_head()
self.abba, self.recovery, self.profiles = producer.write_self_test_abba(self.root, self.revision)
def args(self, original_window):
argv = ["--abba-dir", str(self.abba), "--recovery-window-json", str(self.recovery),
"--out-dir", str(self.root / "out")]
for kind, path in zip(producer.RELEASE_PROFILE_ARTIFACTS, self.profiles):
argv.extend(["--profile-artifact", f"{kind}={path}"])
if original_window:
argv.extend(["--started-at", "2026-01-01T00:00:00Z", "--finished-at", "2026-01-01T02:00:00Z",
"--run-id", "fixture-run-0001", "--measurement-window-id", "fixture-window-0001"])
return producer.parse_args(argv)
def observations(self):
return producer.self_test_observations(
self.revision, dict(zip(producer.RELEASE_PROFILE_ARTIFACTS, self.profiles)))
def test_missing_original_window_is_rejected(self):
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
mock.patch.object(producer.subprocess, "check_call"):
with self.assertRaisesRegex(ValueError, "original measurement window"):
producer.build_descriptor(self.args(original_window=False))
def test_object_counts_do_not_become_segment_oracle_evidence(self):
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
mock.patch.object(producer, "pressure_inputs_verified", return_value=self.observations()), \
mock.patch.object(producer.subprocess, "check_call"):
descriptor = producer.build_descriptor(self.args(original_window=True))
self.assertFalse("P2" in producer.read_json(descriptor)["gates"])
def test_unverified_pressure_inputs_are_rejected_after_abba_verification(self):
with mock.patch.object(producer, "measured_provenance_verified", return_value=True):
with self.assertRaisesRegex(ValueError, "independent verified pressure observations"):
producer.build_descriptor(self.args(original_window=True))
self.assertFalse((self.root / "out").exists())
def test_boolean_claim_cannot_create_pressure_measurements(self):
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
mock.patch.object(producer, "pressure_inputs_verified", return_value=True), \
mock.patch.object(producer.subprocess, "check_call"):
with self.assertRaisesRegex(ValueError, "verified pressure observations"):
producer.build_descriptor(self.args(original_window=True))
self.assertFalse((self.root / "out").exists())
def test_capacity_is_copied_from_verified_observation_without_fabrication(self):
observed = self.observations()
capacity = observed.fields["P3"]["heal_capacity_measurement"]
capacity["completed_heal_objects"] = 0
capacity["heal_capacity"] = {"objects": 0, "versions": 0, "bytes": 0, "completed_objects": 0}
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed), \
mock.patch.object(producer.subprocess, "check_call"):
descriptor = producer.build_descriptor(self.args(original_window=True))
result = producer.read_json(descriptor)["gates"]["P3"]["evidence_fields"]["heal_capacity_measurement"]
self.assertEqual(result["completed_heal_objects"], 0)
self.assertEqual(result["heal_capacity"], capacity["heal_capacity"])
def test_incomplete_verified_fields_do_not_create_output(self):
observed = self.observations()
observed.fields["P3"].pop("heal_capacity_measurement")
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed):
with self.assertRaisesRegex(ValueError, "observations are incomplete"):
producer.build_descriptor(self.args(original_window=True))
self.assertFalse((self.root / "out").exists())
def test_failed_checker_does_not_publish_a_pass_descriptor(self):
observed = self.observations()
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed), \
mock.patch.object(producer.subprocess, "check_call", side_effect=subprocess.CalledProcessError(1, ["fixture"])):
with self.assertRaises(subprocess.CalledProcessError):
producer.build_descriptor(self.args(original_window=True))
self.assertFalse((self.root / "out" / "release-bundle-scheduler-pressure.json").exists())
def test_profile_replacement_after_verification_is_rejected(self):
observed = self.observations()
self.profiles[0].write_bytes(b"changed profile")
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed):
with self.assertRaisesRegex(ValueError, "profile artifact differs"):
producer.build_descriptor(self.args(original_window=True))
self.assertFalse((self.root / "out" / "release-bundle-scheduler-pressure.json").exists())
def test_destination_corruption_is_rejected_before_publication(self):
observed = self.observations()
original_rename = producer.os.rename
def corrupt_after_move(source, destination):
original_rename(source, destination)
(destination / "P3-heal_capacity_measurement.json").write_text("corrupted")
def check_destination(command, **_kwargs):
descriptor = Path(command[-2])
self.assertEqual(descriptor.parent, (self.root / "out").resolve())
self.assertFalse((descriptor.parent / "release-bundle-scheduler-pressure.json").exists())
fields = producer.read_json(descriptor)["gates"][command[-1]]["evidence_fields"]
for field in fields.values():
if producer.digest(descriptor.parent / field["artifact"]) != field["sha256"]:
raise subprocess.CalledProcessError(1, command)
with mock.patch.object(producer, "measured_provenance_verified", return_value=True), \
mock.patch.object(producer, "pressure_inputs_verified", return_value=observed), \
mock.patch.object(producer.os, "rename", side_effect=corrupt_after_move), \
mock.patch.object(producer.subprocess, "check_call", side_effect=check_destination):
with self.assertRaises(subprocess.CalledProcessError):
producer.build_descriptor(self.args(original_window=True))
self.assertFalse((self.root / "out" / "release-bundle-scheduler-pressure.json").exists())
self.assertFalse((self.root / "out" / ".release-bundle-scheduler-pressure.pending.json").exists())
if __name__ == "__main__":
unittest.main()
@@ -6,3 +6,4 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
RUNNER="$SCRIPT_DIR/run_scanner_heal_scheduler_pressure_evidence.py"
"${RUSTFS_PYTHON_BIN:-python3}" "$RUNNER" --self-test
"${RUSTFS_PYTHON_BIN:-python3}" "$SCRIPT_DIR/test_scanner_heal_scheduler_pressure_evidence.py"
@@ -443,4 +443,6 @@ grep -q -- 'unknown arg: --secret-key' "$secret_arg_log"
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_service.py"
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_process_receipt.py"
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_abba.py"
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_client_trace.py"
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_summarize_scanner_heal_perf.py"
"$ROOT_DIR/scripts/python_bin.sh" "$ROOT_DIR/scripts/test_scanner_otlp_window.py"
+259 -36
View File
@@ -16,7 +16,6 @@ from unittest import mock
sys.path.insert(0, str(Path(__file__).resolve().parent))
import summarize_scanner_heal_perf as summary
import check_test_wiring as wiring
def sha(path: Path) -> str:
@@ -118,6 +117,9 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
"status": "pass",
"foreground_p95_ms": 8.0,
"foreground_p99_ms": 10.0,
"foreground_percentiles_ms_by_leg": {
leg: {"p95": 8.0, "p99": 10.0} for leg in summary.LEGS
},
"throughput_ops": 100.0,
"error_rate": 0.0,
"p99_regression": 0.02,
@@ -200,7 +202,8 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
(self.abba / "manifest.json").write_text(json.dumps(self.manifest), encoding="utf-8")
(self.abba / "report.json").write_text(json.dumps(self.report), encoding="utf-8")
def test_measured_pass_writes_quiet_artifacts(self):
@mock.patch.object(summary, "measured_provenance_verified", return_value=True)
def test_measured_pass_writes_quiet_artifacts(self, _verified):
cache_log = self.root / "cache.log"
cache_log.write_text(
"compiler noise\nCACHE_COST "
@@ -237,7 +240,8 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
self.assertIn("w11_running_heal_build_statuses: observed,observed,observed", summary.markdown(result))
self.assertEqual(result["cache_cost"]["max_save_body_amplification"], 2.0)
def test_release_descriptor_binds_g10_p1_p3_measured_artifacts(self):
@mock.patch.object(summary, "measured_provenance_verified", return_value=True)
def test_release_descriptor_rejects_unreconciled_comparison_percentiles(self, _verified):
profile_paths = []
for kind in summary.RELEASE_PROFILE_ARTIFACTS:
artifact = self.root / f"{kind}.artifact"
@@ -252,36 +256,13 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
"release_profile_artifact": profile_paths,
})
result = summary.build_summary(args)
summary.write_release_bundle_descriptor(args, result)
descriptor = summary.read_json(args.release_bundle_descriptor_out)
self.assertEqual(sorted(descriptor["gates"]), ["G10", "P1", "P3"])
self.assertEqual(
descriptor["gates"]["G10"]["evidence_fields"]["scheduler_bound_evidence"]["scheduler_bounds"],
list(summary.RELEASE_SCHEDULER_BOUNDS),
)
profile = descriptor["gates"]["P1"]["evidence_fields"]["profile_evidence"]
self.assertEqual(sorted(profile["profile_artifacts"]), sorted(summary.RELEASE_PROFILE_ARTIFACTS))
self.assertEqual(
profile["measurement_window_id"],
descriptor["gates"]["P3"]["evidence_fields"]["two_hour_pressure_measurement"]["measurement_window_id"],
)
for gate in ("G10", "P1", "P3"):
with mock.patch("subprocess.check_output", return_value="b" * 40):
status = wiring.scanner_heal_release_bundle_gate_status(
Path(__file__).resolve().parents[1],
args.release_bundle_descriptor_out,
gate,
)
self.assertEqual(status["verified_gate"], gate)
save_frequency = profile["profile_artifacts"]["save-frequency"]
wrapper = args.release_bundle_descriptor_out.parent / save_frequency["artifact"]
payload = summary.read_json(wrapper)
raw_profile = args.release_bundle_descriptor_out.parent / payload["raw_profile_artifact"]
self.assertTrue(raw_profile.is_file())
self.assertEqual(payload["saved_bytes"], 8192)
self.assertEqual(payload["raw_profile_sha256"], sha(raw_profile))
with self.assertRaisesRegex(ValueError, "source evidence is unavailable"):
summary.write_release_bundle_descriptor(args, result)
self.assertFalse(args.release_bundle_descriptor_out.exists())
self.assertFalse((self.root / "release-descriptor-artifacts").exists())
def test_release_descriptor_requires_profile_artifacts(self):
@mock.patch.object(summary, "measured_provenance_verified", return_value=True)
def test_release_descriptor_stays_closed_without_profile_artifacts(self, _verified):
args = type("Args", (), {
"abba_dir": self.abba,
"cache_cost_log": None,
@@ -290,8 +271,31 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
"release_source_revision": "b" * 40,
"release_profile_artifact": [],
})
with self.assertRaisesRegex(ValueError, "missing profile artifacts"):
with self.assertRaisesRegex(ValueError, "source evidence is unavailable"):
summary.write_release_bundle_descriptor(args, summary.build_summary(args))
self.assertFalse(args.release_bundle_descriptor_out.exists())
def test_self_reported_measured_pass_has_unverified_provenance(self):
args = type("Args", (), {
"abba_dir": self.abba,
"cache_cost_log": None,
"require_cache_cost": False,
"release_bundle_descriptor_out": self.root / "release-descriptor.json",
"release_source_revision": "b" * 40,
"release_profile_artifact": [],
})
for claim in (None, "verified"):
with self.subTest(claim=claim):
if claim is not None:
self.report["measurement_provenance"] = claim
self.write_inputs()
result = summary.build_summary(args)
self.assertEqual(result["verdict"], "FAIL")
self.assertEqual(result["abba"]["measurement_provenance"], "unverified")
self.assertIn("not independently verified", result["reason"])
self.assertTrue(all(row["status"] == "unverified" for row in result["abba"]["w11_running_heal_build"]))
with self.assertRaisesRegex(ValueError, "requires a measured PASS"):
summary.write_release_bundle_descriptor(args, result)
def test_synthetic_report_fails_as_performance_conclusion(self):
self.report.update(status="synthetic_validated", performance="pending", evidence="synthetic")
@@ -329,9 +333,122 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
self.assertIsNone(result["abba"]["error_rate"])
self.assertIsNone(result["abba"]["foreground_pressure_samples"])
self.assertIsNone(result["abba"]["foreground_pressure_high_samples"])
self.assertIn("collector failed", result["reason"])
self.assertIn("error details omitted", result["reason"])
self.assertIn("- completed_cells: 7", summary.markdown(result))
self.assertIn("- error: collector failed", summary.markdown(result))
self.assertIn("- error: details omitted", summary.markdown(result))
def test_historical_error_does_not_echo_raw_command(self):
sentinel = "TEST_ONLY_PRIVATE_ARG"
self.report = {"status": "failed", "performance": "pending", "evidence": "diagnostic",
"completed_cells": 7, "error": f"Command ['client', '{sentinel}'] timed out"}
self.write_inputs()
args = type("Args", (), {"abba_dir": self.abba, "cache_cost_log": None,
"require_cache_cost": False, "json_out": None, "markdown_out": None})
result = summary.build_summary(args)
self.assertEqual(result["verdict"], "FAIL")
self.assertNotIn(sentinel, json.dumps(result))
self.assertNotIn(sentinel, summary.markdown(result))
out, err = io.StringIO(), io.StringIO()
with mock.patch.object(sys, "argv", ["summarize_scanner_heal_perf.py", "--abba-dir", str(self.abba)]), \
contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
self.assertEqual(summary.main(), 1)
self.assertNotIn(sentinel, out.getvalue() + err.getvalue())
def test_unknown_report_labels_are_not_echoed(self):
sentinel = "TEST_ONLY_PRIVATE_LABEL"
self.report = {"status": sentinel, "performance": sentinel, "evidence": sentinel, "comparisons": []}
self.write_inputs()
result = summary.summarize_abba(self.abba)
self.assertEqual(result["gate_state"], "fail")
self.assertEqual(result["status"], "unknown")
self.assertEqual(result["performance"], "unknown")
self.assertEqual(result["evidence"], "unknown")
self.assertNotIn(sentinel, json.dumps(result))
def test_unverified_provenance_does_not_echo_endpoints_or_flags(self):
sentinel = "TEST_ONLY_PRIVATE_ENDPOINT_OR_ARG"
self.report = {"status": "failed", "performance": "pending", "evidence": "diagnostic", "comparisons": []}
self.manifest["fixed"]["release_flags"] = f"--token={sentinel}"
self.manifest["fixed"]["durability"] = sentinel
self.manifest["baseline"]["revision"] = sentinel
self.manifest["started_at"] = sentinel
self.manifest["release_evidence"]["distributed"]["metrics_endpoints"] = [f"https://{sentinel}@node.example"]
self.write_inputs()
args = type("Args", (), {"abba_dir": self.abba, "cache_cost_log": None,
"require_cache_cost": False, "json_out": None, "markdown_out": None})
result = summary.build_summary(args)
self.assertEqual(result["verdict"], "FAIL")
self.assertNotIn(sentinel, json.dumps(result))
self.assertNotIn(sentinel, summary.markdown(result))
def test_unverified_comparison_labels_do_not_echo_free_text(self):
sentinel = "TEST_ONLY_PRIVATE_COMPARISON"
self.report.update(status="inconclusive", performance="pending")
self.report["comparisons"][0]["scenario"] = sentinel
self.report["comparisons"][0]["p1"]["baseline_walk_objects"] = sentinel
row = next(row for row in self.report["comparisons"]
if row["scenario"] == "running-heal" and row["comparison"] == "build")
row["w10"]["status"] = sentinel
row["w10"]["candidate_delay_events"] = sentinel
row["w11"]["status"] = sentinel
row["w11"]["rss_growth"] = sentinel
self.write_inputs()
result = summary.summarize_abba(self.abba)
self.assertEqual(result["gate_state"], "fail")
self.assertNotIn(sentinel, json.dumps(result))
def test_cache_cost_labels_do_not_echo_free_text(self):
sentinel = "TEST_ONLY_PRIVATE_CACHE_LABEL"
self.report = {"status": "failed", "performance": "pending", "evidence": "diagnostic", "comparisons": []}
self.write_inputs()
cache_log = self.root / "cache.log"
cache_log.write_text("CACHE_COST " + json.dumps({"schema": 1, "scenario": sentinel,
"cache_wire_bytes": 100, "save_body_bytes_per_sample": 200,
"clone": {"max_ns": 10}, "encode": {"max_ns": 20}, "save_inclusive": {"max_ns": 30},
"build": {"source_revision": sentinel, "source_tree": sentinel,
"test_opt_level_override": sentinel}}) + "\n")
args = type("Args", (), {"abba_dir": self.abba, "cache_cost_log": cache_log,
"require_cache_cost": False, "json_out": None, "markdown_out": None})
result = summary.build_summary(args)
self.assertNotIn(sentinel, json.dumps(result))
self.assertNotIn(sentinel, summary.markdown(result))
def test_cli_error_does_not_echo_input_path(self):
sentinel = "TEST_ONLY_PRIVATE_PATH"
out, err = io.StringIO(), io.StringIO()
with mock.patch.object(sys, "argv", ["summarize_scanner_heal_perf.py", "--abba-dir", str(self.root / sentinel)]), \
contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
self.assertEqual(summary.main(), 1)
self.assertNotIn(sentinel, out.getvalue() + err.getvalue())
def test_unknown_cell_counts_are_not_zero_or_free_text(self):
sentinel = "TEST_ONLY_PRIVATE_COUNT"
for cells, completed in ((None, None), (sentinel, sentinel), ("absent", "absent")):
with self.subTest(cells=cells):
self.report = {"status": "failed", "performance": "pending", "evidence": "diagnostic",
"cells": cells, "completed_cells": completed, "comparisons": []}
if cells == "absent":
self.report.pop("cells")
self.report.pop("completed_cells")
self.write_inputs()
result = summary.summarize_abba(self.abba)
self.assertIsNone(result["cells"])
self.assertIsNone(result["completed_cells"])
self.assertNotIn(sentinel, json.dumps(result))
def test_rejected_descriptor_does_not_write_summary_outputs(self):
json_out = self.root / "summary.json"
markdown_out = self.root / "summary.md"
descriptor = self.root / "descriptor.json"
argv = ["summarize_scanner_heal_perf.py", "--abba-dir", str(self.abba),
"--json-out", str(json_out), "--markdown-out", str(markdown_out),
"--release-bundle-descriptor-out", str(descriptor)]
with mock.patch.object(sys, "argv", argv), contextlib.redirect_stdout(io.StringIO()), \
contextlib.redirect_stderr(io.StringIO()):
self.assertEqual(summary.main(), 1)
self.assertFalse(json_out.exists())
self.assertFalse(markdown_out.exists())
self.assertFalse(descriptor.exists())
def test_partial_error_rate_is_unknown(self):
self.report.update(status="inconclusive", performance="pending")
@@ -344,6 +461,111 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
self.write_inputs()
self.assertIsNone(summary.summarize_abba(self.abba)["error_rate"])
def test_partial_w09_measurements_remain_unknown(self):
self.report.update(status="inconclusive", performance="pending")
fields = {"heal_start_p95_ms": "w09_worst_heal_start_p95_ms",
"heal_duplicate_task_count": "w09_duplicate_task_count",
"heal_lock_hold_p95_ms": "w09_worst_lock_hold_p95_ms"}
for field, output in fields.items():
for missing in (None, "short", "absent"):
with self.subTest(field=field, missing=missing):
original = copy.deepcopy(self.report["comparisons"][0]["w09"])
if missing is None:
self.report["comparisons"][0]["w09"][field][1] = None
elif missing == "short":
self.report["comparisons"][0]["w09"][field] = [0]
else:
self.report["comparisons"][0]["w09"].pop(field)
self.write_inputs()
self.assertIsNone(summary.summarize_abba(self.abba)[output])
self.report["comparisons"][0]["w09"] = original
self.report["comparisons"].pop()
self.write_inputs()
incomplete = summary.summarize_abba(self.abba)
for output in fields.values():
self.assertIsNone(incomplete[output])
self.assertIsNone(incomplete["worst_p99_regression"])
self.assertIsNone(incomplete["worst_throughput_loss"])
self.assertIsNone(incomplete["throughput_ops"])
self.assertIsNone(incomplete["max_reported_comparison_throughput_ops"])
def test_partial_candidate_p2_measurements_remain_unknown(self):
self.report.update(status="inconclusive", performance="pending")
for missing in ("b1", "b2", "short", "absent", "comparison"):
with self.subTest(missing=missing):
original = copy.deepcopy(self.report["comparisons"])
row = self.report["comparisons"][0]
if missing == "b1":
row["p2_post_stop_work_multiples"][1] = None
elif missing == "b2":
row["p2_post_stop_work_multiples"][2] = None
elif missing == "short":
row["p2_post_stop_work_multiples"] = [None, 1.0]
elif missing == "absent":
row.pop("p2_post_stop_work_multiples")
else:
self.report["comparisons"].pop()
self.write_inputs()
self.assertIsNone(summary.summarize_abba(self.abba)["p2_worst_post_stop_work_multiple"])
self.report["comparisons"] = original
def test_unverified_reported_percentiles_are_not_a_global_percentile(self):
self.report.update(status="inconclusive", performance="pending")
self.write_inputs()
result = summary.summarize_abba(self.abba)
self.assertIsNone(result["foreground_p95_ms"])
self.assertIsNone(result["foreground_p99_ms"])
self.assertIsNone(result["throughput_ops"])
self.assertEqual(result["max_reported_comparison_throughput_ops"], 100.0)
def test_passing_report_requires_each_leg_percentile(self):
self.report["comparisons"][0]["foreground_percentiles_ms_by_leg"].pop("B2")
self.write_inputs()
with self.assertRaisesRegex(ValueError, "foreground percentiles"):
summary.summarize_abba(self.abba)
def test_passing_report_rejects_invalid_leg_percentiles(self):
for p95, p99 in ((-1, 10), (1, float("inf")), (float("nan"), 10), (11, 10)):
with self.subTest(p95=p95, p99=p99):
self.report["comparisons"][0]["foreground_percentiles_ms_by_leg"]["B2"] = {"p95": p95, "p99": p99}
self.write_inputs()
with self.assertRaisesRegex(ValueError, "foreground percentiles"):
summary.summarize_abba(self.abba)
def test_nonfinite_report_metrics_are_rejected(self):
self.report.update(status="inconclusive", performance="pending")
for value in (float("inf"), float("nan")):
with self.subTest(value=value):
self.report["comparisons"][0]["p99_regression"] = value
self.write_inputs()
with self.assertRaisesRegex(ValueError, "invalid numeric field"):
summary.summarize_abba(self.abba)
def test_per_leg_attempt_ratios_are_not_attempt_samples(self):
self.report.update(status="inconclusive", performance="pending")
self.write_inputs()
result = summary.summarize_abba(self.abba)
self.assertIsNone(result["attempt_cost_samples"])
self.assertGreater(result["reported_attempt_cost_ratio_count"], 0)
self.report["comparisons"][0]["w10_w11"].pop("attempt_cost_per_healed_object")
self.write_inputs()
self.assertIsNone(summary.summarize_abba(self.abba)["reported_attempt_cost_ratio_count"])
def test_error_rate_cannot_be_negative_or_exceed_one(self):
self.report.update(status="inconclusive", performance="pending")
for value in (-0.1, 1.1):
with self.subTest(value=value):
self.report["comparisons"][0]["error_rate"] = value
self.write_inputs()
with self.assertRaisesRegex(ValueError, "error_rate"):
summary.summarize_abba(self.abba)
def test_passing_report_cannot_contain_request_errors(self):
self.report["comparisons"][0]["error_rate"] = 0.01
self.write_inputs()
with self.assertRaisesRegex(ValueError, "error_rate"):
summary.summarize_abba(self.abba)
def test_partial_pressure_samples_are_unknown(self):
self.report.update(status="inconclusive", performance="pending")
self.report["comparisons"][0]["w10_w11"].pop("foreground_pressure_samples")
@@ -563,7 +785,8 @@ class ScannerHealPerfSummaryTest(unittest.TestCase):
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
self.assertEqual(process.returncode, 0, process.stderr)
self.assertEqual(process.returncode, 1, process.stderr)
self.assertTrue(process.stdout.startswith("FAIL scanner_heal_perf "))
self.assertEqual(process.stdout.count("\n"), 1)
self.assertEqual(process.stderr, "")