v6.2.0-rc.5 shipped with an empty Proxmox workloads table and a crashing
Thresholds page (#1663) while the assertions that catch exactly that were
failing in CI: spec 64 red in the non-gating probation tier two hours
before the tag, and the gating Core E2E verdict red on the release commit
itself — which the release pipeline never consults. integration_tests is
also skipped entirely for prereleases, so the builds users test shipped
with no integration coverage at all.
Close the hole with a release_smoke job that runs for every cut,
prereleases included, and blocks create_release and the release verdict:
four interaction-free render assertions (Proxmox nodes+workloads, Docker
hosts+containers, Kubernetes clusters+pods, Alert thresholds) against the
mock-mode image built from the verified frontend bundle. The mock fixture
graph now always contains one freshly provisioned zero-used guest
filesystem, so the omitted-zero-numerics wire shape that crashed rc.5
stays exercised on every mock-backed surface. Verified locally: the suite
passes on main and fails on the rc.5 frontend for exactly the two shipped
regressions.
The Build and Test workflow ran everything in one serial job — frontend
lint/tests/build, the full go test -race sweep, and benchmarks — for a
37-55 minute wall clock on every push. The job now splits into parallel
jobs: frontend (lint, unit tests, type-check, bundle build and size gate),
backend tests sharded three ways with internal/api on its own shard since
it alone takes ~10m under -race, script smoke tests with the backend
build, and benchmarks with the existing baseline cache flow. Backend jobs
satisfy the frontend embed with a stub index.html, which unit tests do
not assert against; the real bundle is still built and size-gated by the
frontend job and by release builds. Expected wall clock is roughly the
slowest single job, 10-15 minutes.
A new changes job skips the build/test jobs when a push touches only
docs/ and Markdown, mirroring the path filters Core E2E already has. The
gitleaks secret scan deliberately stays unconditional since this workflow
is the only per-push scan and docs can leak secrets too.
scripts/dev-prepush.sh runs the checks that most often turn main red,
scoped to the outgoing commits: the canonical completion guard in CI
mode, the registry snapshot tests when registry.json changed, the
mutation registry audits, build plus tests for touched Go packages, and
the frontend type-check. It exists because CI verdicts arrive well after
the push; today's unclassified force-fail route would have failed it
locally in about a minute.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep macOS notarization mandatory for every release candidate while requiring Windows Authenticode only for stable promotion, matching the publish workflow and RC4 release packet.
Mined per-spec failure data from the 32 completed Core E2E main runs
since the 2026-07-18 quarantine delist (failed-shard logs; a green
shard means every spec in it passed). 26 specs failed or retry-flaked
within the 10 most recent completed runs; they seed PROBATION_SPECS in
tests/integration/playwright.config.ts. The remaining 62 files form the
stable tier and are the only specs that can fail the e2e-verdict job.
Mechanism extends the existing quarantine list rather than adding a
parallel one: PULSE_E2E_TIER=stable ignores probation specs,
PULSE_E2E_TIER=probation runs only them, unset runs the full suite
(local behavior unchanged). CI runs both tiers per shard against the
same containers; the probation pass sits behind continue-on-error with
its own report/results dirs and artifacts, so a probation flake is
reported in the shard summary without painting main red.
Promotion rule, documented next to the list: a probation spec promotes
to stable after 10 consecutive green main runs with no failure and no
retry-flake; one incident on main demotes a stable spec back. In the
newest completed run (29729544151) every failure was in a probation
spec — under this split that run's verdict is green.
Contract-Neutral: CI-only E2E tier split: gating semantics of push-CI verdict; local npm test behavior and dev-runtime orchestration unchanged, no public contract delta
Playwright HTML reports uploaded on every run at 30-day retention blew
the Actions storage quota (69.8 GB on 2026-07-20). Reports now upload
only when a shard fails, and all report/video/screenshot artifacts
keep a 3-day retention.
Contract-Neutral: RC4 gate repair only: responsive layout correction, internal test hook, and E2E-only runtime target; no public subsystem contract changes
cancel-in-progress meant a busy main never completed a verdict: four
consecutive runs were cancelled by follow-up pushes this afternoon, so
the workflow produced neither green nor red for hours. Adopt the Core
E2E concurrency policy instead and let the in-progress run finish while
queued runs collapse to the newest pending one.
The 'Prove Patrol lifecycle and permission boundaries' step includes
./internal/api, whose frontend_embed.go declares
'//go:embed all:frontend-modern/dist'. CI checkouts never contain that
gitignored build output, so the package failed to compile and the step
died before a single lifecycle test ran (last four runs all red this
way). These proofs exercise Go logic only, so a one-file stub dist
satisfies the embed without paying for a frontend build.
Canonical Governance failed on nearly every push today (6 of the last 8
commits) because the local pre-commit hook and the CI workflow disagreed
about the canonical completion guard in two ways.
Frontend-only commits skipped the guard entirely: the hook gated ALL
governance checks behind governance-path detection as a perf
optimization, but subsystem contracts name canonical frontend files, so
those commits landed locally and failed the same guard in CI. The hook
now always runs the (cheap) completion guard; only the multi-minute Go
test and audit battery stays path-gated.
Contract-neutral bypasses did not travel: PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT
authorized the commit in the local shell and vanished, so CI re-ran the
guard without it and went red. A new prepare-commit-msg hook records the
reason as a Contract-Neutral trailer in the commit message, and the
workflow now evaluates the guard per commit (each commit's file list
against its own parent), honoring the trailer exactly as the commit-time
hook honored the env var. Per-commit evaluation also stops a compliant
commit from being blamed for range-mates.
The new hook is registered as a worktree-sensitive governance file in
governance_stage_guard.py with a matching unit test assert.
A maintenance cut of an older line (v5.1.36 after v6 GA, or a future
6.0.x patch after 6.1 ships) was allowed to move Docker/GHCR :latest and
the GitHub latest release marker onto itself, silently downgrading every
install that follows latest. Promote :MAJOR and :MAJOR.MINOR
unconditionally, but :latest and make_latest only when the tag is the
highest stable semver. workflow_dispatch gains force_latest as the
explicit rollback escape hatch.
The guard judged substantive contract updates by diffing HEAD against
the index. In CI nothing is staged, the index equals HEAD, so every
contract file piped in via --files-from-stdin looked unchanged and the
guard blocked compliant pushes. Concretely, run 28944317805 blocked
7645965af even though its deployment-installability.md addition sits
inside the Current State section.
The guard now accepts --diff-base <ref> (requires --files-from-stdin),
resolves it to its merge base with HEAD so the comparison anchor
matches the three-dot changed-file list, and compares base vs HEAD
contract texts in that mode. Pre-commit keeps the index comparison.
The canonical-governance workflow passes the push or PR range base.
The weekly release-dry-run schedule failed at 'Resolve rehearsal
metadata' because GitHub does not apply workflow_dispatch input
defaults to schedule events, so rollback_version arrived empty and
resolve_release_promotion.py rejected the run.
Scheduled runs now pass --derive-rollback-latest-stable, which fills
an empty rollback_version with the latest stable tag preceding the
rehearsal version (currently v6.0.4 for 6.0.5-rc.3). Manual dispatches
keep the explicit rollback_version requirement; the stale prefilled
5.1.29 default is removed so operators state the target themselves.
The deployment-installability contract records the scoped scheduled
exception.
Agents push to main every few minutes and a sharded run takes about 30,
so cancel-in-progress meant a busy main could never complete a verdict.
With cancel-in-progress off, the in-flight run finishes and GitHub
collapses queued runs to the newest pending one, so intermediate pushes
still skip without killing the run that is about to report.
rcourtman