mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 12:21:31 +00:00
test(ci): TEST-003 — flip Frontend E2E from informational to merge-gate
Sprint 5 unified-master-audit closure. The Phase 8 E2E workflow at
.github/workflows/e2e.yml shipped with continue-on-error: true and
a header banner that said it would be promoted to required-for-merge
once 1-2 weeks of green runs accumulated. The accumulation happened;
the flip didn't.
Ground-truth via api.github.com/repos/certctl-io/certctl/actions/runs
(2026-05-16): 14 consecutive green runs across 2026-05-14 to
2026-05-15 (heaviest Sprint 1-4 frontend churn in the repo's history,
6 commits touching web/**) confirmed the suite is stable. No flakes,
no flaps, no timeouts.
Fix:
- .github/workflows/e2e.yml continue-on-error: true → false.
- Workflow name strips the '(informational)' tag.
- Header banner rewritten to reflect the new posture + flag the
one operator action still required (adding the job to the
branch-protection required-checks list at
https://github.com/certctl-io/certctl/settings/branches).
- New docs/operator/runbooks/e2e-snapshot-update.md documents the
visual-regression snapshot-bump workflow now that a red E2E
run blocks merge. Includes the standard (one or two affected
tests) + mass-bump (font upgrade / framework migration) paths,
plus an explicit anti-patterns section (do NOT regenerate from
a developer's local machine; do NOT add --update-snapshots to
the always-run step).
Closes TEST-003.
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
# Runbook: regenerating Playwright visual-regression snapshots
|
||||
|
||||
> Last reviewed: 2026-05-16
|
||||
|
||||
Use this when:
|
||||
|
||||
- You've intentionally changed UI shape (added a column, restyled a
|
||||
banner, replaced an icon set) and the next `Frontend E2E` CI run
|
||||
fails with `Screenshot comparison failed:` errors on multiple
|
||||
`04-visual-regression.spec.ts` cases.
|
||||
- A deterministic-but-platform-specific font-rendering difference
|
||||
emerges (Linux runner vs your Mac dev box) and you want to refresh
|
||||
baselines from the canonical CI environment.
|
||||
|
||||
TEST-003 closure (Sprint 5, 2026-05-16) flipped the workflow from
|
||||
`continue-on-error: true` to `false`. Pre-fix you could ignore a
|
||||
red E2E run and ship anyway. Post-fix the run blocks the merge, so
|
||||
any change that legitimately moves pixels needs the snapshot bump
|
||||
captured here.
|
||||
|
||||
Do NOT use this to make a real visual regression disappear. The
|
||||
snapshots are version-controlled evidence — if a pixel diff fires
|
||||
unexpectedly, investigate the rendering change before bumping.
|
||||
|
||||
## What "snapshots" means here
|
||||
|
||||
`web/playwright/04-visual-regression.spec.ts` calls
|
||||
`toHaveScreenshot()`. Playwright stores the canonical PNG at
|
||||
`web/playwright/04-visual-regression.spec.ts-snapshots/<test-name>-<browser>-<platform>.png`
|
||||
on first run. Subsequent runs compare pixel-by-pixel against that
|
||||
file. We commit the PNGs to git so the CI runner and local dev
|
||||
share a single source of truth.
|
||||
|
||||
Two failure modes the diff is designed to catch:
|
||||
|
||||
- **Intentional UI change.** You added a new field to the Targets
|
||||
table. The screenshot now has an extra column. The baseline
|
||||
doesn't. Pixel diff fires — this is the "operator updates
|
||||
baselines" path documented below.
|
||||
- **Regression.** A CSS change inadvertently shifted spacing.
|
||||
Investigate before regenerating; don't paper over the diff.
|
||||
|
||||
## Standard bump (one or two affected tests)
|
||||
|
||||
1. Run the E2E suite locally with the update flag against the
|
||||
same Linux runner image Playwright uses:
|
||||
|
||||
```bash
|
||||
cd web
|
||||
npx playwright test 04-visual-regression.spec.ts --update-snapshots
|
||||
```
|
||||
|
||||
If you're on macOS, run it through Docker against the same image
|
||||
the workflow uses (`mcr.microsoft.com/playwright`); font
|
||||
rendering differs between platforms and Linux baselines must
|
||||
come from a Linux source.
|
||||
|
||||
2. Inspect every regenerated PNG:
|
||||
|
||||
```bash
|
||||
git status web/playwright/*.spec.ts-snapshots/
|
||||
git diff --stat web/playwright/*.spec.ts-snapshots/
|
||||
```
|
||||
|
||||
PNG diffs in `git diff` are unhelpful — open the files in any
|
||||
image viewer and confirm the change matches your intent.
|
||||
|
||||
3. Commit the snapshots alongside the source change in the same
|
||||
PR:
|
||||
|
||||
```bash
|
||||
git add web/playwright/*.spec.ts-snapshots/
|
||||
git commit -m "chore(e2e): refresh visual snapshots after <change>"
|
||||
```
|
||||
|
||||
4. Push and confirm CI's E2E job greens out.
|
||||
|
||||
## Mass bump (font upgrade, framework migration)
|
||||
|
||||
Use the workflow's `workflow_dispatch` input to regenerate from
|
||||
CI's canonical environment:
|
||||
|
||||
1. Go to `Actions` → `Frontend E2E` → `Run workflow`.
|
||||
2. Set `update_snapshots: true`.
|
||||
3. The workflow runs Playwright with `--update-snapshots`, then
|
||||
commits + pushes the regenerated PNGs to a feature branch
|
||||
`playwright/snapshot-update-<run-id>`.
|
||||
4. Open a PR from that branch to master. Review the PNG diffs in
|
||||
the PR view (GitHub renders image diffs side-by-side for
|
||||
committed PNGs).
|
||||
5. Merge.
|
||||
|
||||
## What NOT to do
|
||||
|
||||
- Don't regenerate snapshots from a developer's local machine and
|
||||
push them as the canonical baseline. The Linux runner's font
|
||||
hinting differs from macOS / Windows, so the baselines must come
|
||||
from the same image the CI workflow runs.
|
||||
- Don't add `--update-snapshots` to the always-run e2e step in
|
||||
`.github/workflows/e2e.yml`. That's how snapshot regressions
|
||||
become invisible — every diff gets accepted, every PR ships
|
||||
fine, and the visual-regression layer becomes decorative.
|
||||
- Don't bump snapshots in a "fix typo" PR. Every PNG change is
|
||||
an architectural decision; pair it with the source change that
|
||||
justifies it.
|
||||
Reference in New Issue
Block a user