mirror of https://github.com/shankar0123/certctl.git synced 2026-06-07 16:01:30 +00:00

Files

T

shankar0123 3e09401502 test(ci): TEST-003 — flip Frontend E2E from informational to merge-gate

Sprint 5 unified-master-audit closure. The Phase 8 E2E workflow at
.github/workflows/e2e.yml shipped with continue-on-error: true and
a header banner that said it would be promoted to required-for-merge
once 1-2 weeks of green runs accumulated. The accumulation happened;
the flip didn't.

Ground-truth via api.github.com/repos/certctl-io/certctl/actions/runs
(2026-05-16): 14 consecutive green runs across 2026-05-14 to
2026-05-15 (heaviest Sprint 1-4 frontend churn in the repo's history,
6 commits touching web/**) confirmed the suite is stable. No flakes,
no flaps, no timeouts.

Fix:
  - .github/workflows/e2e.yml continue-on-error: true → false.
  - Workflow name strips the '(informational)' tag.
  - Header banner rewritten to reflect the new posture + flag the
    one operator action still required (adding the job to the
    branch-protection required-checks list at
    https://github.com/certctl-io/certctl/settings/branches).
  - New docs/operator/runbooks/e2e-snapshot-update.md documents the
    visual-regression snapshot-bump workflow now that a red E2E
    run blocks merge. Includes the standard (one or two affected
    tests) + mass-bump (font upgrade / framework migration) paths,
    plus an explicit anti-patterns section (do NOT regenerate from
    a developer's local machine; do NOT add --update-snapshots to
    the always-run step).

Closes TEST-003.

2026-05-16 05:19:38 +00:00

3.9 KiB

Raw Permalink Blame History

Runbook: regenerating Playwright visual-regression snapshots

Last reviewed: 2026-05-16

Use this when:

You've intentionally changed UI shape (added a column, restyled a banner, replaced an icon set) and the next Frontend E2E CI run fails with Screenshot comparison failed: errors on multiple 04-visual-regression.spec.ts cases.
A deterministic-but-platform-specific font-rendering difference emerges (Linux runner vs your Mac dev box) and you want to refresh baselines from the canonical CI environment.

TEST-003 closure (Sprint 5, 2026-05-16) flipped the workflow from continue-on-error: true to false. Pre-fix you could ignore a red E2E run and ship anyway. Post-fix the run blocks the merge, so any change that legitimately moves pixels needs the snapshot bump captured here.

Do NOT use this to make a real visual regression disappear. The snapshots are version-controlled evidence — if a pixel diff fires unexpectedly, investigate the rendering change before bumping.

What "snapshots" means here

web/playwright/04-visual-regression.spec.ts calls toHaveScreenshot(). Playwright stores the canonical PNG at web/playwright/04-visual-regression.spec.ts-snapshots/<test-name>-<browser>-<platform>.png on first run. Subsequent runs compare pixel-by-pixel against that file. We commit the PNGs to git so the CI runner and local dev share a single source of truth.

Two failure modes the diff is designed to catch:

Intentional UI change. You added a new field to the Targets table. The screenshot now has an extra column. The baseline doesn't. Pixel diff fires — this is the "operator updates baselines" path documented below.
Regression. A CSS change inadvertently shifted spacing. Investigate before regenerating; don't paper over the diff.

Standard bump (one or two affected tests)

Run the E2E suite locally with the update flag against the same Linux runner image Playwright uses:
```
cd web
npx playwright test 04-visual-regression.spec.ts --update-snapshots
```
If you're on macOS, run it through Docker against the same image the workflow uses (mcr.microsoft.com/playwright); font rendering differs between platforms and Linux baselines must come from a Linux source.
Inspect every regenerated PNG:
```
git status web/playwright/*.spec.ts-snapshots/
git diff --stat web/playwright/*.spec.ts-snapshots/
```
PNG diffs in git diff are unhelpful — open the files in any image viewer and confirm the change matches your intent.

Commit the snapshots alongside the source change in the same PR:

git add web/playwright/*.spec.ts-snapshots/
git commit -m "chore(e2e): refresh visual snapshots after <change>"

Push and confirm CI's E2E job greens out.

Mass bump (font upgrade, framework migration)

Use the workflow's workflow_dispatch input to regenerate from CI's canonical environment:

Go to Actions → Frontend E2E → Run workflow.
Set update_snapshots: true.
The workflow runs Playwright with --update-snapshots, then commits + pushes the regenerated PNGs to a feature branch playwright/snapshot-update-<run-id>.
Open a PR from that branch to master. Review the PNG diffs in the PR view (GitHub renders image diffs side-by-side for committed PNGs).
Merge.

What NOT to do

Don't regenerate snapshots from a developer's local machine and push them as the canonical baseline. The Linux runner's font hinting differs from macOS / Windows, so the baselines must come from the same image the CI workflow runs.
Don't add --update-snapshots to the always-run e2e step in .github/workflows/e2e.yml. That's how snapshot regressions become invisible — every diff gets accepted, every PR ships fine, and the visual-regression layer becomes decorative.
Don't bump snapshots in a "fix typo" PR. Every PNG change is an architectural decision; pair it with the source change that justifies it.

3.9 KiB Raw Permalink Blame History