Files
pad/internal/store/dashboard_batch_test.go
T
xarmian 3c93472613 perf(store): run the store suite in parallel (TASK-2900 deliverable 3) (#1263)
The SQLite leg was CPU-bound and serial: 154.9s of CPU inside 185.6s of wall on
an eight-core box, which is 0.84 cores. Unlike the Postgres leg there was no
setup cost to reclaim — the unattributed gap between package wall-clock and the
sum of its tests measured MINUS 1.4s, so per-binary overhead was already nil —
and no fat test to find beyond one 48s outlier. The only lever left was the
seven idle cores.

t.Parallel() on 844 of 906 tests. Every test in this package builds its own
isolated store (a SQLite file copy, or since 6f8b6eb9 a Postgres clone), so it
shares no database with any other.

Medians on a quiet box, three SQLite repeats and two Postgres:

  SQLite wall     185.6s -> 111.0s   -40%
  Postgres wall   224.4s -> 121.2s   -46%   (495.8s -> 121.2s, -76%, since 6f8b6eb9)
  -race wall     2006.3s -> 1365.0s  -32%

THE COST, stated rather than folded into the win: SQLite CPU rose 154.9s ->
159.9s, +3.2%. That is contention overhead, it is real, and it is paid by
whoever else is on the box — this change moves WALL, it does not remove work.
A pre-registered prediction that CPU would stay within ~1% is falsified by that
number, though only just.

THAT FIGURE IS ITSELF A CORRECTION. An earlier draft of this message reported
the CPU cost as +10.4%, from ONE run taken while another seat was working the
box. Three repeats on a quiet box put the median at +3.2% — the published
number overstated the cost by more than three times. The direction survived the
correction and the magnitude did not, which is the case for repeating a figure
you intend to publish as a cost just as carefully as one you intend to publish
as a win.

Postgres binary CPU is flat to slightly down (63.9s -> 63.1s), which fits the
mechanism rather than contradicting it: that leg was 82% waiting, so parallelism
fills idle time instead of competing for cycles.

VARIANCE, since it changed shape: the parallel SQLite leg's repeat spread is
5.9% (106.7 / 111.0 / 113.2) against 0.7% for the serial leg it replaces.
Parallel runs are genuinely less reproducible than serial ones — scheduling
decides which tests share a core — so a future before/after on this package
needs more than one run per side to say anything, which the serial suite did
not.

The -race figure was measured because it was a RISK, not for completeness. CI's
Go job runs `go test -race -timeout=45m ./...` and already takes ~42 minutes;
the race detector serialises heavily, so parallelism could plausibly have made
that leg slower and pushed CI toward its own timeout. It does not: 2006s ->
1365s on the same box, measured against the template-only tree.

EXCLUSIONS — 62 tests, each for a mechanism rather than caution:

  19  files measuring ALLOCATIONS (runtime.ReadMemStats). Those counters are
      process-global, so a concurrent test's allocations land in the
      measurement. Excluded by FILE, since the hazard is the process.
   7  migration_guard_test.go, every test of which mutates the EXPORTED
      globals store.AllowSchemaAhead or store.BinaryVersion and restores in a
      t.Cleanup. AllowSchemaAhead=true under parallelism disarms the
      schema-ahead guard for every concurrently-running test — which is the
      property two of those very tests exist to assert.
  23  tests measuring ELAPSED TIME. Wall-clock under contention is not the
      wall-clock the assertion was written against.
   1  test mutating a PACKAGE-LEVEL GLOBAL (see below).
   3  the template mechanism tests, which assert on package-level counters.
  16  already had t.Parallel().

Excluding a test is a real remedy here rather than a hope, and the reason is a
Go semantic worth naming: a top-level test that calls t.Parallel() is HELD and
resumed only after the sequential pass finishes. Every excluded test therefore
still runs alone in a quiet process, which is exactly the property its timing or
allocation assertion depends on.

TWO EXCLUSIONS WERE MISSED BY MY OWN SWEEP, and the shape is the same both
times — an instrument whose question was narrower than the claim resting on it.

FIRST (found before shipping, by grepping the package's own `var` lines):


I recorded that this package has "no per-test global setters", having grepped
t.Setenv, os.Setenv and Set*ForTesting — all genuinely zero. The counts were
true and the conclusion was false. That sweep asked whether a test changes the
ENVIRONMENT; the claim it was carrying was whether a test changes anything
GLOBAL, which is wider.

TestUpdateDocument_CascadeExhaustionRollsBackTheWholeRename assigns the
package-level cascadeRewriteAttempts and restores it in a defer. Under
t.Parallel that window stays open while every other cascade test runs. The
mutating test would not flake; the tests AROUND it would, intermittently, which
is the worst shape to debug. Found by a different question — grep the package's
own non-test `var` declarations, then grep the tests for each name.

SECOND (found by codex, P1, NOT by me): that grep was `^var [a-z]`. It finds
UNEXPORTED package vars and silently skips exported ones, so it never saw
store.AllowSchemaAhead or store.BinaryVersion — both mutated by every test in
migration_guard_test.go. The instrument asked "which unexported globals exist"
while the claim it carried was "which globals exist". The fix that caught the
first miss was itself too narrow to catch the second, which is the argument for
enforcing the invariant in code rather than trusting the sweep that produced it.

Both exclusions are now enforced by guards rather than by the exclusion list:
TestCascadeExhaustionStaysSerial (function-scoped) and
TestMigrationGuardTestsStaySerial (file-scoped). Both fail closed on a missing
file or renamed function, strip line comments so an explanatory comment naming
t.Parallel() can neither satisfy nor trip them, carry a positive control that
the scan can SEE a real t.Parallel() elsewhere, and are mutation-controlled in
both directions.

VERIFICATION — 10 runs, zero failures, zero data races:

  SQLite parallel    x4   exit 0
  Postgres parallel  x3   exit 0
  -race -timeout=45m x1   exit 0, zero DATA RACE reports
  baseline -race     x1   exit 0 (the counterfactual above)

A false alarm not filed as a finding: the first -race run exited 1 with zero
`--- FAIL` lines and zero data races — the shape that reads as a pass to
anything counting FAIL lines. The log said `panic: test timed out after 10m0s`.
That was the invocation, not the code: `-timeout` was omitted so Go's default
applied, and ci.yml already carries a comment explaining that the explicit
-timeout exists because the default is too low.

KNOWN BOUND, not a solved problem: the test container's max_connections is 100,
each store sets MaxOpenConns(25), and -parallel defaults to GOMAXPROCS. On an
eight-core box the worst case permits 200. Three Postgres runs showed no
53300 "too many clients", but the arithmetic still allows it on a larger box or
a heavier test, and that belongs here as a bound rather than as an absence of
observed failures.

Claude-Session: https://claude.ai/code/session_01HeChkgZVYb3NTgTcckF5KR
2026-09-06 16:59:31 -04:00

268 lines
8.2 KiB
Go

package store
import (
"testing"
"github.com/PerpetualSoftware/pad/internal/models"
)
// linkChild wires child -> parent via a `parent` link (the shape
// GetChildItems / GetChildItemsForParents walk).
func linkChild(t *testing.T, s *Store, workspaceID, childID, parentID string) {
t.Helper()
if _, err := s.CreateItemLink(workspaceID, models.ItemLinkCreate{
TargetID: parentID,
LinkType: "parent",
}, childID); err != nil {
t.Fatalf("link child %s -> parent %s: %v", childID, parentID, err)
}
}
// linkBlocks wires blocker -> blocked via a `blocks` link.
func linkBlocks(t *testing.T, s *Store, workspaceID, blockerID, blockedID string) {
t.Helper()
if _, err := s.CreateItemLink(workspaceID, models.ItemLinkCreate{
TargetID: blockedID,
LinkType: "blocks",
}, blockerID); err != nil {
t.Fatalf("link blocker %s -> blocked %s: %v", blockerID, blockedID, err)
}
}
func TestGetChildItemsForParents(t *testing.T) {
t.Parallel()
s := testStore(t)
ws := createTestWorkspace(t, s, "Test")
col := createTestCollection(t, s, ws.ID, "Tasks")
planA := createTestItem(t, s, ws.ID, col.ID, "Plan A", "")
planB := createTestItem(t, s, ws.ID, col.ID, "Plan B", "")
planC := createTestItem(t, s, ws.ID, col.ID, "Plan C (no children)", "")
a1 := createTestItem(t, s, ws.ID, col.ID, "A child 1", "body-a1")
a2 := createTestItem(t, s, ws.ID, col.ID, "A child 2", "body-a2")
b1 := createTestItem(t, s, ws.ID, col.ID, "B child 1", "body-b1")
linkChild(t, s, ws.ID, a1.ID, planA.ID)
linkChild(t, s, ws.ID, a2.ID, planA.ID)
linkChild(t, s, ws.ID, b1.ID, planB.ID)
byParent, err := s.GetChildItemsForParents([]string{planA.ID, planB.ID, planC.ID})
if err != nil {
t.Fatalf("GetChildItemsForParents: %v", err)
}
if got := len(byParent[planA.ID]); got != 2 {
t.Errorf("plan A: expected 2 children, got %d", got)
}
if got := len(byParent[planB.ID]); got != 1 {
t.Errorf("plan B: expected 1 child, got %d", got)
}
if _, ok := byParent[planC.ID]; ok {
t.Errorf("plan C: expected no map entry for childless parent")
}
// Grouping must match the per-parent GetChildItems (same rows).
single, err := s.GetChildItems(planA.ID)
if err != nil {
t.Fatalf("GetChildItems: %v", err)
}
if len(single) != len(byParent[planA.ID]) {
t.Errorf("batch child count %d != single %d for plan A", len(byParent[planA.ID]), len(single))
}
// The projection omits content (heavy body column) but keeps identity +
// computed ref/fields.
for _, child := range byParent[planA.ID] {
if child.Content != "" {
t.Errorf("expected empty content in batch projection, got %q for %s", child.Content, child.Title)
}
if child.Ref == "" {
t.Errorf("expected computed ref on child %s", child.Title)
}
}
}
func TestGetChildItemsForParentsExcludesDeleted(t *testing.T) {
t.Parallel()
s := testStore(t)
ws := createTestWorkspace(t, s, "Test")
col := createTestCollection(t, s, ws.ID, "Tasks")
plan := createTestItem(t, s, ws.ID, col.ID, "Plan", "")
live := createTestItem(t, s, ws.ID, col.ID, "Live child", "")
gone := createTestItem(t, s, ws.ID, col.ID, "Deleted child", "")
linkChild(t, s, ws.ID, live.ID, plan.ID)
linkChild(t, s, ws.ID, gone.ID, plan.ID)
if err := s.DeleteItem(gone.ID); err != nil {
t.Fatalf("DeleteItem: %v", err)
}
byParent, err := s.GetChildItemsForParents([]string{plan.ID})
if err != nil {
t.Fatalf("GetChildItemsForParents: %v", err)
}
if got := len(byParent[plan.ID]); got != 1 {
t.Fatalf("expected 1 live child, got %d", got)
}
if byParent[plan.ID][0].ID != live.ID {
t.Errorf("expected live child, got %s", byParent[plan.ID][0].Title)
}
}
func TestGetChildItemsForParentsEmpty(t *testing.T) {
t.Parallel()
s := testStore(t)
byParent, err := s.GetChildItemsForParents(nil)
if err != nil {
t.Fatalf("GetChildItemsForParents(nil): %v", err)
}
if len(byParent) != 0 {
t.Errorf("expected empty map, got %d entries", len(byParent))
}
}
func TestGetBlocksEdges(t *testing.T) {
t.Parallel()
s := testStore(t)
ws := createTestWorkspace(t, s, "Test")
col := createTestCollection(t, s, ws.ID, "Tasks")
blocker := createTestItem(t, s, ws.ID, col.ID, "Blocker", "")
blocked := createTestItem(t, s, ws.ID, col.ID, "Blocked", "")
// A non-blocks link must NOT appear in the result.
parent := createTestItem(t, s, ws.ID, col.ID, "Parent", "")
child := createTestItem(t, s, ws.ID, col.ID, "Child", "")
linkBlocks(t, s, ws.ID, blocker.ID, blocked.ID)
linkChild(t, s, ws.ID, child.ID, parent.ID)
edges, err := s.GetBlocksEdges(ws.ID)
if err != nil {
t.Fatalf("GetBlocksEdges: %v", err)
}
if len(edges) != 1 {
t.Fatalf("expected 1 blocks edge, got %d: %+v", len(edges), edges)
}
e := edges[0]
if e.TargetID != blocked.ID {
t.Errorf("expected target %s (blocked), got %s", blocked.ID, e.TargetID)
}
if e.SourceID != blocker.ID {
t.Errorf("expected source %s (blocker), got %s", blocker.ID, e.SourceID)
}
if e.SourceTitle != "Blocker" {
t.Errorf("expected source title 'Blocker', got %q", e.SourceTitle)
}
if e.SourceCollectionID != col.ID {
t.Errorf("expected source collection %s, got %s", col.ID, e.SourceCollectionID)
}
}
func TestGetBlocksEdgesExcludesDeleted(t *testing.T) {
t.Parallel()
s := testStore(t)
ws := createTestWorkspace(t, s, "Test")
col := createTestCollection(t, s, ws.ID, "Tasks")
blocker := createTestItem(t, s, ws.ID, col.ID, "Blocker", "")
blocked := createTestItem(t, s, ws.ID, col.ID, "Blocked", "")
linkBlocks(t, s, ws.ID, blocker.ID, blocked.ID)
// Soft-deleting either endpoint drops the edge (join requires both live).
if err := s.DeleteItem(blocker.ID); err != nil {
t.Fatalf("DeleteItem: %v", err)
}
edges, err := s.GetBlocksEdges(ws.ID)
if err != nil {
t.Fatalf("GetBlocksEdges: %v", err)
}
if len(edges) != 0 {
t.Errorf("expected 0 edges after deleting blocker, got %d", len(edges))
}
}
func TestGetItemsByIDsIncludeDeleted(t *testing.T) {
t.Parallel()
s := testStore(t)
ws := createTestWorkspace(t, s, "Test")
col := createTestCollection(t, s, ws.ID, "Tasks")
live := createTestItem(t, s, ws.ID, col.ID, "Live", "body")
gone := createTestItem(t, s, ws.ID, col.ID, "Archived", "body")
if err := s.DeleteItem(gone.ID); err != nil {
t.Fatalf("DeleteItem: %v", err)
}
got, err := s.GetItemsByIDsIncludeDeleted([]string{live.ID, gone.ID, "missing-id"})
if err != nil {
t.Fatalf("GetItemsByIDsIncludeDeleted: %v", err)
}
// Both live and soft-deleted must be present; missing ID absent.
if got[live.ID] == nil {
t.Errorf("expected live item present")
}
if got[gone.ID] == nil {
t.Errorf("expected soft-deleted item present (include-deleted)")
}
if _, ok := got["missing-id"]; ok {
t.Errorf("expected missing id absent")
}
if got[live.ID] != nil {
if got[live.ID].Title != "Live" {
t.Errorf("expected title 'Live', got %q", got[live.ID].Title)
}
if got[live.ID].Ref == "" {
t.Errorf("expected computed ref on hydrated item")
}
if got[live.ID].Content != "" {
t.Errorf("expected empty content in projection, got %q", got[live.ID].Content)
}
}
}
func TestGetItemsByIDsIncludeDeletedEmpty(t *testing.T) {
t.Parallel()
s := testStore(t)
got, err := s.GetItemsByIDsIncludeDeleted(nil)
if err != nil {
t.Fatalf("GetItemsByIDsIncludeDeleted(nil): %v", err)
}
if len(got) != 0 {
t.Errorf("expected empty map, got %d", len(got))
}
}
func TestListItemsNoContentProjection(t *testing.T) {
t.Parallel()
s := testStore(t)
ws := createTestWorkspace(t, s, "Test")
col := createTestCollection(t, s, ws.ID, "Tasks")
createTestItem(t, s, ws.ID, col.ID, "Has body", "the full markdown body")
// Default: content loaded.
full, err := s.ListItems(ws.ID, models.ItemListParams{})
if err != nil {
t.Fatalf("ListItems: %v", err)
}
if len(full) != 1 || full[0].Content != "the full markdown body" {
t.Fatalf("expected content loaded by default, got %+v", full)
}
// NoContent: body omitted, everything else intact.
skinny, err := s.ListItems(ws.ID, models.ItemListParams{NoContent: true})
if err != nil {
t.Fatalf("ListItems NoContent: %v", err)
}
if len(skinny) != 1 {
t.Fatalf("expected 1 item, got %d", len(skinny))
}
if skinny[0].Content != "" {
t.Errorf("expected empty content with NoContent, got %q", skinny[0].Content)
}
if skinny[0].Title != "Has body" {
t.Errorf("expected title preserved, got %q", skinny[0].Title)
}
}