feat(relations): materialised reverse index and Referenced-by (PLAN-2857 U5 / TASK-2997) (#1328)

PLAN-2857 unit U5, closes TASK-2997.

A MATERIALISED reverse index for `relation` field values —
`item_relation_links` per the ratified Q3 shape — plus a write hook, a
startup backfill, a viewer-scoped query, an endpoint, and the
"Referenced by N" section.

THE HOOK LIVES AT THE STORE'S FIELD-WRITING SITES, not at the doors the
row named. A door-level hook is a per-door wiring claim over a population
that GROWS; a store-level one cannot be bypassed by a door that does not
exist yet, and it is the shape migration 061 set for the content index.
The sites were enumerated with a grep rather than by eye, and two of them
are not in U1's resolver population at all.

Three placements are not obvious from the call sites: UPDATE re-reads the
blob FROM THE ROW, because a field write is a server-side MERGE and
indexing the submission would drop every key the caller did not mention;
MOVE indexes against the DESTINATION collection, the one write where blob
and schema change together; and the attachment remap calls the hook even
though it cannot change a relation value, because omitting it would be a
standing claim about what that function can touch.

A SCHEMA CHANGE REINDEXES THE COLLECTION (lead ruling), inside the
schema-update transaction and under the workspace lock, gated on the
schema or the values actually moving — an icon edit must not pay it.
Measured: 947ms-1.8s for 10k items, which is why it is inline and not a
job. The backfill batches by workspace in bounded chunks: 10k items in
1.06s across ~20 transactions, where one-per-item would have been 10,000.

THE COUNT IS THE VIEWER'S. ResolveBacklinksVisibility reused verbatim, so
"Referenced by 3" means "by 3 you can see" — a true count leaks the
existence of items the viewer cannot open. Filtering is in SQL so LIMIT
counts visible rows.

EIGHT CODEX ROUNDS, ELEVEN P1s; the first seven each found something
real. The ones worth keeping:

  - the backfill's guard was "does the table have any row", and rows
    commit per item, so an interrupted pass left it non-empty and
    INCOMPLETE and skipped forever. Completion is recorded explicitly,
    after the work.
  - RESTORE is an eighth hook site. I enumerated writes of one COLUMN;
    the index depends on (blob, schema, LIVENESS). That miss produced the
    input matrix on TASK-2997's trail, which round 4 then validated row
    by row.
  - the backfill SKIPPED zero-edge items as an optimisation — precisely
    the case that preserves a stale row and then marks the pass complete.
  - a source in a SOFT-DELETED COLLECTION was disclosed; its items stay
    live and an Unrestricted viewer has no collection filter to catch it.
  - rounds 3 and 4 were the same miss at two scopes: I locked what I was
    READING and not what whose change invalidates the read. Round 3 also
    refuted a claim I had put in a commit message, and the record was
    corrected rather than left standing.
  - the field-label lookup added in round 6 BECAME round 7's P1: a
    per-collection SELECT while the rows cursor was open, which is
    BUG-2409's deadlock class. One query now, with a MaxOpenConns(1)
    guard that the restored shape kills.

Three times a comment described code that had changed under it,
including one claiming a counterfactual TEST that does not exist — the
counterfactual is a mutation, recorded per round with its verdict.

Q3'S COUNTERFACTUAL, re-run after every rewrite: hook removed, mutant
COMPILES, all legs fail. Per-site too, because "all sites wired" is
otherwise untested.

NOT COVERED, said plainly: the Postgres interleavings are argued from the
isolation level and the lock order, not demonstrated — SQLite serialises
writes and cannot express the race. Two hook sites are uncovered, one
unfalsifiable by construction and one with no caller.

BUG-3017 files the pre-existing attachment-remap lost update this
branch's workspace lock closes, so it is on the record rather than
absorbed here.

Gates: go test ./... 0, vet 0, gofmt clean, lint 0 issues, svelte-check 0
errors, make test-pg 0 with all 17 relation tests confirmed run BY NAME
on Postgres. All 8 CI checks green on 8c0fe775.

Claude-Session: https://claude.ai/code/session_01PRcuviXka4Un1FGrBrX8jc
This commit is contained in:
xarmian
2026-09-11 00:57:47 -04:00
committed by GitHub
parent 162eccb42b
commit 63adbebb76
19 changed files with 2413 additions and 8 deletions
+17
View File
@@ -177,6 +177,23 @@ func serveCmd() *cobra.Command {
"items_scanned", bf.ItemsScanned, "errors", bf.Errors) "items_scanned", bf.ItemsScanned, "errors", bf.Errors)
} }
// The same job for `relation` field values (PLAN-2857 U5).
// Non-fatal for the same reason: a database without the reverse
// index is a database missing a "Referenced by" section, not a
// broken one, and refusing to boot over it would be the worse
// failure.
if rb, err := s.BackfillRelationLinks(); err != nil {
slog.Warn("relation-link backfill failed; non-fatal", "error", err)
} else if rb.LinksInserted > 0 {
slog.Info("Relation-link backfill complete",
"items_scanned", rb.ItemsScanned,
"links_inserted", rb.LinksInserted,
)
} else if !rb.Skipped {
slog.Debug("Relation-link backfill found nothing to index",
"items_scanned", rb.ItemsScanned)
}
// Backfill: populate status_transitions from the historical // Backfill: populate status_transitions from the historical
// activity log (PLAN-1628 / TASK-1637). Idempotent — gated on an // activity log (PLAN-1628 / TASK-1637). Idempotent — gated on an
// empty table, so it replays history exactly once on the first // empty table, so it replays history exactly once on the first
@@ -0,0 +1,104 @@
package server
import (
"net/http"
"strconv"
"github.com/go-chi/chi/v5"
"github.com/PerpetualSoftware/pad/internal/store"
)
// handleGetItemRelationBacklinks answers "which items point at this one
// through a relation field?" (PLAN-2857 U5 / TASK-2997).
//
// Separate from handleGetItemBacklinks rather than folded into it. The two
// answer different questions — a `[[wiki-link]]` in prose is not a typed
// field-valued edge — and they have different shapes: a relation backlink
// carries the FIELD KEY it points through and has no snippet, because a field
// value has no surrounding text to quote. Merging them would mean a response
// where half the entries have a snippet and half have a field key, and every
// consumer branching on which.
//
// THE COUNT IS THE VIEWER'S (ratified, day 55). Visibility comes from the same
// guestResourceFilter the wiki backlinks use, so "referenced by 3" means "by 3
// you can see". A true count would leak the existence of items the viewer has
// no access to.
//
// No cross-workspace tier, unlike wiki backlinks: a relation value names an
// item id, and PLAN-2857 v1 excludes cross-workspace relation targets — a
// carried value is dropped at the boundary rather than resolved. There is
// nothing foreign to page through.
func (s *Server) handleGetItemRelationBacklinks(w http.ResponseWriter, r *http.Request) {
workspaceID, ok := s.getWorkspaceID(w, r)
if !ok {
return
}
itemSlug := chi.URLParam(r, "itemSlug")
// IncludeDeleted, matching the wiki handler: a soft-deleted item still has
// a page, and "who pointed at this?" is a question that outlives it.
item, err := s.store.ResolveItemIncludeDeleted(workspaceID, itemSlug)
if err != nil {
writeInternalError(w, err)
return
}
if item == nil {
writeError(w, http.StatusNotFound, "not_found", "Item not found")
return
}
if !s.requireItemVisible(w, r, workspaceID, item) {
return
}
limit := 50
if v := r.URL.Query().Get("limit"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
limit = n
}
}
if limit > 300 {
limit = 300
}
offset := 0
if v := r.URL.Query().Get("offset"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n >= 0 {
offset = n
}
}
fullCollIDs, grantedItemIDs, gErr := s.guestResourceFilter(r, workspaceID)
if gErr != nil {
writeInternalError(w, gErr)
return
}
vis := store.BacklinksVisibility{Unrestricted: fullCollIDs == nil && grantedItemIDs == nil}
if !vis.Unrestricted {
vis.FullCollectionIDs = fullCollIDs
vis.GrantedItemIDs = grantedItemIDs
}
total, err := s.store.CountRelationBacklinks(item.ID, workspaceID, vis)
if err != nil {
writeInternalError(w, err)
return
}
links, err := s.store.GetRelationBacklinks(item.ID, workspaceID, limit, offset, vis)
if err != nil {
writeInternalError(w, err)
return
}
if links == nil {
links = []store.RelationBacklink{}
}
writeJSON(w, http.StatusOK, map[string]any{
"relation_backlinks": links,
// `total` is what the header renders as "Referenced by N". It is
// computed under the SAME visibility and the SAME filters as the page,
// so the header cannot promise a number the list will not produce.
"total": total,
"limit": limit,
"offset": offset,
})
}
+113
View File
@@ -0,0 +1,113 @@
package server
import (
"encoding/json"
"net/http"
"testing"
"github.com/PerpetualSoftware/pad/internal/models"
)
// The "Referenced by N" door (PLAN-2857 U5 / TASK-2997).
//
// The store tests own the index's correctness. These own the two things only
// the door can answer: that the endpoint is wired to the index at all, and
// that the number it reports is the VIEWER'S — which is the ratified
// visibility decision, and the one with a disclosure consequence.
type relationBacklinksResponse struct {
RelationBacklinks []struct {
SourceItemID string `json:"source_item_id"`
SourceRef string `json:"source_ref"`
SourceTitle string `json:"source_title"`
FieldKey string `json:"field_key"`
} `json:"relation_backlinks"`
Total int `json:"total"`
}
// getRelationBacklinks drives the endpoint as `user` and decodes the payload.
func (f *doorFixture) getRelationBacklinks(user *models.User, role string, target *models.Item) relationBacklinksResponse {
f.t.Helper()
var rr = f.callAs(user, role, f.srv.handleGetItemRelationBacklinks, "GET",
"/api/v1/workspaces/"+f.ws.Slug+"/items/"+target.Slug+"/relation-backlinks",
map[string]string{"itemSlug": target.Slug}, nil)
if rr.Code != http.StatusOK {
f.t.Fatalf("relation-backlinks: expected 200, got %d: %s", rr.Code, rr.Body.String())
}
var out relationBacklinksResponse
if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil {
f.t.Fatalf("decode: %v\nbody: %s", err, rr.Body.String())
}
return out
}
// TestRelationBacklinksDoor_ReportsTheEdgeWithItsFieldKey is the wiring claim:
// the endpoint reads the index, and it carries the field key that makes a
// relation backlink different from a wiki one.
func TestRelationBacklinksDoor_ReportsTheEdgeWithItsFieldKey(t *testing.T) {
f := newDoorFixture(t)
source := f.seed(`{"owner_ref":"` + f.target.ID + `"}`)
got := f.getRelationBacklinks(f.owner, "owner", f.target)
if got.Total != 1 {
t.Fatalf("total = %d, want 1 — the endpoint is not reading the index", got.Total)
}
if len(got.RelationBacklinks) != 1 {
t.Fatalf("want 1 backlink, got %d", len(got.RelationBacklinks))
}
b := got.RelationBacklinks[0]
if b.SourceItemID != source.ID {
t.Errorf("source = %s, want %s", b.SourceItemID, source.ID)
}
if b.FieldKey != "owner_ref" {
t.Errorf("field_key = %q, want %q — without it the panel cannot say WHICH field points here, which is the whole reason this is not a wiki backlink", b.FieldKey, "owner_ref")
}
if b.SourceTitle == "" || b.SourceRef == "" {
t.Errorf("the backlink lost the source's identity: %+v", b)
}
}
// TestRelationBacklinksDoor_TheCountIsTheViewersNotTheTruth is the ratified
// visibility decision, and the reason it needs a test is that the "wrong"
// behaviour here looks more correct: reporting the true count is what a
// naive implementation does, and it leaks the existence of items the viewer
// cannot see.
func TestRelationBacklinksDoor_TheCountIsTheViewersNotTheTruth(t *testing.T) {
f := newDoorFixture(t)
// Two sources point at the same target. The restricted viewer is granted
// exactly one of them.
visibleSource := f.seed(`{"owner_ref":"` + f.target.ID + `"}`)
f.seed(`{"owner_ref":"` + f.target.ID + `"}`) // the one they cannot see
if got := f.getRelationBacklinks(f.owner, "owner", f.target); got.Total != 2 {
t.Fatalf("the OWNER should see both (total %d); without that the restricted leg proves nothing", got.Total)
}
blind := mustUser(t, f.srv, "relation-backlinks-blind@example.com", "relbacklinksblind", "")
if err := f.srv.store.AddWorkspaceMember(f.ws.ID, blind.ID, "editor"); err != nil {
t.Fatalf("AddWorkspaceMember: %v", err)
}
// Can see the target's collection, and among the SOURCES only the one
// granted below.
if err := f.srv.store.SetMemberCollectionAccess(f.ws.ID, blind.ID, "specific",
[]string{f.people.ID}); err != nil {
t.Fatalf("SetMemberCollectionAccess: %v", err)
}
if _, err := f.srv.store.CreateItemGrant(f.ws.ID, visibleSource.ID, blind.ID, "read", f.owner.ID); err != nil {
t.Fatalf("CreateItemGrant: %v", err)
}
got := f.getRelationBacklinks(blind, "editor", f.target)
if got.Total != 1 {
t.Errorf("total = %d, want 1 — the count must be the VIEWER'S; reporting 2 tells them an item they cannot see exists", got.Total)
}
if len(got.RelationBacklinks) != 1 || got.RelationBacklinks[0].SourceItemID != visibleSource.ID {
t.Errorf("the page leaked a source the viewer cannot see: %+v", got.RelationBacklinks)
}
// The count and the page must agree. A header promising a number the list
// cannot produce is the drift CountBacklinks documents for the wiki side.
if got.Total != len(got.RelationBacklinks) {
t.Errorf("total %d but %d rows — the count and the page ran under different filters", got.Total, len(got.RelationBacklinks))
}
}
+1
View File
@@ -1807,6 +1807,7 @@ func (s *Server) setupRouter() {
r.Get("/children", s.handleGetItemChildren) r.Get("/children", s.handleGetItemChildren)
r.Get("/progress", s.handleGetItemProgress) r.Get("/progress", s.handleGetItemProgress)
r.Get("/backlinks", s.handleGetItemBacklinks) r.Get("/backlinks", s.handleGetItemBacklinks)
r.Get("/relation-backlinks", s.handleGetItemRelationBacklinks)
r.Get("/tasks", s.handleGetItemChildren) // deprecated alias r.Get("/tasks", s.handleGetItemChildren) // deprecated alias
r.Get("/grants", s.handleListItemGrants) r.Get("/grants", s.handleListItemGrants)
r.Post("/grants", s.handleCreateItemGrant) r.Post("/grants", s.handleCreateItemGrant)
+36 -7
View File
@@ -1502,6 +1502,25 @@ func (s *Store) RemapAttachmentReferencesInWorkspace(workspaceID string, oldToNe
} }
defer tx.Rollback() defer tx.Rollback()
// Serialise against concurrent item writes in this workspace (codex round
// 5 on PLAN-2857 U5). This function SCANS every item's fields and content,
// then writes the snapshot back — so without the lock a concurrent update
// can commit newer fields in between and this overwrites them.
//
// That lost update predates the relation index and is not caused by it;
// what U5 adds is that the same stale snapshot is now also used to rebuild
// the reverse index, so the staleness reaches a second place. One lock
// closes both.
//
// It is the same advisory lock ITEM writers take, and taking it first here
// preserves their order (workspace, then rows) so the two cannot invert.
// Comment writers do not take it at all — which is safe, because a lock
// nobody else in that path holds cannot participate in a cycle, and this
// transaction takes no lock a comment writer could be waiting on.
if err := s.acquireWorkspaceSeqLock(tx, workspaceID); err != nil {
return err
}
// ORDER BY id (BUG-2778 class sweep): this gathers rows and then UPDATEs // ORDER BY id (BUG-2778 class sweep): this gathers rows and then UPDATEs
// them one by one inside a transaction, so the scan's order IS the lock // them one by one inside a transaction, so the scan's order IS the lock
// order. Two concurrent remaps in one workspace would otherwise be free to // order. Two concurrent remaps in one workspace would otherwise be free to
@@ -1518,26 +1537,27 @@ func (s *Store) RemapAttachmentReferencesInWorkspace(workspaceID string, oldToNe
// where the second stage locks the renamed row itself and ordering the // where the second stage locks the renamed row itself and ordering the
// cascade therefore cannot help, ordering IS sufficient for the cycle // cascade therefore cannot help, ordering IS sufficient for the cycle
// between these per-row updates. // between these per-row updates.
rows, err := tx.Query(s.q(`SELECT id, content, fields FROM items WHERE workspace_id = ? AND deleted_at IS NULL ORDER BY id`), workspaceID) rows, err := tx.Query(s.q(`SELECT id, collection_id, content, fields FROM items WHERE workspace_id = ? AND deleted_at IS NULL ORDER BY id`), workspaceID)
if err != nil { if err != nil {
return fmt.Errorf("scan items for remap: %w", err) return fmt.Errorf("scan items for remap: %w", err)
} }
type rowUpdate struct { type rowUpdate struct {
id string id string
content string collectionID string
fields string content string
fields string
} }
var updates []rowUpdate var updates []rowUpdate
for rows.Next() { for rows.Next() {
var id, content, fields string var id, collectionID, content, fields string
if err := rows.Scan(&id, &content, &fields); err != nil { if err := rows.Scan(&id, &collectionID, &content, &fields); err != nil {
rows.Close() rows.Close()
return fmt.Errorf("scan item: %w", err) return fmt.Errorf("scan item: %w", err)
} }
newContent := remapAttachmentRefs(content, oldToNew) newContent := remapAttachmentRefs(content, oldToNew)
newFields := remapAttachmentRefs(fields, oldToNew) newFields := remapAttachmentRefs(fields, oldToNew)
if newContent != content || newFields != fields { if newContent != content || newFields != fields {
updates = append(updates, rowUpdate{id: id, content: newContent, fields: newFields}) updates = append(updates, rowUpdate{id: id, collectionID: collectionID, content: newContent, fields: newFields})
} }
} }
if err := rows.Err(); err != nil { if err := rows.Err(); err != nil {
@@ -1620,6 +1640,15 @@ func (s *Store) RemapAttachmentReferencesInWorkspace(workspaceID string, oldToNe
u.content, u.fields, u.id); err != nil { u.content, u.fields, u.id); err != nil {
return fmt.Errorf("update item %s: %w", u.id, err) return fmt.Errorf("update item %s: %w", u.id, err)
} }
// This rewrite maps `pad-attachment:` ids and cannot change a relation
// VALUE — but it does write the blob, and the hook is a pure function
// of (blob, schema), so calling it is a no-op while OMITTING it would
// be a standing claim about what remapAttachmentRefs can touch. The
// collection id rides along on the scan above rather than costing a
// lookup per item (PLAN-2857 U5).
if _, err := s.replaceRelationLinks(tx, u.id, workspaceID, u.collectionID, u.fields); err != nil {
return fmt.Errorf("index relation links for item %s: %w", u.id, err)
}
} }
for _, u := range commentUpdates { for _, u := range commentUpdates {
if _, err := tx.Exec(s.q(`UPDATE comments SET body = ? WHERE id = ?`), if _, err := tx.Exec(s.q(`UPDATE comments SET body = ? WHERE id = ?`),
+44 -1
View File
@@ -590,7 +590,26 @@ func (s *Store) UpdateCollection(id string, input models.CollectionUpdate) (*mod
// without inventing a second ordering rule to keep in sync with the first, // without inventing a second ordering rule to keep in sync with the first,
// and it is taken BEFORE the row lock, preserving the order this comment // and it is taken BEFORE the row lock, preserving the order this comment
// exists to protect. // exists to protect.
if len(input.Migrations) > 0 || renaming { // The relation reverse index is rebuilt below whenever the SCHEMA moves,
// and that rebuild must not interleave with item writes: an item
// transaction that read the OLD schema can commit its own relation hook
// after the reindex has run, leaving the index matching neither the old
// shape nor the new one (codex round 2). So the same workspace lock the
// migrations take is taken for a schema change too.
//
// Gated rather than unconditional, and that is a cost decision as much as
// a correctness one: an icon or description edit changes nothing the index
// depends on, and reindexing a 10k-item collection for it would put ~1.8s
// on an update that has no business paying it.
// CHANGE-SENSITIVE, not merely "a schema pointer was supplied". A client
// that round-trips the collection and submits identical schema bytes would
// otherwise pay the whole-collection reindex for a no-op, which is the
// same cost the icon-edit gate above exists to avoid — and my first
// version of this line claimed to test movement while only testing
// presence (codex round 3).
schemaMoved := input.Schema != nil && *input.Schema != existing.Schema
reindexRelations := schemaMoved || len(input.Migrations) > 0
if len(input.Migrations) > 0 || renaming || reindexRelations {
if err := s.acquireWorkspaceSeqLock(tx, existing.WorkspaceID); err != nil { if err := s.acquireWorkspaceSeqLock(tx, existing.WorkspaceID); err != nil {
return nil, err return nil, err
} }
@@ -695,6 +714,21 @@ func (s *Store) UpdateCollection(id string, input models.CollectionUpdate) (*mod
} }
} }
// Rebuild this collection's relation reverse index, LAST in the tx so it
// sees both the new schema and any field-value migration above
// (PLAN-2857 U5, lead ruling). A schema change can move which keys ARE
// relations without touching a single item, so nothing else in this
// transaction would repair the index — and the next per-item write never
// comes for items nobody edits.
//
// Only when the schema or the field VALUES actually moved — see the
// workspace-lock note above for why this is gated.
if reindexRelations {
if err := s.ReindexCollectionRelationLinks(tx, id, existing.WorkspaceID); err != nil {
return nil, fmt.Errorf("reindex relation links: %w", err)
}
}
if err := tx.Commit(); err != nil { if err := tx.Commit(); err != nil {
return nil, fmt.Errorf("commit collection update: %w", err) return nil, fmt.Errorf("commit collection update: %w", err)
} }
@@ -828,6 +862,15 @@ func (s *Store) MigrateItemFieldValues(collectionID string, migrations []models.
return totalAffected, err return totalAffected, err
} }
// This function has no non-test caller today, and the hook is here anyway
// (PLAN-2857 U5). It rewrites field VALUES in bulk, so wiring it to a door
// later without this line would stale the relation index for a whole
// collection — a trap that costs one line to remove now and a debugging
// session to find later.
if err := s.ReindexCollectionRelationLinks(tx, collectionID, workspaceID); err != nil {
return totalAffected, err
}
if err := tx.Commit(); err != nil { if err := tx.Commit(); err != nil {
return totalAffected, err return totalAffected, err
} }
+16
View File
@@ -857,6 +857,22 @@ func (s *Store) ImportWorkspace(data *models.WorkspaceExport, newName string, ow
if err != nil { if err != nil {
return nil, fmt.Errorf("remap item %s: %w", it.Title, err) return nil, fmt.Errorf("remap item %s: %w", it.Title, err)
} }
// Index the relation edges (PLAN-2857 U5). HERE and not at the
// first-pass INSERT: this pass writes the FINAL blob, with
// source-workspace ids already remapped to their new ones, and it
// visits every inserted item (the guard above skips only items that
// were never inserted). Indexing at the insert would index ids
// belonging to the exporting workspace.
//
// A value the bundle carries that is not an id — a title, say — is
// indexed as written and simply matches no target. That row is inert
// for the reverse query, which is the same disposition migration 088
// gives every dangling reference. BUG-3014 tracks the separate
// question of how such a value should READ.
if _, err := s.replaceRelationLinks(tx, newItemID, ws.ID, collMap[it.CollectionID], fields); err != nil {
return nil, fmt.Errorf("index relation links for item %s: %w", it.Title, err)
}
} }
// Import comments // Import comments
+52
View File
@@ -378,6 +378,15 @@ func (s *Store) insertItemTx(tx *sql.Tx, id, workspaceID, collectionID, slug, ts
return fmt.Errorf("index wiki links: %w", err) return fmt.Errorf("index wiki links: %w", err)
} }
// Index this item's `relation` field values, in the same transaction and
// for the same reason as the line above: partial state never lands, and a
// rolled-back write rolls back its index rows. This site covers create AND
// the cross-workspace copy, which reaches it through createItemTxWithID.
// PLAN-2857 U5 / TASK-2997.
if _, err := s.replaceRelationLinks(tx, id, workspaceID, collectionID, fields); err != nil {
return fmt.Errorf("index relation links: %w", err)
}
// Phase 2a (TASK-1595): flip any pre-existing broken `[[Title]]` // Phase 2a (TASK-1595): flip any pre-existing broken `[[Title]]`
// rows that have been waiting for an item with this title to // rows that have been waiting for an item with this title to
// arrive. Cheap when no broken rows match (the common case). // arrive. Cheap when no broken rows match (the common case).
@@ -2780,6 +2789,24 @@ func (s *Store) updateItemWithParentLinkOnce(
} }
} }
// Re-index `relation` values when the blob was written (PLAN-2857 U5).
//
// Read the blob BACK from the row rather than indexing *input.Fields. An
// update's field write is a server-side field-level MERGE (IDEA-1480), so
// what was submitted is not what was stored — indexing the submission
// would drop the edges of every key the caller did not mention. Reading
// the row makes the index match stored truth by construction, which is the
// only version of this that stays true as the merge rules change.
if input.Fields != nil {
var storedFields string
if err := tx.QueryRow(s.q(`SELECT fields FROM items WHERE id = ?`), id).Scan(&storedFields); err != nil {
return nil, fmt.Errorf("re-read fields for relation index: %w", err)
}
if _, err := s.replaceRelationLinks(tx, id, existing.WorkspaceID, existing.CollectionID, storedFields); err != nil {
return nil, fmt.Errorf("index relation links: %w", err)
}
}
// BUG-2013: apply the parent-link mutation INSIDE this tx, after the // BUG-2013: apply the parent-link mutation INSIDE this tx, after the
// field write. A failure here (cycle detected, DB error) rolls the // field write. A failure here (cycle detected, DB error) rolls the
// whole transaction back, so the caller can never observe the field // whole transaction back, so the caller can never observe the field
@@ -3026,6 +3053,21 @@ func (s *Store) restoreItemOnce(id string, opt mutationOptions) (*models.Item, e
return nil, sql.ErrNoRows return nil, sql.ErrNoRows
} }
// Re-derive the relation index for the item coming back (PLAN-2857 U5).
//
// Restore writes no `fields`, which is why this site does not appear in
// the list of field-writing hooks — and it is a hole exactly because the
// index depends on the item being LIVE as well as on (blob, schema).
// ReindexCollectionRelationLinks deliberately skips soft-deleted items, so
// a schema change that happens WHILE an item is deleted never reaches it:
// the item comes back carrying whatever rows it had before, against a
// schema that has moved. Both directions bite — a field that became a
// relation leaves the restored item missing edges, one that stopped being
// a relation leaves it with edges it should not have (codex round 1).
if _, err := s.replaceRelationLinks(tx, id, existing.WorkspaceID, existing.CollectionID, existing.Fields); err != nil {
return nil, fmt.Errorf("index relation links on restore: %w", err)
}
// item.restored carries the POST-restore snapshot (SPEC-3 v1.1). Read // item.restored carries the POST-restore snapshot (SPEC-3 v1.1). Read
// after the UPDATE, in-tx: the row is live again at this point, so // after the UPDATE, in-tx: the row is live again at this point, so
// getItemTx sees it, and the snapshot reflects the state a consumer will // getItemTx sees it, and the snapshot reflects the state a consumer will
@@ -4899,6 +4941,16 @@ func (s *Store) moveItemWithPreCheckOnce(
return nil, fmt.Errorf("move item: %w", err) return nil, fmt.Errorf("move item: %w", err)
} }
// Re-index against the DESTINATION collection (PLAN-2857 U5). A move is
// the one write where the blob and the SCHEMA change together: the
// identical value indexes differently on the other side, because which
// keys are relations is a property of the collection, not of the item. A
// hook that passed the item's old collection here would leave every moved
// item indexed under the shape it just left.
if _, err := s.replaceRelationLinks(tx, itemID, existing.WorkspaceID, targetCollectionID, newFieldsJSON); err != nil {
return nil, fmt.Errorf("index relation links on move: %w", err)
}
// A move can carry a status-changing field override (e.g. // A move can carry a status-changing field override (e.g.
// `pad item move ... --field status=done`), which rewrites `fields` // `pad item move ... --field status=done`), which rewrites `fields`
// outside the UpdateItemWithPreCheck path. Record the transition here // outside the UpdateItemWithPreCheck path. Record the transition here
@@ -0,0 +1,73 @@
-- Migration 088: materialised reverse index for `relation` field values
-- (PLAN-2857 U5 / TASK-2997).
--
-- A relation field stores the target item's id inside the source item's
-- `fields` JSON blob. Asking the reverse question — "which items point at
-- TASK-5?" — has no answer today short of scanning every blob.
--
-- WHY MATERIALISED RATHER THAN DERIVED AT READ (ratified Q3). The derived
-- query is an unindexed `json_extract(fields,'$.color') = ?` scan; no index
-- can be added for a USER-DEFINED key without a per-key migration; and on a
-- `multi_relation` array it silently matches nothing rather than failing.
-- That is the same argument migration 061 made for wiki-links, on the same
-- table, against the same alternative.
--
-- WHY A NEW TABLE rather than extending item_wiki_links: that table's
-- target_kind is CHECK-constrained to three CONTENT-link kinds, and its
-- position / display_text columns are content-shaped. A field-valued edge
-- has no position and no display override — but it does have a source FIELD
-- KEY, which nothing in that schema can carry.
--
-- Field notes:
-- * source_field_key is the schema key the value sits under. It is what
-- makes this table not item_wiki_links, and it is what lets the reverse
-- side say "referenced by X via `owner`" rather than just "referenced".
-- * target_item_id is NOT FK'd, matching 061's reasoning: a dangling
-- reference is a row worth keeping and rendering honestly on the SOURCE
-- side. Note that such a row is INERT for the reverse query — no
-- target-side lookup can ever find it — so it exists for the source
-- side and a future broken-references report, not for the count.
-- * ordinal is the position within a `multi_relation` array (U4, not yet
-- landed). Single-valued relations are always 0. Present up-front so U4
-- does not need an ALTER.
-- * workspace_id is denormalised from the source item so the reverse query
-- can scope without a join, exactly as the visibility filter needs it.
--
--
-- NUL INVARIANT (migration 084's census). All four TEXT columns here are
-- recorded as unprotected rather than given their own triggers, and the reason
-- is a derivation, not an omission:
--
-- * target_item_id is extracted from `items.fields`, and source_field_key
-- from `collections.schema`. BOTH of those columns are classJSON-protected
-- already, so a NUL cannot be in the value this table copies.
-- * source_item_id and workspace_id are server-generated ids.
--
-- The derivation is TRANSACTIONAL, which is the part worth stating: every
-- write hook runs after its own row's INSERT/UPDATE inside the same
-- transaction, so the protecting trigger has already fired and rejected a
-- NUL-bearing blob before this table sees anything derived from it. A hook
-- moved to run BEFORE its row's write would break that, and the census would
-- not notice — it checks columns, not ordering.
-- CASCADE on source_item_id matches every other items-referencing table: an
-- item's OUTGOING edges disappear when it is hard-deleted. A SOFT delete
-- leaves the rows in place and the reverse query filters on the source's
-- deleted_at, so a restore brings the edges back without re-deriving them.
CREATE TABLE IF NOT EXISTS item_relation_links (
source_item_id TEXT NOT NULL,
source_field_key TEXT NOT NULL,
target_item_id TEXT NOT NULL,
workspace_id TEXT NOT NULL,
ordinal INTEGER NOT NULL DEFAULT 0,
FOREIGN KEY (source_item_id) REFERENCES items(id) ON DELETE CASCADE
);
-- The reverse lookup this table exists for.
CREATE INDEX IF NOT EXISTS idx_relation_links_target
ON item_relation_links(target_item_id);
-- The write hook's own key: every write replaces one (source, field) pair's
-- rows wholesale, so this is the delete's index as well as the forward read's.
CREATE INDEX IF NOT EXISTS idx_relation_links_source_field
ON item_relation_links(source_item_id, source_field_key);
@@ -99,6 +99,10 @@ item_links.source_id
item_links.target_id item_links.target_id
item_links.user_id item_links.user_id
item_links.workspace_id item_links.workspace_id
item_relation_links.source_field_key
item_relation_links.source_item_id
item_relation_links.target_item_id
item_relation_links.workspace_id
item_reminders.acked_at item_reminders.acked_at
item_reminders.created_at item_reminders.created_at
item_reminders.fired_at item_reminders.fired_at
@@ -0,0 +1,73 @@
-- Migration 065 (Postgres; SQLite 088): materialised reverse index for `relation` field values
-- (PLAN-2857 U5 / TASK-2997).
--
-- A relation field stores the target item's id inside the source item's
-- `fields` JSON blob. Asking the reverse question — "which items point at
-- TASK-5?" — has no answer today short of scanning every blob.
--
-- WHY MATERIALISED RATHER THAN DERIVED AT READ (ratified Q3). The derived
-- query is an unindexed `json_extract(fields,'$.color') = ?` scan; no index
-- can be added for a USER-DEFINED key without a per-key migration; and on a
-- `multi_relation` array it silently matches nothing rather than failing.
-- That is the same argument migration 061 made for wiki-links, on the same
-- table, against the same alternative.
--
-- WHY A NEW TABLE rather than extending item_wiki_links: that table's
-- target_kind is CHECK-constrained to three CONTENT-link kinds, and its
-- position / display_text columns are content-shaped. A field-valued edge
-- has no position and no display override — but it does have a source FIELD
-- KEY, which nothing in that schema can carry.
--
-- Field notes:
-- * source_field_key is the schema key the value sits under. It is what
-- makes this table not item_wiki_links, and it is what lets the reverse
-- side say "referenced by X via `owner`" rather than just "referenced".
-- * target_item_id is NOT FK'd, matching 061's reasoning: a dangling
-- reference is a row worth keeping and rendering honestly on the SOURCE
-- side. Note that such a row is INERT for the reverse query — no
-- target-side lookup can ever find it — so it exists for the source
-- side and a future broken-references report, not for the count.
-- * ordinal is the position within a `multi_relation` array (U4, not yet
-- landed). Single-valued relations are always 0. Present up-front so U4
-- does not need an ALTER.
-- * workspace_id is denormalised from the source item so the reverse query
-- can scope without a join, exactly as the visibility filter needs it.
--
--
-- NUL INVARIANT (migration 084's census). All four TEXT columns here are
-- recorded as unprotected rather than given their own triggers, and the reason
-- is a derivation, not an omission:
--
-- * target_item_id is extracted from `items.fields`, and source_field_key
-- from `collections.schema`. BOTH of those columns are classJSON-protected
-- already, so a NUL cannot be in the value this table copies.
-- * source_item_id and workspace_id are server-generated ids.
--
-- The derivation is TRANSACTIONAL, which is the part worth stating: every
-- write hook runs after its own row's INSERT/UPDATE inside the same
-- transaction, so the protecting trigger has already fired and rejected a
-- NUL-bearing blob before this table sees anything derived from it. A hook
-- moved to run BEFORE its row's write would break that, and the census would
-- not notice — it checks columns, not ordering.
-- CASCADE on source_item_id matches every other items-referencing table: an
-- item's OUTGOING edges disappear when it is hard-deleted. A SOFT delete
-- leaves the rows in place and the reverse query filters on the source's
-- deleted_at, so a restore brings the edges back without re-deriving them.
CREATE TABLE IF NOT EXISTS item_relation_links (
source_item_id TEXT NOT NULL,
source_field_key TEXT NOT NULL,
target_item_id TEXT NOT NULL,
workspace_id TEXT NOT NULL,
ordinal INTEGER NOT NULL DEFAULT 0,
FOREIGN KEY (source_item_id) REFERENCES items(id) ON DELETE CASCADE
);
-- The reverse lookup this table exists for.
CREATE INDEX IF NOT EXISTS idx_relation_links_target
ON item_relation_links(target_item_id);
-- The write hook's own key: every write replaces one (source, field) pair's
-- rows wholesale, so this is the delete's index as well as the forward read's.
CREATE INDEX IF NOT EXISTS idx_relation_links_source_field
ON item_relation_links(source_item_id, source_field_key);
+441
View File
@@ -0,0 +1,441 @@
package store
import (
"database/sql"
"encoding/json"
"fmt"
"strings"
"github.com/PerpetualSoftware/pad/internal/models"
)
// The materialised reverse index for `relation` field values
// (PLAN-2857 U5 / TASK-2997). See migration 088 for why it is materialised
// and why it is its own table.
//
// THE INDEX IS A FUNCTION OF (BLOB, SCHEMA), and both halves matter. Which
// keys are relations is a property of the COLLECTION, so the same blob
// indexes differently after a move, and a schema change can invalidate every
// item in a collection with no item write at all. That second case is handled
// by ReindexCollectionRelationLinks, called inside the schema-update
// transaction (lead ruling on the U5 contract) — "it fixes itself on the next
// write" is false for items nobody edits, which is the same argument that
// made this index materialised rather than derived.
//
// THIS FUNCTION DOES NOT RESOLVE ANYTHING. Since U6 a relation value may be
// SUPPLIED as a title, but the resolver canonicalises it to an id in place
// before any store write, so the blob reaching here already holds ids. Doing
// its own lookup would be a second, divergent resolver — and would index a
// title on any path where the resolver had not run.
// replaceRelationLinks rebuilds the reverse-index rows for one source item.
//
// Must run inside `tx` — the caller owns transactionality — and is called at
// every store site that writes an item's `fields` blob, not at the doors. A
// door-level hook is a per-door wiring claim and the door population grows; a
// store-level one cannot be bypassed by a door that does not exist yet. That
// is also the shape replaceWikiLinks established for the content index.
//
// Idempotent: the same (item, blob, schema) twice yields the same row set.
// Calling it on a write that did not touch a relation value is a deliberate
// no-op rather than an omission, so no caller has to reason about which
// writes "can" matter.
// Returns the number of edge rows written, so a caller reporting progress
// counts what LANDED rather than re-deriving a prediction from a schema it read
// earlier — those two disagree the moment a schema changes underneath.
func (s *Store) replaceRelationLinks(tx *sql.Tx, sourceItemID, workspaceID, collectionID, fieldsJSON string) (int, error) {
// Delete first so callers never pre-clear, and so a blob that has lost
// its relation values correctly ends with zero rows.
if _, err := tx.Exec(s.q(`DELETE FROM item_relation_links WHERE source_item_id = ?`), sourceItemID); err != nil {
return 0, fmt.Errorf("delete prior relation links: %w", err)
}
schema, err := s.relationSchemaForCollectionTx(tx, collectionID)
if err != nil {
return 0, err
}
if len(schema) == 0 {
// The collection declares no relation fields, so there is nothing to
// index. Not an error, and the delete above still ran — which is what
// makes retyping a field away from `relation` self-healing on the
// next write of each item.
return 0, nil
}
values := relationValuesFromBlob(fieldsJSON, schema)
if len(values) == 0 {
return 0, nil
}
for _, v := range values {
if _, err := tx.Exec(s.q(`
INSERT INTO item_relation_links
(source_item_id, source_field_key, target_item_id, workspace_id, ordinal)
VALUES (?, ?, ?, ?, ?)
`), sourceItemID, v.fieldKey, v.targetID, workspaceID, v.ordinal); err != nil {
return 0, fmt.Errorf("insert relation link %s/%s: %w", v.fieldKey, v.targetID, err)
}
}
return len(values), nil
}
// relationLinkRow is one edge about to be written.
type relationLinkRow struct {
fieldKey string
targetID string
ordinal int
}
// relationValuesFromBlob extracts the relation edges a fields blob carries,
// given the set of keys the collection declares as relations.
//
// A blob that will not parse yields NO edges rather than an error: a corrupt
// blob is a different defect that other code reports, and refusing the write
// here would make this index able to block an item's save.
func relationValuesFromBlob(fieldsJSON string, relationKeys map[string]struct{}) []relationLinkRow {
trimmed := strings.TrimSpace(fieldsJSON)
if trimmed == "" || trimmed == "{}" || trimmed == "null" {
return nil
}
var blob map[string]any
if err := json.Unmarshal([]byte(trimmed), &blob); err != nil {
return nil
}
var out []relationLinkRow
for key := range relationKeys {
raw, present := blob[key]
if !present || raw == nil {
continue
}
switch v := raw.(type) {
case string:
if id := strings.TrimSpace(v); id != "" {
out = append(out, relationLinkRow{fieldKey: key, targetID: id})
}
case []any:
// `multi_relation` (U4, not yet landed). Indexed now so the table
// does not need a second pass when that type arrives, and so a
// hand-written array value is not silently unindexed today.
for i, entry := range v {
id, isStr := entry.(string)
if !isStr {
continue
}
if id = strings.TrimSpace(id); id != "" {
out = append(out, relationLinkRow{fieldKey: key, targetID: id, ordinal: i})
}
}
}
}
return out
}
// relationSchemaForCollectionTx returns the set of field keys the collection
// declares as `relation` (or `multi_relation`).
//
// Reads through the transaction on purpose: a schema change and the reindex it
// triggers happen in ONE transaction, so a pool read here would see the
// pre-change schema and rebuild the index to match the shape being replaced.
func (s *Store) relationSchemaForCollectionTx(tx *sql.Tx, collectionID string) (map[string]struct{}, error) {
var schemaJSON sql.NullString
err := tx.QueryRow(s.q(`SELECT schema FROM collections WHERE id = ?`), collectionID).Scan(&schemaJSON)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("read collection schema for relation index: %w", err)
}
if !schemaJSON.Valid || strings.TrimSpace(schemaJSON.String) == "" {
return nil, nil
}
return relationKeysFromSchemaJSON(schemaJSON.String), nil
}
// relationKeysFromSchemaJSON returns the field keys a collection schema
// declares as `relation` (or `multi_relation`).
//
// ONE function so the write hook and the backfill cannot disagree about what
// counts as a relation field — two copies of this rule would drift the moment
// a third relation-ish type appears, and the symptom would be an index that is
// correct at write time and wrong after a rebuild.
//
// A schema that will not parse yields no keys rather than an error: that is a
// different defect, reported elsewhere, and the index declines to be the
// second voice.
func relationKeysFromSchemaJSON(schemaJSON string) map[string]struct{} {
if strings.TrimSpace(schemaJSON) == "" {
return nil
}
var schema models.CollectionSchema
if err := json.Unmarshal([]byte(schemaJSON), &schema); err != nil {
return nil
}
keys := map[string]struct{}{}
for _, def := range schema.Fields {
if def.Type == "relation" || def.Type == "multi_relation" {
keys[def.Key] = struct{}{}
}
}
if len(keys) == 0 {
return nil
}
return keys
}
// ReindexCollectionRelationLinks rebuilds the reverse index for every live
// item in one collection.
//
// WHY THIS EXISTS (lead ruling on the U5 contract). The index is a function of
// (blob, schema), and a SCHEMA change moves the second half without touching a
// single item. Declare a `relation` field on a collection that already holds
// values under that key, or retype one away, and every existing item is
// mis-indexed. "It fixes itself on the next write of each item" is false for
// items nobody edits — which is the same argument that made this index
// materialised rather than derived at read time, so accepting it here would
// undo the decision.
//
// Runs INSIDE the schema-update transaction, so the property is: after a
// committed schema change, no item in the collection is mis-indexed. A failure
// rolls the schema change back with it rather than committing a schema the
// index does not match.
//
// Bounded by the collection's live item count. The cost at 10k items is
// recorded on TASK-2997's trail, because "a rare owner-only operation" is a
// claim about frequency, not about cost, and the two are worth separating.
func (s *Store) ReindexCollectionRelationLinks(tx *sql.Tx, collectionID, workspaceID string) error {
// Ids first, then per-item work: holding a SELECT cursor open while
// writing upsets some drivers, the same reason the wiki backfill collects
// before it writes.
rows, err := tx.Query(s.q(`
SELECT id, fields FROM items
WHERE collection_id = ? AND workspace_id = ? AND deleted_at IS NULL
`), collectionID, workspaceID)
if err != nil {
return fmt.Errorf("scan collection for relation reindex: %w", err)
}
type row struct{ id, fields string }
var items []row
for rows.Next() {
var r row
if scanErr := rows.Scan(&r.id, &r.fields); scanErr != nil {
rows.Close()
return fmt.Errorf("scan relation reindex row: %w", scanErr)
}
items = append(items, r)
}
if rowsErr := rows.Err(); rowsErr != nil {
rows.Close()
return fmt.Errorf("iterate relation reindex rows: %w", rowsErr)
}
rows.Close()
for _, r := range items {
if _, err := s.replaceRelationLinks(tx, r.id, workspaceID, collectionID, r.fields); err != nil {
return err
}
}
return nil
}
// RelationBacklink is one item that points at the target through a relation
// field, plus the field it points through.
//
// FieldKey is the part that makes this not a wiki backlink: the reverse side
// can say "referenced by CAR-3 via `color`" rather than just "referenced by",
// which is the whole reason this index is its own table.
type RelationBacklink struct {
SourceItemID string `json:"source_item_id"`
SourceRef string `json:"source_ref"`
SourceTitle string `json:"source_title"`
CollectionSlug string `json:"collection_slug"`
FieldKey string `json:"field_key"`
FieldLabel string `json:"field_label,omitempty"`
}
// relationBacklinkCap bounds any single page. Matches GetBacklinks' cap for
// the same reason: a caller that asks for everything should not be able to.
const relationBacklinkCap = 300
// GetRelationBacklinks returns the live items that reference targetItemID
// through a `relation` field, filtered to what this viewer may see.
//
// VISIBILITY IS THE VIEWER'S, NOT THE TRUTH (ratified, day 55). `vis` is built
// from ResolveBacklinksVisibility — the same resolver the wiki-link reverse
// side uses, reused verbatim rather than reinvented — and the consequence is
// on the record deliberately: "Referenced by 3" means "by 3 you can see". A
// true count would leak the existence of items the viewer has no access to,
// which is the exact leak that resolver exists to close.
//
// Filtering happens in SQL, not after the fetch, so LIMIT counts VISIBLE rows.
// Filtering above the limit would let invisible rows consume page slots and
// silently shrink pages — and worse, a page that came back short would itself
// be a signal about how many hidden rows there are.
//
// Soft-deleted SOURCES are excluded by the join, so a deleted item stops
// referencing and a restore brings it back without re-deriving anything. A
// soft-deleted TARGET keeps its rows: the question "who pointed at this?"
// still has an answer after deletion.
func (s *Store) GetRelationBacklinks(targetItemID, workspaceID string, limit, offset int, vis BacklinksVisibility) ([]RelationBacklink, error) {
if limit <= 0 || limit > relationBacklinkCap {
limit = 50
}
if offset < 0 {
offset = 0
}
// Restricted with nothing visible → empty without a query. Postgres
// rejects the `IN ()` form this would otherwise build.
if !vis.Unrestricted && len(vis.FullCollectionIDs) == 0 && len(vis.GrantedItemIDs) == 0 {
return nil, nil
}
args := []any{targetItemID, workspaceID}
visClause := ""
if !vis.Unrestricted {
collClause := "FALSE"
if len(vis.FullCollectionIDs) > 0 {
ph := make([]string, len(vis.FullCollectionIDs))
for i, cid := range vis.FullCollectionIDs {
ph[i] = "?"
args = append(args, cid)
}
collClause = "s.collection_id IN (" + strings.Join(ph, ",") + ")"
}
itemClause := "FALSE"
if len(vis.GrantedItemIDs) > 0 {
ph := make([]string, len(vis.GrantedItemIDs))
for i, iid := range vis.GrantedItemIDs {
ph[i] = "?"
args = append(args, iid)
}
itemClause = "s.id IN (" + strings.Join(ph, ",") + ")"
}
visClause = " AND (" + collClause + " OR " + itemClause + ")"
}
args = append(args, limit, offset)
rows, err := s.db.Query(s.q(`
SELECT rl.source_item_id, rl.source_field_key,
s.title, s.item_number, s.collection_id, c.prefix, c.slug, c.schema
FROM item_relation_links rl
JOIN items s ON s.id = rl.source_item_id
JOIN collections c ON c.id = s.collection_id
WHERE rl.target_item_id = ? AND rl.workspace_id = ?
AND s.deleted_at IS NULL
-- A soft-deleted COLLECTION leaves its items live, so without this a
-- source in a collection the viewer can no longer open still shows
-- up here — including for an Unrestricted viewer, who has no
-- collection filter to catch it (codex round 2).
AND c.deleted_at IS NULL
AND s.id != rl.target_item_id`+visClause+`
ORDER BY s.updated_at DESC, rl.source_item_id, rl.source_field_key
LIMIT ? OFFSET ?
`), args...)
if err != nil {
return nil, fmt.Errorf("query relation backlinks: %w", err)
}
defer rows.Close()
// Field LABELS come from the source collection's schema, which the query
// above already carries on every row — parsed once per distinct collection
// here rather than re-queried.
//
// The first version issued a SELECT per distinct collection WHILE THE ROWS
// CURSOR WAS STILL OPEN. That needs a SECOND pool connection, and when the
// pool is saturated the second connection never arrives — an application
// deadlock no SQLSTATE names, and the same class BUG-2409 closed for the
// collection writers (codex round 7). It was also an N+1. Selecting
// `c.schema` in the one query removes both.
labels := map[string]map[string]string{}
labelFor := func(collectionID, schemaJSON, key string) string {
byKey, known := labels[collectionID]
if !known {
byKey = map[string]string{}
var schema models.CollectionSchema
if json.Unmarshal([]byte(schemaJSON), &schema) == nil {
for _, def := range schema.Fields {
if def.Label != "" {
byKey[def.Key] = def.Label
}
}
}
labels[collectionID] = byKey
}
return byKey[key]
}
var out []RelationBacklink
for rows.Next() {
var b RelationBacklink
// item_number is NULLABLE on legacy rows — scanning it into an int
// errors and would take the whole list down, the same defect U6 shipped
// in hydration (codex round 7 there).
var number sql.NullInt64
var schemaJSON sql.NullString
var prefix, collectionID string
if err := rows.Scan(&b.SourceItemID, &b.FieldKey, &b.SourceTitle, &number, &collectionID, &prefix, &b.CollectionSlug, &schemaJSON); err != nil {
return nil, fmt.Errorf("scan relation backlink: %w", err)
}
if number.Valid {
b.SourceRef = fmt.Sprintf("%s-%d", prefix, number.Int64)
}
b.FieldLabel = labelFor(collectionID, schemaJSON.String, b.FieldKey)
out = append(out, b)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterate relation backlinks: %w", err)
}
return out, nil
}
// CountRelationBacklinks is GetRelationBacklinks' count, under the SAME
// visibility and the SAME filters.
//
// It exists as its own query rather than len() of the page because the UI
// wants "Referenced by N" without fetching N rows. The filters must stay in
// lockstep with GetRelationBacklinks — a drift would let the header promise a
// number the list cannot produce, which is exactly the failure CountBacklinks
// documents for the wiki side.
func (s *Store) CountRelationBacklinks(targetItemID, workspaceID string, vis BacklinksVisibility) (int, error) {
if !vis.Unrestricted && len(vis.FullCollectionIDs) == 0 && len(vis.GrantedItemIDs) == 0 {
return 0, nil
}
args := []any{targetItemID, workspaceID}
visClause := ""
if !vis.Unrestricted {
collClause := "FALSE"
if len(vis.FullCollectionIDs) > 0 {
ph := make([]string, len(vis.FullCollectionIDs))
for i, cid := range vis.FullCollectionIDs {
ph[i] = "?"
args = append(args, cid)
}
collClause = "s.collection_id IN (" + strings.Join(ph, ",") + ")"
}
itemClause := "FALSE"
if len(vis.GrantedItemIDs) > 0 {
ph := make([]string, len(vis.GrantedItemIDs))
for i, iid := range vis.GrantedItemIDs {
ph[i] = "?"
args = append(args, iid)
}
itemClause = "s.id IN (" + strings.Join(ph, ",") + ")"
}
visClause = " AND (" + collClause + " OR " + itemClause + ")"
}
var n int
err := s.db.QueryRow(s.q(`
SELECT COUNT(*)
FROM item_relation_links rl
JOIN items s ON s.id = rl.source_item_id
JOIN collections c ON c.id = s.collection_id
WHERE rl.target_item_id = ? AND rl.workspace_id = ?
AND s.deleted_at IS NULL
-- Same filter as the page query. The two must stay identical or the
-- header promises a number the list cannot produce.
AND c.deleted_at IS NULL
AND s.id != rl.target_item_id`+visClause), args...).Scan(&n)
if err != nil {
return 0, fmt.Errorf("count relation backlinks: %w", err)
}
return n, nil
}
+290
View File
@@ -0,0 +1,290 @@
package store
import (
"database/sql"
"errors"
"fmt"
)
// BackfillRelationLinksResult reports what the backfill did so cmd/pad can log
// a one-line summary at startup.
type BackfillRelationLinksResult struct {
// Skipped is true when the backfill decided it had nothing to do without
// scanning items — see BackfillRelationLinks for the two ways that
// happens.
Skipped bool
// ItemsScanned counts the live items the backfill walked.
ItemsScanned int
// LinksInserted is the number of index rows written.
LinksInserted int
}
// BackfillRelationLinks derives `item_relation_links` for every live item.
// Called from server startup after migrations.
//
// ITS GUARD IS A COMPLETION MARKER, and the two shapes it is not are both
// instructive.
//
// It is not the wiki backfill's per-item EXISTS. That test does not work here:
// most items carry no relation value, so "zero rows" is their correct steady
// state and is indistinguishable from "never indexed" — a per-item EXISTS
// would re-derive nearly every item on every boot, forever.
//
// It is also no longer "does the table have any row", which is what I wrote
// first and which codex round 1 correctly called a P1. Rows commit per item,
// so a crash partway through leaves the table NON-EMPTY AND INCOMPLETE — and
// that guard then skips forever, with the missing edges never derived. The
// "rebuild by deleting the table" story only ever covered deliberate deletion,
// not an interrupted run.
//
// So completion is recorded EXPLICITLY, in platform_settings, and only AFTER a
// full pass finishes. A crash before that leaves no marker and the next boot
// re-derives — which is safe because replaceRelationLinks deletes before it
// inserts, so a repeat pass is idempotent rather than additive. The marker is
// written after the work for the same reason a dedupe token must be: writing
// it first turns a mid-run failure into a permanent silent loss.
//
// A REBUILD is still one line, just a different one: delete the marker (or the
// table and the marker) and restart.
//
// The schema probe stays as a cheap second skip — a database where no
// collection declares a relation field has nothing to derive — and it records
// completion too, because a pass over zero work is a completed pass. A
// relation field arriving later is handled by UpdateCollection's reindex and
// the write hooks, not by this.
//
// TWO SERVERS BOOTING AT ONCE is redundant, not corrupting, and the argument
// is worth writing down because the obvious fix — a startup lock — has worse
// failure modes than the thing it prevents (a crashed holder blocking boot).
//
// Both servers see no marker and both walk items. Each per-item write re-reads
// the blob inside its own transaction and replaces that item's rows wholesale,
// so two passes over one item converge on the same rows rather than doubling
// them. Whichever server finishes first writes the marker, and the marker is
// honest at that moment: a full pass DID complete. The other server's
// remaining writes re-derive rows that are already correct.
//
// The interleaving that WOULD corrupt — an older pass writing a stale blob
// over a newer one — is closed by the in-transaction re-read, not by the
// marker. That is the load-bearing part; the marker only decides whether a
// pass runs at all.
//
// PLAN-2857 U5 / TASK-2997.
func (s *Store) BackfillRelationLinks() (*BackfillRelationLinksResult, error) {
result := &BackfillRelationLinksResult{}
var marker string
err := s.db.QueryRow(s.q(`SELECT value FROM platform_settings WHERE key = ?`), relationLinksBackfilledFlag).Scan(&marker)
if err == nil && marker == "1" {
result.Skipped = true
return result, nil
}
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return nil, fmt.Errorf("probe relation backfill marker: %w", err)
}
// Which collections declare a relation field, and what their relation keys
// are. One pass, so the per-item work below is a map lookup rather than a
// schema parse.
relationKeysByCollection, err := s.collectionsWithRelationFields()
if err != nil {
return nil, err
}
if len(relationKeysByCollection) == 0 {
// A pass over zero work is a completed pass.
if err := s.markRelationLinksBackfilled(); err != nil {
return nil, err
}
result.Skipped = true
return result, nil
}
// IDS ONLY. The scan used to carry every item's `fields` blob, which put
// the whole database's field JSON in memory before a single row was
// written (codex round 6). It was never needed: each write re-reads the
// blob under a lock, precisely because a snapshot written later is a
// snapshot that can be stale. Three ids per row is ~100 bytes, so 100k
// items is ~10MB rather than the sum of every blob.
rows, err := s.db.Query(s.q(`
SELECT id, workspace_id, collection_id
FROM items
WHERE deleted_at IS NULL
`))
if err != nil {
return nil, fmt.Errorf("scan items for relation backfill: %w", err)
}
var items []itemRow
for rows.Next() {
var r itemRow
if scanErr := rows.Scan(&r.id, &r.workspaceID, &r.collectionID); scanErr != nil {
rows.Close()
return nil, fmt.Errorf("scan relation backfill row: %w", scanErr)
}
items = append(items, r)
}
if rowsErr := rows.Err(); rowsErr != nil {
rows.Close()
return nil, fmt.Errorf("iterate relation backfill rows: %w", rowsErr)
}
rows.Close()
// BATCHED BY WORKSPACE, in bounded chunks. One transaction per ITEM was
// the first shape and codex round 5 was right to call it out: a 100k-item
// database meant 100k transactions and 100k advisory-lock acquisitions in
// a synchronous startup pause, which is connection churn nobody asked for.
//
// The other extreme — one transaction per workspace — is worse in a
// different way: at the measured ~1.8s per 10k items, a 100k-item
// workspace would hold its write lock for ~18 seconds at boot. Bounded
// chunks keep both the transaction count and the lock hold time small, and
// the per-chunk failure granularity is what the per-item version was
// really buying.
byWorkspace := map[string][]itemRow{}
for _, it := range items {
if _, ok := relationKeysByCollection[it.collectionID]; !ok {
continue
}
byWorkspace[it.workspaceID] = append(byWorkspace[it.workspaceID], it)
}
for workspaceID, wsItems := range byWorkspace {
for start := 0; start < len(wsItems); start += relationBackfillChunk {
end := start + relationBackfillChunk
if end > len(wsItems) {
end = len(wsItems)
}
inserted, err := s.backfillRelationChunk(workspaceID, wsItems[start:end])
if err != nil {
return nil, err
}
result.ItemsScanned += end - start
result.LinksInserted += inserted
}
}
// ONLY NOW. Every earlier return path is a failure, and none of them
// records completion — so an interrupted run is re-derived on the next
// boot rather than skipped forever.
if err := s.markRelationLinksBackfilled(); err != nil {
return nil, err
}
return result, nil
}
// relationLinksBackfilledFlag is the platform_settings key recording that a
// FULL relation-index derivation has completed. Same mechanism the
// webhook-secret encryption migration uses for the same question.
const relationLinksBackfilledFlag = "relation_links_backfilled"
// markRelationLinksBackfilled records a completed pass.
func (s *Store) markRelationLinksBackfilled() error {
if _, err := s.db.Exec(s.q(`
INSERT INTO platform_settings (key, value, updated_at) VALUES (?, ?, ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at
`), relationLinksBackfilledFlag, "1", now()); err != nil {
return fmt.Errorf("persist relation backfill marker: %w", err)
}
return nil
}
// collectionsWithRelationFields maps collection id -> its relation field keys,
// for every collection that declares at least one.
func (s *Store) collectionsWithRelationFields() (map[string]map[string]struct{}, error) {
rows, err := s.db.Query(s.q(`SELECT id, schema FROM collections`))
if err != nil {
return nil, fmt.Errorf("scan collections for relation keys: %w", err)
}
defer rows.Close()
out := map[string]map[string]struct{}{}
for rows.Next() {
var id, schemaJSON string
if err := rows.Scan(&id, &schemaJSON); err != nil {
return nil, fmt.Errorf("scan collection schema: %w", err)
}
keys := relationKeysFromSchemaJSON(schemaJSON)
if len(keys) > 0 {
out[id] = keys
}
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterate collection schemas: %w", err)
}
return out, nil
}
// itemRow is one row of the backfill's initial scan — identity only. The blob
// is deliberately absent: every write re-reads it under a lock, so carrying it
// here would cost memory proportional to the database's total field JSON and
// buy a value that must be discarded anyway.
type itemRow struct{ id, workspaceID, collectionID string }
// relationBackfillChunk bounds one backfill transaction. Small enough that the
// workspace lock is never held long at boot, large enough that a big database
// does not pay a transaction per item.
const relationBackfillChunk = 500
// backfillRelationChunk indexes one bounded batch of items from ONE workspace,
// in a single transaction holding that workspace's lock.
func (s *Store) backfillRelationChunk(workspaceID string, batch []itemRow) (int, error) {
tx, err := s.db.Begin()
if err != nil {
return 0, fmt.Errorf("begin relation backfill tx: %w", err)
}
// Belt and braces on top of the explicit rollbacks: a driver can return a
// RECOVERABLE error from Commit, which no explicit path covers. A rollback
// after a successful commit is a no-op.
defer tx.Rollback() //nolint:errcheck // no-op after commit
// WORKSPACE LOCK FIRST, then each item row — the order every item write and
// UpdateCollection use, so this cannot invert against them.
//
// The row lock alone serialises against concurrent ITEM writes but not
// against a concurrent SCHEMA change, which is guarded a level up by this
// lock: without it the backfill can read the old schema, watch
// UpdateCollection reindex the collection, then write rows derived from the
// shape it just replaced and commit last (codex rounds 3 and 4).
if err := s.acquireWorkspaceSeqLock(tx, workspaceID); err != nil {
return 0, err
}
inserted := 0
for _, it := range batch {
// LOCKED, not just re-read. The re-read closes the gap between the
// SCAN and this transaction; the lock closes the gap between this read
// and the replacement below. SQLite's write lock already serialises
// and it does not take the clause here, hence the dialect gate — the
// same shape UpdateCollection uses for its own re-read.
lockedRead := `SELECT fields, collection_id FROM items WHERE id = ? AND deleted_at IS NULL`
if s.dialect.Driver() == DriverPostgres {
lockedRead += " FOR UPDATE"
}
var fields, collectionID string
if err := tx.QueryRow(s.q(lockedRead), it.id).Scan(&fields, &collectionID); err != nil {
if errors.Is(err, sql.ErrNoRows) {
// Deleted between the scan and now. Not a match any more.
continue
}
return 0, fmt.Errorf("re-read item %s for backfill: %w", it.id, err)
}
// CALLED UNCONDITIONALLY, even when the blob carries no edges. Skipping
// the zero-edge case was an optimisation that preserved exactly the
// stale rows this pass exists to clear — replaceRelationLinks deletes
// before it inserts, so that case is the one that must run.
// The COUNT comes from the write itself. Deriving it from the
// schema map captured before the pass would disagree with reality
// whenever a schema changed underneath — reporting a number of rows
// that were never written (codex round 6).
n, err := s.replaceRelationLinks(tx, it.id, workspaceID, collectionID, fields)
if err != nil {
return 0, err
}
inserted += n
}
if err := tx.Commit(); err != nil {
return 0, fmt.Errorf("commit relation backfill tx: %w", err)
}
return inserted, nil
}
@@ -0,0 +1,94 @@
package store
import (
"encoding/json"
"fmt"
"os"
"testing"
"time"
"github.com/PerpetualSoftware/pad/internal/models"
)
// TestRelationReindexCost measures ReindexCollectionRelationLinks at scale.
//
// The lead asked for this number on TASK-2997's trail before accepting the
// reindex-on-schema-change ruling, and the reason is worth keeping: "a rare
// owner-only operation" is a claim about FREQUENCY, and it says nothing about
// what the operation costs when it does run. The reindex holds the
// schema-update transaction open for its whole duration, so its cost is a
// user-visible latency on an API call, not background work.
//
// Skipped unless RELATION_COST=1 — it builds 10k items, which is minutes of
// setup nobody wants on every `go test ./...`.
func TestRelationReindexCost(t *testing.T) {
if os.Getenv("RELATION_COST") != "1" {
t.Skip("set RELATION_COST=1 to run the 10k reindex measurement")
}
s := testStore(t)
ws, colors, cars, red := relationFixture(t, s)
_ = colors
const n = 10000
start := time.Now()
for i := 0; i < n; i++ {
blob, _ := json.Marshal(map[string]any{"status": "open", "color": red.ID})
if _, err := s.CreateItem(ws.ID, cars.ID, models.ItemCreate{
Title: fmt.Sprintf("Car %d", i),
Fields: string(blob),
}); err != nil {
t.Fatalf("CreateItem %d: %v", i, err)
}
}
t.Logf("setup: created %d items in %s", n, time.Since(start).Round(time.Millisecond))
tx, err := s.db.Begin()
if err != nil {
t.Fatalf("begin: %v", err)
}
reindexStart := time.Now()
if err := s.ReindexCollectionRelationLinks(tx, cars.ID, ws.ID); err != nil {
tx.Rollback() //nolint:errcheck
t.Fatalf("reindex: %v", err)
}
if err := tx.Commit(); err != nil {
t.Fatalf("commit: %v", err)
}
elapsed := time.Since(reindexStart)
count, err := s.CountRelationBacklinks(red.ID, ws.ID, unrestricted)
if err != nil {
t.Fatalf("count: %v", err)
}
if count != n {
t.Fatalf("reindex produced %d edges, want %d — the measurement below would describe the wrong work", count, n)
}
t.Logf("REINDEX COST: %d items, %d edges, %s", n, count, elapsed.Round(time.Millisecond))
// And the BACKFILL over the same data, which is the startup path and the
// one codex round 5 asked about: it batches by workspace in bounded
// chunks, so the numbers that matter are wall time and transaction COUNT
// rather than time alone.
if _, err := s.db.Exec(s.q(`DELETE FROM item_relation_links`)); err != nil {
t.Fatalf("clear index: %v", err)
}
if _, err := s.db.Exec(s.q(`DELETE FROM platform_settings WHERE key = ?`), relationLinksBackfilledFlag); err != nil {
t.Fatalf("clear marker: %v", err)
}
backfillStart := time.Now()
res, err := s.BackfillRelationLinks()
if err != nil {
t.Fatalf("backfill: %v", err)
}
backfillElapsed := time.Since(backfillStart)
after, err := s.CountRelationBacklinks(red.ID, ws.ID, unrestricted)
if err != nil {
t.Fatalf("count after backfill: %v", err)
}
if after != n {
t.Fatalf("the backfill produced %d edges, want %d — the timing below would describe the wrong work", after, n)
}
txns := (res.ItemsScanned + relationBackfillChunk - 1) / relationBackfillChunk
t.Logf("BACKFILL COST: %d items scanned, %d edges, %s, ~%d transactions (chunk=%d)",
res.ItemsScanned, res.LinksInserted, backfillElapsed.Round(time.Millisecond), txns, relationBackfillChunk)
}
+713
View File
@@ -0,0 +1,713 @@
package store
import (
"encoding/json"
"testing"
"time"
"github.com/PerpetualSoftware/pad/internal/models"
)
// The materialised reverse index for relation values (PLAN-2857 U5 /
// TASK-2997). Q3's proving test, leg for leg.
//
// EVERY LEG HERE IS A WIRING CLAIM. The index is only ever as good as the
// hooks that maintain it, so a test that exercised the helper directly would
// vouch for nothing — these go through the real write paths (CreateItem,
// UpdateItem, DeleteItem, RestoreItem, MoveItem, UpdateCollection) and read
// through the real query.
//
// The counterfactual that proves they are wired is a MUTATION, not a test in
// this file: replaceRelationLinks is stubbed to return early, the mutant is
// checked to COMPILE, and every leg here must fail. It is recorded in the
// commit messages with its verdict. An earlier version of this comment
// pointed at a TestRelationLinks_RemovedHookGoesStale that does not exist —
// a claim about the file's own contents that nothing checked (codex round 7).
// unrestricted is the visibility vector for a viewer who sees everything. The
// viewer-scoped legs build their own.
var unrestricted = BacklinksVisibility{Unrestricted: true}
// relationIndexFixture returns a workspace with Cars (carrying a `color`
// relation) and Colors, plus one colour to point at.
func relationIndexFixture(t *testing.T, s *Store) (*models.Workspace, *models.Collection, *models.Collection, *models.Item) {
t.Helper()
return relationFixture(t, s)
}
// countFor is the reverse count a viewer who sees everything gets.
func countFor(t *testing.T, s *Store, target *models.Item, ws *models.Workspace) int {
t.Helper()
n, err := s.CountRelationBacklinks(target.ID, ws.ID, unrestricted)
if err != nil {
t.Fatalf("CountRelationBacklinks(%s): %v", target.Title, err)
}
return n
}
// carPointingAt creates an item in `cars` whose `color` relation names target.
func carPointingAt(t *testing.T, s *Store, ws *models.Workspace, cars *models.Collection, title, targetID string) *models.Item {
t.Helper()
blob, _ := json.Marshal(map[string]any{"status": "open", "color": targetID})
it, err := s.CreateItem(ws.ID, cars.ID, models.ItemCreate{Title: title, Fields: string(blob)})
if err != nil {
t.Fatalf("CreateItem(%s): %v", title, err)
}
return it
}
// TestRelationLinks_WriteThenReadCountsTheSource is the first half of Q3's
// property: a written relation value shows up on the target's reverse side.
func TestRelationLinks_WriteThenReadCountsTheSource(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
if n := countFor(t, s, red, ws); n != 0 {
t.Fatalf("a fresh target is referenced by %d, want 0 — the fixture is not clean", n)
}
car := carPointingAt(t, s, ws, cars, "Red Car", red.ID)
if n := countFor(t, s, red, ws); n != 1 {
t.Fatalf("after a write the target is referenced by %d, want 1", n)
}
links, err := s.GetRelationBacklinks(red.ID, ws.ID, 50, 0, unrestricted)
if err != nil {
t.Fatalf("GetRelationBacklinks: %v", err)
}
if len(links) != 1 {
t.Fatalf("want 1 backlink, got %d", len(links))
}
if links[0].SourceItemID != car.ID {
t.Errorf("source = %s, want %s", links[0].SourceItemID, car.ID)
}
// The field key is the whole reason this index is its own table: the
// reverse side must be able to say WHICH field points here.
if links[0].FieldKey != "color" {
t.Errorf("field key = %q, want %q", links[0].FieldKey, "color")
}
if links[0].SourceRef != car.Ref || links[0].SourceTitle != "Red Car" {
t.Errorf("backlink lost the source's identity: %+v", links[0])
}
}
// TestRelationLinks_ChangingTheValueMovesTheEdge is Q3's BOTH HALVES leg.
//
// A writer that only ever INSERTs passes "the new target counts it". Only the
// old target dropping to zero proves the edge MOVED rather than accumulated,
// and that is the half a naive hook fails.
func TestRelationLinks_ChangingTheValueMovesTheEdge(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, colors, cars, red := relationIndexFixture(t, s)
blue := createTestItem(t, s, ws.ID, colors.ID, "Blue", "")
car := carPointingAt(t, s, ws, cars, "Repainted Car", red.ID)
if n := countFor(t, s, red, ws); n != 1 {
t.Fatalf("setup: red referenced by %d, want 1", n)
}
blob, _ := json.Marshal(map[string]any{"status": "open", "color": blue.ID})
fields := string(blob)
if _, err := s.UpdateItem(car.ID, models.ItemUpdate{Fields: &fields}); err != nil {
t.Fatalf("UpdateItem: %v", err)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Errorf("the OLD target is still referenced by %d, want 0 — the hook inserts but does not remove, so edges accumulate", n)
}
if n := countFor(t, s, blue, ws); n != 1 {
t.Errorf("the NEW target is referenced by %d, want 1", n)
}
}
// TestRelationLinks_SoftDeletedSourceDropsOutAndRestoreBringsItBack.
//
// store/items.go documents that a restore resurrects an item's relationships;
// the reverse side has to match, or a restored item is silently missing from
// the pages that referenced it.
func TestRelationLinks_SoftDeletedSourceDropsOutAndRestoreBringsItBack(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
car := carPointingAt(t, s, ws, cars, "Disappearing Car", red.ID)
if n := countFor(t, s, red, ws); n != 1 {
t.Fatalf("setup: referenced by %d, want 1", n)
}
if err := s.DeleteItem(car.ID); err != nil {
t.Fatalf("DeleteItem: %v", err)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Errorf("a soft-deleted source still counts (%d); the reverse side must not cite an item nobody can open", n)
}
if _, err := s.RestoreItem(car.ID); err != nil {
t.Fatalf("RestoreItem: %v", err)
}
if n := countFor(t, s, red, ws); n != 1 {
t.Errorf("after restore the source is referenced %d times, want 1 — restore resurrects relationships and the reverse side must agree", n)
}
}
// TestRelationLinks_SoftDeletedTargetKeepsItsRows asserts on the STORE's
// answer rather than on a chip's wording.
//
// Q3's row phrases this leg as "the source's chip renders (deleted)". That
// chip is U2's, and BUG-3013 is that "deleted" overclaims when gone and hidden
// are indistinguishable — so pinning the word here would couple this unit to a
// wording that is already scheduled to change. What U5 owes is that the EDGE
// survives: the question "who pointed at this?" still has an answer after the
// target is deleted.
func TestRelationLinks_SoftDeletedTargetKeepsItsRows(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
carPointingAt(t, s, ws, cars, "Orphaned Car", red.ID)
if err := s.DeleteItem(red.ID); err != nil {
t.Fatalf("DeleteItem(target): %v", err)
}
if n := countFor(t, s, red, ws); n != 1 {
t.Errorf("deleting the TARGET dropped its reverse side to %d; the edge belongs to the source, which still points here", n)
}
}
// TestRelationLinks_MoveReindexesAgainstTheDestinationSchema.
//
// A move is the one write where the blob and the SCHEMA change together. The
// same stored value indexes differently on the other side, because which keys
// are relations belongs to the COLLECTION.
func TestRelationLinks_MoveReindexesAgainstTheDestinationSchema(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
car := carPointingAt(t, s, ws, cars, "Moving Car", red.ID)
if n := countFor(t, s, red, ws); n != 1 {
t.Fatalf("setup: referenced by %d, want 1", n)
}
// A collection with NO relation field: the same blob carries no edge here.
plain, err := s.CreateCollection(ws.ID, models.CollectionCreate{
Name: "Plain",
Schema: `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"}]}`,
})
if err != nil {
t.Fatalf("CreateCollection: %v", err)
}
// The blob travels unchanged; only the collection differs.
moved, _ := json.Marshal(map[string]any{"status": "open", "color": red.ID})
if _, err := s.MoveItem(car.ID, plain.ID, string(moved)); err != nil {
t.Fatalf("MoveItem: %v", err)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Errorf("after moving into a collection with no relation field the target is still referenced %d times — the move indexed against the collection it LEFT", n)
}
}
// TestRelationLinks_SchemaChangeReindexesTheCollection is the lead's D2 ruling
// as a test.
//
// Declaring a relation field over values that already exist must index them.
// Nothing writes those items, so "it fixes itself on the next write" would
// leave the index wrong indefinitely — the same argument that made this index
// materialised rather than derived.
func TestRelationLinks_SchemaChangeReindexesTheCollection(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, _, red := relationIndexFixture(t, s)
// A collection where `color` is plain TEXT, holding what is already a
// valid item id.
garages, err := s.CreateCollection(ws.ID, models.CollectionCreate{
Name: "Garages",
Schema: `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"},{"key":"color","type":"text"}]}`,
})
if err != nil {
t.Fatalf("CreateCollection: %v", err)
}
blob, _ := json.Marshal(map[string]any{"status": "open", "color": red.ID})
if _, err := s.CreateItem(ws.ID, garages.ID, models.ItemCreate{Title: "Garage", Fields: string(blob)}); err != nil {
t.Fatalf("CreateItem: %v", err)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Fatalf("a TEXT field must not be indexed as a relation (count %d)", n)
}
// Now declare it a relation. No item is written.
reshaped := `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"},{"key":"color","type":"relation","collection":"colors"}]}`
if _, err := s.UpdateCollection(garages.ID, models.CollectionUpdate{Schema: &reshaped}); err != nil {
t.Fatalf("UpdateCollection: %v", err)
}
if n := countFor(t, s, red, ws); n != 1 {
t.Errorf("after the field became a relation the target is referenced %d times, want 1 — nothing writes these items, so only the schema-change reindex can find them", n)
}
// And the reverse direction: retyping it away must remove the edges.
back := `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"},{"key":"color","type":"text"}]}`
if _, err := s.UpdateCollection(garages.ID, models.CollectionUpdate{Schema: &back}); err != nil {
t.Fatalf("UpdateCollection(back): %v", err)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Errorf("after retyping the field away from relation the target is still referenced %d times, want 0", n)
}
}
// TestRelationLinks_BackfillDerivesRowsWrittenBeforeTheIndexExisted is Q3's
// backfill leg. The table is emptied to simulate a pre-migration database.
func TestRelationLinks_BackfillDerivesRowsWrittenBeforeTheIndexExisted(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
carPointingAt(t, s, ws, cars, "Pre-index Car", red.ID)
// Simulate the pre-migration state: the rows exist, the index does not,
// and no completed pass has been recorded.
if _, err := s.db.Exec(s.q(`DELETE FROM item_relation_links`)); err != nil {
t.Fatalf("clear index: %v", err)
}
if _, err := s.db.Exec(s.q(`DELETE FROM platform_settings WHERE key = ?`), relationLinksBackfilledFlag); err != nil {
t.Fatalf("clear backfill marker: %v", err)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Fatalf("the index was not actually cleared (count %d), so the leg below would pass without a backfill", n)
}
res, err := s.BackfillRelationLinks()
if err != nil {
t.Fatalf("BackfillRelationLinks: %v", err)
}
if res.Skipped {
t.Fatal("the backfill skipped: it must not short-circuit when the table is empty and a collection declares a relation field")
}
if n := countFor(t, s, red, ws); n != 1 {
t.Errorf("after the backfill the target is referenced %d times, want 1", n)
}
// Second run short-circuits, which is what keeps steady-state boots cheap.
again, err := s.BackfillRelationLinks()
if err != nil {
t.Fatalf("BackfillRelationLinks (second): %v", err)
}
if !again.Skipped {
t.Errorf("the second run re-derived instead of short-circuiting (%+v); every boot would pay a full scan", again)
}
}
// TestRelationLinks_ImportIndexesTheRemappedIDs covers the import hook, which
// none of the legs above reach.
//
// Two things have to be true and only one is obvious. The edge must be indexed
// at all — and it must be indexed with the DESTINATION's id, not the exported
// one. The import rewrites relation values to the new ids in a SECOND pass, so
// a hook at the first-pass INSERT would index ids belonging to the workspace
// that produced the bundle: rows that match nothing, on an index whose whole
// job is matching.
func TestRelationLinks_ImportIndexesTheRemappedIDs(t *testing.T) {
t.Parallel()
s := testStore(t)
owner, err := s.CreateUser(models.UserCreate{
Name: "Owner", Email: "relation-index-import@example.com", Password: "passw0rd!",
})
if err != nil {
t.Fatalf("CreateUser: %v", err)
}
const oldColorID = "old-color-indexed"
export := &models.WorkspaceExport{
Version: 1,
ExportedAt: "2026-09-11T00:00:00Z",
Workspace: models.WorkspaceExportMeta{Name: "Index Archive", Slug: "index-archive"},
Collections: []models.CollectionExport{
{
ID: "old-coll-colors", Name: "Colors", Slug: "colors", Prefix: "COLO",
Schema: `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"}]}`,
CreatedAt: "2026-09-11T00:00:00Z", UpdatedAt: "2026-09-11T00:00:00Z",
},
{
ID: "old-coll-cars", Name: "Cars", Slug: "cars", Prefix: "CAR",
Schema: `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"},{"key":"color","type":"relation","collection":"colors"}]}`,
CreatedAt: "2026-09-11T00:00:00Z", UpdatedAt: "2026-09-11T00:00:00Z",
},
},
Items: []models.ItemExport{
{
ID: oldColorID, CollectionID: "old-coll-colors", Title: "Imported Red", Slug: "imported-red",
Fields: `{}`, Tags: `[]`,
CreatedAt: "2026-09-11T00:00:00Z", UpdatedAt: "2026-09-11T00:00:00Z",
},
{
ID: "old-car-indexed", CollectionID: "old-coll-cars", Title: "Imported Car", Slug: "imported-car",
Fields: `{"color":"` + oldColorID + `"}`, Tags: `[]`,
CreatedAt: "2026-09-11T00:00:00Z", UpdatedAt: "2026-09-11T00:00:00Z",
},
},
}
ws, err := s.ImportWorkspace(export, "Index Archive Target", owner.ID, "")
if err != nil {
t.Fatalf("ImportWorkspace: %v", err)
}
items, err := s.ListItems(ws.ID, models.ItemListParams{})
if err != nil {
t.Fatalf("ListItems: %v", err)
}
var red, car models.Item
for _, it := range items {
switch it.Title {
case "Imported Red":
red = it
case "Imported Car":
car = it
}
}
if red.ID == "" || car.ID == "" {
t.Fatalf("the bundle did not import both items; the assertion below would be vacuous (got %d items)", len(items))
}
n, err := s.CountRelationBacklinks(red.ID, ws.ID, unrestricted)
if err != nil {
t.Fatalf("CountRelationBacklinks: %v", err)
}
if n != 1 {
t.Fatalf("the imported target is referenced %d times, want 1 — the import hook did not run, or ran before the id remap", n)
}
links, err := s.GetRelationBacklinks(red.ID, ws.ID, 50, 0, unrestricted)
if err != nil {
t.Fatalf("GetRelationBacklinks: %v", err)
}
if len(links) != 1 || links[0].SourceItemID != car.ID {
t.Errorf("backlink = %+v, want the DESTINATION's car id %s — an index built at the first-pass insert would hold the exported id instead", links, car.ID)
}
}
// TestRelationLinks_RestoreAfterASchemaChangeReindexes is codex round 1's
// second P1, and it is a hole my own site enumeration could not have found.
//
// I enumerated the sites that WRITE items.fields. Restore writes no fields —
// it clears deleted_at — so it never appeared. But the index depends on
// (blob, schema) AND on the item being live, and the collection reindex
// deliberately skips soft-deleted items. So: delete an item, change the
// collection's schema while it is gone, restore it, and its rows are whatever
// they were before the deletion — stale against a schema that has moved.
//
// Both directions matter. A field that BECAME a relation leaves the restored
// item missing edges; a field that stopped being one leaves it with edges it
// should not have.
func TestRelationLinks_RestoreAfterASchemaChangeReindexes(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
car := carPointingAt(t, s, ws, cars, "Time Traveller", red.ID)
if n := countFor(t, s, red, ws); n != 1 {
t.Fatalf("setup: referenced by %d, want 1", n)
}
// Gone while the schema moves under it.
if err := s.DeleteItem(car.ID); err != nil {
t.Fatalf("DeleteItem: %v", err)
}
plain := `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"},{"key":"color","type":"text"}]}`
if _, err := s.UpdateCollection(cars.ID, models.CollectionUpdate{Schema: &plain}); err != nil {
t.Fatalf("UpdateCollection: %v", err)
}
if _, err := s.RestoreItem(car.ID); err != nil {
t.Fatalf("RestoreItem: %v", err)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Errorf("the restored item is referenced %d times, want 0 — `color` is a TEXT field now, and the reindex that would have noticed skipped this item because it was deleted at the time", n)
}
// The other direction: make it a relation again while the item is gone,
// and the restore must FIND the edge.
if err := s.DeleteItem(car.ID); err != nil {
t.Fatalf("DeleteItem (second): %v", err)
}
back := `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"},{"key":"color","type":"relation","collection":"colors"}]}`
if _, err := s.UpdateCollection(cars.ID, models.CollectionUpdate{Schema: &back}); err != nil {
t.Fatalf("UpdateCollection (back): %v", err)
}
if _, err := s.RestoreItem(car.ID); err != nil {
t.Fatalf("RestoreItem (second): %v", err)
}
if n := countFor(t, s, red, ws); n != 1 {
t.Errorf("the restored item is referenced %d times, want 1 — `color` is a relation again and the restore must derive the edge the reindex could not", n)
}
}
// TestRelationLinks_AnInterruptedBackfillIsRetriedNotSkipped is codex round
// 1's first P1.
//
// The guard used to be "does item_relation_links have any row". Rows commit
// per item, so a crash partway through leaves the table NON-EMPTY AND
// INCOMPLETE — and that guard then skips forever, with the missing edges never
// derived. The rebuild story ("delete the table") only ever covered deliberate
// deletion.
//
// Completion is now recorded explicitly and only after a full pass, so a table
// that is partially populated with no marker must be RE-DERIVED.
func TestRelationLinks_AnInterruptedBackfillIsRetriedNotSkipped(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, colors, cars, red := relationIndexFixture(t, s)
blue := createTestItem(t, s, ws.ID, colors.ID, "Blue", "")
carPointingAt(t, s, ws, cars, "Indexed Car", red.ID)
carPointingAt(t, s, ws, cars, "Missed Car", blue.ID)
// The shape an interrupted run leaves behind: one item's rows present,
// another's missing, and NO completion marker.
if _, err := s.db.Exec(s.q(`DELETE FROM item_relation_links WHERE target_item_id = ?`), blue.ID); err != nil {
t.Fatalf("simulate partial index: %v", err)
}
if _, err := s.db.Exec(s.q(`DELETE FROM platform_settings WHERE key = ?`), relationLinksBackfilledFlag); err != nil {
t.Fatalf("clear marker: %v", err)
}
if n := countFor(t, s, blue, ws); n != 0 {
t.Fatalf("the partial state was not established (blue count %d); the assertion below would pass for the wrong reason", n)
}
if n := countFor(t, s, red, ws); n != 1 {
t.Fatalf("the partial state removed too much (red count %d)", n)
}
res, err := s.BackfillRelationLinks()
if err != nil {
t.Fatalf("BackfillRelationLinks: %v", err)
}
if res.Skipped {
t.Fatal("the backfill SKIPPED a partially populated table; under the old any-row guard the missing edges would never be derived")
}
if n := countFor(t, s, blue, ws); n != 1 {
t.Errorf("the missed edge is still missing (count %d) after a retry", n)
}
// And the already-indexed one is not duplicated — the pass is idempotent
// because replaceRelationLinks deletes before it inserts.
if n := countFor(t, s, red, ws); n != 1 {
t.Errorf("the already-indexed edge is now counted %d times; a repeat pass must replace rather than append", n)
}
}
// TestRelationLinks_BackfillClearsAStaleEdge is codex round 2's first P1.
//
// The backfill used to SKIP an item whose blob carries no edges, as an
// optimisation. That is wrong for exactly the state it exists to repair: an
// interrupted pass can leave a STALE row for an item whose value has since
// been cleared, and skipping it preserves that row and then writes the
// completion marker over it — permanently.
//
// The retry test above covers a MISSING edge. This one covers the opposite,
// which is the case the optimisation hid.
func TestRelationLinks_BackfillClearsAStaleEdge(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
car := carPointingAt(t, s, ws, cars, "Cleared Car", red.ID)
// The item's value is gone, but the index still carries the edge — the
// shape an interrupted pass leaves behind. Written directly so the hook
// does not repair it first.
cleared, _ := json.Marshal(map[string]any{"status": "open"})
if _, err := s.db.Exec(s.q(`UPDATE items SET fields = ? WHERE id = ?`), string(cleared), car.ID); err != nil {
t.Fatalf("clear the value behind the hook's back: %v", err)
}
if _, err := s.db.Exec(s.q(`DELETE FROM platform_settings WHERE key = ?`), relationLinksBackfilledFlag); err != nil {
t.Fatalf("clear marker: %v", err)
}
if n := countFor(t, s, red, ws); n != 1 {
t.Fatalf("the stale state was not established (count %d); the assertion below would pass for the wrong reason", n)
}
if _, err := s.BackfillRelationLinks(); err != nil {
t.Fatalf("BackfillRelationLinks: %v", err)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Errorf("the stale edge survived the backfill (count %d) — an item with zero edges is precisely the one that must still be rewritten, because replaceRelationLinks deletes before it inserts", n)
}
}
// TestRelationLinks_ASourceInADeletedCollectionIsNotCounted is codex round 2's
// disclosure P1.
//
// DeleteCollection soft-deletes the COLLECTION and leaves its items live. The
// reverse query joins collections, so without a deleted_at filter a source
// sitting in a collection the viewer can no longer open still appears — and
// for an Unrestricted viewer there is no collection filter anywhere else to
// catch it.
func TestRelationLinks_ASourceInADeletedCollectionIsNotCounted(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
doomed, err := s.CreateCollection(ws.ID, models.CollectionCreate{
Name: "Doomed",
Schema: `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"},{"key":"color","type":"relation","collection":"colors"}]}`,
})
if err != nil {
t.Fatalf("CreateCollection: %v", err)
}
blob, _ := json.Marshal(map[string]any{"status": "open", "color": red.ID})
if _, err := s.CreateItem(ws.ID, doomed.ID, models.ItemCreate{Title: "Doomed Car", Fields: string(blob)}); err != nil {
t.Fatalf("CreateItem: %v", err)
}
// CONTROL: a source in a LIVE collection, so the assertion below is about
// the deletion rather than about the query returning nothing.
carPointingAt(t, s, ws, cars, "Surviving Car", red.ID)
if n := countFor(t, s, red, ws); n != 2 {
t.Fatalf("setup: referenced by %d, want 2", n)
}
if err := s.DeleteCollection(doomed.ID, ""); err != nil {
t.Fatalf("DeleteCollection: %v", err)
}
if n := countFor(t, s, red, ws); n != 1 {
t.Errorf("referenced by %d, want 1 — a source in a soft-deleted collection is still being disclosed; its items stay live, so only a collections.deleted_at filter hides it", n)
}
links, err := s.GetRelationBacklinks(red.ID, ws.ID, 50, 0, unrestricted)
if err != nil {
t.Fatalf("GetRelationBacklinks: %v", err)
}
if len(links) != 1 {
t.Errorf("the page returned %d rows while the count says 1 — the two ran under different filters", len(links))
}
}
// TestRelationLinks_HardDeleteCascadesTheEdges pins a claim the input matrix
// makes and nothing else checked (codex round 4's nit).
//
// The matrix says a hard delete needs no hook because `ON DELETE CASCADE` on
// source_item_id removes the outgoing edges with the row. That is a claim
// about a FOREIGN KEY being enforced — which on SQLite depends on
// `foreign_keys=on` being set by the connection, not on the DDL alone. A
// matrix row resting on "the schema says so" is worth one test.
func TestRelationLinks_HardDeleteCascadesTheEdges(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
car := carPointingAt(t, s, ws, cars, "Doomed Car", red.ID)
if n := countFor(t, s, red, ws); n != 1 {
t.Fatalf("setup: referenced by %d, want 1", n)
}
// Hard delete, not soft: the row goes, and the FK must take its edges.
if _, err := s.db.Exec(s.q(`DELETE FROM items WHERE id = ?`), car.ID); err != nil {
t.Fatalf("hard delete: %v", err)
}
var orphans int
if err := s.db.QueryRow(s.q(`SELECT COUNT(*) FROM item_relation_links WHERE source_item_id = ?`), car.ID).Scan(&orphans); err != nil {
t.Fatalf("count orphan rows: %v", err)
}
if orphans != 0 {
t.Errorf("%d index row(s) survived a hard delete of their source — the cascade is not firing, and the input matrix's 'no hook needed here' reason does not hold", orphans)
}
if n := countFor(t, s, red, ws); n != 0 {
t.Errorf("referenced by %d after the source was hard-deleted, want 0", n)
}
}
// TestRelationLinks_BacklinkCarriesTheFieldLabel pins codex round 6's nit,
// which was a field declared, documented, consumed by the UI — and never
// populated, so every group rendered its raw key.
//
// The label is what the schema author wrote for humans, so the choice was to
// populate it or delete the field. This asserts the populated version, and the
// fallback case (a field with no label) alongside it, because the UI branches
// on exactly that.
func TestRelationLinks_BacklinkCarriesTheFieldLabel(t *testing.T) {
t.Parallel()
s := testStore(t)
ws, _, _, red := relationIndexFixture(t, s)
// `owner` carries a label; `sidekick` deliberately does not.
labelled, err := s.CreateCollection(ws.ID, models.CollectionCreate{
Name: "Labelled",
Schema: `{"fields":[
{"key":"status","type":"select","options":["open","done"],"default":"open"},
{"key":"owner","label":"Responsible Colour","type":"relation","collection":"colors"},
{"key":"sidekick","type":"relation","collection":"colors"}
]}`,
})
if err != nil {
t.Fatalf("CreateCollection: %v", err)
}
blob, _ := json.Marshal(map[string]any{"status": "open", "owner": red.ID, "sidekick": red.ID})
if _, err := s.CreateItem(ws.ID, labelled.ID, models.ItemCreate{Title: "Two Ways", Fields: string(blob)}); err != nil {
t.Fatalf("CreateItem: %v", err)
}
links, err := s.GetRelationBacklinks(red.ID, ws.ID, 50, 0, unrestricted)
if err != nil {
t.Fatalf("GetRelationBacklinks: %v", err)
}
if len(links) != 2 {
t.Fatalf("want 2 edges (one per relation field), got %d", len(links))
}
byKey := map[string]RelationBacklink{}
for _, l := range links {
byKey[l.FieldKey] = l
}
if got := byKey["owner"].FieldLabel; got != "Responsible Colour" {
t.Errorf("owner's label = %q, want %q — the field was declared and consumed while nothing populated it", got, "Responsible Colour")
}
// No label in the schema means an EMPTY one here, not the key: the client
// owns that fallback, and duplicating it server-side would make the two
// disagree the day either changes.
if got := byKey["sidekick"].FieldLabel; got != "" {
t.Errorf("sidekick's label = %q, want empty — a field with no schema label must not have one invented for it", got)
}
}
// TestRelationBacklinksDoNoSecondPoolRead is the guard for the invariant round
// 7's P1 cost me.
//
// GetRelationBacklinks holds the rows cursor open while it builds the result.
// The first version of the field-LABEL lookup issued a SELECT per distinct
// collection inside that loop — which needs a SECOND pool connection, and when
// the pool is saturated the second connection never arrives. That is an
// application deadlock no SQLSTATE names, the same class BUG-2409 closed for
// the collection writers, and it only shows up under load.
//
// With MaxOpenConns(1) the open cursor owns the only connection, so ANY second
// pool read here deadlocks deterministically instead of rarely. Works on both
// dialects.
func TestRelationBacklinksDoNoSecondPoolRead(t *testing.T) {
s := testStore(t)
ws, _, cars, red := relationIndexFixture(t, s)
// Two sources in DIFFERENT collections, so a per-collection lookup would
// have to issue more than one — a single-collection fixture would let the
// broken shape pass on a lucky cache hit.
carPointingAt(t, s, ws, cars, "Pool Car", red.ID)
other, err := s.CreateCollection(ws.ID, models.CollectionCreate{
Name: "Garages Too",
Schema: `{"fields":[{"key":"status","type":"select","options":["open","done"],"default":"open"},{"key":"color","label":"Paint","type":"relation","collection":"colors"}]}`,
})
if err != nil {
t.Fatalf("CreateCollection: %v", err)
}
blob, _ := json.Marshal(map[string]any{"status": "open", "color": red.ID})
if _, err := s.CreateItem(ws.ID, other.ID, models.ItemCreate{Title: "Pool Garage", Fields: string(blob)}); err != nil {
t.Fatalf("CreateItem: %v", err)
}
// From here the pool has exactly one connection.
s.db.SetMaxOpenConns(1)
done := make(chan error, 1)
go func() {
_, err := s.GetRelationBacklinks(red.ID, ws.ID, 50, 0, unrestricted)
done <- err
}()
select {
case err := <-done:
if err != nil {
t.Fatalf("GetRelationBacklinks: %v", err)
}
case <-time.After(30 * time.Second):
t.Fatal("GetRelationBacklinks did not complete under MaxOpenConns(1): it issued a pool read while holding its own rows cursor, and is waiting for a connection it is itself occupying")
}
}
+27
View File
@@ -7,6 +7,7 @@ import type {
CollectionCreate, CollectionCreate,
CollectionUpdate, CollectionUpdate,
Backlink, Backlink,
RelationBacklinksPage,
Item, Item,
BulkItemsRequest, BulkItemsRequest,
BulkItemsResponse, BulkItemsResponse,
@@ -1370,6 +1371,32 @@ export const api = {
})}` })}`
), ),
/**
* Inbound `relation` FIELD references to an item — the "Referenced
* by" section (PLAN-2857 U5). Distinct from `backlinks`, which is
* `[[...]]` mentions in prose: these are typed field edges and each
* carries the `field_key` it points through.
*
* `total` is the count under the CALLER'S visibility, not the true
* one. A viewer with partial access sees their own number, because a
* true count would disclose the existence of items they cannot read.
*
* No cross-workspace tier: a relation names an item id and PLAN-2857
* v1 excludes cross-workspace targets, so there is nothing foreign to
* page through.
*/
relationBacklinks: (
ws: string,
slug: string,
opts?: { limit?: number; offset?: number }
) =>
request<RelationBacklinksPage>(
`/workspaces/${ws}/items/${slug}/relation-backlinks${qs({
limit: opts?.limit,
offset: opts?.offset,
})}`
),
/** Get child items linked to a parent item */ /** Get child items linked to a parent item */
children: (ws: string, slug: string) => children: (ws: string, slug: string) =>
request<Item[]>(`/workspaces/${ws}/items/${slug}/children`), request<Item[]>(`/workspaces/${ws}/items/${slug}/children`),
@@ -0,0 +1,261 @@
<script lang="ts">
import { SvelteMap } from 'svelte/reactivity';
import { api } from '$lib/api/client';
import type { RelationBacklink } from '$lib/types';
interface Props {
wsSlug: string;
username: string;
itemSlug: string;
itemId: string;
}
let { wsSlug, username, itemSlug, itemId }: Props = $props();
const PAGE_LIMIT = 50;
let rows = $state<RelationBacklink[]>([]);
/**
* The server's count under THIS viewer's visibility — not the true one.
* "Referenced by 3" means "by 3 you can see", which is the ratified
* decision: a true count would tell the viewer that items they cannot open
* exist. Rendered from `total` rather than `rows.length` so the header is
* right when the list is paginated.
*/
let total = $state(0);
let loading = $state(true);
let loadingMore = $state(false);
let error = $state('');
let hasMore = $state(false);
$effect(() => {
void itemSlug;
void itemId;
void wsSlug;
loadFirstPage();
});
async function loadFirstPage() {
// Capture the request identity BEFORE the await. ItemDetail reuses this
// panel across a no-{#key} item switch — the props just change — so a
// slower load for item A must not overwrite B's rows or push A's count
// into the parent's badge. Same fence BacklinksPanel carries, for the
// same reason and the same bug class (PLAN-2105 / TASK-2112).
const reqSlug = itemSlug;
const reqWs = wsSlug;
loading = true;
error = '';
try {
const page = await api.items.relationBacklinks(reqWs, reqSlug, { limit: PAGE_LIMIT });
if (reqSlug !== itemSlug || reqWs !== wsSlug) return;
rows = page.relation_backlinks;
total = page.total;
hasMore = page.relation_backlinks.length === PAGE_LIMIT && page.total > page.relation_backlinks.length;
} catch (err) {
if (reqSlug !== itemSlug || reqWs !== wsSlug) return;
error = err instanceof Error ? err.message : 'Failed to load references';
rows = [];
total = 0;
hasMore = false;
} finally {
if (reqSlug === itemSlug && reqWs === wsSlug) loading = false;
}
}
async function loadMore() {
const reqSlug = itemSlug;
const reqWs = wsSlug;
loadingMore = true;
try {
const page = await api.items.relationBacklinks(reqWs, reqSlug, {
limit: PAGE_LIMIT,
offset: rows.length
});
if (reqSlug !== itemSlug || reqWs !== wsSlug) return;
rows = [...rows, ...page.relation_backlinks];
total = page.total;
hasMore = rows.length < page.total;
} catch {
// A failed "show more" leaves what is already displayed alone —
// dropping loaded rows because a later page failed would be worse
// than stopping.
if (reqSlug === itemSlug && reqWs === wsSlug) hasMore = false;
} finally {
if (reqSlug === itemSlug && reqWs === wsSlug) loadingMore = false;
}
}
/**
* Grouped by the FIELD that points here, not by collection.
*
* That is the difference from "Mentioned in": a relation edge is typed, so
* "three items reference this through `owner`" is the useful reading, and
* an item referenced through two different fields genuinely belongs under
* both. SvelteMap's insertion order preserves the server's ordering.
*/
type Group = { key: string; label: string; rows: RelationBacklink[] };
let groups = $derived.by<Group[]>(() => {
const map = new SvelteMap<string, Group>();
for (const row of rows) {
const key = row.field_key;
let group = map.get(key);
if (!group) {
group = { key, label: row.field_label || row.field_key, rows: [] };
map.set(key, group);
}
group.rows.push(row);
}
return Array.from(map.values());
});
function rowHref(row: RelationBacklink): string {
// A row with no ref is a legacy item with no item_number; its slug is
// not carried here, so it links to the collection rather than 404ing
// on an empty ref segment.
if (!row.source_ref) return `/${username}/${wsSlug}/${row.collection_slug}`;
return `/${username}/${wsSlug}/${row.collection_slug}/${row.source_ref}`;
}
/**
* One source can point at the target through TWO fields, so
* `source_item_id` alone is not unique. Svelte 5 rejects duplicate keys in
* dev and silently reuses DOM in prod, which would render one of the two.
*/
function rowKey(row: RelationBacklink, index: number): string {
return `${row.source_item_id}|${row.field_key}|${index}`;
}
// Collapse entirely on zero — most items are referenced by nothing and an
// empty header is noise. A failed fetch still shows a one-liner, because
// silently dropping it is worse than a small note.
let shouldRender = $derived(loading || error !== '' || rows.length > 0);
</script>
{#if shouldRender}
<div class="relation-backlinks-panel">
<div class="panel-header">
<h3>Referenced by</h3>
{#if !loading && !error && total > 0}
<span class="count">{total}</span>
{/if}
</div>
{#if loading}
<div class="loading">Loading references…</div>
{:else if error}
<div class="error">{error}</div>
{:else}
{#each groups as group (group.key)}
<div class="field-group">
<div class="field-label">{group.label}</div>
<ul class="rows">
{#each group.rows as row, i (rowKey(row, i))}
<li class="row">
<a href={rowHref(row)} class="row-link">
{#if row.source_ref}
<span class="row-ref">{row.source_ref}</span>
{/if}
<span class="row-title">{row.source_title}</span>
</a>
</li>
{/each}
</ul>
</div>
{/each}
{#if hasMore}
<div class="more">
<button type="button" class="more-btn" disabled={loadingMore} onclick={loadMore}>
{loadingMore ? 'Loading…' : 'Show more'}
</button>
</div>
{/if}
{/if}
</div>
{/if}
<style>
.relation-backlinks-panel {
padding: var(--space-4) 0;
border-top: 1px solid var(--border);
}
.panel-header {
display: flex;
align-items: baseline;
gap: var(--space-2);
margin-bottom: var(--space-3);
}
.panel-header h3 {
margin: 0;
font-size: 0.95em;
}
.count {
font-size: 0.85em;
color: var(--text-muted);
}
.loading,
.error {
font-size: 0.85em;
color: var(--text-muted);
}
.field-group {
margin-bottom: var(--space-3);
}
.field-label {
font-size: 0.8em;
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.04em;
margin-bottom: var(--space-1);
}
.rows {
list-style: none;
margin: 0;
padding: 0;
}
.row {
padding: var(--space-1) 0;
}
.row-link {
display: inline-flex;
align-items: baseline;
gap: var(--space-2);
text-decoration: none;
color: inherit;
}
.row-link:hover .row-title {
text-decoration: underline;
}
.row-ref {
font-family: var(--font-mono, monospace);
font-size: 0.8em;
color: var(--text-muted);
}
.more-btn {
background: none;
border: 1px solid var(--border);
border-radius: var(--radius-sm, 4px);
padding: var(--space-1) var(--space-2);
font-size: 0.85em;
cursor: pointer;
color: inherit;
}
.more-btn:disabled {
opacity: 0.6;
cursor: default;
}
</style>
@@ -34,6 +34,7 @@
} from '$lib/components/timeline/feed'; } from '$lib/components/timeline/feed';
import ChildItems from '$lib/components/ChildItems.svelte'; import ChildItems from '$lib/components/ChildItems.svelte';
import BacklinksPanel from '$lib/components/BacklinksPanel.svelte'; import BacklinksPanel from '$lib/components/BacklinksPanel.svelte';
import RelationBacklinksPanel from '$lib/components/RelationBacklinksPanel.svelte';
import { goto } from '$app/navigation'; import { goto } from '$app/navigation';
import { relativeTime, wikiLinksToMarkdown, markdownToWikiLinks, cleanBrokenLinks, unescapeDocLinks } from '$lib/utils/markdown'; import { relativeTime, wikiLinksToMarkdown, markdownToWikiLinks, cleanBrokenLinks, unescapeDocLinks } from '$lib/utils/markdown';
import { toastStore } from '$lib/stores/toast.svelte'; import { toastStore } from '$lib/stores/toast.svelte';
@@ -6068,6 +6069,22 @@
onCountChange={(n) => { if (keyedSlug !== itemSlug) return; backlinksCount = n; }} onCountChange={(n) => { if (keyedSlug !== itemSlug) return; backlinksCount = n; }}
/> />
</div> </div>
<!--
Referenced-by panel (PLAN-2857 U5). Sits beside the
mentioned-in panel rather than inside it: a `[[wiki-link]]` in
prose and a typed `relation` FIELD value are different
relationships, and merging them would produce a list where
half the entries have a snippet and half have a field name.
Its own count is the VIEWER'S count, not the true one.
-->
<div id="item-relation-backlinks">
<RelationBacklinksPanel
{wsSlug}
{username}
{itemSlug}
itemId={item.id}
/>
</div>
{/if} {/if}
</div><!-- /tab-panel Relationships --> </div><!-- /tab-panel Relationships -->
+37
View File
@@ -1339,6 +1339,43 @@ export interface ItemLinkCreate {
* Mirrors `internal/models/backlink.go`. The Phase 3 UI (TASK-1596) renders * Mirrors `internal/models/backlink.go`. The Phase 3 UI (TASK-1596) renders
* these as the "Mentioned in" panel beneath an item's content. * these as the "Mentioned in" panel beneath an item's content.
*/ */
/**
* One item that points at another through a `relation` FIELD — the
* "Referenced by" section's data (PLAN-2857 U5).
*
* Deliberately not a `Backlink`. A wiki backlink is a mention in prose and
* carries a `snippet`; this is a typed field-valued edge and carries the
* `field_key` it points through, which is what lets the section say
* "referenced by CAR-3 via Colour" rather than just "referenced by". There is
* no snippet because a field value has no surrounding text to quote.
*/
export interface RelationBacklink {
source_item_id: string;
/** Empty on a legacy item with no item_number — render the title alone. */
source_ref: string;
source_title: string;
collection_slug: string;
/** The schema key the value sits under. */
field_key: string;
/** The field's human label, when the schema gives one. */
field_label?: string;
}
/**
* The "Referenced by" payload.
*
* `total` is the COUNT UNDER THE VIEWER'S VISIBILITY, not the true one — a
* true count would tell the viewer that items they cannot see exist. So
* "Referenced by 3" means "by 3 you can see", and the number is computed under
* the same filters as the page it accompanies.
*/
export interface RelationBacklinksPage {
relation_backlinks: RelationBacklink[];
total: number;
limit: number;
offset: number;
}
export interface Backlink { export interface Backlink {
source_item_id: string; source_item_id: string;
source_ref: string; source_ref: string;