Files
pad/internal/store/views.go
T
xarmian ec71903be7 feat(store): JSONB NOT NULL hardening on items/views + handler shape validation (IDEA-1486+1488) (#566)
* feat(store): NOT NULL hardening on items.fields/tags + views.config (IDEA-1486)

Paired ship of IDEA-1486 (sibling-table JSONB NOT NULL hardening) and
IDEA-1488 (handler-layer shape validation for ViewUpdate/CollectionUpdate).
Generalizes the IDEA-1484 / collections.settings precedent (PR #562) to the
remaining nullable JSON columns and closes the shape-validation gap that
NOT NULL alone doesn't cover.

Schema layer (IDEA-1486 floor):
- migrations/056_items_jsonb_not_null.sql: rebuild items with
  fields TEXT NOT NULL DEFAULT '{}' and tags TEXT NOT NULL DEFAULT '[]',
  preserving all 7 indexes, recreating the 3 items_fts triggers, and
  rebuilding the FTS5 index. Foreign-keys-off / on bookends are lifted
  outside the IDEA-1485 atomic-tx wrapper.
- migrations/057_views_config_not_null.sql: rebuild views with
  config TEXT NOT NULL DEFAULT '{}'.
- pgmigrations/035 + 036: SET NOT NULL + SET DEFAULT on the three JSONB
  columns. Split per-table to mirror the SQLite per-table file granularity.

Store layer (IDEA-1486 floor):
- items.go UpdateItem and views.go UpdateView normalize "" -> "{}" / "[]"
  before writing. Same boundary pattern as CreateItem and the IDEA-1484
  precedent at collections.go:248.
- export.go ImportWorkspace coerces empty-string AND malformed JSON at
  import time on items.fields, items.tags, and collections.settings.
  Malformed input is coerce-and-log via slog.Warn (length only, never raw
  value) so legacy bundles don't fail-stop on one bad row.
- remapFieldIDs early-returns "{}" on empty input so the second-pass
  UPDATE can't write "" verbatim.
- Migrated the existing fmt.Printf at export.go:329 to slog.Warn for
  consistency.

Handler layer (IDEA-1488 ceiling):
- ViewCreate / ViewUpdate UnmarshalJSON via flexJSONToString with new
  ErrInvalidConfigType sentinel.
- CollectionCreate / CollectionUpdate UnmarshalJSON with new
  ErrInvalidSettingsType sentinel.
- handlers_views.go and handlers_collections.go surface both sentinels
  as 400 with the domain-level message (mirrors the BUG-1144 precedent at
  handlers_items.go:641).

Tests:
- internal/store/items_views_jsonb_test.go: store-coercion + import
  coercion + log-and-coerce-on-malformed + SQLite schema introspection
  (7 indexes + 3 FTS triggers + items_fts virtual table survival) +
  Postgres NOT NULL enforcement + migration re-apply idempotency +
  item_links round-trip after rebuild.
- internal/server/handlers_views_collections_jsonb_test.go: PATCH/POST
  flexible-shape coverage for views.config and collections.settings,
  including domain-level 400 message assertions that the response does
  not leak Go unmarshal internals.

Refs: IDEA-1486, IDEA-1488, IDEA-1484 (precedent), IDEA-1485 (substrate).

* fix(store,models): codex R1 follow-ups for IDEA-1486 / IDEA-1488

Three concrete defects surfaced by codex R1 against the initial paired
ship. All three close holes that defeated parts of the original contract.

P1.1: migration 056 missed the playbook invocation_slug unique index.
- migrations/056_items_jsonb_not_null.sql: recreate the partial UNIQUE
  index idx_items_invocation_slug_per_collection from migration 054
  verbatim after the other 7 indexes. Without it, the application-layer
  pre-check in handlers_items.go:checkUniqueFields would be a TOCTOU
  race with no DB-level guard — the original index that 054 explicitly
  added as the actual uniqueness backstop would be silently dropped
  during the items rebuild.
- items_views_jsonb_test.go: the schema-introspection test now asserts
  8 indexes, not 7. Verified via `grep -rn "ON items(" migrations/`
  that no other items-touching indexes were missed.

P1.2: flexJSONToString didn't validate inner content of JSON-encoded
strings. Pre-fix, `{"config": "[]"}` / `{"settings": "not json"}` /
`{"fields": "[]"}` / `{"tags": "{}"}` slipped past the shape validators
because the `case '"'` branch unmarshalled the envelope and returned
the inner string verbatim — bypassing the whole point of IDEA-1488.
- models/item.go: after unmarshalling the JSON-encoded string, validate
  that the trimmed inner content's first byte matches expectedStart
  ('{' / '[') AND parses as JSON. Empty inner strings still pass
  through to the store-layer empty-string coercion (IDEA-1486 floor),
  so legacy "" → default normalization is preserved.
- The pre-existing ItemUpdate fields/tags path inherits the same
  tightening because it routes through this helper — covered by new
  test file handlers_items_jsonb_inner_shape_test.go.
- Parallel handler tests for views.config and collections.settings
  added to handlers_views_collections_jsonb_test.go.

P2: coerceJSONForImport accepted JSON null as well-formed.
- store/export.go: json.Unmarshal("null", &m) returns err=nil with m
  staying nil; the prior code returned the raw "null" string verbatim,
  which lands as JSONB null on Postgres (satisfies NOT NULL since SQL
  NULL ≠ JSONB null) or text "null" on SQLite. The non-nil check on
  the unmarshalled value routes JSON null to the existing
  log-and-coerce path with the rest of the malformed shapes.
- items_views_jsonb_test.go: extended import test with an item
  carrying fields=null / tags=null; expects both coerced to "{}" /
  "[]" and the structured slog.Warn emitted.

Verified: make test (SQLite) and the full ./... suite against the
existing port-5445 Postgres container both pass cleanly.

Refs: IDEA-1486, IDEA-1488, codex R1 review.

* fix(store,server): codex R2 follow-ups for IDEA-1486 / IDEA-1488

Two defects surfaced by codex R2. P1 is a real ship-breaker; P2 closes
a parity gap that R1 missed.

P1: migration backfill normalized only SQL NULL, not malformed/wrong-
shape JSON.

The four new migrations originally wrote `WHERE x IS NULL`. Rows with
fields = '' / 'null' / '[]' / 'not json' all survived the filter, then
violated the post-migration NOT NULL+shape contract. Concrete ship-
breaker on SQLite: 056 recreates the partial UNIQUE index on
json_extract(fields, '$.invocation_slug') from migration 054, and
json_extract errors on rows whose fields fails json_valid — a single
bad row breaks CREATE INDEX mid-migration. Toggle-verified: with the
NULL-only WHERE, the new SQLite test fails at exactly that CREATE
INDEX with "SQL logic error: malformed JSON (1)".

Widened the backfill clauses:
- migrations/056: UPDATE items WHERE fields IS NULL OR json_valid(fields)=0
  OR json_type(fields)!='object' (same trio for tags with 'array').
- migrations/057: same trio for views.config.
- pgmigrations/035: WHERE fields IS NULL OR jsonb_typeof(fields)!='object'.
  JSONB rejects invalid JSON on write so the json_valid leg isn't
  needed on Postgres; only the shape check matters.
- pgmigrations/036: same shape check on views.config.

Regression tests in internal/store/items_views_jsonb_test.go:
- TestItemsViewsJSONB_SQLiteBackfillRepairsMalformedShapes: applies
  migrations through 053 (skipping 054 which would itself error on
  malformed rows), seeds every observable shape pathology — SQL NULL,
  empty string, JSON null literal, wrong-shape JSON, non-JSON garbage —
  then applies 055/056/057. Asserts every malformed row is repaired AND
  the partial UNIQUE index actually fires on duplicate invocation_slug
  post-rebuild (proving the CREATE INDEX path executed end-to-end).
- TestItemsViewsJSONB_PostgresBackfillRepairsMalformedShapes: parallel
  Postgres coverage; seeds JSONB null / array / primitive via direct
  ::jsonb cast and asserts the widened WHERE clause repairs each.

P2: handleCreateItem didn't unwrap ErrInvalidFieldsType/ErrInvalidTagsType.

R1's flexJSONToString tightening propagated the sentinels through every
UnmarshalJSON path, but handleCreateItem (POST /items) still returned
'invalid JSON: <wrapped>' from decodeJSON. PATCH and the view/collection
POST/PATCH handlers already unwrapped — POST was the outlier.

- internal/server/handlers_items.go: mirror the PATCH-side errors.Is
  handling at the POST path. Brief, three-line diff.
- handlers_items_jsonb_inner_shape_test.go: new TestCreateItem_
  JSONEncodedStringInnerShapeValidated covers POST with fields=`[]`,
  fields=42, tags=`{}`, tags={"x":1}, plus a valid positive control.
  Asserts no "invalid JSON:" wrapper and presence of the sentinel
  message verbatim.

Backfill-pattern audit (codex R2's grep prompt): only 055 / pg-034
(collections.settings, already shipped) exhibits the same NULL-only
WHERE gap. Per the brief: NOT touched — retroactive repair belongs to
a separate IDEA. Other NULL-only backfills (043/pg-023's
oauth_providers, 044/pg-024's expires_at) handle their respective
shapes correctly or aren't JSON columns.

Verified: make test (SQLite) clean. Full ./... suite against the
existing port-5445 Postgres container clean (one unrelated flake in
internal/collab passed on rerun).

Refs: IDEA-1486, IDEA-1488, codex R2 review.
2026-05-16 10:15:59 -04:00

208 lines
5.5 KiB
Go

package store
import (
"database/sql"
"fmt"
"github.com/PerpetualSoftware/pad/internal/models"
)
// CreateView adds a new saved view.
func (s *Store) CreateView(workspaceID string, input models.ViewCreate) (*models.View, error) {
id := newID()
ts := now()
slug := input.Slug
if slug == "" {
slug = slugify(input.Name)
}
slug, err := s.uniqueSlug("views", "workspace_id", workspaceID, slug)
if err != nil {
return nil, fmt.Errorf("generate slug: %w", err)
}
config := input.Config
if config == "" {
config = "{}"
}
viewType := input.ViewType
if viewType == "" {
viewType = "list"
}
_, err = s.db.Exec(s.q(`
INSERT INTO views (id, workspace_id, collection_id, name, slug, view_type, config, sort_order, is_default, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, 0, ?, ?, ?)`),
id, workspaceID, input.CollectionID, input.Name, slug, viewType, config, s.dialect.BoolToInt(false), ts, ts,
)
if err != nil {
return nil, fmt.Errorf("insert view: %w", err)
}
return s.GetView(id)
}
// GetView returns a single view by ID.
func (s *Store) GetView(id string) (*models.View, error) {
var v models.View
var collectionID *string
var isDefault bool
var createdAt, updatedAt string
err := s.db.QueryRow(s.q(`
SELECT id, workspace_id, collection_id, name, slug, view_type, config, sort_order, is_default, created_at, updated_at
FROM views
WHERE id = ?`), id).Scan(
&v.ID, &v.WorkspaceID, &collectionID, &v.Name, &v.Slug, &v.ViewType,
&v.Config, &v.SortOrder, &isDefault, &createdAt, &updatedAt,
)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("get view: %w", err)
}
v.CollectionID = collectionID
v.IsDefault = isDefault
v.CreatedAt = parseTime(createdAt)
v.UpdatedAt = parseTime(updatedAt)
return &v, nil
}
// GetViewBySlug returns a view by its workspace-scoped slug.
func (s *Store) GetViewBySlug(workspaceID, slug string) (*models.View, error) {
var v models.View
var collectionID *string
var isDefault bool
var createdAt, updatedAt string
err := s.db.QueryRow(s.q(`
SELECT id, workspace_id, collection_id, name, slug, view_type, config, sort_order, is_default, created_at, updated_at
FROM views
WHERE workspace_id = ? AND slug = ?`), workspaceID, slug).Scan(
&v.ID, &v.WorkspaceID, &collectionID, &v.Name, &v.Slug, &v.ViewType,
&v.Config, &v.SortOrder, &isDefault, &createdAt, &updatedAt,
)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("get view by slug: %w", err)
}
v.CollectionID = collectionID
v.IsDefault = isDefault
v.CreatedAt = parseTime(createdAt)
v.UpdatedAt = parseTime(updatedAt)
return &v, nil
}
// ListViews returns all views for a collection within a workspace.
func (s *Store) ListViews(workspaceID, collectionID string) ([]models.View, error) {
rows, err := s.db.Query(s.q(`
SELECT id, workspace_id, collection_id, name, slug, view_type, config, sort_order, is_default, created_at, updated_at
FROM views
WHERE workspace_id = ? AND collection_id = ?
ORDER BY sort_order ASC, created_at ASC`), workspaceID, collectionID)
if err != nil {
return nil, fmt.Errorf("list views: %w", err)
}
defer rows.Close()
var views []models.View
for rows.Next() {
var v models.View
var collID *string
var isDefault bool
var createdAt, updatedAt string
if err := rows.Scan(
&v.ID, &v.WorkspaceID, &collID, &v.Name, &v.Slug, &v.ViewType,
&v.Config, &v.SortOrder, &isDefault, &createdAt, &updatedAt,
); err != nil {
return nil, fmt.Errorf("scan view: %w", err)
}
v.CollectionID = collID
v.IsDefault = isDefault
v.CreatedAt = parseTime(createdAt)
v.UpdatedAt = parseTime(updatedAt)
views = append(views, v)
}
return views, rows.Err()
}
// UpdateView modifies an existing view.
func (s *Store) UpdateView(id string, input models.ViewUpdate) (*models.View, error) {
ts := now()
// Build dynamic SET clause
sets := []string{"updated_at = ?"}
args := []interface{}{ts}
if input.Name != nil {
sets = append(sets, "name = ?")
args = append(args, *input.Name)
}
if input.ViewType != nil {
sets = append(sets, "view_type = ?")
args = append(args, *input.ViewType)
}
if input.Config != nil {
// IDEA-1486: normalize the empty-string sentinel to a valid JSON
// object before writing. After the NOT NULL DEFAULT '{}'
// hardening (migration 057 / pgmigrations 036), Postgres rejects
// "" at JSONB type-validation and SQLite would silently store
// invalid JSON. Mirrors CreateView at views.go:24-27 and the
// IDEA-1484 precedent at collections.go:248. Shape validation
// (object vs. array vs. primitive) is handled at the handler
// boundary by ViewUpdate.UnmarshalJSON (IDEA-1488).
config := *input.Config
if config == "" {
config = "{}"
}
sets = append(sets, "config = ?")
args = append(args, config)
}
if input.SortOrder != nil {
sets = append(sets, "sort_order = ?")
args = append(args, *input.SortOrder)
}
args = append(args, id)
query := "UPDATE views SET "
for i, s := range sets {
if i > 0 {
query += ", "
}
query += s
}
query += " WHERE id = ?"
result, err := s.db.Exec(s.q(query), args...)
if err != nil {
return nil, fmt.Errorf("update view: %w", err)
}
n, _ := result.RowsAffected()
if n == 0 {
return nil, sql.ErrNoRows
}
return s.GetView(id)
}
// DeleteView removes a view by ID.
func (s *Store) DeleteView(id string) error {
result, err := s.db.Exec(s.q("DELETE FROM views WHERE id = ?"), id)
if err != nil {
return fmt.Errorf("delete view: %w", err)
}
n, _ := result.RowsAffected()
if n == 0 {
return sql.ErrNoRows
}
return nil
}