xarmian ec71903be7 feat(store): JSONB NOT NULL hardening on items/views + handler shape validation (IDEA-1486+1488) (#566)
* feat(store): NOT NULL hardening on items.fields/tags + views.config (IDEA-1486)

Paired ship of IDEA-1486 (sibling-table JSONB NOT NULL hardening) and
IDEA-1488 (handler-layer shape validation for ViewUpdate/CollectionUpdate).
Generalizes the IDEA-1484 / collections.settings precedent (PR #562) to the
remaining nullable JSON columns and closes the shape-validation gap that
NOT NULL alone doesn't cover.

Schema layer (IDEA-1486 floor):
- migrations/056_items_jsonb_not_null.sql: rebuild items with
  fields TEXT NOT NULL DEFAULT '{}' and tags TEXT NOT NULL DEFAULT '[]',
  preserving all 7 indexes, recreating the 3 items_fts triggers, and
  rebuilding the FTS5 index. Foreign-keys-off / on bookends are lifted
  outside the IDEA-1485 atomic-tx wrapper.
- migrations/057_views_config_not_null.sql: rebuild views with
  config TEXT NOT NULL DEFAULT '{}'.
- pgmigrations/035 + 036: SET NOT NULL + SET DEFAULT on the three JSONB
  columns. Split per-table to mirror the SQLite per-table file granularity.

Store layer (IDEA-1486 floor):
- items.go UpdateItem and views.go UpdateView normalize "" -> "{}" / "[]"
  before writing. Same boundary pattern as CreateItem and the IDEA-1484
  precedent at collections.go:248.
- export.go ImportWorkspace coerces empty-string AND malformed JSON at
  import time on items.fields, items.tags, and collections.settings.
  Malformed input is coerce-and-log via slog.Warn (length only, never raw
  value) so legacy bundles don't fail-stop on one bad row.
- remapFieldIDs early-returns "{}" on empty input so the second-pass
  UPDATE can't write "" verbatim.
- Migrated the existing fmt.Printf at export.go:329 to slog.Warn for
  consistency.

Handler layer (IDEA-1488 ceiling):
- ViewCreate / ViewUpdate UnmarshalJSON via flexJSONToString with new
  ErrInvalidConfigType sentinel.
- CollectionCreate / CollectionUpdate UnmarshalJSON with new
  ErrInvalidSettingsType sentinel.
- handlers_views.go and handlers_collections.go surface both sentinels
  as 400 with the domain-level message (mirrors the BUG-1144 precedent at
  handlers_items.go:641).

Tests:
- internal/store/items_views_jsonb_test.go: store-coercion + import
  coercion + log-and-coerce-on-malformed + SQLite schema introspection
  (7 indexes + 3 FTS triggers + items_fts virtual table survival) +
  Postgres NOT NULL enforcement + migration re-apply idempotency +
  item_links round-trip after rebuild.
- internal/server/handlers_views_collections_jsonb_test.go: PATCH/POST
  flexible-shape coverage for views.config and collections.settings,
  including domain-level 400 message assertions that the response does
  not leak Go unmarshal internals.

Refs: IDEA-1486, IDEA-1488, IDEA-1484 (precedent), IDEA-1485 (substrate).

* fix(store,models): codex R1 follow-ups for IDEA-1486 / IDEA-1488

Three concrete defects surfaced by codex R1 against the initial paired
ship. All three close holes that defeated parts of the original contract.

P1.1: migration 056 missed the playbook invocation_slug unique index.
- migrations/056_items_jsonb_not_null.sql: recreate the partial UNIQUE
  index idx_items_invocation_slug_per_collection from migration 054
  verbatim after the other 7 indexes. Without it, the application-layer
  pre-check in handlers_items.go:checkUniqueFields would be a TOCTOU
  race with no DB-level guard — the original index that 054 explicitly
  added as the actual uniqueness backstop would be silently dropped
  during the items rebuild.
- items_views_jsonb_test.go: the schema-introspection test now asserts
  8 indexes, not 7. Verified via `grep -rn "ON items(" migrations/`
  that no other items-touching indexes were missed.

P1.2: flexJSONToString didn't validate inner content of JSON-encoded
strings. Pre-fix, `{"config": "[]"}` / `{"settings": "not json"}` /
`{"fields": "[]"}` / `{"tags": "{}"}` slipped past the shape validators
because the `case '"'` branch unmarshalled the envelope and returned
the inner string verbatim — bypassing the whole point of IDEA-1488.
- models/item.go: after unmarshalling the JSON-encoded string, validate
  that the trimmed inner content's first byte matches expectedStart
  ('{' / '[') AND parses as JSON. Empty inner strings still pass
  through to the store-layer empty-string coercion (IDEA-1486 floor),
  so legacy "" → default normalization is preserved.
- The pre-existing ItemUpdate fields/tags path inherits the same
  tightening because it routes through this helper — covered by new
  test file handlers_items_jsonb_inner_shape_test.go.
- Parallel handler tests for views.config and collections.settings
  added to handlers_views_collections_jsonb_test.go.

P2: coerceJSONForImport accepted JSON null as well-formed.
- store/export.go: json.Unmarshal("null", &m) returns err=nil with m
  staying nil; the prior code returned the raw "null" string verbatim,
  which lands as JSONB null on Postgres (satisfies NOT NULL since SQL
  NULL ≠ JSONB null) or text "null" on SQLite. The non-nil check on
  the unmarshalled value routes JSON null to the existing
  log-and-coerce path with the rest of the malformed shapes.
- items_views_jsonb_test.go: extended import test with an item
  carrying fields=null / tags=null; expects both coerced to "{}" /
  "[]" and the structured slog.Warn emitted.

Verified: make test (SQLite) and the full ./... suite against the
existing port-5445 Postgres container both pass cleanly.

Refs: IDEA-1486, IDEA-1488, codex R1 review.

* fix(store,server): codex R2 follow-ups for IDEA-1486 / IDEA-1488

Two defects surfaced by codex R2. P1 is a real ship-breaker; P2 closes
a parity gap that R1 missed.

P1: migration backfill normalized only SQL NULL, not malformed/wrong-
shape JSON.

The four new migrations originally wrote `WHERE x IS NULL`. Rows with
fields = '' / 'null' / '[]' / 'not json' all survived the filter, then
violated the post-migration NOT NULL+shape contract. Concrete ship-
breaker on SQLite: 056 recreates the partial UNIQUE index on
json_extract(fields, '$.invocation_slug') from migration 054, and
json_extract errors on rows whose fields fails json_valid — a single
bad row breaks CREATE INDEX mid-migration. Toggle-verified: with the
NULL-only WHERE, the new SQLite test fails at exactly that CREATE
INDEX with "SQL logic error: malformed JSON (1)".

Widened the backfill clauses:
- migrations/056: UPDATE items WHERE fields IS NULL OR json_valid(fields)=0
  OR json_type(fields)!='object' (same trio for tags with 'array').
- migrations/057: same trio for views.config.
- pgmigrations/035: WHERE fields IS NULL OR jsonb_typeof(fields)!='object'.
  JSONB rejects invalid JSON on write so the json_valid leg isn't
  needed on Postgres; only the shape check matters.
- pgmigrations/036: same shape check on views.config.

Regression tests in internal/store/items_views_jsonb_test.go:
- TestItemsViewsJSONB_SQLiteBackfillRepairsMalformedShapes: applies
  migrations through 053 (skipping 054 which would itself error on
  malformed rows), seeds every observable shape pathology — SQL NULL,
  empty string, JSON null literal, wrong-shape JSON, non-JSON garbage —
  then applies 055/056/057. Asserts every malformed row is repaired AND
  the partial UNIQUE index actually fires on duplicate invocation_slug
  post-rebuild (proving the CREATE INDEX path executed end-to-end).
- TestItemsViewsJSONB_PostgresBackfillRepairsMalformedShapes: parallel
  Postgres coverage; seeds JSONB null / array / primitive via direct
  ::jsonb cast and asserts the widened WHERE clause repairs each.

P2: handleCreateItem didn't unwrap ErrInvalidFieldsType/ErrInvalidTagsType.

R1's flexJSONToString tightening propagated the sentinels through every
UnmarshalJSON path, but handleCreateItem (POST /items) still returned
'invalid JSON: <wrapped>' from decodeJSON. PATCH and the view/collection
POST/PATCH handlers already unwrapped — POST was the outlier.

- internal/server/handlers_items.go: mirror the PATCH-side errors.Is
  handling at the POST path. Brief, three-line diff.
- handlers_items_jsonb_inner_shape_test.go: new TestCreateItem_
  JSONEncodedStringInnerShapeValidated covers POST with fields=`[]`,
  fields=42, tags=`{}`, tags={"x":1}, plus a valid positive control.
  Asserts no "invalid JSON:" wrapper and presence of the sentinel
  message verbatim.

Backfill-pattern audit (codex R2's grep prompt): only 055 / pg-034
(collections.settings, already shipped) exhibits the same NULL-only
WHERE gap. Per the brief: NOT touched — retroactive repair belongs to
a separate IDEA. Other NULL-only backfills (043/pg-023's
oauth_providers, 044/pg-024's expires_at) handle their respective
shapes correctly or aren't JSON columns.

Verified: make test (SQLite) clean. Full ./... suite against the
existing port-5445 Postgres container clean (one unrelated flake in
internal/collab passed on rerun).

Refs: IDEA-1486, IDEA-1488, codex R2 review.
2026-05-16 10:15:59 -04:00
2026-03-26 01:52:36 +00:00
2026-03-26 01:52:36 +00:00

Pad

Project Management for the agent era.

CI Release Go Report Card Container image on GHCR License GitHub Sponsors


One binary. Local-first. No accounts required. Pad gives you a CLI, a web UI, and an AI agent skill — all backed by SQLite, all running on your machine. Your project data never leaves your laptop.

Pad dashboard showing collection summaries, active work, an active plan with progress, and a recent activity feed

Quick Start

brew install PerpetualSoftware/tap/pad
cd your-project
pad init                    # configure, auth, workspace, AI skill — all in one
pad server open             # opens the web UI at localhost:7777

pad init is the smart entry point — it auto-detects what's needed, walks you through each step, and is safe to re-run anytime (it skips finished steps and prints a status summary).

Then, in a fresh agent session in your project, say:

use pad to get IDEA-1

Your new workspace ships with a thoughtful first idea — IDEA-1 — that the agent reads and uses to help you get set up around your actual project. It's the fastest way to go from empty workspace to "okay, this is mine."

Why Pad?

Tools like Linear, Jira, and Notion are built for teams on the cloud. Pad is built for developers on their machine — and for the AI agents working alongside them.

Pad Linear / Jira Notion
Setup pad init Create account, invite team, configure Create account, pick template
AI agents Native /pad skill for 7+ tools Third-party integrations Third-party integrations
Data Local SQLite, you own it Their cloud Their cloud
Offline Full functionality Read-only cache at best Limited
CLI First-class Afterthought None
Price Free, open source Per-seat pricing Per-seat pricing

Features

For Developers

CLI that doesn't get in your way. Create tasks, search items, check status — without leaving the terminal.

pad item create task "Fix OAuth redirect" --priority high
pad item create idea "Real-time collaboration" --category infrastructure
pad item list tasks --status in-progress
pad item search "authentication"
pad project dashboard                   # Project dashboard
pad project next                        # What should I work on?
pad server info                         # How this client is connected to Pad

Web UI that stays out of your way. A clean, dark-themed interface at localhost:7777 with:

  • Board, list, and table views — drag-and-drop between status columns
  • Keyboard navigationj/k to move, Enter to open, Esc to go back, Cmd+K to search
  • Rich text editor — Tiptap-based with markdown, formatting toolbar, and auto-save
  • Wiki-links — type [[Title]] to link between items
  • Real-time updates — agent creates a task in the terminal, it appears in the browser instantly (via SSE)
  • Dashboard — collection overview, active work, plan tracking, activity feed

Pad tasks board view: kanban columns for Open, In-Progress, Done, Cancelled with task cards in each

For AI Agents

Your agent becomes a project partner. Install the /pad skill once, and your AI coding tool can read, create, and update project items through natural language.

pad agent install        # Auto-detects your tools and installs the skill

Works with Claude Code, Cursor, Windsurf, Codex, GitHub Copilot, Amazon Q, and JetBrains Junie.

Then just talk to your project:

> /pad what should I work on next?
> /pad I finished the OAuth fix
> /pad create a task to add rate limiting
> /pad let's brainstorm about the API redesign

Conventions and playbooks teach agents how your project works:

  • Conventions — trigger-based rules like "run tests before marking a task done" or "use conventional commits"
  • Playbooks — multi-step workflows like "when implementing a feature: read the spec, create a branch, write tests first, then implement". Playbooks can declare a kebab-case invocation_slug so users can invoke them directly: /pad ship PLAN-42, /pad release 0.5.0. Fresh startup workspaces ship a generic ship playbook out of the box.
pad item create convention "Run tests before completing tasks" \
  --field trigger=on-task-complete \
  --field scope=all \
  --field priority=must

Agents load relevant conventions automatically. All agent actions are attributed in the activity feed, so you always know what the AI changed.

Onboard agents to a new codebase:

pad workspace onboard    # Analyzes project structure, saves workspace context, and suggests conventions

Collections & Custom Fields

Pad organizes work into collections — typed containers with structured fields.

Built-in collections:

Collection Purpose
Tasks Work items with status, priority, assignee, effort, due date
Ideas Feature ideas with impact and category
Plans Project milestones with progress tracking
Docs Documentation, decisions, reference material
Conventions Project rules that guide agent behavior
Playbooks Multi-step workflows for agents to follow

Create your own with typed fields — select, text, date, number, url, relation, checkbox:

pad collection create "Bug Reports" \
  --fields "severity:select:low,medium,high,critical; browser:text; reproducible:checkbox"

Items get reference numbers automatically (TASK-5, BUG-12) and can be moved between collections with field migration.

Installation

Homebrew (macOS and Linux)

brew install PerpetualSoftware/tap/pad

Build from Source

git clone https://github.com/PerpetualSoftware/pad
cd pad
make build
cp pad ~/.local/bin/   # or /usr/local/bin/

Requires Go 1.26+ and Node.js 22+.

The go install github.com/PerpetualSoftware/pad/cmd/pad@latest path is not supported for the full Pad binary, because the web UI must be built and embedded during the source build.

Docker

docker run -p 127.0.0.1:7777:7777 -v pad-data:/data ghcr.io/perpetualsoftware/pad

This publishes Pad to localhost:7777 on the host machine, which is the recommended default for local use.

Single user, more than one device? Publish to all interfaces so you can reach Pad from your phone, tablet, or another machine on the same LAN, Tailscale network, or home VPN:

docker run -p 7777:7777 -v pad-data:/data ghcr.io/perpetualsoftware/pad

For multi-instance deployments, Pad supports Postgres + Redis via docker-compose.yml — see docs/deployment.md for the full setup.

Binary Download

Pre-built binaries for macOS, Linux, and Windows are available on the releases page.

Getting Started

1. Set up Pad

cd ~/projects/myapp
pad init "My App"

pad init is the smart entry point that handles everything in one command:

  • Configures this client's connection (local server, remote, or Docker)
  • Auto-starts the local server
  • Creates the first admin account on a fresh local install (Docker / remote hosts run pad auth setup on the server instead)
  • Logs you in if needed
  • Creates or links a workspace for the current directory (writes .pad.toml)
  • Installs the /pad skill for any AI tools detected in the project

Run from your project root. Safe to re-run anytime — it skips finished steps and prints a status summary if nothing's needed.

Choose a template with --template, or omit it for an interactive picker grouped by category (Software / People / …):

pad workspace init --list-templates                   # See the full catalog grouped by category
pad init "My App" --template scrum                    # Scrum-style with sprints
pad init "My App" --template product                  # Product management focused
pad init "My Hiring" --template hiring                # Company-side: requisitions, candidates, interview loops, feedback
pad init "Job Search" --template interviewing         # Candidate-side: applications, interviews, companies, contacts

Pad ships templates for software (startup / scrum / product), people workflows (hiring, interviewing), and has reserved categories for research, content, operations, and personal use so the same project-management primitives fit well beyond code projects.

2. Start working

# From the CLI
pad item create task "Set up CI pipeline" --priority high
pad item create idea "Add WebSocket support" --category infrastructure
pad project dashboard

# From the web UI
pad server open              # Opens localhost:7777 in your browser

# From your AI agent
# Just use /pad in Claude Code, Cursor, etc.

3. Teach your agents the rules

pad workspace onboard        # Auto-analyze project, save workspace context, and suggest conventions
# Or browse the convention library
pad library list --type conventions  # Pre-built conventions you can adopt
pad library list --type playbooks    # Pre-built multi-step workflows

4. Optional — connect a desktop AI app via MCP

Pad ships an MCP (Model Context Protocol) server so Claude Desktop, Cursor, or Windsurf can manage items, plans, ideas, and dependencies as native tools, read workspace state by URL, and load multi-step workflows as prompts.

pad mcp install claude-desktop   # or: cursor, windsurf, --all
# Restart the client; pad shows up as the "pad" MCP server.

Tool catalog (v0.4) — eight resource × action tools plus pad_set_workspace (nine total), no flat verb explosion:

Tool Actions
pad_item create, update, delete, get, list, move, link, unlink, deps, star, unstar, starred, comment, list-comments, bulk-update, note, decide
pad_workspace list, members, invite, storage, audit-log
pad_collection list, create
pad_project dashboard, next, standup, changelog
pad_role list, create, delete
pad_search query
pad_playbook list, get, run
pad_meta server-info, version, tool-surface, bootstrap
pad_set_workspace session-default workspace pinning (response embeds the bootstrap blob)

Plus resources at pad://workspaces, pad://workspace/{ws}/dashboard, pad://workspace/{ws}/items, pad://workspace/{ws}/items/{ref}, pad://workspace/{ws}/collections, pad://workspace/{ws}/bootstrap, and pad://_meta/version.

Stability contract — two version constants, both advertised in the initialize handshake under capabilities.experimental.padCmdhelp and capabilities.experimental.padToolSurface (and queryable at pad://_meta/version):

  • cmdhelp_version: "0.1" — CLI help-tree contract (used at dispatch time)
  • tool_surface_version: "0.4" — MCP tool catalog contract (PLAN-1410 trimmed the bootstrap-response shape by ~40%; see internal/mcp/version.go for the full v0.4 changelog)

External agents pin against these so a future rename doesn't break them silently. Errors come back as structured envelopes ({error: {code, message, hint, available_workspaces, ...}}) with a closed eight-code taxonomy.

Full guide at getpad.dev/mcp/local — install paths, action enums per tool, error taxonomy, troubleshooting.

CLI Reference

pad auth configure                    Configure how this client connects to Pad
pad auth setup                        Initialize the first admin account
pad auth login                        Sign in
pad auth whoami                       Show current user

pad server start                      Start the Pad API server
pad server stop                       Stop the Pad server
pad server info                       Show client, connection, and local server status
pad server open                       Open web UI in browser

pad workspace init [name]             Initialize workspace in current directory
pad workspace link <workspace>        Link current directory to an existing workspace
pad workspace list                    List all workspaces
pad workspace switch <workspace>      Switch active workspace
pad workspace context                 Show structured workspace context
pad workspace context set --file X    Update structured workspace context from JSON
pad workspace onboard                 Analyze project, save workspace context, and suggest conventions
pad workspace members                 List workspace members
pad workspace invite <email>          Invite a workspace member
pad workspace join <code>             Accept an invitation
pad workspace export                  Export workspace data
pad workspace import <file>           Import workspace data

pad project dashboard                 Project dashboard
pad project next                      Recommended next task
pad project ready                     Query actionable next items
pad project stale                     Query stalled or attention-worthy items
pad project standup [--days N]        Daily standup report
pad project changelog [--days N]      Release notes from completed items
pad project watch                     Real-time activity stream
pad project reconcile                 Reconcile item and PR state

pad item create <coll> "title"        Create item (task, idea, plan, doc, ...)
pad item list [collection]            List items (filters: --status, --priority, --all)
pad item show <ref>                   Show item detail
pad item update <ref>                 Update item fields
pad item delete <ref>                 Delete item
pad item move <ref> <collection>      Move item between collections
pad item edit <ref>                   Open item in $EDITOR
pad item search "query"               Full-text search across all items
pad item comment <ref> "text"         Add comment to an item
pad item comments <ref>               View item comments
pad item note <ref> "summary"         Append an implementation note to an item
pad item decide <ref> "decision"      Append a decision log entry to an item
pad item block <src> <target>         Create dependency
pad item blocked-by <item> <blk>      Mark item as blocked
pad item deps <ref>                   Show dependencies
pad item unblock <src> <target>       Remove dependency
pad item related <ref>                Show direct relationships for an item
pad item implemented-by <ref>         Show incoming implementers for an item
pad item bulk-update --status X       Batch update multiple items

pad collection list                   List collections with item counts
pad collection create <name>          Create a custom collection

pad library list                      Browse convention and playbook library
pad library activate <title>          Activate a convention or playbook

pad agent install [tool]              Install /pad skill for AI coding tools
pad agent status                      Show supported tools and installation status
pad agent update                      Update installed tool integrations

pad github link [item-ref]            Link current branch's PR to item
pad github status [item-ref]          Show PR status for linked items
pad github unlink <item-ref>          Remove PR link from item

pad webhook list             List workspace webhooks
pad webhook create <url>     Create webhook

All commands accept --format json for machine-readable output and --workspace to target a specific workspace.

Authentication

Pad runs without authentication by default for frictionless local use. For local installs, pad init creates the first admin account inline. The lower-level commands are useful when you're hosting a Pad server (Docker / remote) and need to set up auth on the server host directly:

pad auth setup         # Initialize the first admin account (server host, non-local mode)
pad auth login         # Sign in
pad auth whoami        # Show current user
pad auth logout        # Sign out

Once a user exists, all API requests and web UI access require authentication. Credentials are stored in ~/.pad/credentials.json. Multiple users can be invited to workspaces with role-based access control (owner, editor, viewer).

pad workspace members               # List workspace members
pad workspace invite user@example.com
pad workspace join <code>

Architecture

┌──────────────────────────────────────────────┐
│              pad (single binary)              │
│                                               │
│  ┌──────────┐  ┌──────────┐  ┌────────────┐  │
│  │   CLI    │  │  REST    │  │  Embedded  │  │
│  │ (Cobra)  │  │  API     │  │  Web UI    │  │
│  └────┬─────┘  └────┬─────┘  │ (SvelteKit)│  │
│       │    HTTP      │        └────────────┘  │
│       └──────────────┤                        │
│                ┌─────▼─────┐                  │
│                │  SQLite   │                  │
│                │  + FTS5   │                  │
│                └───────────┘                  │
└───────────────────────────────────────────────┘
  • Go backend — chi router, SQLite via modernc.org/sqlite (pure Go, no CGO), FTS5 full-text search, SSE for real-time updates
  • SvelteKit frontend — Svelte 5, Tiptap editor, drag-and-drop, adapter-static, embedded via go:embed
  • Single binary — serves the API and web UI, runs on macOS, Linux, and Windows
  • Workspace-per-project — each project gets its own workspace linked by a .pad.toml file

All data lives in ~/.pad/pad.db. Your data. Your machine. No telemetry, no cloud, no accounts required.

Contributing

See CONTRIBUTING.md for the development guide.

make build      # Build web UI + Go binary
make test       # Run Go tests
make dev-web    # SvelteKit dev server with hot reload
make install    # Build, install to ~/.local/bin, restart server

Security

See SECURITY.md for reporting vulnerabilities.

License

Apache License 2.0

Languages
Go 65.1%
TypeScript 21.5%
Svelte 12.8%
Shell 0.3%
CSS 0.1%