pad/internal/models/item.go

package models

import (
	"bytes"
	"database/sql"
	"encoding/json"
	"errors"
	"fmt"
	"net/url"
	"sort"
	"strings"
	"time"
	"unicode/utf8"
)

const (
	ItemFieldGitHubPR            = "github_pr"
	ItemFieldImplementationNotes = "implementation_notes"
	ItemFieldDecisionLog         = "decision_log"
	ItemFieldConvention          = "convention"
)

// reservedItemFieldKeys is the canonical set of field keys Pad itself writes
// into an item's fields blob. They are deliberately NOT declared by any
// collection schema — each is rendered from its own dedicated surface rather
// than as a generic field — which means every code path that reasons about
// fields by consulting a schema is, by construction, blind to them.
//
// That blindness is not hypothetical: it is the shared root of BUG-2627 (the
// CLI types a --field value by schema lookup, so these keys fall through to a
// raw string and become unreadable) and BUG-2674 (MigrateFields drops any key
// absent from the target schema, so a move destroyed them outright). Both were
// code paths that did not know these keys are special.
//
// The set lives here, once, so a caller can ASK instead of re-listing. Before
// this existed there were four constants and a single inline || chain in a CLI
// display path — a shape where the next reserved field added lands in the
// constants, gets wired into whichever surface prompted it, and silently misses
// every other.
//
// ADDING A KEY HERE IS NOT THE WHOLE JOB, and pretending otherwise would
// recreate the drift this set exists to stop. Membership tests inherit it for
// free — MigrateFields' carry, SchemaForMigratedFields, the collection-schema
// gate, the copy preflight's enumeration, the CLI's display filter. Three
// places still need a hand edit, because each needs something a set cannot
// supply:
//
//   - referentialItemFieldKeys below — does the new key point OUT of the item?
//   - reservedFieldLabel (handlers_items_copy_preflight.go) — a human label
//   - RESERVED_FIELD_KEYS (web/src/lib/components/collections/
//     field-editor-types.ts) — the client-side gate, deliberately a separate
//     list because it lowercases and is therefore stricter than this one
//
// TestReservedItemFieldKeysAreStableAndComplete fails on any change to this
// set, which is the reminder to visit all three.
var reservedItemFieldKeys = map[string]struct{}{
	ItemFieldGitHubPR:            {},
	ItemFieldImplementationNotes: {},
	ItemFieldDecisionLog:         {},
	ItemFieldConvention:          {},
}

// ItemWriteWarnings carries advisory notes about a write that succeeded
// (BUG-2850). Nothing here is an error: every field named has been stored.
//
// UndeclaredFields lists keys written into the item's fields blob that the
// collection's schema does not declare. Those keys are ACCEPTED — a census of
// this deployment found 168 live values under 14 such keys, and refusing them
// would have broken read-modify-write on items nobody edited wrongly — but a
// typo and a deliberate extra field are indistinguishable once stored, so the
// write says which keys it did not recognize instead of staying silent.
// DroppedFields lists schema-declared keys the write DISCARDED, in ONE
// disposition covering two passes.
//
// The general case (BUG-3079): an injected schema DEFAULT that fails the same
// `validateFieldType` a supplied value takes. It used to be assigned and
// `continue`d past that check, so the same bytes were refused through one door
// and stored through the other, decided only by who put them there.
//
// The relation case (TASK-2878) is narrower and still has its own pass, because
// "is this a string" and "does this string name a live, visible item" are
// different questions and only the first can be answered in the DB-free
// validator. NOTE for anyone reading that unit's prose: it says an injected
// default is the only route by which a non-string reaches a relation field.
// That was true when written and is no longer — BUG-3079 closed the shape half
// at the validator, so what arrives at the relation pass is already a string.
//
// Dropping rather than refusing, in both cases, because nobody in the request
// typed the value and refusing would make every write into that collection fail
// on a schema defect its author must fix elsewhere. The exception is a REQUIRED
// field, where dropping leaves it absent and the write is refused as required —
// naming the default as the cause. Additive and omitempty, so a clean write is
// byte-identical.
// UnresolvedRelations lists relation keys whose value was STORED without
// resolving to a live item. Only an IMPORT produces these: Dave's ruling
// (day 57) is that import must carry junk relation values rather than refuse
// them, because refusing would break the import of any artifact written before
// referent validation existed. Distinct from DroppedFields — the value is kept,
// not discarded — and from UndeclaredFields, which is about the KEY rather than
// what it points at.
type ItemWriteWarnings struct {
	UndeclaredFields    []string `json:"undeclared_fields,omitempty"`
	DroppedFields       []string `json:"dropped_fields,omitempty"`
	UnresolvedRelations []string `json:"unresolved_relations,omitempty"`
	// OptionsAdded names the select options this write added to its
	// collection's schema (BUG-3446): a library trigger or scope a `blank`
	// workspace's system collection did not list yet. Keyed by field.
	OptionsAdded map[string][]string `json:"options_added,omitempty"`
	// ContentOutcome names where a content write ended up, in the same vocabulary
	// the content_not_applied error uses. On a 200 it takes exactly one value,
	// "applied_pending_flush": the content went to the collaborative document rather
	// than to items.content — the row is updated only by a later
	// ?source=collab-snapshot write, which nothing guarantees happens — so the
	// `content` on this response is the markdown as SENT rather than as stored
	// (BUG-2995).
	//
	// It describes THIS WRITE, not the row's state when you read the response: a
	// concurrent flush may already have updated the row, and the server does not
	// re-read to find out. Either way the caller's move is to re-read rather than
	// to re-send.
	//
	// A caller that reads `content` back to confirm its own write needs this: the
	// stored form may still differ from the sent form after the row is updated,
	// because the markdown makes a round trip through the editor on its way there
	// (measured on BUG-2995's trail: setext headings, bullet markers, emphasis
	// characters, list renumbering and blank-line runs normalise, while markdown
	// already in the editor's preferred form survives unchanged).
	ContentOutcome string `json:"content_outcome,omitempty"`

	// PrunedPendingEdits counts the content-bearing op-log rows above the flush
	// watermark that a direct content write DELETED (BUG-3133). With no writer
	// tab connected, a content write replaces items.content and prunes the
	// op-log, and unflushed edits from a tab that closed before flushing exist
	// nowhere else. A write without a version token, or with
	// overwrite_pending_edits, still does that; this names how many it destroyed.
	PrunedPendingEdits int `json:"pruned_pending_edits,omitempty"`
	// UnguardedContentWrite marks a content write from an agent door (anything
	// but a browser session) that carried neither a version token nor replace
	// (PLAN-3555 phase 1). It replaced whatever was stored since the caller
	// read the item. Phase 2 refuses such a write; send expected_seq (the seq
	// you read) or replace: true.
	UnguardedContentWrite bool `json:"unguarded_content_write,omitempty"`

	// NotUnique describes each value a move or copy DROPPED because the
	// destination collection declares that field unique and another item
	// already holds the value (BUG-2367). Its keys are also in DroppedFields;
	// this says why and what, because for invocation_slug the drop has a
	// consequence the reader must be able to act on — the item stops answering
	// to that slug.
	NotUnique []NotUniqueDrop `json:"not_unique,omitempty"`
}

// NotUniqueDrop is one carried value dropped for colliding on a unique field.
// Holder is the ref of the item holding the value, present ONLY when the caller
// may see that item; Message is the one sentence every surface prints.
type NotUniqueDrop struct {
	Key     string `json:"key"`
	Value   string `json:"value"`
	Holder  string `json:"holder,omitempty"`
	Message string `json:"message"`
}

// ImportUniqueDrop is one unique value a workspace import removed from an
// item because an item earlier in the archive already held it (BUG-3825).
// Item and Holder are refs in the NEW workspace; the importer owns all of it,
// so naming the holder discloses nothing.
type ImportUniqueDrop struct {
	Item   string `json:"item"`
	Key    string `json:"key"`
	Value  string `json:"value"`
	Holder string `json:"holder"`
}

// RelationTarget is one hydrated `relation` value: the item an ID points at,
// in the smallest shape a caller needs to RENDER it without a second request
// (PLAN-2857 U6).
//
// `ID` is always set — it is the stored value, and it is all there is when the
// target is gone or invisible. `Ref` and `Title` are present only when the
// target resolved and the requester may see it, so a consumer must treat them
// as optional rather than assuming a hydrated entry is complete.
//
// `StoredAsText` (BUG-3014, ToolSurfaceVersion 0.50) is set when the stored
// value is not UUID-shaped, so it was never an id to look up: a title a bundle
// import carried verbatim, or legacy free text. An ID-only entry WITHOUT it
// still means "gone or not visible", deliberately indistinguishable. The flag is
// decided from the stored bytes alone, never from a lookup, so it says nothing
// about any target and cannot serve as an existence oracle. Its predicate is
// syntactic on purpose: tying it to the resolver's ladder would let its meaning
// change as the ladder does, without a version bump.
type RelationTarget struct {
	ID           string `json:"id"`
	Ref          string `json:"ref,omitempty"`
	Title        string `json:"title,omitempty"`
	StoredAsText bool   `json:"stored_as_text,omitempty"`
}

// RelationTargetSet is what one `relation_targets` KEY carries: a single target
// object for a `relation` field, or a JSON ARRAY of them, in stored order, for a
// `multi_relation` field (PLAN-2857 U4, lead ruling day 64).
//
// ONE map, not two. A parallel `multi_relation_targets` key was the obvious
// alternative and was ruled against: a consumer rendering "what does this field
// point at?" would have to ask two questions and merge the answers, and the
// field's key already says which shape to expect because the consumer has the
// schema. The cost is this custom marshaller; the benefit is that there is
// exactly one place to look.
//
// A SCALAR ENTRY MARSHALS BYTE-IDENTICALLY to what v0.31 emitted — the whole
// compatibility claim rests on that, and a test asserts it against a literal
// rather than against another call to this code.
type RelationTargetSet struct {
	// One is set for a scalar `relation` key.
	One *RelationTarget
	// List is set for a `multi_relation` key. A non-nil EMPTY list marshals as
	// `[]`, which is distinct from the key being absent: absent means "nothing
	// hydrated this field", `[]` means "this field holds no references".
	List []RelationTarget
}

// NewRelationTargetSet wraps one scalar target.
func NewRelationTargetSet(t RelationTarget) RelationTargetSet {
	return RelationTargetSet{One: &t}
}

// NewRelationTargetList wraps an ordered list of targets.
func NewRelationTargetList(ts []RelationTarget) RelationTargetSet {
	if ts == nil {
		ts = []RelationTarget{}
	}
	return RelationTargetSet{List: ts}
}

func (r RelationTargetSet) MarshalJSON() ([]byte, error) {
	if r.List != nil {
		return json.Marshal(r.List)
	}
	if r.One != nil {
		return json.Marshal(*r.One)
	}
	// Neither set. `null` rather than `{}`: an empty object would read as a
	// target with an empty id, which is a claim, and this state is the absence
	// of one.
	return []byte("null"), nil
}

// UnmarshalJSON accepts both shapes, because the CLI decodes item JSON the
// server produced and must round-trip whichever one it is handed.
func (r *RelationTargetSet) UnmarshalJSON(b []byte) error {
	trimmed := bytes.TrimSpace(b)
	if len(trimmed) == 0 || string(trimmed) == "null" {
		r.One, r.List = nil, nil
		return nil
	}
	if trimmed[0] == '[' {
		var list []RelationTarget
		if err := json.Unmarshal(trimmed, &list); err != nil {
			return err
		}
		if list == nil {
			list = []RelationTarget{}
		}
		r.One, r.List = nil, list
		return nil
	}
	var one RelationTarget
	if err := json.Unmarshal(trimmed, &one); err != nil {
		return err
	}
	r.One, r.List = &one, nil
	return nil
}

// ContentOutcomeAppliedPendingFlush is the only ContentOutcome value that rides on
// a 2xx: the content reached the collaborative document and items.content has not
// caught up. It lives here rather than beside the error-path outcome constants
// because two packages read it — the server writes it, the CLI renders a warning
// on it — and a duplicated string literal is a rename away from one side silently
// never matching (BUG-2995).
const ContentOutcomeAppliedPendingFlush = "applied_pending_flush"

// ContentStatePendingFlush is Item.ContentState's value when the op-log holds
// unflushed content-bearing rows: the live document is ahead of the row, and a
// tab opening the item can catch the row up. The same string as the write
// outcome, for the reason given on ContentOutcomeAppliedPendingFlush.
const ContentStatePendingFlush = ContentOutcomeAppliedPendingFlush

// ContentStateSetAside is Item.ContentState's value when edits the row never
// received were SET ASIDE by a collab schema-version rebuild (BUG-3244): they
// were written under an older editor schema, so they can no longer replay into
// the live document and opening the item does NOT recover them. They are kept,
// readable as raw updates, until someone recovers or discards them. Advice that
// names a remedy must branch on this value; a door that only marks the body
// stale should ask IsContentStateStale instead.
const ContentStateSetAside = "superseded_set_aside"

// The version-row attribution of an op-log recovery (ItemUpdate.Recovered,
// TASK-2198 U4). item_versions.created_by and .source carry no CHECK
// constraint on either dialect; "system" is already the created_by of
// template-seeded items. The web timeline labels them "System" / "Recovered";
// `pad item history` prints them as stored.
const (
	VersionCreatedBySystem = "system"
	VersionSourceRecovery  = "recovery"
	// VersionSourceChatGPT labels a version saved before a content change
	// made through the ChatGPT catalog (TASK-3321 U1b). A client may not
	// claim it.
	VersionSourceChatGPT  = "chatgpt"
	RecoveryChangeSummary = "recovered from an unsaved editor session"
)

// IsContentStateStale reports whether an Item.ContentState value says the
// stored body is behind edits that exist elsewhere, for any reason. It is the
// one helper for doors that mark a body stale without saying what to do.
func IsContentStateStale(state string) bool {
	return state == ContentStatePendingFlush || state == ContentStateSetAside
}

// IsReservedItemField reports whether key is system-written metadata rather than
// a user-facing schema field. Callers that filter, migrate, or render an item's
// fields map should consult this rather than enumerating the constants.
func IsReservedItemField(key string) bool {
	_, ok := reservedItemFieldKeys[key]
	return ok
}

// referentialItemFieldKeys are the reserved keys whose VALUE points at
// something outside the item — a resource whose meaning depends on the
// surrounding workspace's context rather than on the item itself.
//
// The distinction decides how far they travel (BUG-2674, lead ruling). The
// carry rule is one sentence: system-minted NON-REFERENTIAL data carries;
// referential system data carries only where its referent's context still
// holds. implementation_notes and decision_log describe the item's own history
// and are true wherever the item is. github_pr names a repository that is a
// property of the SOURCE workspace's context — carried into a different
// workspace it renders as a live PR link on an item whose project may have no
// relationship to that repo, which is a false statement rather than a preserved
// one.
//
// So this is not an exception to the rule; it is the rule's own qualifier doing
// its job. A same-workspace move leaves the referent's context unchanged, so
// these carry there.
var referentialItemFieldKeys = map[string]struct{}{
	ItemFieldGitHubPR: {},
}

// IsReferentialItemField reports whether a reserved key's value depends on the
// workspace context around it. See referentialItemFieldKeys.
func IsReferentialItemField(key string) bool {
	_, ok := referentialItemFieldKeys[key]
	return ok
}

// ReservedItemFieldKeys returns the reserved keys in a stable order, for
// callers that need to enumerate rather than test membership (schema-key
// validation, error messages). Sorted so the output is deterministic — an
// error message that lists these must not reorder between runs.
func ReservedItemFieldKeys() []string {
	keys := make([]string, 0, len(reservedItemFieldKeys))
	for k := range reservedItemFieldKeys {
		keys = append(keys, k)
	}
	sort.Strings(keys)
	return keys
}

type Item struct {
	ID           string `json:"id"`
	WorkspaceID  string `json:"workspace_id"`
	CollectionID string `json:"collection_id"`
	Title        string `json:"title"`
	Slug         string `json:"slug"`
	Ref          string `json:"ref,omitempty"` // computed: e.g. "TASK-5", "BUG-8"
	Content      string `json:"content"`

	// ContentState reports that the `Content` above is known to be BEHIND the
	// item's live collaborative document (BUG-3000). Empty — and so omitted —
	// whenever the row is current, which is the overwhelmingly common case, so a
	// consumer that does not know this field sees byte-identical responses.
	//
	// Its pending value is ItemWriteWarnings.ContentOutcome's, deliberately: a
	// caller should have ONE story about where content is, whether it learned it
	// from a write response or from a read. The name differs because a write
	// warning describes what a REQUEST did, while this describes what the ROW is.
	// It has a second value no write produces, superseded_set_aside (BUG-3244,
	// ContentStateSetAside): edits a schema rebuild set aside, which opening the
	// item does not restore. It wins when both apply; see IsContentStateStale.
	//
	// The predicate is "the op-log holds a CONTENT-BEARING row above
	// items.content_flushed_op_log_id", i.e. the document is ahead of the row.
	// Rows that provably cannot change the document (SyncStep1 frames, empty
	// updates, byte-identical re-sends) are persisted but not counted (BUG-3124;
	// store/yjs_content_bearing.go). That is deliberately broader than "an
	// applier-path API write is pending flush": it also covers a user typing in an
	// open tab whose edits have not flushed. Both serve stale content, the row does
	// not record WHY the op-log is ahead, and the narrow reading is therefore not
	// expressible — see the BUG-3000 trail.
	//
	// It is emitted ONLY where a real `content` value is, and is built alongside
	// that column for exactly that reason: a query selecting an empty-string literal
	// for content must not claim its empty body is stale.
	ContentState string `json:"content_state,omitempty"`

	Fields         string     `json:"fields"` // JSON string
	Tags           string     `json:"tags"`   // JSON array string
	Pinned         bool       `json:"pinned"`
	SortOrder      int        `json:"sort_order"`
	ParentID       *string    `json:"parent_id,omitempty"`
	CreatedBy      string     `json:"created_by"`
	LastModifiedBy string     `json:"last_modified_by"`
	Source         string     `json:"source"`
	CreatedAt      time.Time  `json:"created_at"`
	UpdatedAt      time.Time  `json:"updated_at"`
	DeletedAt      *time.Time `json:"deleted_at,omitempty"`

	// Assignment: (user, role) pair
	AssignedUserID *string `json:"assigned_user_id,omitempty"`
	AgentRoleID    *string `json:"agent_role_id,omitempty"`
	RoleSortOrder  int     `json:"role_sort_order"`

	// Auto-assigned sequential number within collection
	ItemNumber *int `json:"item_number,omitempty"`

	// Seq is a workspace-scoped monotonically-increasing sequence number
	// stamped on every mutation (create / update / soft-delete /
	// restore). It is the cursor mechanic for the local-first read
	// model's delta sync (PLAN-1343, DOC-1342 design decision #1).
	// Clients track the max seq they have seen and request
	// `?since=<seq>` deltas to resume. Robust against clock-skew /
	// same-millisecond-write / NTP-step correctness holes that an
	// `updated_at` watermark would carry.
	// SERIALISED ALWAYS, no omitempty (BUG-3037): seq is now a write token a
	// client round-trips as `expected_seq`, so a value the server declines to
	// send is a token the client cannot return. A live row's seq is >= 1, so
	// dropping omitempty changes no real payload — it removes the case where a
	// 0 would vanish and be read as "this item has no seq" by a client that
	// then silently writes without a token.
	Seq int64 `json:"seq"`

	// Populated by joins (not stored)
	AssignedUserName  string `json:"assigned_user_name,omitempty"`
	AssignedUserEmail string `json:"assigned_user_email,omitempty"`
	AgentRoleName     string `json:"agent_role_name,omitempty"`
	AgentRoleSlug     string `json:"agent_role_slug,omitempty"`
	AgentRoleIcon     string `json:"agent_role_icon,omitempty"`
	CollectionSlug    string `json:"collection_slug,omitempty"`
	CollectionName    string `json:"collection_name,omitempty"`
	CollectionIcon    string `json:"collection_icon,omitempty"`
	CollectionPrefix  string `json:"collection_prefix,omitempty"`

	// Parent link (populated by enrichItemForResponse / enrichItemsWithParent)
	ParentLinkID         string `json:"parent_link_id,omitempty"`
	ParentRef            string `json:"parent_ref,omitempty"`
	ParentTitle          string `json:"parent_title,omitempty"`
	ParentSlug           string `json:"parent_slug,omitempty"`
	ParentCollectionSlug string `json:"parent_collection_slug,omitempty"`

	// RelationTargets hydrates `relation` field values for READ responses
	// (PLAN-2857 U6): field key -> the item that value names.
	//
	// It sits BESIDE `fields` rather than inside it, by lead ruling. `fields` is
	// a write shape as well as a read shape — a caller reads an item, changes
	// one key and writes the object back — so replacing a relation's ID string
	// with an object there would break that round trip for every existing
	// client. Here it is additive and omitempty: a consumer that does not know
	// the key ignores it.
	//
	// A value that cannot be hydrated appears as an ID-ONLY entry rather than
	// being omitted. Omitting it would read as "this item has no relation",
	// which is a different and false statement; the honest one is "it points at
	// this id and I cannot tell you more". Two cases produce that: a DANGLING
	// value whose target is gone, and a target the requester may not see —
	// hydrating a ref or title for the latter would leak exactly what the
	// write-side visibility collapse exists to withhold.
	RelationTargets map[string]RelationTargetSet `json:"relation_targets,omitempty"`

	// Warnings is populated on WRITE responses only (create / update), never
	// on reads, and is never stored. It carries things the write did that the
	// caller might not have meant, or could not otherwise learn: field keys the
	// collection's schema does not declare (BUG-2850), keys the write dropped or
	// kept unresolved (TASK-2878), and where the content ended up when it went
	// to a live collaborative document rather than the row (BUG-2995). See
	// ItemWriteWarnings for the members; this comment names the kinds rather
	// than enumerating them, so adding one does not falsify it.
	//
	// Additive and omitempty: a client that does not know the key ignores it,
	// and a write with nothing to report is byte-identical to before. The
	// alternative — wrapping the response as {item, warnings} — would have
	// broken every existing parser of the item write response.
	Warnings *ItemWriteWarnings `json:"warnings,omitempty"`

	// Appended is populated on an UPDATE response that carried
	// append_implementation_note / append_decision, and never on reads or in
	// storage. See ItemAppendedEntries.
	Appended *ItemAppendedEntries `json:"appended,omitempty"`

	// HasChildren is true if this item has child items linked to it.
	// Populated by enrichment, not stored in the DB.
	HasChildren bool `json:"has_children,omitempty"`

	// IsUnparented is populated only on unrestricted local-first index and
	// delta responses. A pointer preserves the distinction between a
	// structurally-parented item (false) and metadata that was deliberately
	// omitted for a restricted caller (nil).
	IsUnparented *bool `json:"is_unparented,omitempty"`

	// MovedTo names the destination(s) an ARCHIVED item was moved to by a
	// cross-workspace move (PLAN-2357 / TASK-2359). Populated on the single-item
	// GET response only, and only for destinations the caller has independently
	// been authorized to READ — see (*server.Server).movedToDestinations.
	//
	// `omitempty` is load-bearing, not cosmetic. A caller who may not read the
	// destination must receive a response byte-identical to one for an archived
	// item that was never moved: no key, no null, no empty array. A
	// structurally distinguishable response is itself the disclosure the ACL
	// gate exists to prevent. Never populate this from a list, search, activity
	// or share-link path, and never emit a placeholder when the gate denies.
	MovedTo []ItemMovedTo `json:"moved_to,omitempty"`

	// Lease is the item's LIVE execution lease (#1221) — who is actively
	// executing this item right now, distinct from AssignedUserID's
	// longer-term ownership. Populated on the single-item GET and on list
	// responses; nil when unclaimed OR expired (an expired lease is absent
	// on every read path — expiry is the reaper). Never persisted through
	// item update paths: claiming/releasing goes through the dedicated
	// claim/release endpoints and deliberately bumps neither updated_at
	// nor the version history.
	Lease *ItemLease `json:"lease,omitempty"`

	// Decisions are the item's latest typed-decision answers, one per
	// question, each marked current or not (TASK-3117). Populated ONLY on the
	// single-item GET, and absent when there are none — including on every
	// instance with no decision provider configured, so the item shape there
	// is unchanged. Never persisted through item paths.
	Decisions []ItemDecision `json:"decisions,omitempty"`

	DerivedClosure      *ItemDerivedClosure      `json:"derived_closure,omitempty"`
	CodeContext         *ItemCodeContext         `json:"code_context,omitempty"`
	Convention          *ItemConventionMetadata  `json:"convention,omitempty"`
	ImplementationNotes []ItemImplementationNote `json:"implementation_notes,omitempty"`
	DecisionLog         []ItemDecisionLogEntry   `json:"decision_log,omitempty"`

	// LastMutation carries the status/assignment delta that THIS specific
	// store call (UpdateItemWithParentLink / MoveItemWithPreCheck) produced,
	// computed synchronously inside the same transaction that already writes
	// the canonical status_transitions row (TASK-2533). It exists so the
	// watch-notification pipeline can classify an update's kind
	// (status-change / assignment) from a race-free, in-tx source of truth
	// instead of a separate before/after snapshot pair racing against
	// concurrent writers of the same item.
	//
	// Nil when the update touched neither the collection's done-field nor
	// assigned_user_id. Never persisted, never populated by GetItem or any
	// list/search path — only the two store functions above set it, and
	// only on the *models.Item they hand back to their caller. `json:"-"`
	// is deliberate: this is an internal signal for one in-process
	// consumer, not a wire contract the API is committing to.
	LastMutation *ItemMutationSignal `json:"-"`

	// PreUpdate is the item as it stood immediately BEFORE this store call
	// wrote it, read inside the same transaction and under the same locks as
	// the write (BUG-2776). It exists so a caller can describe what IT
	// changed rather than what changed since the caller last looked: the
	// handler's own pre-read happens before permission checks and before the
	// store's locks, so any concurrent writer's change lands between the two
	// and, diffed naively, gets attributed to this request's author.
	//
	// Same contract as LastMutation above and for the same reasons: set only
	// by UpdateItemWithParentLink, on the *models.Item it hands back, never
	// persisted, never populated by GetItem or any list/search path, and
	// `json:"-"` because it is an in-process signal rather than a wire
	// contract. It is a COPY taken at re-read time, so it stays the
	// pre-write view even though the store keeps reading `existing` after.
	//
	// Non-nil on every successful UpdateItemWithParentLink return. A consumer
	// that finds it nil is looking at an item from some other path and must
	// NOT silently fall back to its own stale snapshot — that fallback IS the
	// defect this field exists to remove.
	PreUpdate *Item `json:"-"`

	// CreatedVersionID is the id of the version row this update wrote, empty
	// when the throttle wrote none (PLAN-3578: the activity entry names it).
	// Internal, like PreUpdate.
	CreatedVersionID string `json:"-"`
	// BodyEditedWithoutVersion is set by UpdateItem when the update changed
	// the body and the per-actor throttle wrote no version row, so the edit
	// has no version to show it (PLAN-2348 U2, for U3's "edited the body").
	BodyEditedWithoutVersion bool `json:"-"`

	// ViaApp and ViaAppName name the app that CREATED the item (created_via_app),
	// absent for a person's or agent's (SPEC-6 U9c, TASK-3413). The name
	// is the app's bot display name, else its origin, and survives
	// uninstall. The server fills both on read; no scan sets them.
	ViaApp     string `json:"via_app,omitempty"`
	ViaAppName string `json:"via_app_name,omitempty"`
}

// ItemMutationSignal is the race-free status/assignment delta attached to
// models.Item.LastMutation. See that field's doc comment for why it exists
// and who populates it.
type ItemMutationSignal struct {
	StatusChanged  bool
	StatusFieldKey string
	FromStatus     string
	ToStatus       string

	AssignmentChanged  bool
	FromAssignedUserID string // "" = was unassigned
	ToAssignedUserID   string // "" = now unassigned
}

// ComputeRef sets the Ref field from CollectionPrefix and ItemNumber.
// Call this after populating the item from a database query.
func (item *Item) ComputeRef() {
	if item.CollectionPrefix != "" && item.ItemNumber != nil {
		item.Ref = fmt.Sprintf("%s-%d", item.CollectionPrefix, *item.ItemNumber)
	}
}

// ItemLease is a live execution lease on an item (#1221): holder is a
// freeform identity string (defaulting server-side to the authenticated
// user), and the lease is live iff ExpiresAt is in the future. Expired
// leases are never returned — expiry is absence.
type ItemLease struct {
	Holder string `json:"holder"`
	// Worker is the SHORT form of the holder's worker id ("wk_3f2a",
	// TASK-3613), empty for a name-only lease. Display only: the full id
	// is never returned on a read.
	Worker     string    `json:"worker,omitempty"`
	AcquiredAt time.Time `json:"acquired_at"`
	ExpiresAt  time.Time `json:"expires_at"`
}

// ItemMovedTo is one destination an archived item was moved to, rendered in
// DISPLAYABLE terms. It deliberately carries no UUIDs: the consumer must be
// able to render (and link to) the destination without a second call, and
// exposing internal IDs of a resource in another workspace buys nothing the
// slug/ref pair does not already give.
//
// Every field describes a resource the caller has already been authorized to
// read, and authorization is all-or-nothing per entry: an entry is never
// partially REDACTED, it is dropped. (Individual fields may still be empty for
// ordinary reasons — a workspace with no resolvable owner username, an item
// whose collection has no prefix — which is what the omitempty tags are for.
// Absence here never means "withheld".)
type ItemMovedTo struct {
	// WorkspaceSlug is the destination workspace's CANONICAL slug — the same
	// value the token consent allow-list was tested against.
	WorkspaceSlug string `json:"workspace_slug"`
	WorkspaceName string `json:"workspace_name,omitempty"`
	// WorkspaceOwnerUsername completes the /{username}/{workspace}/... web
	// route. Empty when the join did not resolve one; the consumer degrades to
	// a non-linked label rather than guessing.
	WorkspaceOwnerUsername string `json:"workspace_owner_username,omitempty"`

	CollectionSlug string `json:"collection_slug,omitempty"`
	// Ref is the destination item's issue ID ("TASK-5"). Empty only for the
	// rare item whose collection has no prefix or number.
	Ref      string `json:"ref,omitempty"`
	ItemSlug string `json:"item_slug"`
	Title    string `json:"title"`

	// MovedAt is when the move was recorded (RFC3339 UTC), matching the
	// provenance row's created_at.
	MovedAt string `json:"moved_at,omitempty"`
}

type ItemRelationRef struct {
	ID             string `json:"id"`
	Slug           string `json:"slug,omitempty"`
	Ref            string `json:"ref,omitempty"`
	Title          string `json:"title"`
	CollectionSlug string `json:"collection_slug,omitempty"`
	Status         string `json:"status,omitempty"`
}

type ItemDerivedClosure struct {
	IsClosed     bool              `json:"is_closed"`
	Kind         string            `json:"kind"`
	Summary      string            `json:"summary"`
	RelatedItems []ItemRelationRef `json:"related_items,omitempty"`
}

type ItemCodeContext struct {
	Provider    string                   `json:"provider"`
	Repo        string                   `json:"repo,omitempty"`
	Branch      string                   `json:"branch,omitempty"`
	PullRequest *ItemPullRequestMetadata `json:"pull_request,omitempty"`
}

type ItemConventionMetadata struct {
	Category    string   `json:"category,omitempty"`
	Trigger     string   `json:"trigger,omitempty"`
	Surfaces    []string `json:"surfaces,omitempty"`
	Enforcement string   `json:"enforcement,omitempty"`
	Commands    []string `json:"commands,omitempty"`
}

type ItemPullRequestMetadata struct {
	Number    int    `json:"number"`
	URL       string `json:"url"`
	Title     string `json:"title"`
	State     string `json:"state"`
	UpdatedAt string `json:"updated_at,omitempty"`
	MergeSHA  string `json:"merge_sha,omitempty"`
	MergedAt  string `json:"merged_at,omitempty"`
}

type ItemImplementationNote struct {
	ID        string `json:"id,omitempty"`
	Summary   string `json:"summary"`
	Details   string `json:"details,omitempty"`
	CreatedAt string `json:"created_at,omitempty"`
	CreatedBy string `json:"created_by,omitempty"`
}

type ItemDecisionLogEntry struct {
	ID        string `json:"id,omitempty"`
	Decision  string `json:"decision"`
	Rationale string `json:"rationale,omitempty"`
	CreatedAt string `json:"created_at,omitempty"`
	CreatedBy string `json:"created_by,omitempty"`
}

// ItemGitHubPR is the stored shape of the `github_pr` system metadata key,
// and the typed update member that writes it (BUG-2696). The JSON tags are
// the stored shape ExtractItemCodeContext decodes.
type ItemGitHubPR struct {
	Number    int    `json:"number"`
	URL       string `json:"url"`
	Title     string `json:"title,omitempty"`
	State     string `json:"state,omitempty"`
	Branch    string `json:"branch,omitempty"`
	Repo      string `json:"repo,omitempty"`
	UpdatedAt string `json:"updated_at,omitempty"`
	// MergeSHA and MergedAt are set once the PR has merged (TASK-3617): the
	// merge commit's full SHA and the RFC3339 merge time, as gh reports them.
	// A verification recorded at an older code revision is flagged
	// possibly_stale by them.
	MergeSHA string `json:"merge_sha,omitempty"`
	MergedAt string `json:"merged_at,omitempty"`
}

// ValidateGitHubPR refuses a PR object that would not render as a link: it
// needs a positive number and an absolute http(s) URL.
func ValidateGitHubPR(pr *ItemGitHubPR) error {
	if pr == nil {
		return errors.New("github_pr is empty")
	}
	if pr.Number <= 0 {
		return fmt.Errorf("github_pr.number must be a positive PR number, got %d", pr.Number)
	}
	u, err := url.Parse(strings.TrimSpace(pr.URL))
	if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" {
		return fmt.Errorf("github_pr.url must be an absolute http(s) URL, got %q", pr.URL)
	}
	if pr.MergeSHA != "" && !IsFullCommitSHA(pr.MergeSHA) {
		return fmt.Errorf("github_pr.merge_sha must be a full 40-hex commit SHA, got %q", pr.MergeSHA)
	}
	if pr.MergedAt != "" {
		if _, err := time.Parse(time.RFC3339, pr.MergedAt); err != nil {
			return fmt.Errorf("github_pr.merged_at must be an RFC3339 time, got %q", pr.MergedAt)
		}
	}
	return nil
}

// IsFullCommitSHA reports whether s is a full 40-hex lowercase commit SHA.
func IsFullCommitSHA(s string) bool {
	if len(s) != 40 {
		return false
	}
	for _, c := range s {
		if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
			return false
		}
	}
	return true
}

type githubPRFields struct {
	Number    int    `json:"number"`
	URL       string `json:"url"`
	Title     string `json:"title"`
	State     string `json:"state"`
	Branch    string `json:"branch"`
	Repo      string `json:"repo"`
	UpdatedAt string `json:"updated_at"`
	MergeSHA  string `json:"merge_sha"`
	MergedAt  string `json:"merged_at"`
}

type conventionFields struct {
	Category    string   `json:"category"`
	Trigger     string   `json:"trigger"`
	Surfaces    []string `json:"surfaces"`
	Enforcement string   `json:"enforcement"`
	Commands    []string `json:"commands"`
}

func ExtractItemCodeContext(fieldsJSON string) *ItemCodeContext {
	fieldsMap, ok := parseItemFields(fieldsJSON)
	if !ok {
		return nil
	}

	raw, ok := fieldsMap[ItemFieldGitHubPR]
	if !ok {
		return nil
	}

	payload, err := json.Marshal(raw)
	if err != nil {
		return nil
	}

	var githubPR githubPRFields
	if err := json.Unmarshal(payload, &githubPR); err != nil {
		return nil
	}
	if githubPR.Number == 0 && githubPR.URL == "" && githubPR.Branch == "" && githubPR.Repo == "" {
		return nil
	}

	context := &ItemCodeContext{
		Provider: "github",
		Repo:     githubPR.Repo,
		Branch:   githubPR.Branch,
	}
	if githubPR.Number != 0 || githubPR.URL != "" || githubPR.Title != "" || githubPR.State != "" {
		context.PullRequest = &ItemPullRequestMetadata{
			Number:    githubPR.Number,
			URL:       githubPR.URL,
			Title:     githubPR.Title,
			State:     githubPR.State,
			UpdatedAt: githubPR.UpdatedAt,
			MergeSHA:  githubPR.MergeSHA,
			MergedAt:  githubPR.MergedAt,
		}
	}

	return context
}

func ExtractItemConventionMetadata(fieldsJSON string) *ItemConventionMetadata {
	fieldsMap, ok := parseItemFields(fieldsJSON)
	if !ok {
		return nil
	}

	var metadata ItemConventionMetadata
	hasMetadata := false

	// hasConventionShape tracks whether we've found a Convention-
	// SPECIFIC marker — the structured convention field, or one of
	// trigger / surfaces / scope / commands / direct enforcement.
	// `category` alone is NOT a Convention marker (Ideas, Bugs, Roadmap
	// items also use category). Used to gate the priority→enforcement
	// legacy fallback below; without this gate every Task/Idea with a
	// `priority` field got a phantom `convention.enforcement` surfaced
	// on its response (BUG-987 bug 13).
	hasConventionShape := false

	if raw, ok := fieldsMap[ItemFieldConvention]; ok {
		payload, err := json.Marshal(raw)
		if err == nil {
			var structured conventionFields
			if err := json.Unmarshal(payload, &structured); err == nil {
				metadata = ItemConventionMetadata{
					Category:    structured.Category,
					Trigger:     structured.Trigger,
					Surfaces:    append([]string(nil), structured.Surfaces...),
					Enforcement: structured.Enforcement,
					Commands:    append([]string(nil), structured.Commands...),
				}
				hasMetadata = true
				hasConventionShape = true
			}
		}
	}

	if metadata.Category == "" {
		if category, ok := fieldsMap["category"].(string); ok {
			metadata.Category = category
			hasMetadata = true
			// Note: category alone does NOT flip hasConventionShape —
			// many non-Convention collections legitimately use it.
		}
	}
	if metadata.Trigger == "" {
		if trigger, ok := fieldsMap["trigger"].(string); ok {
			metadata.Trigger = trigger
			hasMetadata = true
			hasConventionShape = true
		}
	}
	// Direct enforcement only — the priority fallback runs at the
	// END so surfaces/scope/commands have a chance to flip
	// hasConventionShape first. Without that ordering, a legacy
	// Convention like `{scope:"all", priority:"must"}` (no trigger)
	// would silently drop enforcement because the fallback ran
	// before scope set hasConventionShape.
	if metadata.Enforcement == "" {
		if value, ok := fieldsMap["enforcement"].(string); ok {
			metadata.Enforcement = value
			hasMetadata = true
			hasConventionShape = true
		}
	}
	if len(metadata.Surfaces) == 0 {
		if surfaces := extractStringList(fieldsMap["surfaces"]); len(surfaces) > 0 {
			metadata.Surfaces = surfaces
			hasMetadata = true
			hasConventionShape = true
		} else if scope, ok := fieldsMap["scope"].(string); ok && scope != "" {
			metadata.Surfaces = []string{scope}
			hasMetadata = true
			hasConventionShape = true
		}
	}
	if len(metadata.Commands) == 0 {
		if commands := extractStringList(fieldsMap["commands"]); len(commands) > 0 {
			metadata.Commands = commands
			hasMetadata = true
			hasConventionShape = true
		}
	}

	// Legacy priority→enforcement fallback. Runs AFTER all other
	// markers because hasConventionShape only flips once we've seen
	// a Convention-specific signal. Without this ordering, a legacy
	// Convention with only `{scope, priority}` would lose its
	// enforcement value because scope hadn't been processed yet
	// (Codex review on PR #361 caught this).
	if metadata.Enforcement == "" && hasConventionShape {
		if priority, ok := fieldsMap["priority"].(string); ok {
			metadata.Enforcement = priority
		}
	}

	if !hasMetadata {
		return nil
	}
	// Final guard: if we ONLY matched on `category` (no Convention-
	// specific markers), the item isn't a Convention. Suppress the
	// metadata entirely — surfacing { category } on a non-Convention
	// item just for category alone produced confusing responses.
	if !hasConventionShape {
		return nil
	}
	return normalizeItemConventionMetadata(&metadata)
}

func ExtractItemImplementationNotes(fieldsJSON string) []ItemImplementationNote {
	fieldsMap, ok := parseItemFields(fieldsJSON)
	if !ok {
		return nil
	}
	raw, ok := fieldsMap[ItemFieldImplementationNotes]
	if !ok {
		return nil
	}

	payload, err := json.Marshal(raw)
	if err != nil {
		return nil
	}

	var notes []ItemImplementationNote
	if err := json.Unmarshal(payload, &notes); err != nil {
		return nil
	}
	if len(notes) == 0 {
		return nil
	}
	return notes
}

func ExtractItemDecisionLog(fieldsJSON string) []ItemDecisionLogEntry {
	fieldsMap, ok := parseItemFields(fieldsJSON)
	if !ok {
		return nil
	}
	raw, ok := fieldsMap[ItemFieldDecisionLog]
	if !ok {
		return nil
	}

	payload, err := json.Marshal(raw)
	if err != nil {
		return nil
	}

	var entries []ItemDecisionLogEntry
	if err := json.Unmarshal(payload, &entries); err != nil {
		return nil
	}
	if len(entries) == 0 {
		return nil
	}
	return entries
}

// ErrStructuredFieldUnreadable is returned by the Append* helpers when the item
// already carries a structured-entry field whose stored value cannot be decoded
// into its entry slice. Callers should surface it rather than retry: the append
// is refused precisely because completing it would destroy the stored value.
var ErrStructuredFieldUnreadable = errors.New("structured field is present but unreadable")

// assertStructuredFieldAppendable refuses an append when key holds a value that
// does not decode into []T.
//
// BUG-2627. The Append* helpers below build the new slice from Extract*, then
// assign it over the key unconditionally. Extract* returns nil for FOUR
// different reasons and only one of them is a defect, so the nil itself cannot
// be the refusal condition:
//
//  1. the key is absent          — the first append on an item. Must proceed.
//  2. the key holds an empty array — well-formed, just empty. Must proceed.
//  3. the key holds an explicit JSON null — carries no entries. Must proceed.
//  4. the key holds a value that does not decode — the defect. Must REFUSE,
//     because the unconditional assign would overwrite that value with a
//     one-element slice and report success. Observed live: an item whose
//     implementation_notes was a JSON-ENCODED STRING lost its stored note to a
//     single `pad item note` call, with no warning on any surface.
//
// So this checks decodability directly against the raw value instead of reusing
// Extract*'s nil, which cannot distinguish (4) from (1), (2) or (3).
//
// T must be the entry type the MATCHING Extract* decodes into. A wrong-but-
// compiling instantiation is silently destructive rather than merely wrong:
// encoding/json ignores unknown fields, so ItemImplementationNote ACCEPTS
// `{"decision":{...}}` while ExtractItemDecisionLog rejects it — the guard would
// permit an append that the extractor then reads as empty, which is the exact
// divergence this function exists to prevent. Both directions are pinned by
// tests; see TestAppendDecisionLogRefusesEntriesOnlyItsOwnTypeRejects.
func assertStructuredFieldAppendable[T any](fieldsMap map[string]any, key string) error {
	raw, ok := fieldsMap[key]
	if !ok {
		return nil // case 1 — absent, nothing to lose.
	}
	if raw == nil {
		return nil // an explicit JSON null carries no entries either.
	}

	payload, err := json.Marshal(raw)
	if err != nil {
		return fmt.Errorf("%w: %q could not be re-encoded for inspection: %w", ErrStructuredFieldUnreadable, key, err)
	}

	var entries []T
	if err := json.Unmarshal(payload, &entries); err != nil {
		// Deliberately does NOT name a repair command. The remedy has to be
		// one that works in THIS state, and every append path is exactly what
		// is being refused here; `pad item show --format json` is read-only
		// and does surface the raw value, so it is the one action safe to
		// suggest.
		return fmt.Errorf(
			"%w: %q holds a value that is not a list of entries, so appending would overwrite and destroy it; "+
				"inspect the stored value with `pad item show <ref> --format json` and repair it before appending (BUG-2627)",
			ErrStructuredFieldUnreadable, key)
	}
	return nil
}

// unreadableFieldsBlob tags an append's fields-parse failure as
// ErrStructuredFieldUnreadable (Codex round 5).
//
// The classification, not the failure, is what this changes. An item whose
// whole fields column will not parse is unreadable in exactly the sense
// BUG-2675's code exists for — deterministic, unfixable by the caller, and
// pointless to retry — but the bare parse error carried none of that, so it
// reached agents as `server_error` and invited the retry loop the code was
// added to stop. Same reasoning, one level out from the per-key guard.
func unreadableFieldsBlob(err error) error {
	return fmt.Errorf("%w: the item's fields blob does not parse, so nothing can be appended to it; "+
		"inspect it with `pad item show <ref> --format json` and repair it: %w", ErrStructuredFieldUnreadable, err)
}

// StructuredFieldIsAppendable reports whether an append to key would be
// ACCEPTED on an item carrying fieldsJSON — i.e. whether the Append* helpers
// would proceed rather than refusing with ErrStructuredFieldUnreadable.
//
// It exists so a caller that wants to TALK about appendability asks the same
// question the guard answers, instead of re-deriving it. BUG-2627 part 2's
// refusal message names `pad item note` as the remedy, which is a lie whenever
// that command would itself refuse; the message therefore has to agree with the
// guard EXACTLY, and a second decode written to look equivalent is not exact.
// Codex round 2 caught a first version of it that decoded into
// []json.RawMessage: a stored `[1]` passed there and failed the real guard, so
// the message prescribed a command that refuses.
//
// The entry type per key matches the matching Extract*, for the reason
// assertStructuredFieldAppendable documents at length: json ignores unknown
// fields, so the WRONG-but-compiling instantiation silently permits what the
// extractor rejects.
//
// A fieldsJSON that will not parse AT ALL returns false, because that is the
// honest answer to the question asked: the Append* helpers bail on the same
// parse and return an error, so an append would not be accepted. An earlier
// version returned true on the reasoning that a broken outer blob is "a
// different problem" — which is true of the CAUSE and irrelevant to the
// CALLER, who would have been told to run a command that cannot succeed
// (Codex round 4).
func StructuredFieldIsAppendable(fieldsJSON, key string) bool {
	fieldsMap, err := parseMutableItemFields(fieldsJSON)
	if err != nil {
		return false
	}
	switch key {
	case ItemFieldImplementationNotes:
		return assertStructuredFieldAppendable[ItemImplementationNote](fieldsMap, key) == nil
	case ItemFieldDecisionLog:
		return assertStructuredFieldAppendable[ItemDecisionLogEntry](fieldsMap, key) == nil
	}
	// No append helper owns this key, so nothing can refuse an append to it.
	return true
}

func AppendImplementationNote(fieldsJSON string, note ItemImplementationNote) (string, error) {
	fieldsMap, err := parseMutableItemFields(fieldsJSON)
	if err != nil {
		return "", unreadableFieldsBlob(err)
	}
	if err := assertStructuredFieldAppendable[ItemImplementationNote](fieldsMap, ItemFieldImplementationNotes); err != nil {
		return "", err
	}

	notes := ExtractItemImplementationNotes(fieldsJSON)
	notes = append(notes, note)
	fieldsMap[ItemFieldImplementationNotes] = notes
	return marshalWithPersistedEntryIDs(fieldsMap)
}

// marshalWithPersistedEntryIDs marshals an append helper's result and gives
// any sibling entry without a usable, unique id a persisted one (BUG-2788).
// The helpers re-marshal the whole array, so this is where an idless entry
// carried in from an import or a legacy row gets fixed on the item's next
// append, rather than being re-written idless.
func marshalWithPersistedEntryIDs(fieldsMap map[string]any) (string, error) {
	out, err := marshalItemFields(fieldsMap)
	if err != nil {
		return "", err
	}
	fixed, _, err := EnsureStructuredEntryIDs(out)
	if err != nil {
		return "", err
	}
	return fixed, nil
}

func AppendDecisionLogEntry(fieldsJSON string, entry ItemDecisionLogEntry) (string, error) {
	fieldsMap, err := parseMutableItemFields(fieldsJSON)
	if err != nil {
		return "", unreadableFieldsBlob(err)
	}
	if err := assertStructuredFieldAppendable[ItemDecisionLogEntry](fieldsMap, ItemFieldDecisionLog); err != nil {
		return "", err
	}

	entries := ExtractItemDecisionLog(fieldsJSON)
	entries = append(entries, entry)
	fieldsMap[ItemFieldDecisionLog] = entries
	return marshalWithPersistedEntryIDs(fieldsMap)
}

func ApplyItemConventionMetadata(fieldsJSON string, metadata *ItemConventionMetadata) (string, error) {
	fieldsMap, err := parseMutableItemFields(fieldsJSON)
	if err != nil {
		return "", err
	}

	normalized := normalizeItemConventionMetadata(metadata)
	if normalized == nil {
		delete(fieldsMap, ItemFieldConvention)
		delete(fieldsMap, "category")
		delete(fieldsMap, "trigger")
		delete(fieldsMap, "scope")
		delete(fieldsMap, "priority")
		delete(fieldsMap, "enforcement")
		delete(fieldsMap, "surfaces")
		delete(fieldsMap, "commands")
		return marshalItemFields(fieldsMap)
	}

	fieldsMap[ItemFieldConvention] = normalized
	fieldsMap["category"] = normalized.Category
	fieldsMap["trigger"] = normalized.Trigger
	fieldsMap["enforcement"] = normalized.Enforcement
	fieldsMap["priority"] = normalized.Enforcement
	fieldsMap["surfaces"] = normalized.Surfaces
	fieldsMap["commands"] = normalized.Commands
	if len(normalized.Surfaces) > 0 {
		fieldsMap["scope"] = normalized.Surfaces[0]
	}

	return marshalItemFields(fieldsMap)
}

func BuildConventionItemFields(status string, metadata *ItemConventionMetadata) (string, error) {
	fieldsJSON, err := ApplyItemConventionMetadata("{}", metadata)
	if err != nil {
		return "", err
	}
	fieldsMap, err := parseMutableItemFields(fieldsJSON)
	if err != nil {
		return "", err
	}
	if status != "" {
		fieldsMap["status"] = status
	}
	return marshalItemFields(fieldsMap)
}

func parseItemFields(fieldsJSON string) (map[string]any, bool) {
	if fieldsJSON == "" || fieldsJSON == "{}" {
		return nil, false
	}
	var fieldsMap map[string]any
	if err := json.Unmarshal([]byte(fieldsJSON), &fieldsMap); err != nil {
		return nil, false
	}
	return fieldsMap, true
}

func parseMutableItemFields(fieldsJSON string) (map[string]any, error) {
	if fieldsJSON == "" || fieldsJSON == "{}" {
		return map[string]any{}, nil
	}
	var fieldsMap map[string]any
	if err := DecodeJSONKeepingNumbers([]byte(fieldsJSON), &fieldsMap); err != nil {
		return nil, fmt.Errorf("parse item fields: %w", err)
	}
	if fieldsMap == nil {
		// A literal `null` unmarshals into a NIL map with no error, and every
		// caller here goes on to assign into what it gets back — so `pad item
		// note` against an item whose fields column holds "null" panicked with
		// "assignment to entry in nil map" rather than appending (reproduced;
		// Codex round 3 on BUG-2627). An absent blob and a null blob mean the
		// same thing to every caller, so they get the same empty map.
		return map[string]any{}, nil
	}
	return fieldsMap, nil
}

func marshalItemFields(fieldsMap map[string]any) (string, error) {
	payload, err := json.Marshal(fieldsMap)
	if err != nil {
		return "", fmt.Errorf("marshal item fields: %w", err)
	}
	return string(payload), nil
}

func normalizeItemConventionMetadata(metadata *ItemConventionMetadata) *ItemConventionMetadata {
	if metadata == nil {
		return nil
	}
	normalized := &ItemConventionMetadata{
		Category:    metadata.Category,
		Trigger:     metadata.Trigger,
		Enforcement: metadata.Enforcement,
		Surfaces:    uniqueStrings(metadata.Surfaces),
		Commands:    uniqueStrings(metadata.Commands),
	}
	if normalized.Category == "" && normalized.Trigger == "" && normalized.Enforcement == "" && len(normalized.Surfaces) == 0 && len(normalized.Commands) == 0 {
		return nil
	}
	return normalized
}

func extractStringList(raw any) []string {
	switch value := raw.(type) {
	case []string:
		return uniqueStrings(value)
	case []any:
		var out []string
		for _, entry := range value {
			if str, ok := entry.(string); ok && str != "" {
				out = append(out, str)
			}
		}
		return uniqueStrings(out)
	default:
		return nil
	}
}

func uniqueStrings(values []string) []string {
	if len(values) == 0 {
		return nil
	}
	seen := map[string]struct{}{}
	out := make([]string, 0, len(values))
	for _, value := range values {
		if value == "" {
			continue
		}
		if _, ok := seen[value]; ok {
			continue
		}
		seen[value] = struct{}{}
		out = append(out, value)
	}
	if len(out) == 0 {
		return nil
	}
	return out
}

// UniqueCheck asks the store to refuse a write, inside its transaction and
// after its workspace lock, when a `unique_scope: workspace_collection` field's
// value in Fields is already held by another live item of the collection
// (BUG-3812). The server's check before the write has no lock, so two writers
// could both pass it. Internal-only.
type UniqueCheck struct {
	Schema []FieldDef
	Fields map[string]any
}

type ItemCreate struct {
	// ActorUserID is the request's user, stamped on the create's version row
	// (PLAN-2348 U2). Internal-only.
	ActorUserID string `json:"-"`
	// OpKey is the idempotency key in the body (TASK-3585), for callers that
	// cannot set the Idempotency-Key header. Not part of the request hash;
	// the store ignores it.
	OpKey string `json:"op_key,omitempty"`
	// AfterWriteTx, when set, runs inside the create's transaction after the
	// row is inserted and re-read, before COMMIT (TASK-3585: the idempotency
	// key's answer). An error rolls the create back. Internal.
	AfterWriteTx   func(tx *sql.Tx, created *Item) error `json:"-"`
	Title          string                                `json:"title"`
	Content        string                                `json:"content,omitempty"`
	Fields         string                                `json:"fields,omitempty"`
	Tags           string                                `json:"tags,omitempty"`
	Pinned         bool                                  `json:"pinned,omitempty"`
	ParentID       *string                               `json:"parent_id,omitempty"`
	AssignedUserID *string                               `json:"assigned_user_id,omitempty"`
	AgentRoleID    *string                               `json:"agent_role_id,omitempty"`
	CreatedBy      string                                `json:"created_by,omitempty"`
	Source         string                                `json:"source,omitempty"`
	// Convention is the typed door for the `convention` system metadata key
	// (BUG-3163). Create's `fields` refuses every reserved key, so this is how
	// convention activation writes it: the handler validates the object and
	// lowers it into the fields blob. Like ItemUpdate.GitHubPR (BUG-2696), what
	// it buys is VALIDITY, not sender identity — any caller may send it, but
	// only a well-formed object gets in. The sibling schema fields a convention
	// also carries (trigger, scope, priority, …) stay ordinary `fields`.
	Convention *ItemConventionMetadata `json:"convention,omitempty"`
	// ValidatedSchema is the collection schema, as the exact bytes read from
	// its row, that the caller validated these fields against (BUG-3407).
	// When set, the store compares it with the schema it reads under the
	// workspace seq lock; if the schema moved in between, the keys this write
	// sets are re-validated against the new schema and a failure refuses the
	// write. Nil (internal callers) keeps the store's contract unchanged.
	ValidatedSchema *string `json:"-"`
	// UniqueCheck: see the type (BUG-3812). Internal-only.
	UniqueCheck *UniqueCheck `json:"-"`
	// BuiltinOrigin, when set, records the built-in convention or playbook
	// this item is made from (TASK-3462), in the create's own transaction.
	// Internal-only: set by template seeding and library activation, never
	// by a request body, so a caller cannot claim an origin for its own text.
	BuiltinOrigin *BuiltinOrigin `json:"-"`
}

// ValidateConventionMetadata normalizes a typed convention member and refuses
// one that carries nothing: an empty object would lower to a key the extractor
// reads back as "no metadata", so accepting it would report a write that
// stored nothing readable.
func ValidateConventionMetadata(metadata *ItemConventionMetadata) (*ItemConventionMetadata, error) {
	normalized := normalizeItemConventionMetadata(metadata)
	if normalized == nil {
		return nil, errors.New("convention is empty: set at least one of category, trigger, surfaces, enforcement, commands")
	}
	return normalized, nil
}

// BuildConventionItemCreate is BuildConventionItemFields for the create door
// (BUG-3163): the same fields blob MINUS the reserved `convention` key, plus
// the normalized metadata to send as ItemCreate.Convention.
func BuildConventionItemCreate(status string, metadata *ItemConventionMetadata) (string, *ItemConventionMetadata, error) {
	fieldsJSON, err := BuildConventionItemFields(status, metadata)
	if err != nil {
		return "", nil, err
	}
	fieldsMap, err := parseMutableItemFields(fieldsJSON)
	if err != nil {
		return "", nil, err
	}
	delete(fieldsMap, ItemFieldConvention)
	out, err := marshalItemFields(fieldsMap)
	if err != nil {
		return "", nil, err
	}
	return out, normalizeItemConventionMetadata(metadata), nil
}

type ItemUpdate struct {
	Title   *string `json:"title,omitempty"`
	Content *string `json:"content,omitempty"`
	Fields  *string `json:"fields,omitempty"`
	// FieldsPatch, when non-nil, applies a SHALLOW field-level merge onto
	// the item's CURRENT fields JSON instead of replacing the whole blob
	// (IDEA-1480 / TASK-2022). Each key sets its field; a key mapped to a
	// JSON null DELETES that field; every other stored field is left
	// untouched. The merge runs INSIDE UpdateItem's transaction against the
	// row read under the write lock, so two concurrent single-field patches
	// can no longer clobber each other the way the full-blob
	// read-modify-write does (the lost-write race IDEA-1480 describes).
	//
	// Mutually exclusive with Fields — the HTTP handler rejects a request
	// carrying both. In-process callers should set exactly one.
	FieldsPatch    map[string]interface{} `json:"fields_patch,omitempty"`
	Tags           *string                `json:"tags,omitempty"`
	Pinned         *bool                  `json:"pinned,omitempty"`
	SortOrder      *int                   `json:"sort_order,omitempty"`
	ParentID       *string                `json:"parent_id,omitempty"`
	AssignedUserID *string                `json:"assigned_user_id,omitempty"`
	AgentRoleID    *string                `json:"agent_role_id,omitempty"`
	LastModifiedBy string                 `json:"last_modified_by,omitempty"`
	Source         string                 `json:"source,omitempty"`
	// VersionSource overrides the per-version-row Source attribution
	// without mutating `items.source`. When unset (the common case),
	// the version row inherits the same value as `items.source`
	// (whatever Source ends up being). When set, the version row
	// uses VersionSource and `items.source` is left alone — used by
	// the collab 5s-flush PATCH handler so an auto-flush doesn't
	// re-attribute a CLI/MCP-created item to "collab-snapshot" and
	// silently flip it out of `WorkspaceHasAgentActivity`'s count.
	// Per Codex review round 3 of TASK-1267 [P2].
	VersionSource string `json:"version_source,omitempty"`
	ChangeSummary string `json:"change_summary,omitempty"`
	// ForceVersion bypasses the per-(actor, source) version throttle
	// (VersionThrottleInterval) so a version snapshot is ALWAYS created when
	// content changes. Set by deliberate, infrequent operations that must
	// leave an undo point regardless of recent edit cadence — chiefly a
	// version RESTORE, which changes items.content out from under the existing
	// reverse-patch chain and would corrupt that chain (and lose the restore's
	// own undo point) if a throttled write moved content forward without a
	// bracketing version. Internal-only (`json:"-"`): the store honours it, but
	// no HTTP client can set it. Only consulted when content actually changes.
	ForceVersion bool `json:"-"`
	// ExternalContent is the body an update sends to the item's LIVE
	// collaborative document instead of to items.content: the designated-
	// applier path, taken while a tab has the item open (BUG-3327). The row
	// write runs with Content nil, so without this the store never reached
	// its version write and the edit was in no version. The store versions
	// ExternalContent exactly as it would Content (same throttle, attribution
	// and reverse patch of the body before the edit) and never writes it to
	// the row; the tab's next flush does. Internal-only, and ignored when
	// Content is set.
	ExternalContent *string `json:"-"`
	// ChatGPTDoor marks an update made through the ChatGPT catalog
	// (TASK-3321 U1b), set by the handler from an in-process context marker.
	// Refusals then speak to a chat user rather than to an API caller.
	// Internal-only.
	ChatGPTDoor bool `json:"-"`
	// ActorUserID is the request's user, stamped on a version row this update
	// writes (PLAN-2348 U2). Internal-only; empty for system writers.
	ActorUserID string `json:"-"`
	// MarkRestoreBoundary stamps items.last_restore_seq with this update's newly
	// assigned seq, INSIDE the same transaction as the content write + op-log
	// prune (BUG-2264). Set by version RESTORE and, since BUG-3746, by every
	// prune-and-replace content write (fenceSeedsBefore). That durable per-item
	// boundary is what Join reads to force_refresh a client whose ?content_seq
	// seed predates the restore — surviving a server restart, unlike the
	// in-memory fast-path. Internal-only (`json:"-"`); no HTTP client can set it.
	MarkRestoreBoundary bool `json:"-"`
	// Recovered marks the op-log materializer's write (TASK-2198 U4): the
	// server rebuilt items.content from the item's collaborative op-log after
	// the tab holding those edits went away without flushing them. The store
	// attributes the version row to the system (VersionCreatedBySystem /
	// VersionSourceRecovery) rather than to a guessed author, because op-log
	// rows carry no user id, and advances the flush watermark only to
	// OpLogCursor and only when that cursor is still MAX(op-log id), exactly
	// as a collab-snapshot flush does. Internal-only (`json:"-"`); no HTTP
	// client can set it.
	Recovered bool    `json:"-"`
	Comment   *string `json:"comment,omitempty"`
	// OpLogCursor is the highest item_yjs_updates.id the calling client
	// has applied into its local Y.Doc (TASK-1319). Used by the
	// collab-snapshot flush PATCH to advance the op-log GC watermark
	// (`items.content_flushed_op_log_id`) when, and only when, the
	// cursor matches the current MAX(item_yjs_updates.id) for the item.
	//
	// **Why a pointer.** A nil cursor means "the caller didn't claim
	// to know"; the watermark stays put. *0 means "I have nothing"
	// (e.g. a fresh editor whose op-log is empty); when MAX is also 0
	// the watermark advances to 0 (a no-op stamp) — but practical
	// flushes always have *some* op-log id, so this branch rarely
	// matters.
	//
	// Only honoured when VersionSource == "collab-snapshot". Other
	// content writes (CLI / MCP / version restore / PruneAndApply)
	// already advance the watermark to MAX(op-log.id) at write time
	// because they reconstruct or replace items.content wholesale.
	// Per TASK-1319.
	OpLogCursor *int64 `json:"op_log_cursor,omitempty"`
	// ClearAssignedUser / ClearAgentRole allow explicitly setting to NULL
	// (since nil pointer means "don't change" in partial updates)
	ClearAssignedUser bool `json:"clear_assigned_user,omitempty"`
	ClearAgentRole    bool `json:"clear_agent_role,omitempty"`

	// Force, when true, overrides the server-side open-children guard
	// (IDEA-1494) that otherwise rejects a non-terminal → terminal
	// done-field transition while the item still has non-terminal
	// children. Transport-only: the store layer never sees it (the
	// handler consumes it before calling UpdateItem). Used by `pad item
	// update --force` and MCP `pad_item.action: update` with
	// `force: true`.
	Force bool `json:"force,omitempty"`

	// GitHubPR / ClearGitHubPR are the typed door for the `github_pr` system
	// metadata key (BUG-2696). `fields_patch` refuses `github_pr` like every
	// other reserved key, so this is the ONLY way to write it on update, and
	// the point is VALIDITY rather than who the sender is: any HTTP caller
	// may send it, but it must be a well-formed PR object (ValidateGitHubPR),
	// so the double-encoded strings a `--field github_pr=...` write used to
	// store cannot be written any more. `pad github link` / `unlink` /
	// `project reconcile` use it. Not exposed on the MCP catalog. Mutually
	// exclusive with each other and with a full `fields` write. Transport-
	// only: the handler lowers it into the field patch.
	GitHubPR      *ItemGitHubPR `json:"github_pr,omitempty"`
	ClearGitHubPR bool          `json:"clear_github_pr,omitempty"`

	// RefuseUndeclaredFields is a strict opt-in (BUG-3156): when true, a
	// write that would store a field key the item's collection does not
	// declare is REFUSED with 400 validation_error instead of being accepted
	// and named in warnings.undeclared_fields. It refuses exactly the keys
	// that warning would have named. Transport-only, like Force. A server
	// that predates it ignores the member, which is the accept-and-warn.
	RefuseUndeclaredFields bool `json:"refuse_undeclared_fields,omitempty"`

	// ExpectedUpdatedAt, when non-empty, enables optimistic-concurrency
	// checking (TASK-2022). UpdateItem parses it as RFC3339 and compares it
	// against the item's current updated_at read under the write lock;
	// on mismatch it returns *store.UpdateConflictError so the handler can
	// surface the pad-structured-error/v1 conflict envelope (HTTP 409,
	// code "update_conflict"). Empty = no check (last-writer-wins, the
	// historical default). Round-trip the `updated_at` you last read.
	ExpectedUpdatedAt string `json:"expected_updated_at,omitempty"`

	// ExpectedSeq, when non-nil, is the STRONG optimistic-concurrency token
	// (BUG-3037). Round-trip the `seq` you last read; UpdateItem compares it
	// against the row's current seq under the write lock and returns
	// *store.UpdateConflictError on mismatch, exactly as ExpectedUpdatedAt
	// does.
	//
	// Why a second token rather than a finer ExpectedUpdatedAt: `updated_at`
	// has ONE-SECOND resolution, so two writes inside one second both match it
	// and NEITHER conflicts — the loser is accepted and silently overwrites the
	// winner. Raising that column's resolution is not available: it is TEXT on
	// both dialects and is compared LEXICALLY in SQL (`WHERE updated_at >= ?`
	// on the since-cursor read, plus eight `ORDER BY` sites), and Go's
	// RFC3339Nano omits trailing zeros, so `…:11.5Z` sorts BEFORE `…:11Z` and a
	// cursor would start skipping rows. `seq` has none of those problems: it is
	// an integer, already stamped on every mutation under the workspace seq
	// lock, and already returned to clients.
	//
	// A POINTER, deliberately: absent means "no token" (last-writer-wins, the
	// historical default), and that must stay distinguishable from a caller who
	// sends 0. A live row's seq is always >= 1 — every INSERT stamps
	// MAX(seq)+1 and the backfill migrations numbered from 1 — so a sent 0
	// never matches and is a conflict rather than a bypass.
	ExpectedSeq *int64 `json:"expected_seq,omitempty"`

	// OverwritePendingEdits lifts the BUG-3133 refusal. A content write that
	// carries a version token (ExpectedSeq or ExpectedUpdatedAt) is refused
	// with `content_pending_flush` while the item's op-log holds unflushed
	// collaborative edits, because the token guards the row and those edits
	// are not in it. Setting this says the caller means to replace them.
	// It has no effect on a write without a token or without content.
	OverwritePendingEdits bool `json:"overwrite_pending_edits,omitempty"`

	// RefusePendingEdits asks for the BUG-3133 refusal WITHOUT a version token
	// (BUG-3230 U0). A content write carrying it is refused with
	// `content_pending_flush` while the item's op-log holds unflushed
	// collaborative edits, exactly as a token-carrying write is, but no row
	// version is compared. It exists for writers that cannot carry a token: the
	// item pane's raw-markdown saves overlap (the debounce does not wait for the
	// previous PATCH) and its teardown writes are ordered by ClientWrite, which
	// forbids a token (BUG-3080). OverwritePendingEdits still lifts it.
	// Transport-only, like RefuseUndeclaredFields: a server that predates it
	// ignores the member, which is the old replace-and-prune.
	RefusePendingEdits bool `json:"refuse_pending_edits,omitempty"`

	// AfterWriteTx, when set, runs inside the update's transaction after the
	// row and its version are written and re-read, just before COMMIT
	// (TASK-3585: the idempotency key's answer is written there, so it
	// commits exactly when the write does). An error rolls everything back.
	// Internal.
	AfterWriteTx func(tx *sql.Tx, updated *Item) error `json:"-"`

	// PatchRecord, when set, is written to item_patches in the update's own
	// transaction (PLAN-3578 step 3), so the patch can be undone. Internal.
	PatchRecord *ItemPatchRecord `json:"-"`

	// ContentTransform, when set, computes the new body from the row as it
	// stands inside the update's transaction (PLAN-3578 patch writes) and
	// replaces Content with it. An error aborts the write. Internal: no
	// request body can set it.
	ContentTransform func(existing *Item) (string, error) `json:"-"`

	// Replace is the deliberate-overwrite opt-out (PLAN-3555): the caller means
	// to replace the body whatever was written since it read it, so the write
	// carries no version token on purpose. It changes no outcome in phase 1; it
	// keeps the write out of the unguarded count and its warning, and in phase 2
	// it is what an agent door sends instead of expected_seq. Distinct from
	// OverwritePendingEdits, which lifts the unflushed-edits refusal: a
	// deliberate replace over a tab's pending edits needs both. A version token
	// makes it moot.
	Replace bool `json:"replace,omitempty"`

	// ClientWrite orders ONE browser tab's content writes to an item
	// (BUG-3080). The pane stamps every content PATCH with a random id minted
	// per page load and a counter that only rises, and the server refuses a
	// write whose counter is below one it has already applied for the same
	// (item, tab) — 409 `superseded_write`, nothing written. Two teardown
	// PATCHes are both dispatched before the tab sees either result, so no
	// version token can order them (both carry the same one); this can.
	//
	// Optional and web-only: the CLI and MCP never send it, and a write
	// without it is not compared against any mark. Best-effort by design —
	// see `clientWriteMarks` for the bound, the age and what is forgotten.
	ClientWrite *ClientWriteStamp `json:"client_write,omitempty"`

	// BlankRelationKeys names the collection's scalar `relation` keys for the
	// store to normalise on a FieldsPatch write (BUG-3028): after the patch is
	// merged onto the row read under the write lock, any of these still holding
	// a blank string is deleted, so a legacy "" or " " the patch did not touch
	// is carried through as key-absent. It is decided under the lock rather than
	// from the handler's earlier read, which could delete a value a concurrent
	// write had just set. Server-internal; never read from a request body.
	BlankRelationKeys []string `json:"-"`

	// AppendImplementationNote / AppendDecision append ONE structured entry
	// to the item's `implementation_notes` / `decision_log` (BUG-3056). The
	// server re-reads the row under its write lock and appends there, so a
	// concurrent write to any other key cannot be reverted by it — which is
	// what the GET, append, full-`fields` PATCH these callers used to send
	// did. The server mints the entry's id, created_at and created_by; the
	// caller supplies only the text. Refused alongside `fields` (a full
	// replace has no single meaning combined with an append); allowed
	// alongside `fields_patch`, since both are merges under the same lock.
	//
	// An OLDER server ignores these keys and answers 200 having written
	// nothing, so a client must check GET /server/capabilities for
	// `item_field_append` before sending them — see the CLI's
	// ServerSupportsItemFieldAppend.
	AppendImplementationNote *ItemImplementationNoteAppend `json:"append_implementation_note,omitempty"`
	AppendDecision           *ItemDecisionLogAppend        `json:"append_decision,omitempty"`
	// OpKey is an idempotency key for an APPEND-ONLY update (TASK-3585):
	// append_implementation_note and/or append_decision, plus source. Any
	// other member alongside it is refused 400. Not part of the request hash.
	OpKey string `json:"op_key,omitempty"`

	// ImplementationNoteToAppend / DecisionToAppend are the MINTED entries
	// the handler builds from the two request members above, for the store
	// to append under its lock. Server-internal; never read from a request
	// body, so a caller cannot supply its own id or attribution.
	ImplementationNoteToAppend *ItemImplementationNote `json:"-"`
	DecisionToAppend           *ItemDecisionLogEntry   `json:"-"`
	// ValidatedSchema is the collection schema, as the exact bytes read from
	// its row, that the caller validated these fields against (BUG-3407).
	// When set, the store compares it with the schema it reads under the
	// workspace seq lock; if the schema moved in between, the keys this write
	// sets are re-validated against the new schema and a failure refuses the
	// write. Nil (internal callers) keeps the store's contract unchanged.
	ValidatedSchema *string `json:"-"`
	// UniqueCheck: see the type (BUG-3812). Internal-only.
	UniqueCheck *UniqueCheck `json:"-"`
	// BuiltinSeed, when set, moves the item's built-in origin to this seed
	// inside the update's own transaction (TASK-3462, codex r1): the item and
	// the record of which version of Pad's text it holds commit together.
	// Internal-only: set by the built-in update, never by a request body.
	BuiltinSeed *BuiltinOrigin `json:"-"`
}

// ItemImplementationNoteAppend is the request shape of
// ItemUpdate.AppendImplementationNote: the text only.
type ItemImplementationNoteAppend struct {
	Summary string `json:"summary"`
	Details string `json:"details,omitempty"`
}

// ItemDecisionLogAppend is the request shape of ItemUpdate.AppendDecision.
type ItemDecisionLogAppend struct {
	Decision  string `json:"decision"`
	Rationale string `json:"rationale,omitempty"`
}

// ItemAppendedEntries echoes, on an UPDATE response, the entries that write
// appended (BUG-3056) — including the id the server minted, which the caller
// has no other way to learn.
type ItemAppendedEntries struct {
	ImplementationNote *ItemImplementationNote `json:"implementation_note,omitempty"`
	Decision           *ItemDecisionLogEntry   `json:"decision,omitempty"`
}

// NewStructuredEntryID mints the id of an implementation-note or decision-log
// entry: the prefix and a unix-nano timestamp, the shape every writer has used.
func NewStructuredEntryID(prefix string) string {
	return fmt.Sprintf("%s-%d", prefix, time.Now().UTC().UnixNano())
}

// ErrInvalidFieldsType / ErrInvalidTagsType are returned by
// ItemUpdate.UnmarshalJSON AND ItemCreate.UnmarshalJSON when the
// inbound `fields` / `tags` value is neither a JSON-encoded string
// nor the natural object/array shape. Wire handlers surface the
// sentinel's message verbatim (without the "invalid JSON: ..." wrapper
// from decodeJSON) so callers see a clean domain-level error instead
// of leaked Go internals. See BUG-1144 (Update) and BUG-1432 (Create).
var (
	ErrInvalidFieldsType = errors.New(`"fields" must be a JSON object or a JSON-encoded string`)
	ErrInvalidTagsType   = errors.New(`"tags" must be a JSON array or a JSON-encoded string`)
)

// UnmarshalJSON for ItemCreate mirrors the flexible-shape behaviour
// ItemUpdate gained under BUG-1144: accept `fields` / `tags` either as
// the canonical JSON-encoded string shape (matches models.Item storage)
// OR as the natural nested object/array shape any reasonable HTTP
// client would send.
//
// Pre-BUG-1432 the Create path was brittle in two specific ways agents
// hit on Pad Cloud:
//
//   - `tags: ["foo","bar"]` (natural JSON-array shape) was rejected by
//     the default unmarshaler because the struct field is `string`,
//     yielding `"cannot unmarshal array into Go struct field
//     ItemCreate.tags of type string"` — an HTTP 400 the MCP
//     dispatcher then surfaced as a validation_failed envelope.
//
//   - `fields: {status: "open"}` (natural nested-object shape) hit the
//     same brittleness with `cannot unmarshal object into ... fields of
//     type string`.
//
// The asymmetry with ItemUpdate (which already handled both shapes
// cleanly per BUG-1144) was Codex's tip in the BUG-1432 investigation
// — fixing it here closes the create/update gap and gives agents a
// uniform shape contract across both verbs.
//
// In-process callers that construct ItemCreate{} literals never hit
// this path; only the JSON decode boundary changes.
func (c *ItemCreate) UnmarshalJSON(data []byte) error {
	// Use an alias to inherit every other field's default unmarshal
	// behaviour, while shadowing fields/tags with json.RawMessage so we
	// can inspect the raw shape. The outer fields shadow the embedded
	// alias's same-named fields because they are less deeply nested.
	type alias ItemCreate
	aux := struct {
		Fields json.RawMessage `json:"fields,omitempty"`
		Tags   json.RawMessage `json:"tags,omitempty"`
		*alias
	}{alias: (*alias)(c)}

	if err := json.Unmarshal(data, &aux); err != nil {
		return err
	}

	// flexJSONToString returns *string (nil when absent / null / empty).
	// ItemCreate.Fields/Tags are plain strings, not pointers, so deref
	// when present and leave at zero value otherwise.
	if fieldsStr, err := flexJSONToString(aux.Fields, '{', ErrInvalidFieldsType); err != nil {
		return err
	} else if fieldsStr != nil {
		c.Fields = *fieldsStr
	}

	if tagsStr, err := flexJSONToString(aux.Tags, '[', ErrInvalidTagsType); err != nil {
		return err
	} else if tagsStr != nil {
		c.Tags = *tagsStr
	}

	return nil
}

// UnmarshalJSON accepts `fields` / `tags` either as the canonical
// JSON-encoded string shape (matches models.Item.Fields/Tags storage)
// OR as the natural nested object/array shape any reasonable HTTP
// client would send. The struct fields stay `*string` and the rest
// of the pipeline (validation, store writes, web/CLI consumers) is
// unchanged — we just normalize the wire input here.
//
// In-process callers that construct ItemUpdate{} literals never hit
// this path, so no internal call site needs to change.
//
// See BUG-1144 (input side) and BUG-991 (the symmetric response-side
// dual-emit, fixed at the MCP boundary in PR #364).
// ClientWriteStamp is one tab's position in its own sequence of content
// writes (BUG-3080). Tab is a per-PAGE-LOAD random id — never a session or
// user id, so two tabs of one user never share a sequence.
type ClientWriteStamp struct {
	Tab string `json:"tab"`
	N   int64  `json:"n"`
}

func (u *ItemUpdate) UnmarshalJSON(data []byte) error {
	// Use an alias to inherit every other field's default unmarshal
	// behaviour, while shadowing fields/tags with json.RawMessage so we
	// can inspect the raw shape. The outer fields shadow the embedded
	// alias's same-named fields because they are less deeply nested.
	type alias ItemUpdate
	aux := struct {
		Fields      json.RawMessage `json:"fields,omitempty"`
		Tags        json.RawMessage `json:"tags,omitempty"`
		FieldsPatch json.RawMessage `json:"fields_patch,omitempty"`
		*alias
	}{alias: (*alias)(u)}

	if err := json.Unmarshal(data, &aux); err != nil {
		return err
	}
	// The raw capture below keeps only the LAST fields_patch, where the plain
	// map decode before BUG-3202 merged repeats. Refuse a repeat rather than
	// drop part of the write (BUG-3219).
	if err := RefuseRepeatedMember(data, "fields_patch"); err != nil {
		return err
	}

	// fields_patch is decoded keeping number literals (BUG-3202): its values
	// are merged into the stored blob and written, so a float64 here rounds
	// an integer above 2^53 that the caller sent. Absent and null leave it
	// nil, as the plain decode did.
	u.FieldsPatch = nil
	if trimmed := bytes.TrimSpace(aux.FieldsPatch); len(trimmed) > 0 && !bytes.Equal(trimmed, []byte("null")) {
		patch, err := DecodeFieldsJSON(trimmed)
		if err != nil {
			return fmt.Errorf("fields_patch: %w", err)
		}
		u.FieldsPatch = patch
	}

	// The alias decode leaves u.Fields / u.Tags nil (the raw bytes were
	// captured at the outer level). Re-populate them from the flex parser.
	fieldsStr, err := flexJSONToString(aux.Fields, '{', ErrInvalidFieldsType)
	if err != nil {
		return err
	}
	u.Fields = fieldsStr

	tagsStr, err := flexJSONToString(aux.Tags, '[', ErrInvalidTagsType)
	if err != nil {
		return err
	}
	u.Tags = tagsStr

	return nil
}

// flexJSONToString accepts a raw JSON value and returns it as a
// canonical JSON-encoded string. Acceptable inbound shapes:
//
//   - absent / empty / null → nil (caller leaves the field unchanged)
//   - JSON-encoded string whose INNER content is either empty (the
//     legacy empty-string sentinel handled downstream by store-layer
//     coercion) OR a JSON value whose shape matches expectedStart
//     ('{' or '[')
//   - JSON object or array (matching expectedStart '{' or '[') → re-
//     marshal to string
//
// Any other shape returns errInvalid so the handler surfaces a clean
// domain-level error instead of a leaked Go unmarshal message.
//
// IDEA-1488 R1 codex hardening: the `case '"'` branch validates the
// INNER content's shape (not just the JSON-encoded-string envelope).
// Without this, `{"config": "[]"}` or `{"settings": "not json"}` would
// slip past — the outer string-shape check accepted any inner content
// verbatim, which defeated the shape-validation ceiling that IDEA-1488
// is supposed to add. The pre-existing ItemUpdate fields/tags path
// inherits the same tightening because it routes through this helper.
func flexJSONToString(raw json.RawMessage, expectedStart byte, errInvalid error) (*string, error) {
	trimmed := bytes.TrimSpace(raw)
	if len(trimmed) == 0 || bytes.Equal(trimmed, []byte("null")) {
		return nil, nil
	}
	switch trimmed[0] {
	case '"':
		// JSON-encoded string envelope — unmarshal to the underlying Go
		// string and then validate the inner content's shape matches
		// expectedStart. Subsequent code that does
		// json.Unmarshal([]byte(*s), ...) sees the inner JSON, not a
		// re-quoted string.
		var s string
		if err := json.Unmarshal(trimmed, &s); err != nil {
			return nil, errInvalid
		}
		// Empty inner string is the empty-string sentinel; store-layer
		// coercion handles it (IDEA-1486). Don't reject here so callers
		// retain the legacy "" → default normalization shape.
		innerTrimmed := bytes.TrimSpace([]byte(s))
		if len(innerTrimmed) == 0 {
			return &s, nil
		}
		if innerTrimmed[0] != expectedStart {
			return nil, errInvalid
		}
		// Confirm the inner content actually parses as JSON of the
		// expected shape. Catches strings that start with the right
		// brace but are otherwise garbage (e.g. `"{not valid"`).
		var inner any
		if err := json.Unmarshal(innerTrimmed, &inner); err != nil {
			return nil, errInvalid
		}
		return &s, nil
	case expectedStart:
		// Object or array — re-marshal to a canonical JSON string so
		// the downstream string-typed pipeline can json.Unmarshal it
		// back to a map/slice exactly as if the caller had stringified.
		// Decoded keeping number literals (BUG-3202): this re-marshal is
		// what a caller sending `fields` as an object gets stored, and a
		// float64 round trip rounded every integer above 2^53 in it.
		var v any
		if err := DecodeJSONKeepingNumbers(trimmed, &v); err != nil {
			return nil, errInvalid
		}
		b, err := json.Marshal(v)
		if err != nil {
			return nil, errInvalid
		}
		s := string(b)
		return &s, nil
	default:
		return nil, errInvalid
	}
}

type ItemListParams struct {
	CollectionSlug string
	// ScopeCollectionID restricts to exactly one collection by ID, ANDed
	// unconditionally. It is a SCOPE, not a permission filter: it never
	// combines with CollectionIDs/ItemIDs, which are an OR-ed permission
	// pair ("in a fully granted collection, OR specifically granted"), so
	// scoping through CollectionIDs would widen an item-grant caller to the
	// whole collection. Callers that resolved or gated a collection by ID
	// set this instead of CollectionSlug, so the check and the query agree
	// on identity: a slug can be freed by a rename or delete and re-taken
	// by another collection between the two (BUG-2631).
	ScopeCollectionID string
	CollectionIDs     []string // permission filter: restrict to these collection IDs (nil = no filter)
	ItemIDs           []string // permission filter: additionally restrict to these item IDs (for item-level grants)
	// Fields are EXACT-match field filters: key=value, compared verbatim. A
	// value containing a comma is matched as that literal string (BUG-3167);
	// it used to be split into an OR, which every in-process caller (the
	// uniqueness check, playbook routing, schema-driven completed-work and
	// bootstrap queries) had never meant.
	Fields map[string]string
	// FieldsAnyOf are OR filters: key matches ANY of the listed values. The
	// one producer of an OR from caller input is parseItemListParams, which
	// lowers a `?key=a,b` query value here to keep the documented list
	// contract; everything else asks for an OR by name.
	FieldsAnyOf map[string][]string
	// FieldKeyPresent keeps only items whose fields store this key with a
	// non-null value. Used by the list handlers' check of an undeclared
	// filter key (BUG-3480), through ListItems so it shares the
	// scope/permission clauses.
	FieldKeyPresent string
	Sort            string // e.g. "priority:desc,created_at:asc"
	GroupBy         string
	Search          string // FTS query
	ParentID        string
	Tag             string
	AssignedUserID  string // filter by assigned user
	AgentRoleID     string // filter by agent role (ID or slug)
	ParentLinkID    string // filter by parent link (item ID of the parent)
	// Unparented keeps only items with neither the legacy parent_id column nor
	// an outgoing parent/implements item_link. Incoming links do not count.
	Unparented      bool
	IncludeArchived bool
	// IncludeDeletedCollections returns items whose collection is
	// soft-deleted. Every read leaves them out by default (BUG-3425): a
	// collection's delete leaves its items live. Only the account data
	// export sets it, so a user's export still holds everything they own.
	IncludeDeletedCollections bool
	// NonTerminal, when true, restricts results to items whose resolved
	// done-field value is NOT one of their collection's terminal options.
	// Each collection is evaluated against its OWN terminal_options (falling
	// back to DefaultTerminalStatuses when the schema declares none), so
	// collections with custom status vocabularies (e.g. todo/drafting/
	// scheduled) are handled correctly rather than against a hardcoded
	// global status allowlist (BUG-2001).
	NonTerminal bool
	// NoContent omits the (potentially large) rich-text body column from
	// the projection — callers that only count/summarize/scan structured
	// fields (e.g. the dashboard builder) don't pay to load every item's
	// full markdown. item.Content comes back empty when set. See BUG-2002.
	NoContent bool
	Limit     int
	Offset    int
}

// TagCount is a distinct tag used within a workspace and the number of items
// carrying it. Returned by Store.ListWorkspaceTags and the
// GET /workspaces/{ws}/tags endpoint, ordered by Count desc then Tag asc.
type TagCount struct {
	Tag   string `json:"tag"`
	Count int    `json:"count"`
}

type ItemLink struct {
	ID          string    `json:"id"`
	WorkspaceID string    `json:"workspace_id"`
	SourceID    string    `json:"source_id"`
	TargetID    string    `json:"target_id"`
	LinkType    string    `json:"link_type"`
	CreatedBy   string    `json:"created_by"`
	CreatedAt   time.Time `json:"created_at"`

	// Populated by joins
	SourceTitle          string `json:"source_title,omitempty"`
	TargetTitle          string `json:"target_title,omitempty"`
	SourceSlug           string `json:"source_slug,omitempty"`
	TargetSlug           string `json:"target_slug,omitempty"`
	SourceRef            string `json:"source_ref,omitempty"`
	TargetRef            string `json:"target_ref,omitempty"`
	SourceCollectionSlug string `json:"source_collection_slug,omitempty"`
	TargetCollectionSlug string `json:"target_collection_slug,omitempty"`
	SourceStatus         string `json:"source_status,omitempty"`
	TargetStatus         string `json:"target_status,omitempty"`
}

type ItemLinkCreate struct {
	TargetID  string `json:"target_id"`
	LinkType  string `json:"link_type,omitempty"`
	CreatedBy string `json:"created_by,omitempty"`
	// UserID is the account writing the link, stored in item_links.user_id
	// (BUG-3372). Internal-only: the handler sets it from the request.
	UserID string `json:"-"`
}

// MaxItemTitleRunes bounds an item title at write time.
//
// 255 matches MaxDocumentTitleRunes, but for items the number is not borrowed
// — it is mechanically sufficient, which is why BUG-2831's "there is no single
// safe number, it depends on compressibility" does not apply once the bound
// exists:
//
//   - items carries UNIQUE(workspace_id, slug), which Postgres implements as a
//     btree, and a btree index tuple has a size cap. That cap, not the title
//     column, is what refuses a long title on Postgres while SQLite takes it
//     (BUG-2831).
//   - store.slugify emits ONLY [a-z0-9-] — one output BYTE per input rune at
//     most, and it truncates nothing. So an N-rune title yields at most N bytes
//     of slug.
//
// THE CAP IS 2704 BYTES, NOT 8191, and this is measured rather than inherited.
// BUG-2831 quoted 8191 — the absolute maximum — and the first version of this
// comment repeated it. Driving high-entropy slugs of 2000 / 4000 / 9000 bytes
// through ImportWorkspace against Postgres 17 (the container the PG gate uses)
// with the coercion disabled gives:
//
//	2000 -> accepted
//	4000 -> ERROR: index row size 4056 exceeds btree version 4 maximum 2704
//	        for index "items_workspace_id_slug_key" (SQLSTATE 54000)
//	9000 -> ERROR: index row requires 9056 bytes, maximum size is 8191 (SQLSTATE 54000)
//
// So 2704 is what a regular index tuple is held to (a third of a page) and
// 8191 is the hard ceiling reached only past it. Both are post-compression:
// a REPETITIVE 9000-byte slug is accepted, which is why the filing's threshold
// was unquotable as a single number and why the test fixture uses a
// deterministic high-entropy slug rather than strings.Repeat.
//
// 255 runes therefore bounds the slug at ~255 bytes against the 2704 that
// actually fires: ~10x headroom, so compressibility stops mattering. The
// uniqueness suffix (-2, -3, ...) adds a handful of bytes and does not change
// that.
//
// RUNES, not bytes, because "255 characters" is what a user and a UI counter
// mean. The byte-level residual on the TITLE column is up to 4x this number,
// which the title column (TEXT) carries without complaint — the slug is the
// constrained derivative, and it is ASCII by construction.
//
// Enforced at WRITE time only, and deliberately NOT retroactive: an item whose
// stored title already exceeds the bound keeps working, and an update that does
// not set a title never validates one. Same grandfathering rule as documents
// (Dave's ruling, day-63).
//
// TWO PATHS DO NOT VALIDATE, both by ruling and both carrying legacy data
// rather than caller input (codex round 5 — an earlier version of this comment
// said every door normalizes then validates, which is not true of either):
//
//   - ImportWorkspace COERCES rather than refuses, so restoring an archive
//     cannot die on a row this product itself once accepted.
//   - Cross-workspace copy PROPAGATES the source row's title verbatim, for the
//     same reason; it takes no title from the caller, so it cannot mint one.
//
// The guarantee that holds across every path is therefore narrower than "every
// stored title satisfies this bound": no CALLER-SUPPLIED title is stored
// without being validated against it. Legacy titles at rest are out of scope
// here; making the bound global would be a count-then-repair sweep over
// existing rows, not a change to any door.
//
// The web UI checks user-typed titles against a copy of this bound before
// sending them (web/src/lib/items/titleLimit.ts, BUG-3115). Change both
// together: testdata/item_title_limit.json is asserted by a test in each
// language, so changing only one fails a test.
const MaxItemTitleRunes = 255

// NormalizeItemTitle is the canonical normalization for an item title:
// leading/trailing whitespace is not part of a title.
//
// It exists as its own function because normalize-then-validate must not drift
// between doors — the whole of BUG-2833 is create and update disagreeing about
// one field, and BUG-2831 is create and Postgres disagreeing about the same
// field.
//
// Every door that takes a title FROM A CALLER pairs this with
// ValidateItemTitle and stores what it validated, never the raw input. The two
// legacy-data paths do not, by ruling: ImportWorkspace coerces (it normalizes,
// then truncates or substitutes rather than refusing) and cross-workspace copy
// carries the source row's title through untouched. See MaxItemTitleRunes for
// both. An earlier version of this comment said "every door", which reads as
// covering those two and does not (codex round 6).
func NormalizeItemTitle(title string) string {
	return strings.TrimSpace(title)
}

// ValidateItemTitle checks an ALREADY-NORMALIZED item title at write time.
// Returns a message suitable for a 400 response, or "" when acceptable.
//
// Callers pass NormalizeItemTitle's output. Validating the normalized value is
// the point rather than an implementation detail: a title of "   " is untitled
// wearing a costume, and before this function existed it was legal on create
// (which tested title == "" exactly) and refused on artifact import (which
// trimmed first) — two doors, two rules, and the import door's comment claimed
// to mirror the create gate it was in fact stricter than.
//
// Deliberately NOT covering wikiTitleRoundTripFailure, which ValidateDocumentTitle
// does: the item rename cascade escapes its emission (BUG-2805), so an item
// title containing wiki-link syntax round-trips. Documents' cascade does not
// escape, which is why the check lives there and not here.
func ValidateItemTitle(title string) string {
	if title == "" {
		return "Title is required"
	}
	if n := utf8.RuneCountInString(title); n > MaxItemTitleRunes {
		return fmt.Sprintf("Title is too long: %d characters, maximum %d", n, MaxItemTitleRunes)
	}
	return ""
}

// ItemPatchRecord is a stored patch (PLAN-3578 step 3): its id and its
// applied edits as JSON ([{old, new, ctx_before, ctx_after}]).
type ItemPatchRecord struct {
	ID     string
	ItemID string
	Edits  string
}