Files
pad/internal/store/attachments.go
T
xarmian 09ecc5e02f fix(diff): version writers never diff text that is not valid UTF-8 (BUG-3839) (#2200)
* fix(diff): version writers never diff text that is not valid UTF-8 (BUG-3839)

go-diff works on runes and slices its input at byte offsets derived from
them, so text that is not valid UTF-8 near an edit either panics in
CreatePatch (slice bounds out of range) or builds a patch that does not
reproduce the old text: "hello world" -> "hello \xff world" reverse-applied
to "hello d", with no error. Any invalid byte does it, lone \xff and \x80 as
well as truncated sequences.

diff.ReversePatch returns (patch, ok), false when either side is not valid
UTF-8, and every version writer stores the full body in that case: the item
edit and the throttled rebase (items.go), the import's attachment remap
(attachments.go) and the legacy document edit (documents.go). The full body
is stored losslessly on SQLite; Postgres refuses invalid UTF-8 in the
column, so there the body write fails first and nothing is versioned. No
existing row is rewritten.

Requests cannot deliver such text (encoding/json coerces it, artifact import
rejects it, the materializer emits U+FFFD for lone surrogates), but SQLite
rows from older releases or direct edits can hold it, and an item like that
500ed on every save.

Tests: diff unit pins for both shapes, a census that production code calls
only ReversePatch, and a store pin per caller on SQLite and Postgres. With
the UTF-8 check removed every SQLite leg panics or stores a wrong version,
and the census names the reintroduced call.

Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS

* test(diff): LineCounts, FormatDiffSummary and Unified never panic on invalid UTF-8 (BUG-3839)

Pins the 200k-case fuzz measurement deterministically: the panic shapes as
seeds plus 20k generated pairs from the same seed-1 generator, with an
instrument showing the inputs reach go-diff's PatchApply panic.

Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS

* fix(diff): ApplyPatch returns an error where go-diff would panic (BUG-3839)

ApplyPatch refuses a base or a decoded patch text that is not valid UTF-8,
and turns a go-diff PatchApply panic into an error. The second covers valid
text: a well-formed patch applied over a base it was not made from panics in
about 1.5% of measured pairs, and the version walk does that after it skips
a row. Every caller (the item and document walks, the throttled-edit rebase,
the attachment remap, the tie diagnostic) already treats the error as a
patch that failed, so the row reads as patch_failed instead of a 500.

Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS

* test(store): the remap pin asserts the row it leaves over an invalid body (BUG-3839)

With ApplyPatch refusing an invalid base, the remap cannot resolve a diff
row over the invalid current body: it remaps the patch text in place, and
the row reads as patch_failed. The pin asserts exactly that; it is red when
the base check is removed (the row is then re-diffed from a fuzzy body).

Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS

* refactor(diff): recover wraps only the go-diff call, and the remap pin is named for patch_failed (BUG-3839)

Lead rulings on BUG-3839 checkpoint 11.

Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS

* test(diff): the unchecked-patch census resolves the import name (BUG-3839)

Matches any selector on the file's local name for the diff package (alias,
method value), bare identifiers under a dot import, and skips web,
node_modules, .git and vendor only at the module root. A self-test runs the
matcher over each form.

Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS

* test(store): History past a broken row does not panic (BUG-3839)

The main-reproduction pin: a broken newest row, then an older row applied
over a base it was not made from. Red on main (d95f90850, slice bounds
panic), green here.

Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS

* fix(diff): ApplyPatch refuses a result that is not valid UTF-8 (BUG-3839)

Over a base it was not made from, go-diff can succeed and cut a rune in
half: valid text in, "\x80" out (#2200 review r2, hunter). The rebase would
write that body (Postgres refuses it, failing the edit) and History would
show it. Pinned in the diff fuzz leg and in a mangled_result leg of the item
History, document History and throttled-rebase store tests, on both
backends; red when the check is removed. The doc comment now says an empty
patch returns its base unchecked, which is exact.

Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
2026-10-15 10:24:24 -04:00

2425 lines
100 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package store
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
"strings"
"time"
"github.com/PerpetualSoftware/pad/internal/attachments"
"github.com/PerpetualSoftware/pad/internal/diff"
"github.com/PerpetualSoftware/pad/internal/kernelevents"
"github.com/PerpetualSoftware/pad/internal/models"
)
// WorkspaceStorageInfo is the consolidated quota summary for a workspace:
// used bytes (from the attachments table), the effective limit for the
// owner's plan (after override + platform-setting + hardcoded fallback),
// the plan name, and whether the owner has a per-user storage_bytes
// override configured. -1 in LimitBytes means unlimited.
//
// The override_active flag tracks whether the workspace owner's
// PlanOverrides JSON contains a storage_bytes key — independent of
// whether that override actually changes the effective limit. (On the
// self-hosted plan the limit is unlimited regardless; the flag still
// surfaces the admin's intent in the Settings → Storage UI.)
type WorkspaceStorageInfo struct {
UsedBytes int64 `json:"used_bytes"`
LimitBytes int64 `json:"limit_bytes"` // -1 = unlimited
Plan string `json:"plan"`
OverrideActive bool `json:"override_active"`
// Scope says what UsedBytes sums (TASK-3619): "workspace" (this
// workspace's attachments) or "account" (every live workspace its owner
// owns, for a plan whose limit is pooled; see storagePooledPerAccount).
Scope string `json:"scope"`
}
// Storage scopes (WorkspaceStorageInfo.Scope).
const (
StorageScopeWorkspace = "workspace"
StorageScopeAccount = "account"
)
// storagePooledPerAccount reports whether a plan's storage limit is one pool
// for the whole account rather than a limit per workspace (TASK-3619, the
// PLAN-3605 cost analysis): Pro has unlimited workspaces, so a per-workspace
// cap would have no ceiling. Free has a workspace limit and stays per
// workspace.
func storagePooledPerAccount(plan string) bool { return plan == "pro" }
// attachmentColumns is the canonical column list. Keep the column names in
// alignment with migrations/047_attachments.sql + pgmigrations/026_attachments.sql.
const attachmentColumns = `id, workspace_id, item_id, uploaded_by, storage_key, content_hash,
mime_type, size_bytes, filename, width, height, parent_id, variant, created_at, deleted_at,
filename_source, imported`
// attachmentFilenameSource is the filename_source an insert writes (BUG-2819).
// An unset value is stored as "unknown" — the honest answer for a row whose
// writer did not classify its name — and never as "caller". A value outside
// the enum is refused here rather than by the CHECK constraint, so the error
// names the field instead of arriving as a driver constraint failure.
func attachmentFilenameSource(a *models.Attachment) (string, error) {
if a.FilenameSource == "" {
a.FilenameSource = string(attachments.FilenameSourceUnknown)
}
if !attachments.ValidFilenameSource(a.FilenameSource) {
return "", fmt.Errorf("create attachment: invalid filename_source %q", a.FilenameSource)
}
return a.FilenameSource, nil
}
// importedFlag is attachments.imported for a row being inserted (BUG-3379).
func importedFlag(a *models.Attachment) int {
if a.Imported {
return 1
}
return 0
}
// scanAttachment scans a single row into a models.Attachment, handling
// nullables via *string / *int.
func scanAttachment(row interface {
Scan(dest ...any) error
}) (*models.Attachment, error) {
var a models.Attachment
var itemID, parentID, variant, deletedAt *string
var width, height *int
var createdAt string
var imported int
err := row.Scan(
&a.ID, &a.WorkspaceID, &itemID, &a.UploadedBy, &a.StorageKey, &a.ContentHash,
&a.MimeType, &a.SizeBytes, &a.Filename, &width, &height,
&parentID, &variant, &createdAt, &deletedAt,
&a.FilenameSource, &imported,
)
if err != nil {
return nil, err
}
a.ItemID = itemID
a.ParentID = parentID
a.Variant = variant
a.Width = width
a.Height = height
a.CreatedAt = parseTime(createdAt)
a.DeletedAt = parseTimePtr(deletedAt)
a.Imported = imported == 1
return &a, nil
}
// CreateAttachment inserts an attachment row. id and created_at are
// generated by the store. The caller is expected to have already written
// the blob via AttachmentStore.Put — store-level code does not touch the
// filesystem.
//
// An empty storage_key is rejected. The column is NOT NULL but "" passes
// that constraint, and a row with no key is a live attachment the registry
// cannot resolve: it fails at download time, long after the insert, with
// nothing to point at. The cross-backend arm of PlanAttachmentCopy emits
// exactly that shape deliberately — StorageKey blank until the caller
// transfers the bytes and writes back Put's key — so this guard is what
// turns "the caller must Put first" from a comment into an invariant.
//
// WithPlanLimit makes the insert enforce the owner's storage_bytes limit in
// its own transaction, as CreateAttachmentForLiveItem does, for a
// user-visible original only (ParentID nil). The bundle import passes it in
// cloud mode (BUG-3653). The refusal is a *PlanLimitError.
func (s *Store) CreateAttachment(a *models.Attachment, opts ...MintOption) error {
if !resolveMintOptions(opts).planLimit || a.ParentID != nil {
return s.createAttachmentOn(s.db, a)
}
for attempt := 1; ; attempt++ {
err := s.createLimitedAttachmentTx(a)
if errors.Is(err, errStorageOwnerMoved) && attempt < 3 {
continue
}
return err
}
}
// createLimitedAttachmentTx is one transaction of a limited CreateAttachment.
func (s *Store) createLimitedAttachmentTx(a *models.Attachment) error {
tx, err := s.db.Begin()
if err != nil {
return fmt.Errorf("begin create attachment tx: %w", err)
}
defer tx.Rollback() // no-op after Commit
// The storage lock comes first: acquirePlanLimitLock's ordering rule.
if err := s.enforceStorageLimitTx(tx, a.WorkspaceID, a.SizeBytes); err != nil {
return err
}
if err := s.createAttachmentOn(tx, a); err != nil {
return err
}
return tx.Commit()
}
// CreateAttachmentTx inserts an attachment row inside an existing transaction.
//
// It exists for the cross-workspace copy (PLAN-2357 / DR-9 / DR-11), which has
// to write the clone rows in the SAME transaction as the destination item —
// otherwise a rollback after the self-committing CreateAttachment leaves live
// attachment rows in workspace B pointing at an item that never existed, and a
// failure between the two leaves the copied body's rewritten refs dangling.
// RecordItemWorkspaceMoveTx is the shape this follows.
//
// Identical semantics to CreateAttachment, including the empty-storage_key
// refusal and the "stamp now() when CreatedAt is zero" rule that
// AttachmentCopyRow relies on (its rows carry a deliberately zero CreatedAt).
// Ordering is the caller's contract: attachments has no parent_id foreign key,
// so nothing stops a variant being inserted before its original — insert
// AttachmentCopyPlan.Rows in the order the planner emitted them.
func (s *Store) CreateAttachmentTx(tx *sql.Tx, a *models.Attachment) error {
return s.createAttachmentOn(tx, a)
}
// ErrAttachmentParentItemGone reports that the item an attachment is being
// written against is no longer a live item of the attachment's workspace —
// archived, hard-gone, or never in this workspace at all.
//
// It is a caller-facing sentinel, not an internal failure: handlers turn it
// into the same not-found response every other attachment denial writes, so
// it must not be wrapped in a way that hides it from errors.Is.
var ErrAttachmentParentItemGone = errors.New("attachment parent item is not live in this workspace")
// CreateAttachmentForLiveItem inserts an attachment row, refusing when its
// item_id does not name a LIVE item of the same workspace — re-checked and
// pinned inside the insert's own transaction.
//
// Why a transaction rather than a check in the handler (PLAN-2391 DR-14):
// producing an attachment is check-then-work. A handler that validates the
// parent up front and inserts afterwards leaves a window — item deletion
// commits in a separate transaction (DeleteItem) and can land in the middle,
// so the insert writes a quota-counted live row hanging off an item that is
// already archived, whose bytes the read gate (DR-13) then refuses to serve.
// Locking the item row for the duration of the insert closes the window
// instead of narrowing it.
//
// Postgres takes the row lock with FOR NO KEY UPDATE; a concurrent DeleteItem
// blocks on it and, once this transaction commits, its UPDATE ... WHERE
// deleted_at IS NULL still matches, so archival is delayed but never lost. In
// the other interleaving DeleteItem commits first, and the re-read —
// re-evaluated after the lock is released — no longer matches the deleted_at
// IS NULL predicate, so this call fails closed.
//
// FOR NO KEY UPDATE rather than FOR UPDATE: it is the exact strength needed
// and no more. DeleteItem's UPDATE touches no key column, so Postgres takes
// FOR NO KEY UPDATE for it, and two FOR NO KEY UPDATE holders conflict — the
// archival still blocks. What it does NOT block is FOR KEY SHARE, the lock an
// INSERT into any of the many tables with a `REFERENCES items(id)` foreign key
// takes on the parent row (comments, stars, the Yjs op-log — the last of which
// writes continuously while someone is editing the item). Plain FOR UPDATE
// would stall those for the duration of this transaction to buy nothing.
//
// SQLite skips the lock: its DSN sets _txlock=immediate, so every db.Begin()
// is a BEGIN IMMEDIATE and writers already serialize, and the row-locking
// clause is a syntax error there — the dialect gate is not an optimization.
//
// An orphan row (nil/empty ItemID) has no parent to pin and takes the plain
// insert path, identical to CreateAttachment.
//
// WithPlanLimit makes the insert enforce the owner's storage_bytes limit
// (TASK-3619) in the same transaction, for a user-visible original only: a
// derived row (ParentID set) never refuses, because it is made from bytes the
// limit already admitted. The refusal is a *PlanLimitError.
func (s *Store) CreateAttachmentForLiveItem(a *models.Attachment, opts ...MintOption) error {
limited := resolveMintOptions(opts).planLimit && a.ParentID == nil
orphan := a.ItemID == nil || *a.ItemID == ""
if orphan && !limited {
return s.createAttachmentOn(s.db, a)
}
// An ownership change between the storage check's owner read and its lock
// is retried in a fresh transaction (codex r3 on TASK-3619): it is rare,
// and the next attempt locks the new owner.
for attempt := 1; ; attempt++ {
err := s.createAttachmentForLiveItemTx(a, limited, orphan)
if errors.Is(err, errStorageOwnerMoved) && attempt < 3 {
continue
}
return err
}
}
// createAttachmentForLiveItemTx is one transaction of CreateAttachmentForLiveItem.
func (s *Store) createAttachmentForLiveItemTx(a *models.Attachment, limited, orphan bool) error {
tx, err := s.db.Begin()
if err != nil {
return fmt.Errorf("begin create attachment tx: %w", err)
}
defer tx.Rollback() // no-op after Commit
// The storage lock comes first: acquirePlanLimitLock's ordering rule.
if limited {
if err := s.enforceStorageLimitTx(tx, a.WorkspaceID, a.SizeBytes); err != nil {
return err
}
}
if orphan {
if err := s.createAttachmentOn(tx, a); err != nil {
return err
}
return tx.Commit()
}
query := `SELECT workspace_id FROM items WHERE id = ? AND deleted_at IS NULL`
if s.dialect.Driver() == DriverPostgres {
query += ` FOR NO KEY UPDATE`
}
var itemWorkspaceID string
switch err := tx.QueryRow(s.q(query), *a.ItemID).Scan(&itemWorkspaceID); {
case err == sql.ErrNoRows:
// Missing or archived — indistinguishable on purpose.
return ErrAttachmentParentItemGone
case err != nil:
return fmt.Errorf("lock attachment parent item: %w", err)
}
// Workspace identity is part of the invariant, not a redundant check:
// attachments.item_id carries no FK or same-workspace constraint, so a
// malformed association naming a live item in ANOTHER workspace would
// otherwise pass the liveness re-check and write a cross-workspace row.
if itemWorkspaceID != a.WorkspaceID {
return ErrAttachmentParentItemGone
}
if err := s.createAttachmentOn(tx, a); err != nil {
return err
}
// The choke point (SPEC-3 / TASK-2658): attachment.added, written in the
// same transaction as the row and under the same parent-item lock, so the
// event cannot outlive a refused insert.
//
// GATED TO USER-VISIBLE ORIGINALS, and the gate is the whole subtlety
// here. Variants are ATTACHMENT ROWS too — a thumbnail carries ParentID
// plus Variant "thumb-sm"/"thumb-md" (models.Attachment) — so an ungated
// emit fires three attachment.added events for one image upload, two of
// them announcing files no user added and no binding wants. A derived row
// always has a parent, so "no parent, and not a non-original variant" is
// the test.
//
// What this deliberately still ADMITS: a transform output (rotate/crop),
// which is a new top-level attachment with no ParentID — a user did add
// it, it is independently addressable, and calling it derived would be a
// judgment about provenance the row itself does not make.
if a.ParentID == nil && (a.Variant == nil || *a.Variant == models.AttachmentVariantOriginal) {
if err := s.emitAttachmentEventTx(tx, kernelevents.AttachmentAdded, a); err != nil {
return err
}
}
return tx.Commit()
}
// ErrAttachmentNotAttachable is AttachAttachmentToItem's refusal: the row is
// not a live, unattached original in this workspace at the moment of the
// write. That covers a row attached by someone else first and a row the
// never-attached GC claimed first, so a caller that checked beforehand
// still has to handle it.
var ErrAttachmentNotAttachable = errors.New("attachment is not a live unattached original in this workspace")
// AttachAttachmentToItem binds an UNATTACHED attachment (item_id NULL, as
// `pad attachment upload -` leaves it) to a live item in the same
// workspace, and its live variants with it, because a variant carries its
// parent's item_id (TASK-2247). It never reassigns: every UPDATE requires
// item_id IS NULL, which also makes it and ClaimNeverAttachedAttachment
// mutually exclusive on the row. Authorization is the caller's.
//
// The parent item is pinned the way CreateAttachmentForLiveItem pins it, so
// an archival cannot commit between the liveness check and the bind.
func (s *Store) AttachAttachmentToItem(workspaceID, attachmentID, itemID string) error {
tx, err := s.db.Begin()
if err != nil {
return fmt.Errorf("begin attach attachment tx: %w", err)
}
defer tx.Rollback() // no-op after Commit
query := `SELECT workspace_id FROM items WHERE id = ? AND deleted_at IS NULL`
if s.dialect.Driver() == DriverPostgres {
query += ` FOR NO KEY UPDATE`
}
var itemWorkspaceID string
switch err := tx.QueryRow(s.q(query), itemID).Scan(&itemWorkspaceID); {
case err == sql.ErrNoRows:
return ErrAttachmentParentItemGone
case err != nil:
return fmt.Errorf("lock attach target item: %w", err)
}
if itemWorkspaceID != workspaceID {
return ErrAttachmentParentItemGone
}
res, err := tx.Exec(s.q(`
UPDATE attachments SET item_id = ?
WHERE id = ? AND workspace_id = ? AND item_id IS NULL
AND deleted_at IS NULL AND parent_id IS NULL
`), itemID, attachmentID, workspaceID)
if err != nil {
return fmt.Errorf("attach attachment: %w", err)
}
n, err := res.RowsAffected()
if err != nil {
return fmt.Errorf("attach attachment rows: %w", err)
}
if n != 1 {
return ErrAttachmentNotAttachable
}
if _, err := tx.Exec(s.q(`
UPDATE attachments SET item_id = ?
WHERE parent_id = ? AND workspace_id = ? AND item_id IS NULL AND deleted_at IS NULL
`), itemID, attachmentID, workspaceID); err != nil {
return fmt.Errorf("attach attachment variants: %w", err)
}
return tx.Commit()
}
// createAttachmentOn is the shared insert body, parameterized over the pool or
// a transaction so CreateAttachment and CreateAttachmentTx cannot drift.
func (s *Store) createAttachmentOn(ex sqlExecer, a *models.Attachment) error {
if a.StorageKey == "" {
return fmt.Errorf("create attachment: storage_key is required (write the blob via AttachmentStore.Put first)")
}
if a.ID == "" {
a.ID = newID()
}
ts := now()
if a.CreatedAt.IsZero() {
a.CreatedAt = parseTime(ts)
} else {
ts = a.CreatedAt.UTC().Format("2006-01-02T15:04:05Z07:00")
}
source, err := attachmentFilenameSource(a)
if err != nil {
return err
}
_, err = ex.Exec(s.q(`
INSERT INTO attachments (`+attachmentColumns+`)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`),
a.ID, a.WorkspaceID, a.ItemID, a.UploadedBy, a.StorageKey, a.ContentHash,
a.MimeType, a.SizeBytes, a.Filename, a.Width, a.Height,
a.ParentID, a.Variant, ts, nil, source, importedFlag(a),
)
if err != nil {
return fmt.Errorf("create attachment: %w", err)
}
return nil
}
// GetAttachment returns the attachment with the given id, or (nil, nil)
// if no such row exists. Soft-deleted rows ARE returned (so download
// handlers can distinguish "deleted" from "never existed" and return a
// 410 instead of a 404 if they want to in the future); callers that
// want only live rows should check a.DeletedAt.
func (s *Store) GetAttachment(id string) (*models.Attachment, error) {
a, err := scanAttachment(s.db.QueryRow(s.q(`
SELECT `+attachmentColumns+` FROM attachments WHERE id = ?
`), id))
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("get attachment: %w", err)
}
return a, nil
}
// GetAttachmentVariant returns the derived attachment row for parentID
// with the given variant key (e.g. "thumb-sm"), or (nil, nil) if no such
// row exists. Used by the download handler when a client passes
// ?variant=thumb-sm — TASK-878 will populate these rows; TASK-872
// implements the lookup so the handler degrades gracefully when no
// thumbnail exists yet.
//
// workspaceID scopes the lookup (PLAN-2391 DR-16). parent_id has no FK or
// same-workspace constraint, so a variant row in workspace B can carry a
// parent_id belonging to workspace A — the copy planner demonstrates that
// shape (internal/store/attachments_copy_plan_test.go). Without the scope,
// the download handler authorizes A's original and then serves B's child,
// which defeats the entire item-visibility gate. The scope lives here rather
// than in the handler because the OTHER caller — thumbnail derivation — has
// its own stake in it: an unscoped "does this variant already exist?" probe
// would let a foreign row suppress generation of a legitimate local one.
func (s *Store) GetAttachmentVariant(workspaceID, parentID, variant string) (*models.Attachment, error) {
a, err := scanAttachment(s.db.QueryRow(s.q(`
SELECT `+attachmentColumns+` FROM attachments
WHERE workspace_id = ? AND parent_id = ? AND variant = ? AND deleted_at IS NULL
LIMIT 1
`), workspaceID, parentID, variant))
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("get attachment variant: %w", err)
}
return a, nil
}
// AttachmentListFilters narrow the WorkspaceAttachments result set.
// Zero values mean "no filter on this dimension". The handler turns
// query-string parameters into this struct so the SQL builder stays
// pure and unit-testable.
type AttachmentListFilters struct {
// MimeCategory restricts to a single MIME category bucket
// (matches attachments.Category — "image", "document", etc.).
// Empty = all categories. Translated to MIME predicates per
// mimePredicateForCategory; categories without a clean prefix
// (document, text, archive, other) use an explicit IN list.
MimeCategory string
// Attached restricts to attachments associated with an item.
Attached bool
// Unattached restricts to orphan attachments (item_id IS NULL).
// Mutually exclusive with Attached — handler validates upstream.
Unattached bool
// ItemID restricts to attachments whose parent item has this
// UUID. Empty = no per-item filter. Mutually exclusive with
// Unattached (an orphan can't match a specific item) — handler
// validates upstream. Used by `pad attachment list --item REF`
// after the CLI resolves a TASK-5-style ref to its UUID.
ItemID string
// CollectionID restricts to attachments belonging to items in
// this collection. Empty = no collection filter.
CollectionID string
// Sort field. Accepts "size", "filename", "created_at" with an
// optional " desc" suffix. Empty = "created_at desc" (newest first).
Sort string
// Limit caps the page size. Clamped to [1, 200] by the handler.
Limit int
// Offset pages forward. Combined with Limit + Total for the UI's
// classic page navigator. Negative values clamped to 0.
Offset int
// Restricted, FullCollectionIDs, GrantedItemIDs together encode
// per-user collection + item visibility. When Restricted is true
// the list filters to attachments whose parent item is either
// in one of FullCollectionIDs or has its id in GrantedItemIDs.
// Orphans (item_id IS NULL) are excluded from restricted views
// so a member who only sees one collection can't enumerate
// filenames of unattached uploads from collections they don't
// have access to.
//
// Restricted=false → no filter (admin / full-access member).
// Restricted=true with empty Full+Granted → zero rows.
// Restricted=true with one or both populated → SQL OR of the
// two predicates.
//
// Mirrors the (fullCollIDs, grantedItemIDs) tuple returned by
// Server.guestResourceFilter — keep the semantics in sync.
Restricted bool
FullCollectionIDs []string
GrantedItemIDs []string
}
// AttachmentListItem is a row from WorkspaceAttachments enriched with
// the parent item's title + slug + collection slug so the UI can render
// a clickable link without a follow-up GET. Item fields are nullable
// for orphan rows.
//
// URL construction: the item route is /{user}/{ws}/{collection_slug}/{item_slug},
// so the UI uses ItemSlug — never a synthetic "TASK-5"-style ref. The
// ref shape isn't 1:1 with the route, and exposing it here led to a
// double-collection-slug bug in an earlier draft.
//
// ItemDeleted is true when the parent item exists but has been soft-
// deleted. The row is still surfaced because the bytes still consume
// quota; the UI uses the flag to render "(deleted)" instead of a
// clickable link to a 404'd item.
type AttachmentListItem struct {
models.Attachment
ItemTitle *string `json:"item_title,omitempty"`
ItemSlug *string `json:"item_slug,omitempty"`
ItemDeleted bool `json:"item_deleted,omitempty"`
CollectionSlug *string `json:"collection_slug,omitempty"`
}
// allowedAttachmentSorts pins the columns + directions the list
// endpoint accepts, so a hand-crafted sort= query can't smuggle in
// arbitrary SQL. Map values are the literal SQL fragment we splice in.
var allowedAttachmentSorts = map[string]string{
"size": "a.size_bytes ASC",
"size_desc": "a.size_bytes DESC",
"filename": "a.filename ASC",
"filename_desc": "a.filename DESC",
"created_at": "a.created_at ASC",
"created_at_desc": "a.created_at DESC",
}
// mimePredicateForCategory maps an attachments.Category value to a
// SQL fragment + matching argument list that selects all MIMEs in
// that bucket. Categories with a clean type prefix (image/, video/,
// audio/) use a LIKE; the rest use an explicit IN list mirroring
// the entries in internal/attachments/mime.go.
//
// Returns (frag, args, true) when the category is known. The frag
// uses ? placeholders that the caller splices into the WHERE. ok=false
// for unknown categories — caller must skip the filter so the UI
// shows everything rather than zero rows for typos.
//
// Keep the literal MIME lists in lockstep with internal/attachments/
// mime.go: any time a new MIME is added to the allowlist there, mirror
// it here so the Settings → Storage filter stays useful.
func mimePredicateForCategory(category string) (frag string, args []any, ok bool) {
switch category {
case "image":
return "a.mime_type LIKE ?", []any{"image/%"}, true
case "video":
return "a.mime_type LIKE ?", []any{"video/%"}, true
case "audio":
return "a.mime_type LIKE ?", []any{"audio/%"}, true
case "document":
return mimeInPredicate([]string{
"application/pdf",
"application/msword",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"application/vnd.ms-excel",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/vnd.ms-powerpoint",
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
"application/vnd.oasis.opendocument.text",
"application/vnd.oasis.opendocument.spreadsheet",
"application/vnd.oasis.opendocument.presentation",
"application/rtf",
})
case "text":
// Wider than the upload allowlist on purpose. This filters ROWS THAT
// EXIST by display category; the allowlist decides what may be
// created. application/xml, text/yaml and application/javascript left
// the allowlist in BUG-2963 F6 as spellings nothing could produce, and
// they are kept here because a filter that stops matching a type costs
// something (a row nobody can find) while one that matches a type no
// row carries costs nothing.
return mimeInPredicate([]string{
"text/plain", "text/markdown", "text/csv", "text/tab-separated-values",
"application/json", "application/xml", "text/xml",
"application/yaml", "text/yaml", "application/toml",
"text/html", "text/javascript", "application/javascript",
})
case "archive":
return mimeInPredicate([]string{
"application/zip", "application/x-tar", "application/gzip",
"application/x-bzip2", "application/x-7z-compressed",
})
case "other":
// "Other" is the negation of every named bucket. Easier to
// build by exclusion: NOT in the union of all known MIMEs.
// Listing the categories explicitly keeps this in lockstep
// with the named buckets above without a second source of
// truth.
all := []string{
"application/pdf", "application/msword",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"application/vnd.ms-excel",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/vnd.ms-powerpoint",
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
"application/vnd.oasis.opendocument.text",
"application/vnd.oasis.opendocument.spreadsheet",
"application/vnd.oasis.opendocument.presentation",
"application/rtf",
"text/plain", "text/markdown", "text/csv", "text/tab-separated-values",
"application/json", "application/xml", "text/xml",
"application/yaml", "text/yaml", "application/toml",
"text/html", "text/javascript", "application/javascript",
"application/zip", "application/x-tar", "application/gzip",
"application/x-bzip2", "application/x-7z-compressed",
}
placeholders := make([]string, len(all))
args := make([]any, len(all))
for i, m := range all {
placeholders[i] = "?"
args[i] = m
}
frag := "a.mime_type NOT LIKE 'image/%' AND a.mime_type NOT LIKE 'video/%' AND a.mime_type NOT LIKE 'audio/%' AND a.mime_type NOT IN (" + strings.Join(placeholders, ",") + ")"
return frag, args, true
}
return "", nil, false
}
// mimeInPredicate builds a `a.mime_type IN (?,?,...)` fragment with
// matching args. Helper used by mimePredicateForCategory for the
// non-prefix categories.
func mimeInPredicate(mimes []string) (string, []any, bool) {
placeholders := make([]string, len(mimes))
args := make([]any, len(mimes))
for i, m := range mimes {
placeholders[i] = "?"
args[i] = m
}
return "a.mime_type IN (" + strings.Join(placeholders, ",") + ")", args, true
}
// WorkspaceAttachments lists original (non-derived) attachments in a
// workspace, with optional filtering + sorting + pagination. Returns
// the page rows and the total count of matching rows so the UI can
// render a paginator without a second round-trip.
//
// Intentionally hides derived blobs (thumbnails — rows where
// parent_id IS NOT NULL): they're managed automatically and showing
// them in the list would clutter the page with rows the user didn't
// upload. They still count against storage quota via
// WorkspaceStorageUsage; the totals match the bar even when the list
// shows only originals.
//
// LEFT JOIN to items + collections gives the UI everything it needs
// to render an "in [[Task X]]" link. Soft-deleted items are returned
// with item fields nulled out — the attachment is still visible
// (a deleted item could still be restored), but the link target
// isn't reachable.
//
// Both joins are scoped to the attachment's own workspace, so an
// attachment whose item_id points at another workspace's item lists
// with NULL item/collection metadata rather than leaking a foreign
// title (TASK-2399). The row itself still lists — it consumes quota
// and must remain visible and repairable.
func (s *Store) WorkspaceAttachments(workspaceID string, filters AttachmentListFilters) ([]AttachmentListItem, int, error) {
// Build the WHERE clause incrementally. Every branch parameter
// goes through the placeholder slice — no string concatenation of
// user input. Sort is the only user-controllable splice and it
// goes through the allowedAttachmentSorts allowlist.
var conds []string
var args []any
conds = append(conds, "a.workspace_id = ?")
args = append(args, workspaceID)
conds = append(conds, "a.deleted_at IS NULL")
conds = append(conds, "a.parent_id IS NULL") // hide derived blobs
if filters.Attached {
conds = append(conds, "a.item_id IS NOT NULL")
}
if filters.Unattached {
conds = append(conds, "a.item_id IS NULL")
}
if filters.ItemID != "" {
conds = append(conds, "a.item_id = ?")
args = append(args, filters.ItemID)
}
if frag, mimeArgs, ok := mimePredicateForCategory(filters.MimeCategory); ok {
conds = append(conds, frag)
args = append(args, mimeArgs...)
}
if filters.CollectionID != "" {
conds = append(conds, "i.collection_id = ?")
args = append(args, filters.CollectionID)
}
// Collection + item-level visibility enforcement. Two sources of
// access: collections the user can see in full (FullCollectionIDs)
// and individual items granted to them (GrantedItemIDs). The
// predicate ORs them so an item-grant in a hidden collection still
// resolves; orphans (item_id IS NULL) are excluded entirely so a
// restricted user can't enumerate orphan filenames.
//
// Restricted=false → no filter (admin / full-access member).
// Restricted=true with both lists empty → zero rows.
if filters.Restricted {
var ors []string
if len(filters.FullCollectionIDs) > 0 {
ph := make([]string, len(filters.FullCollectionIDs))
for i, id := range filters.FullCollectionIDs {
ph[i] = "?"
args = append(args, id)
}
ors = append(ors, "i.collection_id IN ("+strings.Join(ph, ",")+")")
}
if len(filters.GrantedItemIDs) > 0 {
ph := make([]string, len(filters.GrantedItemIDs))
for i, id := range filters.GrantedItemIDs {
ph[i] = "?"
args = append(args, id)
}
ors = append(ors, "a.item_id IN ("+strings.Join(ph, ",")+")")
}
if len(ors) == 0 {
conds = append(conds, "1 = 0")
} else {
conds = append(conds, "("+strings.Join(ors, " OR ")+")")
}
}
where := strings.Join(conds, " AND ")
// Count total before applying limit/offset so the UI can render
// "showing 1–25 of 312".
//
// The items LEFT JOIN intentionally does NOT filter on
// items.deleted_at — attachments survive a soft-deleted parent
// (they still consume quota), so the storage list must surface
// them and the collection-level visibility predicate
// (i.collection_id IN ...) must keep working. The handler's
// delete path uses GetItemIncludeDeleted for the same reason.
//
// The workspace predicate belongs in the ON clause, NOT in WHERE:
// in WHERE the LEFT JOIN degenerates into an inner join and a row
// whose item_id points at another workspace's item would vanish
// from the listing entirely — hiding quota-consuming rows and
// making them unrepairable. In ON, a mismatched parent simply
// yields NULL item metadata and the row survives (TASK-2399).
var total int
if err := s.db.QueryRow(s.q(`
SELECT COUNT(*) FROM attachments a
LEFT JOIN items i ON i.id = a.item_id AND i.workspace_id = a.workspace_id
WHERE `+where), args...).Scan(&total); err != nil {
return nil, 0, fmt.Errorf("count workspace attachments: %w", err)
}
orderBy, ok := allowedAttachmentSorts[filters.Sort]
if !ok {
orderBy = "a.created_at DESC" // sensible default — newest first
}
limit := filters.Limit
if limit <= 0 {
limit = 50
}
if limit > 200 {
limit = 200
}
offset := filters.Offset
if offset < 0 {
offset = 0
}
// Column list is the same as attachmentColumns but prefixed with
// `a.` so SQLite doesn't choke on the ambiguous `id` shared with
// the joined items table.
const aliasedAttachmentColumns = `a.id, a.workspace_id, a.item_id, a.uploaded_by, a.storage_key, a.content_hash,
a.mime_type, a.size_bytes, a.filename, a.width, a.height, a.parent_id, a.variant, a.created_at, a.deleted_at,
a.filename_source, a.imported`
// Same rationale as the count query above: include soft-deleted
// parent items so the row is still visible for users who would
// be allowed to see the live item, and so the collection-level
// ACL predicate sees a non-NULL i.collection_id. The response
// surfaces deleted_at on the joined item via item_deleted so the
// UI can render a "(deleted)" tag instead of a clickable link.
//
// Same workspace scoping as the count query above, and for the
// same reason — the two must stay consistent or a restricted
// user's count would diverge from their results. Collections are
// reached through the now-scoped item join, so a foreign parent
// nulls out the collection columns too.
//
// The collections join carries its own workspace predicate as
// well: items.collection_id has no composite workspace foreign
// key (migrations/005_collections.sql), so an item can reference
// a collection in another workspace and would otherwise surface
// that collection's slug/name here. Same ON-clause rule — a
// mismatch nulls the collection columns, it does not drop the
// row.
q := `
SELECT ` + aliasedAttachmentColumns + `,
i.title, i.slug, i.deleted_at,
c.slug, c.name
FROM attachments a
LEFT JOIN items i ON i.id = a.item_id AND i.workspace_id = a.workspace_id
LEFT JOIN collections c ON c.id = i.collection_id AND c.workspace_id = i.workspace_id
WHERE ` + where + `
ORDER BY ` + orderBy + `
LIMIT ? OFFSET ?`
args = append(args, limit, offset)
rows, err := s.db.Query(s.q(q), args...)
if err != nil {
return nil, 0, fmt.Errorf("list workspace attachments: %w", err)
}
defer rows.Close()
var out []AttachmentListItem
for rows.Next() {
var a models.Attachment
var itemID, parentID, variant, deletedAt *string
var width, height *int
var createdAt string
var imported int
// Item + collection columns from the LEFT JOIN. All nullable.
var itemTitle, itemSlug, itemDeletedAt *string
var collSlug, collName *string
if err := rows.Scan(
&a.ID, &a.WorkspaceID, &itemID, &a.UploadedBy, &a.StorageKey, &a.ContentHash,
&a.MimeType, &a.SizeBytes, &a.Filename, &width, &height,
&parentID, &variant, &createdAt, &deletedAt,
&a.FilenameSource, &imported,
&itemTitle, &itemSlug, &itemDeletedAt,
&collSlug, &collName,
); err != nil {
return nil, 0, fmt.Errorf("scan workspace attachment: %w", err)
}
a.ItemID = itemID
a.ParentID = parentID
a.Variant = variant
a.Width = width
a.Height = height
a.CreatedAt = parseTime(createdAt)
a.DeletedAt = parseTimePtr(deletedAt)
a.Imported = imported == 1
row := AttachmentListItem{Attachment: a}
if itemTitle != nil {
row.ItemTitle = itemTitle
}
if itemSlug != nil {
row.ItemSlug = itemSlug
}
if itemDeletedAt != nil && *itemDeletedAt != "" {
row.ItemDeleted = true
}
if collSlug != nil {
row.CollectionSlug = collSlug
}
out = append(out, row)
}
if err := rows.Err(); err != nil {
return nil, 0, fmt.Errorf("iterate workspace attachments: %w", err)
}
return out, total, nil
}
// OrphanedAttachments returns rows eligible for orphan GC reclamation
// (TASK-886). Two cases qualify:
//
// - Never-attached uploads: item_id IS NULL AND deleted_at IS NULL
// AND created_at < grace cutoff. The editor uploads first and
// PATCHes the item content second; a tab-close in between leaves
// a row in this state. 30 days is comfortable headroom for that
// race plus any deferred-attachment workflow we add later.
//
// - Soft-deleted past grace: deleted_at IS NOT NULL AND
// deleted_at < grace cutoff. Delete handlers tombstone rows
// immediately so undelete is possible; GC reclaims after the
// grace period.
//
// Both filters compare the timestamp column (TEXT, ISO 8601 UTC)
// against the cutoff string lexicographically — UTC RFC3339 collates
// in chronological order without parsing. The cutoff is computed by
// the caller so tests can inject a deterministic time.
//
// Returned rows include thumbnail variants (parent_id != NULL) when
// they meet either criterion — soft-deleting an original cascades
// to its thumbnails (SoftDeleteAttachment), so they all share the
// same deleted_at and reach the GC together.
func (s *Store) OrphanedAttachments(graceCutoff time.Time) ([]models.Attachment, error) {
cutoffStr := graceCutoff.UTC().Format(time.RFC3339)
rows, err := s.db.Query(s.q(`
SELECT `+attachmentColumns+`
FROM attachments
WHERE
(item_id IS NULL AND deleted_at IS NULL AND created_at < ?)
OR
(deleted_at IS NOT NULL AND deleted_at < ?)
OR
(deleted_at IS NULL AND parent_id IS NOT NULL AND NOT EXISTS (
SELECT 1 FROM attachments p
WHERE p.id = attachments.parent_id
AND p.workspace_id = attachments.workspace_id
AND p.deleted_at IS NULL
))
ORDER BY created_at, id
`), cutoffStr, cutoffStr)
if err != nil {
return nil, fmt.Errorf("orphaned attachments: %w", err)
}
defer rows.Close()
var out []models.Attachment
for rows.Next() {
a, err := scanAttachment(rows)
if err != nil {
return nil, fmt.Errorf("scan orphan: %w", err)
}
out = append(out, *a)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterate orphans: %w", err)
}
return out, nil
}
// HardDeleteAttachment removes the attachments row outright. Never call
// from a request handler — soft-delete is the safe default for
// user-facing flows. The orphan GC no longer uses it (BUG-2415): its
// row deletions go through the conditional Claim* methods below so the
// delete itself is the atomic claim.
func (s *Store) HardDeleteAttachment(id string) error {
_, err := s.db.Exec(s.q(`DELETE FROM attachments WHERE id = ?`), id)
if err != nil {
return fmt.Errorf("hard delete attachment: %w", err)
}
return nil
}
// stampAttachmentRefsTx bumps attachments.last_referenced_at for every
// `pad-attachment:<id>` reference found in the given texts, inside the
// caller's transaction (BUG-2415). Content writers call this alongside
// their content/fields/body write so the stamp commits atomically with
// the reference: either both are visible to the GC sweep's conditional
// claim, or neither is.
//
// Scoped to the workspace so a pasted foreign-workspace reference can't
// refresh another workspace's rows. Unknown / already-deleted ids simply
// match zero rows. Errors are returned, not swallowed: on Postgres any
// failed statement poisons the transaction anyway, and a save whose
// stamp did not commit would not be protected — failing the save is the
// honest outcome.
//
// ORDERING (codex round 3): call this BEFORE the content statement, as
// early in the writer transaction as the texts are known. On Postgres
// the stamp UPDATE row-locks the attachment rows for the rest of the
// transaction, so a concurrent GC claim DELETE blocks until commit and
// then re-evaluates its predicate against the fresh stamp — refusing.
// Stamping late would leave a window where the claim slips between the
// content write and the stamp. Two residuals, both accepted and
// bounded: (1) the claim COMMITTING before the writer's stamp executes
// at all — the row is gone, the stamp matches zero rows, and the
// committed text carries a dangling ref, the same outcome as
// referencing any already-deleted attachment; deliberately NOT turned
// into a save failure because zero-rows also describes legitimate
// unknown / foreign / typo ids. (2) a writer transaction whose
// stamp-to-commit span EXCEEDS orphanGCRefStaleWindow — the stamp is
// already stale when a blocked claim re-evaluates, so the window's
// sizing (see its comment) is what bounds this case, not the lock
// (codex round 4 P2).
func stampAttachmentRefsTx(tx *sql.Tx, s *Store, workspaceID string, texts ...string) error {
if workspaceID == "" {
return nil
}
seen := map[string]struct{}{}
ids := make([]string, 0, 4)
for _, text := range texts {
if text == "" || !strings.Contains(text, attachmentRefPrefix) {
continue
}
for _, m := range attachmentRefRE.FindAllStringSubmatch(text, -1) {
id := m[1]
if _, dup := seen[id]; dup {
continue
}
seen[id] = struct{}{}
ids = append(ids, id)
}
}
if len(ids) == 0 {
return nil
}
// Chunked like the attachment-copy planner's id walks: a hostile
// payload can carry thousands of refs, and an unbounded IN list
// would blow the bind-parameter limit and roll back the whole
// write (codex round 3 P2).
const stampChunk = 400
ts := time.Now().UTC().Format(time.RFC3339)
for start := 0; start < len(ids); start += stampChunk {
end := start + stampChunk
if end > len(ids) {
end = len(ids)
}
chunk := ids[start:end]
placeholders := make([]string, len(chunk))
args := make([]interface{}, 0, 2*len(chunk)+2)
args = append(args, ts, workspaceID)
for i, id := range chunk {
placeholders[i] = "?"
args = append(args, id)
}
for _, id := range chunk {
args = append(args, id)
}
in := strings.Join(placeholders, ",")
// `parent_id IN (...)` stamps (and on Postgres, ROW-LOCKS) the
// referenced originals' VARIANTS too: a variant's claim DELETE
// only re-evaluates predicates on the row it deletes, so a
// fresh stamp on the parent alone cannot make a concurrently
// claimed thumbnail wait — locking the variant row itself does
// (codex round 4 P1). The claim's parent NOT EXISTS stays as
// the belt for variants created AFTER the stamp landed.
if _, err := tx.Exec(s.q(
`UPDATE attachments SET last_referenced_at = ? WHERE workspace_id = ? AND (id IN (`+
in+`) OR parent_id IN (`+in+`))`), args...); err != nil {
return fmt.Errorf("stamp attachment refs: %w", err)
}
}
return nil
}
// ClaimNeverAttachedAttachment is the orphan GC's atomic claim for a
// never-attached row (BUG-2415): a conditional DELETE that re-asserts
// the row is still unattached, still live, and has no FRESH reference
// stamp — all inside the delete statement itself, so a writer's
// in-transaction stamp (stampAttachmentRefsTx) and this delete
// serialize at the database: whichever commits first wins, and the
// loser observes it (the writer's stamp matches zero rows, or this
// claim deletes zero rows).
//
// Returns whether the row was claimed (deleted). The caller must
// reclaim the blob ONLY on a true return — row-before-bytes is the
// ordering that makes a surviving row imply surviving bytes.
//
// A true return is IRREVOCABLE: the row and its metadata are gone, so
// no later recovery surface (PLAN-2411/PLAN-2416) can restore this
// attachment or retry a failed blob delete — the caller's retained
// StorageKey is the only remaining handle for operator cleanup.
//
// refCutoff bounds stamp freshness: a stamp at-or-after it refuses the
// claim. The content LIKE scan (AttachmentReferenced) still runs before
// this in the sweep, so the stamp only needs to cover references that
// landed AFTER that scan — the window is sized in orphan_gc.go, not
// here.
//
// Variants: content references an ORIGINAL's id, so stamps land on the
// parent row — a fresh PARENT stamp refuses the variant's claim too
// (the NOT EXISTS below). A parent already claimed in the same sweep
// has no row, so its variants claim normally, which is correct: a
// legitimately reclaimed original takes its thumbnails with it.
func (s *Store) ClaimNeverAttachedAttachment(id string, refCutoff time.Time) (bool, error) {
cutoff := refCutoff.UTC().Format(time.RFC3339)
res, err := s.db.Exec(s.q(`
DELETE FROM attachments
WHERE id = ?
AND item_id IS NULL
AND deleted_at IS NULL
AND (last_referenced_at IS NULL OR last_referenced_at < ?)
AND NOT EXISTS (
SELECT 1 FROM attachments p
WHERE p.id = attachments.parent_id
AND p.last_referenced_at >= ?
)
`), id, cutoff, cutoff)
if err != nil {
return false, fmt.Errorf("claim never-attached attachment: %w", err)
}
n, err := res.RowsAffected()
if err != nil {
return false, fmt.Errorf("claim never-attached attachment rows: %w", err)
}
return n == 1, nil
}
// ClaimOrphanedVariantAttachment is the orphan GC's atomic claim for a
// LIVE variant row whose parent original is tombstoned or gone
// (BUG-2388) — the leak artifact of the old non-transactional delete
// cascade racing thumbnail derivation, and the retro-cleanup for rows
// already leaked. The DELETE re-asserts the whole shape at delete
// time: still live, still a variant, and the parent still NOT live —
// so a parent restore committing first makes this match zero rows and
// the variant survives (a restored original keeps its thumbnails).
// Same row-before-bytes and irrevocability contract as the other
// Claim methods.
func (s *Store) ClaimOrphanedVariantAttachment(id string) (bool, error) {
// Transaction + parent row-lock so a concurrent RESTORE (clearing
// the parent's deleted_at) serializes with this claim instead of
// racing its NOT EXISTS snapshot (codex round 1 P1): restore commits
// first → the locked re-read sees a live parent and the claim
// refuses; claim commits first → the restore proceeds against a
// parent whose thumbnail is gone (regenerable state, consistent
// order). SQLite serializes writers via _txlock=immediate; the lock
// clause is Postgres-only, same dialect gate as its siblings.
tx, err := s.db.Begin()
if err != nil {
return false, fmt.Errorf("begin variant claim tx: %w", err)
}
defer tx.Rollback()
var parentID *string
var workspaceID string
if err := tx.QueryRow(s.q(`SELECT parent_id, workspace_id FROM attachments WHERE id = ? AND deleted_at IS NULL`), id).Scan(&parentID, &workspaceID); err != nil {
if errors.Is(err, sql.ErrNoRows) {
return false, nil // already gone / tombstoned
}
return false, fmt.Errorf("read variant row: %w", err)
}
if parentID == nil || *parentID == "" {
return false, nil // not a variant
}
// SAME-WORKSPACE parents only (BUG-2622), matching the candidate
// SELECT's scope: a parent_id resolving into another workspace is
// malformed data (no FK, no same-workspace constraint — the class
// PLAN-2397 repairs), and per DR-11a's rule one level down a foreign
// row is not a legitimate parent, so it must not shield the variant
// from this class. ErrNoRows therefore covers hard-gone AND foreign
// alike, and a foreign parent's restore needs no serialization here —
// its liveness is irrelevant to this row either way.
lockQ := `SELECT deleted_at FROM attachments WHERE id = ? AND workspace_id = ?`
if s.dialect.Driver() == DriverPostgres {
lockQ += ` FOR NO KEY UPDATE`
}
var parentDeletedAt *string
switch err := tx.QueryRow(s.q(lockQ), *parentID, workspaceID).Scan(&parentDeletedAt); {
case errors.Is(err, sql.ErrNoRows):
// Parent hard-gone or foreign — the variant is orphaned; claim below.
case err != nil:
return false, fmt.Errorf("lock variant parent: %w", err)
default:
if parentDeletedAt == nil {
// Parent is LIVE (e.g. restored between the candidate
// SELECT and this claim) — refuse.
return false, nil
}
}
res, err := tx.Exec(s.q(`
DELETE FROM attachments
WHERE id = ? AND deleted_at IS NULL
`), id)
if err != nil {
return false, fmt.Errorf("claim orphaned variant attachment: %w", err)
}
n, err := res.RowsAffected()
if err != nil {
return false, fmt.Errorf("claim orphaned variant rows: %w", err)
}
if err := tx.Commit(); err != nil {
return false, fmt.Errorf("commit variant claim: %w", err)
}
return n == 1, nil
}
// ClaimSoftDeletedAttachment is the orphan GC's atomic claim for a
// soft-deleted row past its grace window (BUG-2415): the DELETE
// re-asserts deleted_at is still set and still past the cutoff at
// delete time, so a concurrent restore (which clears deleted_at)
// commits either before this — claim matches zero rows, row survives —
// or after — the restore's own predicate sees the row gone. Same
// row-before-bytes contract and same IRREVOCABILITY note as
// ClaimNeverAttachedAttachment: once true, no recovery surface can
// bring the row back.
// TASK-2658: a successful claim also emits the ref-only attachment.removed
// event, in the SAME transaction as the delete.
//
// Only THIS claim path emits, and the asymmetry with
// ClaimNeverAttachedAttachment is deliberate rather than an oversight. That
// one reclaims rows that were NEVER attached to an item, and attachment.added
// fires only for attachments written against a live item — so those rows never
// announced their arrival, and announcing their removal would hand a consumer
// a deletion for an id it has never seen.
//
// THE IMPLICATION RUNS ONE WAY ONLY, which is the whole reason the gate below
// exists. Never-attached implies never-announced: a row claimable by
// ClaimNeverAttachedAttachment has item_id IS NULL, no path anywhere sets
// attachments.item_id back to NULL on an existing row, and every birth path
// producing a NULL item_id (CreateAttachment, CreateAttachmentTx,
// CreateAttachmentForLiveItem's own orphan branch) is non-emitting.
//
// The CONVERSE is false: plenty of rows reach THIS path having never announced
// themselves — variants (written by the thumbnail/derivation paths and
// tombstoned by their original's cascade, so two per image upload), attachments
// cloned by a cross-workspace copy, and attachments created by workspace
// import.
//
// So the emit carries the SAME gate as attachment.added — a user-visible
// original, attached to an item — and the two are symmetric by construction
// rather than by argument. That closes the variant route, which is the
// systematic one.
//
// RESIDUE, stated because it is real: an import- or copy-created attachment
// still passes that gate while never having emitted an addition, so its removal
// announces a subject the consumer never saw. The failure mode is noise rather
// than harm — an unknown id in a delete is ignorable, where the reverse
// (announced, never retracted) would leave stale state — and the deeper cause
// is the deliberate silence of the import and copy paths, not this one. A soft-deleted row, by contrast, WAS
// attached and did emit, so its removal closes a loop the consumer is holding
// open.
//
// The transaction does not weaken the claim protocol (BUG-2415): the claim's
// conditionality lives entirely in the DELETE's WHERE clause, which is
// unchanged, and wrapping one conditional statement plus one INSERT in a
// transaction leaves the row-before-bytes contract and the IRREVOCABILITY note
// exactly as they were.
func (s *Store) ClaimSoftDeletedAttachment(id string, graceCutoff time.Time) (bool, error) {
tx, err := s.db.Begin()
if err != nil {
return false, fmt.Errorf("claim soft-deleted attachment: %w", err)
}
defer tx.Rollback()
// Read the refs BEFORE the delete — afterwards there is no row to read
// them from. A missing row is not an error here: it means another sweep
// (or a restore) got there first, and the claim below will match zero rows
// and report false, which is the existing contract.
var workspaceID string
var itemID, parentID, variant sql.NullString
refsKnown := true
switch err := tx.QueryRow(s.q(`SELECT workspace_id, item_id, parent_id, variant FROM attachments WHERE id = ?`), id).
Scan(&workspaceID, &itemID, &parentID, &variant); {
case errors.Is(err, sql.ErrNoRows):
refsKnown = false
case err != nil:
return false, fmt.Errorf("claim soft-deleted attachment refs: %w", err)
}
res, err := tx.Exec(s.q(`
DELETE FROM attachments
WHERE id = ?
AND deleted_at IS NOT NULL
AND deleted_at < ?
`), id, graceCutoff.UTC().Format(time.RFC3339))
if err != nil {
return false, fmt.Errorf("claim soft-deleted attachment: %w", err)
}
n, err := res.RowsAffected()
if err != nil {
return false, fmt.Errorf("claim soft-deleted attachment rows: %w", err)
}
if n != 1 {
// Nothing claimed: commit the (empty) transaction and report false,
// preserving the existing contract exactly.
if err := tx.Commit(); err != nil {
return false, fmt.Errorf("claim soft-deleted attachment: %w", err)
}
return false, nil
}
// Same gate as attachment.added: a user-visible ORIGINAL attached to an
// item. Symmetry by construction — if it could not have announced its
// arrival, it does not announce its removal.
announceable := refsKnown && itemID.Valid && itemID.String != "" &&
!parentID.Valid && (!variant.Valid || variant.String == models.AttachmentVariantOriginal)
if announceable {
if err := s.emitRefOnlyDeletionTx(tx, kernelevents.AttachmentRemoved, workspaceID, id, itemID.String, ""); err != nil {
return false, err
}
}
if err := tx.Commit(); err != nil {
return false, fmt.Errorf("claim soft-deleted attachment: %w", err)
}
return true, nil
}
// AttachmentHashesWithRows reports which of the given content hashes have
// at least one attachments row — in ANY state. Soft-deleted rows count,
// including ones already past the GC grace. That is deliberately BROADER
// than CountProtectingAttachmentsForHash, whose grace-window semantics end
// a row's protection when its own grace expires: the row-driven machinery
// may do that because its claim protocol (row before bytes, BUG-2415)
// coordinates row and blob fates within one sweep. The rowless sweep has
// no claim on any row and no such coordination, so it draws the simplest
// safe line instead — a hash with ANY row is that row's business, handled
// by the row sweep's own claim path within a tick; the rowless sweep
// (BUG-2406) takes only blobs NO row references.
//
// Chunked like the copy planner's lookups so a large listing cannot blow
// the host-parameter limit.
func (s *Store) AttachmentHashesWithRows(hashes []string) (map[string]bool, error) {
out := make(map[string]bool, len(hashes))
for _, chunk := range chunkStrings(hashes, attachmentPlanChunk) {
args := make([]any, 0, len(chunk))
for _, h := range chunk {
args = append(args, h)
}
query := `SELECT DISTINCT content_hash FROM attachments
WHERE content_hash IN (` + sqlPlaceholderList(len(chunk)) + `)`
rows, err := s.db.Query(s.q(query), args...)
if err != nil {
return nil, fmt.Errorf("attachment hashes with rows: %w", err)
}
for rows.Next() {
var h string
if err := rows.Scan(&h); err != nil {
rows.Close()
return nil, fmt.Errorf("attachment hashes with rows: %w", err)
}
out[h] = true
}
if err := rows.Err(); err != nil {
rows.Close()
return nil, fmt.Errorf("attachment hashes with rows: %w", err)
}
rows.Close()
}
return out, nil
}
// AttachmentRowsExistForHash is the single-hash form of
// AttachmentHashesWithRows — the rowless-blob sweep's delete-time
// re-check, run under the in-flight mutex immediately before a blob
// delete so a row inserted after the batched subtraction still protects
// its bytes (see runRowlessBlobSweep's TOCTOU note).
func (s *Store) AttachmentRowsExistForHash(hash string) (bool, error) {
return s.AttachmentRowsExistForHashContext(context.Background(), hash)
}
// AttachmentRowsExistForHashContext is AttachmentRowsExistForHash bounded by
// ctx, for a caller that runs it under a process-wide lock (BUG-3652).
func (s *Store) AttachmentRowsExistForHashContext(ctx context.Context, hash string) (bool, error) {
var n int
err := s.db.QueryRowContext(ctx, s.q(`SELECT COUNT(*) FROM attachments WHERE content_hash = ?`), hash).Scan(&n)
if err != nil {
return false, fmt.Errorf("attachment rows exist for hash: %w", err)
}
return n > 0, nil
}
// CountProtectingAttachmentsForHash returns the number of rows
// pointing at the given content_hash whose presence requires the
// on-disk blob to stay. A row protects the blob when it is either:
//
// - live (deleted_at IS NULL), or
// - soft-deleted but still inside the grace window
// (deleted_at >= graceCutoff)
//
// excludeID is the row currently being GC'd; we don't count it
// against itself. Codex P2 round 3 caught the earlier version's
// gap: counting only deleted_at IS NULL would have GC reclaim the
// blob from row A (soft-deleted 31d ago) even though row B is
// also soft-deleted but still 1 day old — within grace, so its
// blob must stay reachable until its own grace expires.
func (s *Store) CountProtectingAttachmentsForHash(hash, excludeID string, graceCutoff time.Time) (int, error) {
var n int
cutoff := graceCutoff.UTC().Format(time.RFC3339)
err := s.db.QueryRow(s.q(`
SELECT COUNT(*) FROM attachments
WHERE content_hash = ? AND id <> ?
AND (deleted_at IS NULL OR deleted_at >= ?)
`), hash, excludeID, cutoff).Scan(&n)
if err != nil {
return 0, fmt.Errorf("count protecting attachments for hash: %w", err)
}
return n, nil
}
// AttachmentReferenced returns true when anything in the workspace
// mentions "pad-attachment:<id>" — either a live item's content/fields
// JSON or a comment body. The editor and comment-composer upload flows
// both leave attachments.item_id NULL: the canonical association is the
// markdown reference inside the item's content or a comment, NOT a
// column in the attachments table — so the orphan GC has to look at
// that text directly before reclaiming a "never-attached" row,
// otherwise it'd hard-delete attachments that markdown still points at.
//
// Comments are covered because a pasted screenshot (IDEA-1650) may be
// referenced ONLY from a comment body; without the comments scan the GC
// would reclaim it after the grace period and break the embed. Documents
// are covered for the same reason (BUG-2614) — the legacy v1 documents
// API is still mounted, so a direct API consumer can put the only
// reference to an attachment in a document body.
//
// THE SET OF SCANNED SURFACES IS THE CONTRACT. Anything that persists
// user-authored text which can contain a `pad-attachment:` token belongs
// here, and adding such a surface without adding it here silently makes
// its references invisible to the GC. Both of the additions above were
// found that way, after the fact.
//
// Scoped to one workspace because a "pad-attachment:UUID" reference
// only resolves within the workspace where the attachment lives;
// cross-workspace references are intentionally not supported.
//
// Dialect note: items.fields is TEXT on SQLite but JSONB on
// PostgreSQL (see migrations + pgmigrations). LIKE doesn't work on
// JSONB so the Postgres path casts to text first. Codex P1 round 2
// caught this — without the cast, the GC's reference scan errored
// on Postgres and every never-attached row got skipped. comments.body
// is TEXT on both dialects, so it needs no cast.
func (s *Store) AttachmentReferenced(workspaceID, attachmentID string) (bool, error) {
if workspaceID == "" || attachmentID == "" {
return false, nil
}
needle := "pad-attachment:" + attachmentID
pattern := "%" + needle + "%"
fieldsExpr := "fields"
if s.dialect.Driver() == DriverPostgres {
fieldsExpr = "fields::text"
}
// The documents leg covers the legacy v1 `/workspaces/{ws}/documents`
// surface (BUG-2614). It has no first-party client left — nothing in the
// web API client or the CLI calls it — but the CRUD routes are mounted and
// authenticated, so a direct API consumer can still put a
// `pad-attachment:` reference in a document body. Scanning items and
// comments only made such an attachment reclaimable while genuinely
// referenced. Live rows only, mirroring items; comments carry no
// deleted_at at all, which is why that leg has no such filter.
var n int
err := s.db.QueryRow(s.q(`
SELECT
(SELECT COUNT(*) FROM items
WHERE workspace_id = ? AND deleted_at IS NULL
AND (content LIKE ? OR `+fieldsExpr+` LIKE ?))
+ (SELECT COUNT(*) FROM comments
WHERE workspace_id = ? AND body LIKE ?)
+ (SELECT COUNT(*) FROM documents
WHERE workspace_id = ? AND deleted_at IS NULL AND content LIKE ?)
`), workspaceID, pattern, pattern, workspaceID, pattern, workspaceID, pattern).Scan(&n)
if err != nil {
return false, fmt.Errorf("attachment referenced: %w", err)
}
return n > 0, nil
}
// SoftDeleteWorkspaceAttachments tombstones every live attachment
// row (originals AND thumbnail variants) under a workspace in a
// single bulk UPDATE. Used by the bundle import handler to roll back
// a partial workspace when a validation reject fires AFTER blobs
// have already been rehydrated — without this cascade the
// attachment rows stay live (deleted_at IS NULL), pin their blobs
// from orphan-GC reclamation, and continue counting toward the
// importing user's storage usage. Codex P1 on PR #308.
//
// Both originals and thumbnails carry the same workspace_id, so a
// single WHERE workspace_id = ? clause covers both. Returns the
// number of rows tombstoned.
//
// The blob bytes on disk are NOT touched — content-addressed dedupe
// means another live row may still reference the same hash, so
// reclamation is the orphan-GC's job after the grace period.
func (s *Store) SoftDeleteWorkspaceAttachments(workspaceID string) (int64, error) {
ts := now()
res, err := s.db.Exec(s.q(`
UPDATE attachments
SET deleted_at = ?
WHERE workspace_id = ? AND deleted_at IS NULL
`), ts, workspaceID)
if err != nil {
return 0, fmt.Errorf("soft delete workspace attachments: %w", err)
}
n, err := res.RowsAffected()
if err != nil {
return 0, fmt.Errorf("soft delete workspace attachments rows affected: %w", err)
}
return n, nil
}
// SoftDeleteAttachment marks the given attachment row deleted (and
// every variant whose parent_id points at it) so the orphan GC will
// reclaim the bytes after the grace period. Returns sql.ErrNoRows if
// no live row matches.
//
// We mark variants via a separate UPDATE keyed on parent_id rather
// than letting a foreign-key cascade do it — the attachments table
// has no FK on parent_id, by design (DOC-865: thumbnails are
// independent rows so a missing original doesn't break a list query).
//
// The blob on disk is NOT removed here; the same content_hash may be
// referenced by other rows (content-addressed dedupe), so reclamation
// is the GC's job once it can prove no live row references the hash.
func (s *Store) SoftDeleteAttachment(id string) error {
ts := now()
// One TRANSACTION for original + variants (BUG-2388): the old two
// separate statements let a crash — or a concurrent thumbnail
// derivation — land between them, leaving half-tombstoned families.
// (The old comment claimed orphan GC would reach stragglers "via the
// deleted-parent path"; no such path existed — that is exactly the
// leak this bug fixes, and the sweep now has a real orphaned-variant
// class as the belt.)
tx, err := s.db.Begin()
if err != nil {
return fmt.Errorf("begin soft delete tx: %w", err)
}
defer tx.Rollback()
res, err := tx.Exec(s.q(`
UPDATE attachments
SET deleted_at = ?
WHERE id = ? AND deleted_at IS NULL
`), ts, id)
if err != nil {
return fmt.Errorf("soft delete attachment: %w", err)
}
n, err := res.RowsAffected()
if err != nil {
return fmt.Errorf("soft delete attachment rows affected: %w", err)
}
if n == 0 {
return sql.ErrNoRows
}
// Tombstone the thumbnail variants atomically with the original —
// synthetic rows with no reason to outlive it.
if _, err := tx.Exec(s.q(`
UPDATE attachments
SET deleted_at = ?
WHERE parent_id = ? AND deleted_at IS NULL
`), ts, id); err != nil {
return fmt.Errorf("soft delete attachment variants: %w", err)
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("commit soft delete: %w", err)
}
return nil
}
// CreateAttachmentVariantIfParentLive inserts a derived-variant row
// ONLY while its parent original is still live — the liveness check is
// part of the INSERT statement itself (BUG-2388, the same
// claim-by-statement discipline as the orphan-GC claims): a delete
// cascade committing between the derivation's early parent check and
// this insert makes the WHERE EXISTS fail and the insert report false,
// instead of minting a live variant row under a tombstoned parent.
//
// Returns whether the row was inserted. The caller owns the just-Put
// blob on a false return (clean it up under the in-flight hash fence
// it already holds).
func (s *Store) CreateAttachmentVariantIfParentLive(a *models.Attachment) (bool, error) {
if a.ParentID == nil || *a.ParentID == "" {
return false, fmt.Errorf("variant insert requires parent_id")
}
if a.StorageKey == "" {
return false, fmt.Errorf("attachment storage_key must not be empty")
}
if a.ID == "" {
a.ID = newID()
}
ts := now()
if a.CreatedAt.IsZero() {
a.CreatedAt = parseTime(ts)
} else {
ts = a.CreatedAt.UTC().Format("2006-01-02T15:04:05Z07:00")
}
// Same transaction + row-lock shape as CreateAttachmentForLiveItem
// (which closed the identical check-then-insert race against ITEM
// deletion, PLAN-2391 DR-14): on Postgres a plain WHERE EXISTS reads
// a snapshot and a concurrent SoftDeleteAttachment can commit under
// it — the lock makes the delete cascade wait, and whichever commits
// first, the other observes it (codex round 1 P1). SQLite's
// _txlock=immediate already serializes writers, and the lock clause
// is a syntax error there — the dialect gate is not an optimization.
tx, err := s.db.Begin()
if err != nil {
return false, fmt.Errorf("begin variant insert tx: %w", err)
}
defer tx.Rollback()
// The lock also READS the parent's binding (BUG-3385). The caller built
// the variant from a parent snapshot taken before decoding and
// resizing; if the original was attached to an item since, that
// snapshot's item_id is stale. AttachAttachmentToItem updates only the
// variants that already exist, so a variant inserted from the snapshot
// stayed unbound under a bound parent until orphan GC reaped it. Under
// this lock no bind can land between the read and the insert.
//
// The parent's uploader and via_app are read the same way and copied
// (TASK-3396, DOC-3371 §4 "variants inherit their parent's binding"), so
// a thumbnail of an app upload is attributed and app-visible exactly as
// its original is, whatever the caller's snapshot said.
query := `SELECT item_id, uploaded_by, via_app FROM attachments WHERE id = ? AND deleted_at IS NULL`
if s.dialect.Driver() == DriverPostgres {
query += ` FOR NO KEY UPDATE`
}
var parentItemID, parentViaApp sql.NullString
var parentUploader string
switch err := tx.QueryRow(s.q(query), *a.ParentID).Scan(&parentItemID, &parentUploader, &parentViaApp); {
case errors.Is(err, sql.ErrNoRows):
// Parent gone or tombstoned — refuse, no row minted.
return false, nil
case err != nil:
return false, fmt.Errorf("lock variant parent: %w", err)
}
if parentItemID.Valid {
id := parentItemID.String
a.ItemID = &id
} else {
a.ItemID = nil
}
a.UploadedBy = parentUploader
source, err := attachmentFilenameSource(a)
if err != nil {
return false, err
}
if _, err := tx.Exec(s.q(`
INSERT INTO attachments (`+attachmentColumns+`, via_app)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`), a.ID, a.WorkspaceID, a.ItemID, a.UploadedBy, a.StorageKey, a.ContentHash,
a.MimeType, a.SizeBytes, a.Filename, a.Width, a.Height, a.ParentID, a.Variant,
ts, nil, source, importedFlag(a), parentViaApp); err != nil {
return false, fmt.Errorf("create variant row: %w", err)
}
if err := tx.Commit(); err != nil {
return false, fmt.Errorf("commit variant insert: %w", err)
}
return true, nil
}
// WorkspaceItemSlugMap returns slug → id for every live item in a
// workspace. Used by the bundle import path (TASK-885) to remap an
// old attachment's item_id (from the manifest) to the freshly-
// generated id, via item.slug which ImportWorkspace preserves.
//
// Soft-deleted items are excluded; the import path can't realistically
// recreate an attachment under a deleted parent without also
// resurrecting the parent, and the manifest's item_id only has
// meaning for live items at export time.
func (s *Store) WorkspaceItemSlugMap(workspaceID string) (map[string]string, error) {
rows, err := s.db.Query(s.q(`
SELECT id, slug FROM items WHERE workspace_id = ? AND deleted_at IS NULL
`), workspaceID)
if err != nil {
return nil, fmt.Errorf("workspace item slug map: %w", err)
}
defer rows.Close()
out := map[string]string{}
for rows.Next() {
var id, slug string
if err := rows.Scan(&id, &slug); err != nil {
return nil, fmt.Errorf("scan slug map: %w", err)
}
out[slug] = id
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterate slug map: %w", err)
}
return out, nil
}
// RemapAttachmentReferencesInWorkspace rewrites every
// "pad-attachment:OLD" reference in items.content + items.fields,
// in comment bodies (BUG-2615) AND in version history (BUG-3655) to
// "pad-attachment:NEW" for every (old, new) pair in the map. Run after a bundle import has
// rehydrated attachments so the imported content points at the new
// attachment ids instead of the source workspace's ids.
//
// CALLER PRECONDITION — read this before adding a second caller.
// Every surface here is read-modify-write across a scan and a later
// UPDATE, with no row locks and no old-value predicate, and the whole
// population is loaded inside one transaction. That is safe for the
// ONE existing caller (the bundle import) for a reason that is about
// the caller, not this function: it runs against a workspace the
// import itself just created, which no other session can reach yet,
// so there are no concurrent writers to lose an edit to and no
// contention to hold up. Called against a LIVE workspace it would
// clobber a concurrent edit committed between the scan and the write,
// and would hold a long transaction across the entire item and
// comment population. Both would need fixing first — the pre-existing
// items walk has the same shape, so this is a property of the
// function, not of the comment leg added for BUG-2615.
//
// Implementation: a single transaction that loads every item's
// content+fields, every comment body and every live item's version
// chain (planVersionChainRemapTx), runs remapAttachmentRefs over each,
// stamps the attachment rows the content and comment rewrites will
// point at, then writes back only what changed. That helper tokenizes with
// attachmentRefRE and matches whole ids — NOT strings.ReplaceAll,
// which used to rewrite a mapped id sitting as the PREFIX of a
// longer one (Codex round 26). See its doc.
//
// Search needs no separate pass: on SQLite each UPDATE fires the
// items_fts_update / comments_fts_update triggers inside the
// transaction, and on Postgres search_vector is trigger-maintained. A
// post-commit rebuild that used to follow re-inserted every item and
// double-indexed it (BUG-2758).
func (s *Store) RemapAttachmentReferencesInWorkspace(workspaceID string, oldToNew map[string]string) error {
if len(oldToNew) == 0 {
return nil
}
tx, err := s.db.Begin()
if err != nil {
return fmt.Errorf("begin remap tx: %w", err)
}
defer tx.Rollback()
// Serialise against concurrent item writes in this workspace (codex round
// 5 on PLAN-2857 U5). This function SCANS every item's fields and content,
// then writes the snapshot back — so without the lock a concurrent update
// can commit newer fields in between and this overwrites them.
//
// That lost update predates the relation index and is not caused by it;
// what U5 adds is that the same stale snapshot is now also used to rebuild
// the reverse index, so the staleness reaches a second place. One lock
// closes both.
//
// It is the same advisory lock ITEM writers take, and taking it first here
// preserves their order (workspace, then rows) so the two cannot invert.
// Comment writers do not take it at all — which is safe, because a lock
// nobody else in that path holds cannot participate in a cycle, and this
// transaction takes no lock a comment writer could be waiting on.
if err := s.acquireWorkspaceSeqLock(tx, workspaceID); err != nil {
return err
}
// ORDER BY id (BUG-2778 class sweep): this gathers rows and then UPDATEs
// them one by one inside a transaction, so the scan's order IS the lock
// order. Two concurrent remaps in one workspace would otherwise be free to
// take the same row locks in different orders — Postgres does not promise
// a stable seq-scan order, and synchronize_seqscans deliberately starts
// concurrent scans at different points.
//
// SCOPE OF THE CLAIM (codex round 6): this makes the CONTENT-ROW updates
// below take their locks in one order for every caller. It does not order
// the attachment rows stampAttachmentRefsTx locks elsewhere in the same
// transaction, so it is not "the whole deadlock class removed" — it
// removes the cycle between two concurrent remaps' own row updates, which
// is the one this sweep found. Unlike the rename path in documents.go,
// where the second stage locks the renamed row itself and ordering the
// cascade therefore cannot help, ordering IS sufficient for the cycle
// between these per-row updates.
rows, err := tx.Query(s.q(`SELECT id, collection_id, content, fields FROM items WHERE workspace_id = ? AND deleted_at IS NULL ORDER BY id`), workspaceID)
if err != nil {
return fmt.Errorf("scan items for remap: %w", err)
}
type rowUpdate struct {
id string
collectionID string
content string
fields string
oldContent string
}
var updates []rowUpdate
for rows.Next() {
var id, collectionID, content, fields string
if err := rows.Scan(&id, &collectionID, &content, &fields); err != nil {
rows.Close()
return fmt.Errorf("scan item: %w", err)
}
newContent := remapAttachmentRefs(content, oldToNew)
newFields := remapAttachmentRefs(fields, oldToNew)
if newContent != content || newFields != fields {
updates = append(updates, rowUpdate{id: id, collectionID: collectionID, content: newContent, fields: newFields, oldContent: content})
}
}
if err := rows.Err(); err != nil {
rows.Close()
return fmt.Errorf("iterate items for remap: %w", err)
}
rows.Close()
// Comment bodies carry `pad-attachment:` references exactly as item
// content does, and a bundle import re-inserts them (export.go's comment
// import), so walking items alone left every comment pointing at the
// SOURCE workspace's ids — which resolve to nothing in the destination,
// while the freshly-cloned rows they should have pointed at end up
// referenced by nothing and eventually reclaimed (BUG-2615). Comments have
// no deleted_at column, hence no filter here.
crows, err := tx.Query(s.q(`SELECT id, body FROM comments WHERE workspace_id = ? ORDER BY id`), workspaceID)
if err != nil {
return fmt.Errorf("scan comments for remap: %w", err)
}
type commentUpdate struct {
id string
body string
}
var commentUpdates []commentUpdate
for crows.Next() {
var id, body string
if err := crows.Scan(&id, &body); err != nil {
crows.Close()
return fmt.Errorf("scan comment: %w", err)
}
if newBody := remapAttachmentRefs(body, oldToNew); newBody != body {
commentUpdates = append(commentUpdates, commentUpdate{id: id, body: newBody})
}
}
if err := crows.Err(); err != nil {
crows.Close()
return fmt.Errorf("iterate comments for remap: %w", err)
}
crows.Close()
// Version history carries references too (BUG-3655): a full snapshot holds
// a past body whole, and a reverse patch can hold an id in the text it
// re-inserts. Restoring either brought back the SOURCE id. Rewrite the
// whole chain here, against the remapped bodies.
versionUpdates, rechained, err := s.planVersionChainRemapTx(tx, workspaceID, oldToNew)
if err != nil {
return err
}
// Both scans are done and nothing is written yet: the lost-update window
// the workspace seq lock above closes (BUG-2797).
if s.afterRemapScan != nil {
s.afterRemapScan(workspaceID)
}
// Stamp what the rewrites now point AT, inside this same transaction.
// The import stamped each comment body at insert time, but the body still
// held the SOURCE ids then, so those stamps landed on nothing that ends up
// referenced here. Without this the destination's clones are freshly
// created, referenced only by text this transaction just wrote, and
// carrying no stamp — the exact shape the orphan GC's never-attached claim
// reclaims (BUG-2615).
//
// ORDERING: this runs BEFORE the content UPDATEs, per
// stampAttachmentRefsTx's own contract and for the same two reasons every
// other writer follows it. On Postgres the stamp row-locks the attachment
// rows for the rest of the transaction, so a concurrent GC claim blocks
// and re-evaluates against the fresh stamp; stamping last would let a
// claim delete the target while the rewritten text sat uncommitted, and
// the stamp would then match zero rows and commit a dangling reference.
// It also keeps this path's lock order identical to the item and comment
// writers (attachments first, then content rows) — the reverse order
// deadlocks against them (codex round 2 P1).
//
// The texts are already known here: both scans have run and produced the
// exact strings the UPDATEs below will write.
//
// The REWRITTEN TEXTS are passed rather than every id in oldToNew, so only
// ids something actually references get stamped. Stamping the whole map
// would also refresh clones nothing points at, keeping genuinely
// unreferenced rows alive for an extra GC window.
stampTexts := make([]string, 0, len(updates)*2+len(commentUpdates))
for _, u := range updates {
stampTexts = append(stampTexts, u.content, u.fields)
}
for _, u := range commentUpdates {
stampTexts = append(stampTexts, u.body)
}
if err := stampAttachmentRefsTx(tx, s, workspaceID, stampTexts...); err != nil {
return fmt.Errorf("stamp remapped refs: %w", err)
}
for _, u := range updates {
// BUMPS seq (BUG-3037). This write changes `content` and `fields` — the
// state `expected_seq` exists to protect — and its SELECT above is
// workspace-wide, so it can reach a row a client is holding a token for.
// Today its only caller is bundle import, where every row was created
// moments earlier in the same request and nobody holds a token; the bump
// is here anyway so that stays true of the CODE rather than of the caller
// list, which is the kind of claim that goes stale unread. The workspace
// seq lock this transaction already took above is the one the subquery
// needs, so this costs an indexed MAX(seq) per row and no new locking.
// NOT a decision enqueue door (TASK-3117 ruling 3). This is a system
// fan-out rewrite: one import remap touches every matching item, so an enqueue
// per row would be an unbounded provider bill for one action. The rewrite
// DOES change hashed decision state, so each touched item's stored
// answers read current=false until its next direct write re-evaluates.
if _, err := tx.Exec(s.q(`UPDATE items SET content = ?, fields = ?, seq = `+nextWorkspaceSeqSubquery+` WHERE id = ?`),
u.content, u.fields, workspaceID, u.id); err != nil {
return fmt.Errorf("update item %s: %w", u.id, err)
}
// The body changed with no version, so the newest version's reverse
// patch is rebased onto it (BUG-3745). A chain rewritten whole above
// already encodes its newest version against the new body.
if u.content != u.oldContent && !rechained[u.id] {
if err := s.rebaseNewestVersionPatch(tx, u.id, u.oldContent, u.content); err != nil {
return fmt.Errorf("rebase newest version of %s: %w", u.id, err)
}
}
// This rewrite maps `pad-attachment:` ids and cannot change a relation
// VALUE — but it does write the blob, and the hook is a pure function
// of (blob, schema), so calling it is a no-op while OMITTING it would
// be a standing claim about what remapAttachmentRefs can touch. The
// collection id rides along on the scan above rather than costing a
// lookup per item (PLAN-2857 U5).
if _, err := s.replaceRelationLinks(tx, u.id, workspaceID, u.collectionID, u.fields); err != nil {
return fmt.Errorf("index relation links for item %s: %w", u.id, err)
}
}
for _, u := range commentUpdates {
if _, err := tx.Exec(s.q(`UPDATE comments SET body = ? WHERE id = ?`),
u.body, u.id); err != nil {
return fmt.Errorf("update comment %s: %w", u.id, err)
}
}
for _, u := range versionUpdates {
if _, err := tx.Exec(s.q(`UPDATE item_versions SET content = ?, is_diff = ?, body_sha256 = ? WHERE id = ?`),
u.content, s.dialect.BoolToInt(u.isDiff), nullIfEmpty(u.bodySHA256), u.id); err != nil {
return fmt.Errorf("update version %s: %w", u.id, err)
}
}
// The UPDATEs above fired items_fts_update, which already reindexed the
// rewritten content; there is no separate FTS pass (BUG-2758).
if err := tx.Commit(); err != nil {
return fmt.Errorf("commit remap: %w", err)
}
return nil
}
// versionRemap is one item_versions row's rewritten storage.
type versionRemap struct {
id string
content string
isDiff bool
// bodySHA256 is the row's body_sha256 after the rewrite ("" = NULL):
// the remapped body's hash when the old body was verified against one,
// otherwise the row's hash as it was (BUG-3820).
bodySHA256 string
}
// planVersionChainRemapTx plans the item_versions rewrite for
// RemapAttachmentReferencesInWorkspace (BUG-3655). Versions are reverse
// patches resolved from the item's body backwards, so a reference in history
// cannot be rewritten in place: each live item's chain is resolved against
// its body as it stands (the remap has not written yet), every resolved body
// is remapped, and each row is re-encoded against its remapped successor. A
// full row stays full; a patch row is re-diffed and kept only while smaller
// than the body, the rule every version writer uses. Only rows whose body or
// successor changed are rewritten. Line counts are left alone: a remap
// swaps one id for another and changes no line.
//
// It returns the rewrites and the items whose newest row it re-encoded
// against the remapped body, so the caller skips its newest-only rebase for
// them. A patch that fails to apply has no body; its text is remapped as
// written, the rows below it resolve past it as the reader's walk does, and
// the item is logged.
func (s *Store) planVersionChainRemapTx(tx *sql.Tx, workspaceID string, oldToNew map[string]string) ([]versionRemap, map[string]bool, error) {
// ORDER BY id for the lock order, as the item scan above.
rows, err := tx.Query(s.q(`
SELECT i.id, i.content FROM items i
WHERE i.workspace_id = ? AND i.deleted_at IS NULL
AND EXISTS (SELECT 1 FROM item_versions v WHERE v.item_id = i.id)
ORDER BY i.id`), workspaceID)
if err != nil {
return nil, nil, fmt.Errorf("scan versioned items for remap: %w", err)
}
type versioned struct{ id, content string }
var items []versioned
for rows.Next() {
var it versioned
if err := rows.Scan(&it.id, &it.content); err != nil {
rows.Close()
return nil, nil, fmt.Errorf("scan versioned item: %w", err)
}
items = append(items, it)
}
if err := rows.Err(); err != nil {
rows.Close()
return nil, nil, fmt.Errorf("iterate versioned items: %w", err)
}
rows.Close()
var out []versionRemap
rechained := map[string]bool{}
for _, it := range items {
// Newest first, the order the chain resolves in (ListItemVersionsPage).
vrows, err := tx.Query(s.q(`SELECT id, content, is_diff, COALESCE(body_sha256, '') FROM item_versions
WHERE item_id = ? ORDER BY created_at DESC, version_seq DESC`), it.id)
if err != nil {
return nil, nil, fmt.Errorf("scan versions of %s: %w", it.id, err)
}
type row struct {
id, stored, body, hash string
isDiff bool
}
var chain []row
for vrows.Next() {
var r row
if err := vrows.Scan(&r.id, &r.stored, &r.isDiff, &r.hash); err != nil {
vrows.Close()
return nil, nil, fmt.Errorf("scan version: %w", err)
}
chain = append(chain, r)
}
if err := vrows.Err(); err != nil {
vrows.Close()
return nil, nil, fmt.Errorf("iterate versions of %s: %w", it.id, err)
}
vrows.Close()
// The walk ListItemVersionsResolvedPage does, so every body here is
// the one a reader is shown: a patch that fails to apply, or whose
// result does not match the row's body_sha256 (BUG-3820), is skipped,
// and the row below it resolves against the same successor.
known := make([]bool, len(chain))
successor := it.content
for i := range chain {
if !chain[i].isDiff {
chain[i].body, known[i] = chain[i].stored, true
} else if body, err := diff.ApplyPatch(successor, chain[i].stored); err == nil {
chain[i].body, known[i] = body, true
}
if known[i] && chain[i].hash != "" && versionBodySHA256(chain[i].body) != chain[i].hash {
chain[i].body, known[i] = "", false
}
if known[i] {
successor = chain[i].body
}
}
if len(chain) > 0 && known[0] {
rechained[it.id] = true
}
oldSucc, newSucc := it.content, remapAttachmentRefs(it.content, oldToNew)
broken := false
for i, r := range chain {
if !known[i] {
// No body to re-diff. Remap the patch text itself: an id it
// carries whole is replaced by one of the same length, so the
// hunk counts still hold. It resolved no better at the source.
// The successors stay as they are, as in the walk above.
broken = true
if t := remapAttachmentRefs(r.stored, oldToNew); t != r.stored {
out = append(out, versionRemap{id: r.id, content: t, isDiff: r.isDiff, bodySHA256: r.hash})
}
continue
}
newBody := remapAttachmentRefs(r.body, oldToNew)
if newBody != r.body || (r.isDiff && newSucc != oldSucc) {
stored, isDiff := newBody, false
if r.isDiff {
if patch, ok := diff.ReversePatch(newBody, newSucc); ok && diff.IsDiffSmaller(patch, newBody) {
stored, isDiff = patch, true
}
}
// A hash is re-minted only for a body that was verified
// against one: an unhashed row's body was reconstructed, not
// verified, and stays unhashed (BUG-3820).
hash := r.hash
if hash != "" {
hash = versionBodySHA256(newBody)
}
if stored != r.stored || isDiff != r.isDiff || hash != r.hash {
out = append(out, versionRemap{id: r.id, content: stored, isDiff: isDiff, bodySHA256: hash})
}
}
oldSucc, newSucc = r.body, newBody
}
if broken {
slog.Warn("attachment remap: part of an item's version history does not resolve; its patch text was remapped as written",
"workspace_id", workspaceID, "item_id", it.id)
}
}
return out, rechained, nil
}
// remapAttachmentRefs replaces "pad-attachment:OLD" with
// "pad-attachment:NEW" for every (old, new) pair in the map. Pure
// string operation — kept private so callers go through
// RemapAttachmentReferencesInWorkspace, whose UPDATEs fire the FTS
// triggers.
//
// IT TOKENIZES WITH attachmentRefRE, the SAME pattern the copy
// planner enumerates references with, and rewrites only when the
// captured id matches a map key EXACTLY.
//
// A plain strings.ReplaceAll over "pad-attachment:"+old was the
// obvious implementation and was subtly wrong (Codex round 26). The
// regex is greedy — `pad-attachment:<uuid>x` captures `<uuid>x`, one
// id, which resolves to nothing and is deliberately left alone — but a
// substring replace happily rewrote the `<uuid>` PREFIX inside it,
// producing text that matched neither what the planner enumerated nor
// what the user wrote. PLAN-2357 DR-11a is explicit that an
// unresolvable reference keeps its literal text "so the copy renders
// exactly as broken as the source did", and
// items_cross_workspace_copy.go claims the rewrite "covers precisely
// the reference set the plan cloned". Sharing the tokenizer is what
// makes both true rather than nearly true.
func remapAttachmentRefs(s string, oldToNew map[string]string) string {
if len(oldToNew) == 0 || !strings.Contains(s, attachmentRefPrefix) {
return s
}
return attachmentRefRE.ReplaceAllStringFunc(s, func(match string) string {
id := strings.TrimPrefix(match, attachmentRefPrefix)
fresh, ok := oldToNew[id]
if !ok || fresh == "" || fresh == id {
return match
}
return attachmentRefPrefix + fresh
})
}
// workspaceAttachmentsForExportQ returns every original (non-derived,
// non-deleted) attachment in the workspace so the export bundler
// can stream them into the tar. Derived rows (thumbnails) are
// excluded — they're re-derived on import via the existing pipeline.
//
// Soft-deleted parents are NOT excluded here: the attachment row is
// still live, the bytes still exist on disk, and the user explicitly
// chose to export the workspace. Surfacing them lets a round-trip
// preserve the audit trail (the import path will recreate the row
// as an orphan, or the soft-deleted item gets restored separately).
//
// There is no pooled form. The only caller is ExportWorkspaceSnapshot, so the
// manifest and the JSON it ships beside always come from one snapshot
// (BUG-3654).
func (s *Store) workspaceAttachmentsForExportQ(q Queryer, workspaceID string) ([]models.Attachment, error) {
rows, err := q.Query(s.q(`
SELECT `+attachmentColumns+`
FROM attachments
WHERE workspace_id = ? AND deleted_at IS NULL AND parent_id IS NULL
ORDER BY created_at, id
`), workspaceID)
if err != nil {
return nil, fmt.Errorf("export attachments: %w", err)
}
defer rows.Close()
var out []models.Attachment
for rows.Next() {
a, err := scanAttachment(rows)
if err != nil {
return nil, fmt.Errorf("scan export attachment: %w", err)
}
out = append(out, *a)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterate export attachments: %w", err)
}
return out, nil
}
// WorkspaceStorageUsage returns the total bytes consumed by non-deleted
// attachments in the workspace. Includes derived blobs (thumbnails) —
// those are real bytes on disk and count against quota.
func (s *Store) WorkspaceStorageUsage(workspaceID string) (int64, error) {
return s.workspaceStorageUsageQ(s.db, workspaceID)
}
// accountStorageUsageQ is the bytes of non-deleted attachments across every
// workspace ownerID owns: the pool a per-account limit is checked against.
//
// A soft-deleted workspace COUNTS until it is purged (lead ruling, TASK-3619):
// its bytes are still on disk, and the cap exists to stop a full disk.
// Leaving it out would let upload, soft-delete, upload, restore walk past the
// cap, and restore needs no check of its own because nothing changes on disk
// then. Attachments on a soft-deleted ITEM already count: their rows keep
// deleted_at NULL (see WorkspaceAttachments).
func (s *Store) accountStorageUsageQ(q Queryer, ownerID string) (int64, error) {
var total sql.NullInt64
err := q.QueryRow(s.q(`
SELECT COALESCE(SUM(a.size_bytes), 0)
FROM attachments a
JOIN workspaces w ON w.id = a.workspace_id
WHERE w.owner_id = ? AND a.deleted_at IS NULL
`), ownerID).Scan(&total)
if err != nil {
return 0, fmt.Errorf("account storage usage: %w", err)
}
return total.Int64, nil
}
func (s *Store) workspaceStorageUsageQ(q Queryer, workspaceID string) (int64, error) {
var total sql.NullInt64
err := q.QueryRow(s.q(`
SELECT COALESCE(SUM(size_bytes), 0)
FROM attachments
WHERE workspace_id = ? AND deleted_at IS NULL
`), workspaceID).Scan(&total)
if err != nil {
return 0, fmt.Errorf("workspace storage usage: %w", err)
}
return total.Int64, nil
}
// WorkspaceStorageLimit returns the effective storage limit (bytes) for
// the workspace owner's plan, or -1 if the plan is unlimited. Thin
// wrapper around WorkspaceStorageInfo for the upload-time quota check
// path that doesn't care about used bytes / plan / override metadata.
//
// Resolution matches the CheckLimit three-tier order:
// 1. Per-user override (user.PlanOverrides JSON, key "storage_bytes")
// 2. Platform setting (plan_limits_<plan>_storage_bytes)
// 3. Hardcoded fallback (DefaultFreeLimits / DefaultProLimits)
//
// CheckLimit itself can't be reused here because its featureCount path
// only knows about row-counted features (items, members, webhooks, …) —
// storage_bytes is byte-counted via WorkspaceStorageUsage.
func (s *Store) WorkspaceStorageLimit(workspaceID string) (int64, error) {
info, err := s.WorkspaceStorageInfo(workspaceID)
if err != nil {
return 0, err
}
return info.LimitBytes, nil
}
// WorkspaceStorageInfo returns the consolidated quota summary for a
// workspace (used + limit + plan + override flag) in a single call.
// Used by the storage usage API (TASK-881) and the Settings → Storage
// page (TASK-882) so the UI can render the usage bar and the
// "(override)" badge from one fetch.
//
// Limit resolution mirrors WorkspaceStorageLimit. The self-hosted plan
// returns -1 (unlimited); free and pro resolve through resolveLimitQ, so pro
// gets DefaultProLimits.StorageBytes unless a platform setting or an override
// says otherwise (TASK-3619: pro was unlimited before). Workspaces with no
// owner (fresh install, deleted owner) also resolve to unlimited rather than
// blocking uploads.
func (s *Store) WorkspaceStorageInfo(workspaceID string) (*WorkspaceStorageInfo, error) {
info, _, err := s.workspaceStorageInfoQ(s.db, workspaceID)
return info, err
}
// workspaceStorageInfoQ is WorkspaceStorageInfo on a caller-supplied executor,
// so the limited insert reads usage and limit on its own transaction, after
// its lock (BUG-2409). It also returns the owner whose plan set the limit
// (nil when there is none), whom a refusal names as the payer (TASK-3601).
func (s *Store) workspaceStorageInfoQ(q Queryer, workspaceID string) (*WorkspaceStorageInfo, *models.User, error) {
used, err := s.workspaceStorageUsageQ(q, workspaceID)
if err != nil {
return nil, nil, err
}
info := &WorkspaceStorageInfo{
UsedBytes: used,
LimitBytes: -1, // default: no enforceable limit (no owner / self-hosted)
Scope: StorageScopeWorkspace,
}
var ownerID sql.NullString
err = q.QueryRow(s.q(`SELECT owner_id FROM workspaces WHERE id = ?`), workspaceID).Scan(&ownerID)
if err == sql.ErrNoRows {
return info, nil, nil
}
if err != nil {
return nil, nil, fmt.Errorf("workspace storage info: get owner: %w", err)
}
// Workspaces created on a fresh install (no users yet) and legacy
// workspaces from before owner_id was added have no owner. In both
// cases there is no plan to consult — treat as unlimited rather
// than rejecting uploads.
if !ownerID.Valid || ownerID.String == "" {
return info, nil, nil
}
user, err := s.GetUserQ(q, ownerID.String)
if err != nil {
return nil, nil, fmt.Errorf("workspace storage info: get user: %w", err)
}
if user == nil {
// Owner row was deleted but the workspace still references it.
return info, nil, nil
}
plan := user.EffectivePlan(time.Now()) // expiry enforced (BUG-3356)
info.Plan = plan
// Detect whether a per-user storage_bytes override is configured.
// We surface this as a flag for the UI even when the plan is
// self-hosted (where the override has no effect on the effective
// limit) so admins can see the configured value.
if user.PlanOverrides != "" {
var overrides map[string]int
if err := json.Unmarshal([]byte(user.PlanOverrides), &overrides); err == nil {
if _, ok := overrides["storage_bytes"]; ok {
info.OverrideActive = true
}
}
}
// Self-hosted has no enforced limit.
if plan == "self-hosted" {
return info, user, nil
}
// resolveLimitQ returns int, which is 64-bit on every supported host
// (32-bit production hosts are not a supported deployment target).
info.LimitBytes = int64(s.resolveLimitQ(q, plan, "storage_bytes", user.PlanOverrides))
if storagePooledPerAccount(plan) {
info.Scope = StorageScopeAccount
if info.UsedBytes, err = s.accountStorageUsageQ(q, user.ID); err != nil {
return nil, nil, err
}
}
return info, user, nil
}
// errStorageOwnerMoved: the workspace's owner changed between the owner read
// that chose the storage lock and the check under it. Every caller
// (CreateAttachmentForLiveItem, a limited CreateAttachment, the cross-workspace
// copy) retries in a fresh transaction.
var errStorageOwnerMoved = errors.New("storage limit: workspace owner changed during the check")
// ErrStorageOwnerMoved is errStorageOwnerMoved for a caller outside this
// package that drives its own transaction (the app upload, BUG-3653): retry
// in a fresh one.
var ErrStorageOwnerMoved = errStorageOwnerMoved
// testHookAfterStorageOwnerRead, when set by a test, runs between the owner
// read that chooses the storage lock and the lock itself: the window an
// ownership change must not slip through. It gets the transaction because on
// SQLite nothing else can write while it is open. Nil in production.
var testHookAfterStorageOwnerRead func(tx *sql.Tx, workspaceID string)
// testHookCopyAfterStorageCheck, when set by a test, runs in a cross-workspace
// copy after its storage check passed and before it inserts anything, with
// the storage lock held: the window a concurrent upload must wait out
// (TASK-3779). Nil in production.
var testHookCopyAfterStorageCheck func()
// enforceStorageLimitTx is the authoritative storage check (TASK-3619): it
// refuses an insert of adding bytes that would take the workspace past its
// owner's storage_bytes limit. Usage is what WorkspaceStorageInfo sums: the
// workspace, or the owner's whole account where the plan pools it (Pro).
// Being over the limit only stops ADDING bytes: nothing is deleted or
// hidden, and deleting files brings it back under.
//
// It takes acquirePlanLimitLock for storage_bytes first, keyed on the
// workspace's OWNER (the workspace itself when it has none), so two uploads
// into any of one account's workspaces count one after the other, which a
// pooled limit needs (on SQLite, BEGIN IMMEDIATE already serialises them).
// The owner_id read before it is a plain read and takes no lock. The caller
// must take this before any other lock in the transaction, as that lock's
// ordering rule requires. Reading the limit and the usage on tx, after the
// lock, is what makes the check hold under concurrency; a check on the pool
// would not.
//
// An upload that lands exactly on the limit is allowed: the limit is how many
// bytes the workspace may hold.
func (s *Store) enforceStorageLimitTx(tx *sql.Tx, workspaceID string, adding int64) error {
lock, err := s.lockStorageLimitTx(tx, workspaceID)
if err != nil {
return err
}
return s.checkStorageLimitTx(tx, lock, adding)
}
// storageLimitLock is what lockStorageLimitTx locked, for the
// checkStorageLimitTx that follows it in the same transaction.
type storageLimitLock struct {
workspaceID string
ownerID sql.NullString
key string
}
// lockStorageLimitTx is the first half of enforceStorageLimitTx: it reads the
// workspace's owner and takes the storage_bytes plan-limit lock on it. A
// caller that only learns how many bytes it adds after taking other locks
// (the cross-workspace copy, BUG-3653) takes this first, as the lock's
// ordering rule requires, and checks with checkStorageLimitTx once it knows.
func (s *Store) lockStorageLimitTx(tx *sql.Tx, workspaceID string) (storageLimitLock, error) {
lock := storageLimitLock{workspaceID: workspaceID, key: workspaceID}
switch err := tx.QueryRow(s.q(`SELECT owner_id FROM workspaces WHERE id = ?`), workspaceID).Scan(&lock.ownerID); {
case err == sql.ErrNoRows:
case err != nil:
return lock, fmt.Errorf("storage limit: get owner: %w", err)
case lock.ownerID.Valid && lock.ownerID.String != "":
lock.key = lock.ownerID.String
}
if testHookAfterStorageOwnerRead != nil {
testHookAfterStorageOwnerRead(tx, workspaceID)
}
if err := s.acquirePlanLimitLock(tx, lock.key, "storage_bytes"); err != nil {
return lock, err
}
return lock, nil
}
// checkStorageLimitTx is the second half of enforceStorageLimitTx: under the
// lock, it refuses adding bytes that would take the workspace past its
// owner's limit. It returns errStorageOwnerMoved when the owner changed
// since the lock was chosen; the caller retries in a fresh transaction.
func (s *Store) checkStorageLimitTx(tx *sql.Tx, lock storageLimitLock, adding int64) error {
workspaceID, ownerID, lockKey := lock.workspaceID, lock.ownerID, lock.key
info, owner, err := s.workspaceStorageInfoQ(tx, workspaceID)
if err != nil {
return err
}
// The pool checked must be the one locked (codex r3): if the owner moved
// since the read that chose the lock, the check would hold the old
// owner's lock while counting the new owner's pool, and two uploads could
// both pass it. A move after this point is an ownership transfer, which
// carries its bytes unchecked by ruling.
if ownerID.Valid && ownerID.String != "" {
if owner == nil || owner.ID != lockKey {
return errStorageOwnerMoved
}
} else if owner != nil {
return errStorageOwnerMoved
}
if info.LimitBytes < 0 || info.UsedBytes+adding <= info.LimitBytes {
return nil
}
res := LimitResult{
Allowed: false,
Feature: "storage_bytes",
Limit: int(info.LimitBytes),
Current: int(info.UsedBytes),
Plan: info.Plan,
Requested: int(adding),
Scope: info.Scope,
}
if owner != nil {
res.PayerID, res.PayerName = owner.ID, owner.Name
}
return &PlanLimitError{Result: res}
}