Files
pad/internal/attachments/store.go
T
xarmian 08dfbdb318 fix(server): rowless-blob GC sweep — reclaim blobs no attachments row references (BUG-2406) (#1137)
* fix(server): rowless-blob GC sweep — reclaim blobs no attachments row references (BUG-2406)

Every attachment write path calls AttachmentStore.Put BEFORE inserting
the attachments row, so a failure (or crash) between the two leaves a
blob on disk that nothing references — and the row-driven orphan sweep,
which walks Store.OrphanedAttachments, can never see it. Disk that is
never returned; the upload handler's failure comment even claimed the
GC would reclaim it.

Fix: a rowless-blob sweep that runs after the row sweep on the same GC
tick. attachments.Lister is a new OPTIONAL backend capability
(ListBlobs → key/hash/size/mtime); FSStore implements it via one
WalkDir of the sharded tree with a base-name validHash gate (excludes
Put's dot-prefixed temp files and anything the store didn't write).
Backends without the capability are skipped with a once-per-process
notice. Candidate = blob whose content hash has ZERO rows in ANY state
(soft-deleted rows still own their bytes under the row sweep's
row-before-bytes claim protocol, BUG-2415) AND whose mtime predates the
same operator-configured GC grace the row sweep uses — a young rowless
blob is just an upload whose insert hasn't happened yet. Delete-time
guards run under inFlightHashesMu: the in-flight fence plus a
single-hash row RE-CHECK that closes the subtraction-to-delete TOCTOU
(the writer that marked, inserted, and released entirely inside the
gap). Cost: O(blobs) per tick, 24h cadence, never on a request path.
Also retro-reclaims blobs stranded by past row-sweep delete failures.

The wrong claim in handleUploadAttachment's failure path is corrected
to point at this sweep.

Tests: FSStore.ListBlobs impostor coverage; five sweep legs
(aged-rowless reclaimed with a row-sweep-can't-see-it counterfactual,
young kept, live/soft-deleted-row kept, in-flight kept then reclaimed
after release, hook-injected delete-time row kept) — mutation-verified:
removing the re-check, the age gate, or the in-flight fence each fails
its leg; the store-level subtraction contract is pinned separately.

Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V

* docs(store): state the any-row rule's real rationale per Codex review (round 1)

Codex flagged the thumbnail refusal-cleanup's grace-window protection as
inconsistent with the sweep comment's claim that deleting bytes under any
existing row violates the claim protocol. The cleanup (and the row sweep
itself) deliberately end a row's hash-protection when its own grace
expires — CountProtectingAttachmentsForHash documents exactly that, and
the row machinery may do it because its claim protocol coordinates row
and blob fates within a sweep. The overstatement was mine: the rowless
sweep's any-row rule is chosen because it holds no claim on any row and
has no such coordination, not because past-grace stranding is forbidden
to the machinery that does. Comment corrected; no behavior change on
either path.

Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
2026-08-17 09:03:44 -04:00

95 lines
4.2 KiB
Go

// Package attachments provides the storage backend abstraction for
// attachment blobs (images and files uploaded into items). See DOC-865
// "Attachments — architecture & migration design" for the full design.
//
// Storage is content-addressed: every blob is keyed by sha256(content).
// Identical bytes → identical key → one physical copy. Each AttachmentStore
// implementation maps a hash to a concrete location (filesystem path, S3
// object key, …) and namespaces its keys with a backend prefix
// ("fs:<hash>", "s3:<bucket>/<hash>", …) so a Registry can route a key to
// the right backend by prefix alone. This decouples item content (which
// stores opaque "pad-attachment:<uuid>" references) from the backend the
// blob actually lives in, and enables backend migrations (FS → S3) that
// touch zero item content.
package attachments
import (
"context"
"errors"
"io"
"time"
)
// AttachmentStore is the backend abstraction every storage implementation
// must satisfy. Methods are safe for concurrent use.
type AttachmentStore interface {
// Put writes the blob from r into the backend, addressable by hash.
// The implementation MUST verify that the bytes streamed from r hash
// to the supplied hash and return an error if they do not — this is
// the integrity guarantee callers rely on.
//
// Put is idempotent: writing the same hash twice is a no-op on the
// second call. Concurrent Puts of the same hash converge — both
// callers receive the same key and the on-disk content is unchanged.
//
// The returned key is the "<backend>:<...>" form that Registry uses to
// route subsequent Get/Stat/Delete calls back to this store.
//
// mime is informational (some backends, e.g. S3, store it as object
// metadata). The FS backend ignores it; MIME for serving comes from
// the attachments DB row.
Put(ctx context.Context, hash, mime string, r io.Reader) (key string, err error)
// Get returns a ReadCloser positioned at the start of the blob.
// Callers MUST close the returned reader. If key is unknown to this
// store, the returned error wraps ErrNotFound.
Get(ctx context.Context, key string) (io.ReadCloser, error)
// Stat returns the size in bytes of the blob identified by key. If
// key is unknown, the returned error wraps ErrNotFound.
Stat(ctx context.Context, key string) (size int64, err error)
// Delete removes the blob identified by key. Deleting a missing key
// is NOT an error — callers (e.g. the orphan GC) treat it as the
// success case.
Delete(ctx context.Context, key string) error
}
// ErrNotFound is returned (or wrapped) by Get/Stat when the requested key
// is not present in the backend. Callers can compare with errors.Is.
var ErrNotFound = errors.New("attachment not found")
// BlobInfo describes one stored blob as enumerated by a Lister.
type BlobInfo struct {
// Key is the routable "<backend>:<...>" form, exactly what Put
// returned when the blob was written — valid input to Get/Stat/Delete.
Key string
// Hash is the content sha256 the blob is addressed by.
Hash string
// Size in bytes.
Size int64
// ModTime is when the blob landed in the backend (filesystem mtime,
// object-store LastModified). The rowless-blob sweep uses it as the
// age gate: a blob younger than the GC grace period may simply be a
// row insert that has not happened yet.
ModTime time.Time
}
// Lister is an OPTIONAL capability of an AttachmentStore: enumerating
// every blob it holds. The rowless-blob GC sweep (BUG-2406) needs it to
// find blobs that no attachments row references — a leak class the
// row-driven orphan sweep cannot see, produced when anything fails
// between AttachmentStore.Put and the row insert (or the process dies
// there). Kept separate from AttachmentStore rather than added to it so
// a backend without an affordable enumeration (or one not yet wired for
// it) still satisfies the core interface; the sweep detects the
// capability by type assertion and skips — with a logged notice — any
// backend that lacks it.
//
// ListBlobs walks the ENTIRE backend: cost is O(blobs) per call, which
// is why its only caller runs on the orphan-GC cadence (default 24h),
// never on a request path.
type Lister interface {
ListBlobs(ctx context.Context) ([]BlobInfo, error)
}