mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-27 06:47:07 +00:00
08dfbdb318
* fix(server): rowless-blob GC sweep — reclaim blobs no attachments row references (BUG-2406) Every attachment write path calls AttachmentStore.Put BEFORE inserting the attachments row, so a failure (or crash) between the two leaves a blob on disk that nothing references — and the row-driven orphan sweep, which walks Store.OrphanedAttachments, can never see it. Disk that is never returned; the upload handler's failure comment even claimed the GC would reclaim it. Fix: a rowless-blob sweep that runs after the row sweep on the same GC tick. attachments.Lister is a new OPTIONAL backend capability (ListBlobs → key/hash/size/mtime); FSStore implements it via one WalkDir of the sharded tree with a base-name validHash gate (excludes Put's dot-prefixed temp files and anything the store didn't write). Backends without the capability are skipped with a once-per-process notice. Candidate = blob whose content hash has ZERO rows in ANY state (soft-deleted rows still own their bytes under the row sweep's row-before-bytes claim protocol, BUG-2415) AND whose mtime predates the same operator-configured GC grace the row sweep uses — a young rowless blob is just an upload whose insert hasn't happened yet. Delete-time guards run under inFlightHashesMu: the in-flight fence plus a single-hash row RE-CHECK that closes the subtraction-to-delete TOCTOU (the writer that marked, inserted, and released entirely inside the gap). Cost: O(blobs) per tick, 24h cadence, never on a request path. Also retro-reclaims blobs stranded by past row-sweep delete failures. The wrong claim in handleUploadAttachment's failure path is corrected to point at this sweep. Tests: FSStore.ListBlobs impostor coverage; five sweep legs (aged-rowless reclaimed with a row-sweep-can't-see-it counterfactual, young kept, live/soft-deleted-row kept, in-flight kept then reclaimed after release, hook-injected delete-time row kept) — mutation-verified: removing the re-check, the age gate, or the in-flight fence each fails its leg; the store-level subtraction contract is pinned separately. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * docs(store): state the any-row rule's real rationale per Codex review (round 1) Codex flagged the thumbnail refusal-cleanup's grace-window protection as inconsistent with the sweep comment's claim that deleting bytes under any existing row violates the claim protocol. The cleanup (and the row sweep itself) deliberately end a row's hash-protection when its own grace expires — CountProtectingAttachmentsForHash documents exactly that, and the row machinery may do it because its claim protocol coordinates row and blob fates within a sweep. The overstatement was mine: the rowless sweep's any-row rule is chosen because it holds no claim on any row and has no such coordination, not because past-grace stranding is forbidden to the machinery that does. Comment corrected; no behavior change on either path. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
95 lines
4.2 KiB
Go
95 lines
4.2 KiB
Go
// Package attachments provides the storage backend abstraction for
|
|
// attachment blobs (images and files uploaded into items). See DOC-865
|
|
// "Attachments — architecture & migration design" for the full design.
|
|
//
|
|
// Storage is content-addressed: every blob is keyed by sha256(content).
|
|
// Identical bytes → identical key → one physical copy. Each AttachmentStore
|
|
// implementation maps a hash to a concrete location (filesystem path, S3
|
|
// object key, …) and namespaces its keys with a backend prefix
|
|
// ("fs:<hash>", "s3:<bucket>/<hash>", …) so a Registry can route a key to
|
|
// the right backend by prefix alone. This decouples item content (which
|
|
// stores opaque "pad-attachment:<uuid>" references) from the backend the
|
|
// blob actually lives in, and enables backend migrations (FS → S3) that
|
|
// touch zero item content.
|
|
package attachments
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"io"
|
|
"time"
|
|
)
|
|
|
|
// AttachmentStore is the backend abstraction every storage implementation
|
|
// must satisfy. Methods are safe for concurrent use.
|
|
type AttachmentStore interface {
|
|
// Put writes the blob from r into the backend, addressable by hash.
|
|
// The implementation MUST verify that the bytes streamed from r hash
|
|
// to the supplied hash and return an error if they do not — this is
|
|
// the integrity guarantee callers rely on.
|
|
//
|
|
// Put is idempotent: writing the same hash twice is a no-op on the
|
|
// second call. Concurrent Puts of the same hash converge — both
|
|
// callers receive the same key and the on-disk content is unchanged.
|
|
//
|
|
// The returned key is the "<backend>:<...>" form that Registry uses to
|
|
// route subsequent Get/Stat/Delete calls back to this store.
|
|
//
|
|
// mime is informational (some backends, e.g. S3, store it as object
|
|
// metadata). The FS backend ignores it; MIME for serving comes from
|
|
// the attachments DB row.
|
|
Put(ctx context.Context, hash, mime string, r io.Reader) (key string, err error)
|
|
|
|
// Get returns a ReadCloser positioned at the start of the blob.
|
|
// Callers MUST close the returned reader. If key is unknown to this
|
|
// store, the returned error wraps ErrNotFound.
|
|
Get(ctx context.Context, key string) (io.ReadCloser, error)
|
|
|
|
// Stat returns the size in bytes of the blob identified by key. If
|
|
// key is unknown, the returned error wraps ErrNotFound.
|
|
Stat(ctx context.Context, key string) (size int64, err error)
|
|
|
|
// Delete removes the blob identified by key. Deleting a missing key
|
|
// is NOT an error — callers (e.g. the orphan GC) treat it as the
|
|
// success case.
|
|
Delete(ctx context.Context, key string) error
|
|
}
|
|
|
|
// ErrNotFound is returned (or wrapped) by Get/Stat when the requested key
|
|
// is not present in the backend. Callers can compare with errors.Is.
|
|
var ErrNotFound = errors.New("attachment not found")
|
|
|
|
// BlobInfo describes one stored blob as enumerated by a Lister.
|
|
type BlobInfo struct {
|
|
// Key is the routable "<backend>:<...>" form, exactly what Put
|
|
// returned when the blob was written — valid input to Get/Stat/Delete.
|
|
Key string
|
|
// Hash is the content sha256 the blob is addressed by.
|
|
Hash string
|
|
// Size in bytes.
|
|
Size int64
|
|
// ModTime is when the blob landed in the backend (filesystem mtime,
|
|
// object-store LastModified). The rowless-blob sweep uses it as the
|
|
// age gate: a blob younger than the GC grace period may simply be a
|
|
// row insert that has not happened yet.
|
|
ModTime time.Time
|
|
}
|
|
|
|
// Lister is an OPTIONAL capability of an AttachmentStore: enumerating
|
|
// every blob it holds. The rowless-blob GC sweep (BUG-2406) needs it to
|
|
// find blobs that no attachments row references — a leak class the
|
|
// row-driven orphan sweep cannot see, produced when anything fails
|
|
// between AttachmentStore.Put and the row insert (or the process dies
|
|
// there). Kept separate from AttachmentStore rather than added to it so
|
|
// a backend without an affordable enumeration (or one not yet wired for
|
|
// it) still satisfies the core interface; the sweep detects the
|
|
// capability by type assertion and skips — with a logged notice — any
|
|
// backend that lacks it.
|
|
//
|
|
// ListBlobs walks the ENTIRE backend: cost is O(blobs) per call, which
|
|
// is why its only caller runs on the orphan-GC cadence (default 24h),
|
|
// never on a request path.
|
|
type Lister interface {
|
|
ListBlobs(ctx context.Context) ([]BlobInfo, error)
|
|
}
|