4 Commits

Author	SHA1	Message	Date
xarmian	756d91acad	fix(ci): gofmt + bump race-detector timeout to 30m (#299) ... CI on main has been failing since the PLAN-866 attachment work landed. Two independent issues: 1. gofmt failures (golangci-lint) — seven files in the attachments path had trailing-comment alignment that gofmt wanted nudged a column. Pure whitespace; ran `gofmt -w` across the affected files. golangci-lint's gofmt linter caught it on every PR / push since TASK-870 but we hadn't been watching those signals. Files cleaned: internal/attachments/{fs_store_test,mime, mime_test,processor_test}.go, internal/server/{ handlers_attachments_download_test,handlers_attachments_transform, render/attachments_test}.go. Local guard: `gofmt -l ./...` now exits clean. 2. Race-detector tests timed out at 20m on the GitHub-hosted runner. Two contributors: - PostgreSQL adds latency on every CREATE/DROP plus on the bcrypt hash inside auth/bootstrap (~3s per call under -race on the runner). Tests that bootstrap a fresh user (e.g. TestSessionIPChange_*) pay the full cost each time. - The PLAN-866 image-processing tests (thumbnail derivation, rotate / crop transform) added ~2-3 minutes of decode/encode work on top of the existing suite. The previous "20m gives margin without papering over a hang" comment was right at the time it was written; we now genuinely need more headroom. Bumped to 30m on both the SQLite and PostgreSQL race steps. Genuine deadlocks would still trip this and produce the goroutine-dump panic — we just stop confusing "slow but progressing" with "permanently hung". Reference points before / after: - TASK-875 main run #294: Go (PostgreSQL) finished in 17m48s ✓ - TASK-880 main run #298: Go (PostgreSQL) hit 20m timeout ✗ - Local: my new tests under -race add ~63s on a developer laptop (TestThumbnails + TestTransform + TestProcessor combined). Verification: go test ./... — pass go vet ./... — clean gofmt -l (recursively) — clean	2026-04-29 15:41:12 -04:00
xarmian	02be33902f	feat(attachments): ImageProcessor interface + pure-Go impl + thumbnail pipeline (TASK-878) (#295) ... * feat(attachments): ImageProcessor interface + pure-Go impl + thumbnail pipeline (TASK-878) Adds the abstraction Phase 1 needs to derive thumbnail variants on upload, with a pure-Go default implementation that keeps Pad's single-binary distribution intact (no cgo). The libvips-tagged build (Phase 2 / Pad Cloud Docker) will replace processor_purego.go with a vips-backed implementation behind the same Processor interface — see DOC-865. internal/attachments/processor.go: Processor interface — Decode(io.Reader)→(image.Image, format), Resize(img, maxLong), Rotate(img, deg), Crop(img, rect), Encode(img, format, w), Capabilities(). Capabilities struct (image_formats, can_transcode, max_pixels) surfaces what the editor needs to gate per-format rotate/crop UI on (TASK-879/880). ErrUnsupportedFormat + ErrImageTooLarge are separate sentinels so callers can distinguish "format not supported" from "image dimensions too big". internal/attachments/processor_purego.go (//go:build !libvips): Uses github.com/disintegration/imaging plus the stdlib decoders. Supports PNG/JPEG/GIF/BMP/TIFF for all ops. WebP/AVIF/HEIC reach Decode and bounce out via ErrUnsupportedFormat — uploads still succeed (the MIME allowlist is the upload gate), but thumbnails skip and the editor disables rotate/crop UI per Capabilities. Memory ceiling: Decode peeks via image.DecodeConfig (header only) before allocating any pixel buffer and rejects images whose width*height exceeds MaxPixelsDefault (8000² = 64MP). At 4 bytes per pixel that caps the decode buffer at ~256 MiB and prevents an attacker uploading a forged 100kx100k claim from OOMing the server. The forged-CRC test exercises this gate. internal/server/handlers_attachments_thumbnails.go: deriveThumbnails(parentID) runs in goAsync after every image upload. Generates thumb-sm (256px long edge) + thumb-md (1024px), each as its own attachments row with parent_id pointing at the original. Server.Stop() drains the goroutine before SQLite closes, so tests can assert post-conditions deterministically. Skip cases: parent deleted (race), source format not supported (logged at debug), source already smaller than the variant's bound, variant already exists (idempotent reruns). Variants count toward workspace storage usage — DOC-865 is explicit about this and TestThumbnails_CountsTowardWorkspaceUsage proves it. Output format policy: PNG inputs stay PNG to preserve transparency; everything else encodes as JPEG q=85. internal/server/handlers_capabilities.go: GET /api/v1/server/capabilities returns the Processor's static capability profile under {image: {...}}. Public route — the editor needs it before login (e.g. shared-item preview surfaces). Reports an empty image-formats list when no processor is wired, signalling the editor to disable rotate/crop UI rather than 500-ing the editor mount. cmd/pad/main.go: wires SetImageProcessor(NewProcessor()) alongside SetAttachments at startup; logs the supported formats so operators know whether they're on the pure-Go or libvips build. Tests: - processor_test.go: 12 unit tests covering capability profile, decode round-trip for PNG/JPEG/GIF, rejection of unsupported formats and oversized images (forged-CRC PNG), resize aspect preservation + pass-through for already-small inputs, rotate multiples-of-90 + negative + 360-modulo handling, crop with bounds clipping + empty-intersection rejection, encode round- trip for PNG/JPEG, ThumbnailFormat/Mime/Ext policy. - handlers_attachments_thumbnails_test.go: 5 integration tests covering thumb-sm + thumb-md generation on PNG/JPEG uploads, skip-when-source-already-small, ?variant=thumb-md serving via the existing GET handler, workspace usage accounting. - handlers_capabilities tests cover the happy path + the no-processor degraded path. Parent: PLAN-866. Closes the thumbnail-fallback gap that TASK-874 / TASK-876 left open (thumb-md URLs were falling back to original because no thumbnails existed). Unblocks TASK-879 (rotation tool) and TASK-880 (crop tool) — both will reuse Processor.Rotate / Processor.Crop with the same Capabilities-driven UI gating. * fix(attachments): make /server/capabilities public per Codex review (round 1) Codex flagged that GET /api/v1/server/capabilities was registered inside the auth-gated API group but missing from isPublicAPIPath, so once any user existed the editor's pre-login fetch would 401 — contradicting the route's "public" register-time intent and breaking the share-preview surface. Fix: add the path to isPublicAPIPath. The handler is read-only, returns a static profile, and has no per-user state, so making it public has no security implication. Added TestServerCapabilities_PublicAfterBootstrap as a regression guard: it bootstraps an admin (so RequireAuth is active) and then fetches the endpoint with no auth cookie, asserting 200. * fix(attachments): make -tags libvips compile per Codex review (round 2) Codex flagged that build tag !libvips on processor_purego.go meant NewProcessor + the Thumbnail* helpers were absent under \`go build -tags libvips\`, so cmd/pad/main.go and the thumbnail handler — which call them unconditionally — broke that build. Two minimal fixes preserving the documented Phase 2 split: 1. Move ThumbnailFormat / ThumbnailMime / ThumbnailExt out of the tagged file and into processor.go (untagged). They're pure format-name policy, not implementation specifics, so both backends share the same definitions. 2. Add processor_libvips.go (//go:build libvips) with a stub NewProcessor that panics at runtime with a clear "Phase 2 hasn't shipped libvips yet" message. The libvips build now compiles; anyone actually instantiating the processor under that tag gets a loud failure rather than a silent degradation. Phase 2 will replace the body with the real govips-v2-backed implementation. Verified: \`go build ./...\` and \`go build -tags libvips ./...\` both clean. Existing tests still pass on the default tag. * fix(attachments): make tests compile under -tags libvips per Codex review (round 3) Codex flagged that running \`go test -tags libvips ./internal/attachments\` or \`./internal/server\` panicked through the libvips NewProcessor stub: processor_test.go and the thumbnail/capability server tests all called NewProcessor() unconditionally, even though the libvips build's stub is intentionally panicking until Phase 2 ships the real implementation. Three minimal fixes: 1. Tag processor_test.go !libvips. It tests the pure-Go implementation specifically — there's no value in running it under libvips, and the stub processor would explode the moment NewProcessor() ran. 2. Tag handlers_attachments_thumbnails_test.go !libvips. Same reasoning — these integration tests assert thumbnail derivation against a working processor. 3. Split testServerWithAttachments's processor wiring into two build-tagged helper files: * testimageprocessor_purego_test.go (//go:build !libvips) wires the real pure-Go processor. * testimageprocessor_libvips_test.go (//go:build libvips) is a no-op so the rest of the server test surface (uploads, downloads, auth, etc.) compiles + runs cleanly under -tags libvips. Verification: go build ./... — OK go build -tags libvips ./... — OK go test ./internal/attachments ./internal/server (default) — pass go test -tags libvips ./internal/server -run "TestUpload|TestDownload" — pass Phase 2 will introduce a real libvips test backend and drop the !libvips tags on the thumbnail tests. * fix(attachments): libvips binary boots cleanly per Codex review (round 4) Codex flagged that the libvips build still crashed at \`pad serve\` startup: cmd/pad/main.go calls attachments.NewProcessor() unconditionally, and the libvips stub was panicking — so any operator who built with -tags libvips today (Phase 2 isn't shipped yet) lost the entire server, not just image processing. Two minimal changes: 1. processor_libvips.go: stop panicking. Return nil + slog.Warn instead. Every call site already nil-checks the processor (the upload handler skips thumbnail derivation, the capabilities endpoint reports a degraded empty formats list), so the libvips-tagged binary now has the same runtime profile as a self-host build that opted out of image processing entirely — uploads succeed, originals display, only derived transformations are unavailable. The slog.Warn keeps the "this build doesn't have it yet" signal loud. 2. cmd/pad/main.go: skip srv.SetImageProcessor when NewProcessor returns nil, and log a "not wired" message in that branch. Distinguishes the wired vs. unwired states cleanly in the boot log. Phase 2 will replace processor_libvips.go's body with the real govips-v2-backed implementation; main.go's wiring is already shape- correct for that transition. Verification: go build ./... — OK go build -tags libvips ./... — OK go test ./... — pass (74s server tests included) go test -tags libvips ./internal/server -run "TestUpload|TestDownload|TestServerCapabilities_Public" — pass	2026-04-29 14:35:49 -04:00
xarmian	48b9e18d34	feat(attachments): upload API with MIME sniff + dedupe + quota tracking (TASK-871) (#288) ... * feat(attachments): upload API with MIME sniff + dedupe + quota tracking (TASK-871) Wires the upload endpoint that turns a multipart POST into an attachments row plus a stored blob. Auth-gated (editor+), per-file size cap, hash-streaming, MIME allowlist with extension blocklist, fire-and-forget quota warning. POST /api/v1/workspaces/{slug}/attachments Multipart "file" field. Optional ?item_id=… or form item_id to associate at upload time. Returns {id, url, mime, size, width?, height?, filename, category, render_mode}. Errors: 400 bad multipart, 400 empty file, 401 unauthorized, 403 insufficient role, 413 over per-file cap, 415 MIME or extension rejection, 503 attachments not configured. internal/attachments/mime.go MIMEEntry + RenderMode + Category typed allowlist mirroring DOC-865. Default-deny. SniffMIME wraps http.DetectContentType. ValidateUpload cross-checks the sniff result against the filename extension and: (a) rejects when the extension maps to a *blocked* MIME — covers .svg (sniffs as text/xml; .svg ext makes the browser run embedded <script>) and .exe family (sniffs vary; extension is unambiguous); (b) rejects when the extension maps to an allowed MIME but the sniff's category disagrees — the "exe pretending to be png" case. Tests cover normalize/lookup/sniff plus happy path, exe-as-png, extension mismatch, SVG, .exe-by-extension-alone, text/plain accept, HTML force-download. internal/store/attachments.go CreateAttachment / GetAttachment / WorkspaceStorageUsage. Pointer scan for nullables; SUM(size_bytes) excludes soft-deleted rows but includes derived blobs (thumbnails are real bytes on disk). internal/server/handlers_attachments.go Body capped via http.MaxBytesReader BEFORE ParseMultipartForm spools any of it. Streams "file" part into an os.CreateTemp file, sha256ing in one io.MultiWriter pass — multi-GB POST never reaches RAM. Sniff on first 512 bytes; image dimension probe via stdlib image.DecodeConfig (PNG/JPEG/GIF). WebP/AVIF/HEIC accepted but width/height nil — matches the "pure-Go gracefully degrades" decision in DOC-865. Calls AttachmentStore.Put (which hash-verifies via the dedup fast path) and inserts the row. Quota check (CheckLimit + WorkspaceStorageUsage) runs in a goroutine — Phase 1 logs only; Phase 2 will enforce. Anonymous uploads on a fresh install (RequireWorkspaceAccess grants implicit owner without a current user) get uploaded_by="system". internal/server/server.go Server.attachments + attachmentMaxBytes fields and SetAttachments setter. Route POST /workspaces/{slug}/attachments wired inside the authenticated workspace block. cmd/pad/main.go Boot wiring: NewFSStore(<DataDir>/attachments) → Registry registered under "fs" → SetAttachments. PAD_ATTACHMENT_MAX_BYTES env override for the per-file cap. Tests internal/server/handlers_attachments_test.go covers: happy path PNG (1x1, dimensions resolve to 1×1) exe bytes with .png filename → 415 PNG bytes with .pdf filename → 415 (extension mismatch) empty body → 400 missing file part → 400 over the size cap → 413 same content uploaded twice → two rows, same content_hash + storage_key, WorkspaceStorageUsage = 2 × bytes (dedupe is at the blob layer, not the row layer) 8 concurrent uploads of identical bytes → all 201, no corruption no registry wired → 503 Verification go build ./... — clean go vet ./... — clean go test ./... — all packages pass make install — server restarts on the new binary Parent: PLAN-866. * fix(attachments): three Codex round-1 findings — drop premature url, accept Office docs, real quota probe 1. Upload response no longer returns "url". TASK-872 wires GET so any URL we return today is a 404 — pulling it out keeps clients from baking in the broken endpoint. 2. Office Open XML docs (.docx/.xlsx/.pptx) and OpenDocument formats (.odt/.ods/.odp) are zipped XML — http.DetectContentType correctly sniffs them as application/zip. Previously the validator's extension-vs-sniff category check rejected them as "mime_extension_mismatch" (archive vs document). Now: when the sniffed type is exactly application/zip and the extension maps to a document MIME, trust the extension and route to the document entry. Plain .zip with the same bytes still routes to archive. Test covers all six office/odf extensions plus the plain-zip case. 3. CheckLimit("storage_bytes") returned "unknown workspace feature" because featureCount only knows row-counted features (items, members, webhooks). The warning path silently dropped every probe. Added Store.WorkspaceStorageLimit which does the same three-tier resolution (user override → platform setting → hardcoded fallback) but returns the limit only — usage is computed separately via the existing WorkspaceStorageUsage. Self-hosted/pro plans return -1 (unlimited). Workspaces without an owner_id (fresh installs and legacy rows) also return -1, so a fresh-install upload no longer logs "owner not found". Switched maybeWarnStorageQuota to use WorkspaceStorageLimit + WorkspaceStorageUsage directly. Now also spawned via Server.goAsync so Stop() drains it (BUG-842 hygiene). Tests - TestValidateUpload_AcceptsOfficeOpenXMLAsZipBytes covers all six extensions + plain .zip - TestUpload_QuotaCheckResolves regression-tests finding 3: both storage helpers return non-error after a real upload - TestUpload_HappyPathPNG asserts the response no longer carries url Verification go build ./... — clean go vet ./... — clean go test ./... — all packages pass * fix(attachments): trim trailing blank line at EOF in mime.go per Codex review (round 2) Round 2 LOW: git diff --check flagged a "new blank line at EOF" on internal/attachments/mime.go. Cosmetic but addressed because the ship-tasks workflow requires zero findings (HIGH/MEDIUM/LOW alike) — leaving LOWs unfixed compounds across PRs and prevents the loop from ever converging clean on later work. * fix(attachments): alias stdlib MIME-sniff quirks per Codex review (round 3) http.DetectContentType returns names that don't match modern IANA conventions for two formats on the allowlist: audio/wave → audio/wav (.wav uploads) application/x-gzip → application/gzip (.gz uploads) Without aliasing, valid uploads of either format hit "mime_not_allowed" because the allowlist uses canonical names. Added a sniffAliases map applied inside SniffMIME so allowlist lookups always see the canonical form. Allowlist stays single-sourced; the fix is one map entry per quirk we discover. Tests: - TestSniffMIME_AliasesStdlibQuirks pins both aliases at the sniff layer - TestValidateUpload_AcceptsWAV / TestValidateUpload_AcceptsGzip verify the end-to-end accept path with real WAV (RIFF/WAVE) and gzip headers	2026-04-29 12:19:06 -04:00
xarmian	de4d28d576	feat(attachments): AttachmentStore interface + FSStore (TASK-870) (#287) ... * feat(attachments): AttachmentStore interface + FSStore (TASK-870) Introduces the storage backend abstraction described in DOC-865 and ships its first concrete implementation. No call sites yet — TASK-871 (upload API) wires it in. internal/attachments/store.go AttachmentStore interface (Put/Get/Stat/Delete) and ErrNotFound sentinel. Put is documented as idempotent — concurrent Puts of the same hash converge — and required to verify that the streamed bytes actually hash to the supplied value. internal/attachments/registry.go Registry routes "<prefix>:<rest>" keys to the store registered for that prefix (Phase 1 = "fs"; Phase 2 will register "s3" alongside). Convenience Get/Stat/Delete helpers resolve + forward in one call so callers don't have to spell out the two-step pattern everywhere. Register panics if the prefix contains ':' since that would make the store unreachable. internal/attachments/fs_store.go FSStore writes to <baseDir>/<aa>/<bb>/<full-hash> with the first 4 hex chars sharding the directory tree two levels deep. Atomic writes: stream + hash to a randomized .tmp in the destination dir, fsync, then intra-directory rename. The streaming sha256 is verified against the supplied hash before the rename, so a mismatch never leaves a visible file. Idempotent fast path: if the canonical file already exists Put short-circuits (and drains the reader so callers don't get a stuck stream). Get returns wrapped ErrNotFound on missing keys; Delete on a missing key is a no-op (matches what the orphan GC needs). Tests cover put/get/stat/delete, hash mismatch, invalid hash format, idempotency, 16-goroutine concurrent Put of the same hash converging to one on-disk file with no orphan tmp files, registry routing, forward-error semantics, and the prefix-with-colon panic. Parent: PLAN-866. * fix(attachments): validate hash on every FSStore key + verify on fast path per Codex review (round 1) Round 1 raised two issues — both real, both fixed. 1. Path traversal in Get/Stat/Delete. extractHash only checked that the key began with "fs:" and the suffix was non-empty before passing it to pathFor(), which used the suffix as a path component. A key like "fs:../../etc/passwd" would escape baseDir for reads/stats/deletes. Fix: extractHash now requires the canonical 64-char lowercase-hex sha256 form via validHash. Same gate that Put already used; now it covers every public method. 2. Idempotent Put fast path skipped hash verification. If the canonical target file already existed, Put returned the key without checking that the supplied reader's bytes hashed to the supplied hash — violating the AttachmentStore.Put contract that implementations MUST verify on every call. A buggy upload path could associate the wrong bytes with an existing hash and silently succeed. Fix: stream r through a hasher when the target exists (no disk I/O), compare against the supplied hash, and reject on mismatch. Also dropped the dead "_short" branch in pathFor — every caller now goes through validHash. Tests added: - TestFSStore_GetStatDeleteRejectBadKeys covers empty/wrong-prefix/empty- hash/non-hex/wrong-length/path-traversal/path-separator/uppercase keys across all three read methods. - TestFSStore_PutFastPathStillVerifiesHash confirms the contract holds on the fast path: a second Put that lies about the hash is rejected with no corruption of the existing file.	2026-04-29 11:44:58 -04:00