2 Commits

Author	SHA1	Message	Date
xarmian	6a5eb3dee0	fix(attachments): HEIC, HEIF and AVIF are recognised from their bytes, not refused (BUG-2961) (#1291) ... fix(attachments): HEIC, HEIF and AVIF are recognised from their bytes, not refused (BUG-2961) image/heic, image/heif and image/avif have been on the upload allowlist all along and every real file of those types was rejected 415 mime_not_allowed: http.DetectContentType implements the WHATWG mimesniff table, which has no signature for ISO base media file format still images, so they sniffed as application/octet-stream and ValidateUpload's first rule refused them before the extension cross-check was ever consulted. HEIC is the iPhone camera default, so that was every photo shared straight from an Apple device, through the web UI today and the mobile share sheet being built. The fix stays inside rule 1 rather than around it. sniffISOBMFFImage reads the ftyp box and returns the allowlisted MIME, so the sniffed type is still what the allowlist is consulted with; it returns "" for everything else, so it can only add detections. The alternative was to trust the filename for these three, the way ValidateUpload already trusts it for zip-based Office documents, and that was refused in triage because a renamed .heic would then pass on a name alone. Both tests layers pin that: an executable named .heic is still refused, and real HEIC bytes named .jpg are still stored as image/heic. Four decisions, each argued at its site: - Every brand in the box, not just the major one. The measured Apple HEIC carries major heic with compatible mif1 MiPr miaf MiHB heic; the measured Apple AVIF carries major avif with compatible MiPr avif miaf mif1; libheif writes major mif1. A check on bytes 8..12 alone is right for one encoder and wrong for the next. A specific brand wins wherever it appears; mif1 alone means image/heif. - Sequence brands (hevc/hevx/hevm/hevs, avis, msf1) are deliberately absent: they name image/*-sequence types, none of which is on the allowlist, so recognising them would produce a sniff ValidateUpload then refuses with a stranger message. - A container carrying an mp4 brand stays a video and is left to the stdlib. - A size-1 (64-bit largesize) ftyp box is declined, because the brands shift eight bytes and the fixed offsets would be reading the size field. Fixtures are real encoder output with provenance recorded in testdata/README.md (sips on macOS 26 for the Apple pair, libheif 1.20.2 here for the other two); hand-built byte vectors appear only for shapes no available encoder produces. Verified by negative control — with the pre-check disabled all four new acceptance tests fail — and by a nine-mutant matrix in which the major-only scan, the dropped mp4 yield, generic-over-specific preference, the minor version read as a brand, the missing box bound, the removed largesize bail, a dropped brand table entry and an unconsulted sniffer are each killed by a named test. The one survivor is equivalent: an 8..11-byte buffer yields no whole brand either way. Codex found two more refusals this owed, both now guarded and mutation-checked: a declared box size of 2..15 is impossible for an ftyp box, and a buffer that stops inside the 16-byte header is a truncated file — in either case the bytes at offset 8 are not brands of a box that exists, and classifying from them let a caller-controlled payload be stored and served as an image. Also corrects processor.go's claim that "display always works (browsers handle WebP / AVIF / HEIC natively)", which the client's own viewer table contradicts and which this change would have made load-bearing (CONVE-23). What it means for rendering on builds that cannot derive a thumbnail is filed as BUG-2964; the wider class of allowlist entries the sniffer cannot produce is BUG-2963. Claude-Session: https://claude.ai/code/session_01HeChkgZVYb3NTgTcckF5KR	2026-09-08 13:30:42 -04:00
xarmian	02be33902f	feat(attachments): ImageProcessor interface + pure-Go impl + thumbnail pipeline (TASK-878) (#295) ... * feat(attachments): ImageProcessor interface + pure-Go impl + thumbnail pipeline (TASK-878) Adds the abstraction Phase 1 needs to derive thumbnail variants on upload, with a pure-Go default implementation that keeps Pad's single-binary distribution intact (no cgo). The libvips-tagged build (Phase 2 / Pad Cloud Docker) will replace processor_purego.go with a vips-backed implementation behind the same Processor interface — see DOC-865. internal/attachments/processor.go: Processor interface — Decode(io.Reader)→(image.Image, format), Resize(img, maxLong), Rotate(img, deg), Crop(img, rect), Encode(img, format, w), Capabilities(). Capabilities struct (image_formats, can_transcode, max_pixels) surfaces what the editor needs to gate per-format rotate/crop UI on (TASK-879/880). ErrUnsupportedFormat + ErrImageTooLarge are separate sentinels so callers can distinguish "format not supported" from "image dimensions too big". internal/attachments/processor_purego.go (//go:build !libvips): Uses github.com/disintegration/imaging plus the stdlib decoders. Supports PNG/JPEG/GIF/BMP/TIFF for all ops. WebP/AVIF/HEIC reach Decode and bounce out via ErrUnsupportedFormat — uploads still succeed (the MIME allowlist is the upload gate), but thumbnails skip and the editor disables rotate/crop UI per Capabilities. Memory ceiling: Decode peeks via image.DecodeConfig (header only) before allocating any pixel buffer and rejects images whose width*height exceeds MaxPixelsDefault (8000² = 64MP). At 4 bytes per pixel that caps the decode buffer at ~256 MiB and prevents an attacker uploading a forged 100kx100k claim from OOMing the server. The forged-CRC test exercises this gate. internal/server/handlers_attachments_thumbnails.go: deriveThumbnails(parentID) runs in goAsync after every image upload. Generates thumb-sm (256px long edge) + thumb-md (1024px), each as its own attachments row with parent_id pointing at the original. Server.Stop() drains the goroutine before SQLite closes, so tests can assert post-conditions deterministically. Skip cases: parent deleted (race), source format not supported (logged at debug), source already smaller than the variant's bound, variant already exists (idempotent reruns). Variants count toward workspace storage usage — DOC-865 is explicit about this and TestThumbnails_CountsTowardWorkspaceUsage proves it. Output format policy: PNG inputs stay PNG to preserve transparency; everything else encodes as JPEG q=85. internal/server/handlers_capabilities.go: GET /api/v1/server/capabilities returns the Processor's static capability profile under {image: {...}}. Public route — the editor needs it before login (e.g. shared-item preview surfaces). Reports an empty image-formats list when no processor is wired, signalling the editor to disable rotate/crop UI rather than 500-ing the editor mount. cmd/pad/main.go: wires SetImageProcessor(NewProcessor()) alongside SetAttachments at startup; logs the supported formats so operators know whether they're on the pure-Go or libvips build. Tests: - processor_test.go: 12 unit tests covering capability profile, decode round-trip for PNG/JPEG/GIF, rejection of unsupported formats and oversized images (forged-CRC PNG), resize aspect preservation + pass-through for already-small inputs, rotate multiples-of-90 + negative + 360-modulo handling, crop with bounds clipping + empty-intersection rejection, encode round- trip for PNG/JPEG, ThumbnailFormat/Mime/Ext policy. - handlers_attachments_thumbnails_test.go: 5 integration tests covering thumb-sm + thumb-md generation on PNG/JPEG uploads, skip-when-source-already-small, ?variant=thumb-md serving via the existing GET handler, workspace usage accounting. - handlers_capabilities tests cover the happy path + the no-processor degraded path. Parent: PLAN-866. Closes the thumbnail-fallback gap that TASK-874 / TASK-876 left open (thumb-md URLs were falling back to original because no thumbnails existed). Unblocks TASK-879 (rotation tool) and TASK-880 (crop tool) — both will reuse Processor.Rotate / Processor.Crop with the same Capabilities-driven UI gating. * fix(attachments): make /server/capabilities public per Codex review (round 1) Codex flagged that GET /api/v1/server/capabilities was registered inside the auth-gated API group but missing from isPublicAPIPath, so once any user existed the editor's pre-login fetch would 401 — contradicting the route's "public" register-time intent and breaking the share-preview surface. Fix: add the path to isPublicAPIPath. The handler is read-only, returns a static profile, and has no per-user state, so making it public has no security implication. Added TestServerCapabilities_PublicAfterBootstrap as a regression guard: it bootstraps an admin (so RequireAuth is active) and then fetches the endpoint with no auth cookie, asserting 200. * fix(attachments): make -tags libvips compile per Codex review (round 2) Codex flagged that build tag !libvips on processor_purego.go meant NewProcessor + the Thumbnail* helpers were absent under \`go build -tags libvips\`, so cmd/pad/main.go and the thumbnail handler — which call them unconditionally — broke that build. Two minimal fixes preserving the documented Phase 2 split: 1. Move ThumbnailFormat / ThumbnailMime / ThumbnailExt out of the tagged file and into processor.go (untagged). They're pure format-name policy, not implementation specifics, so both backends share the same definitions. 2. Add processor_libvips.go (//go:build libvips) with a stub NewProcessor that panics at runtime with a clear "Phase 2 hasn't shipped libvips yet" message. The libvips build now compiles; anyone actually instantiating the processor under that tag gets a loud failure rather than a silent degradation. Phase 2 will replace the body with the real govips-v2-backed implementation. Verified: \`go build ./...\` and \`go build -tags libvips ./...\` both clean. Existing tests still pass on the default tag. * fix(attachments): make tests compile under -tags libvips per Codex review (round 3) Codex flagged that running \`go test -tags libvips ./internal/attachments\` or \`./internal/server\` panicked through the libvips NewProcessor stub: processor_test.go and the thumbnail/capability server tests all called NewProcessor() unconditionally, even though the libvips build's stub is intentionally panicking until Phase 2 ships the real implementation. Three minimal fixes: 1. Tag processor_test.go !libvips. It tests the pure-Go implementation specifically — there's no value in running it under libvips, and the stub processor would explode the moment NewProcessor() ran. 2. Tag handlers_attachments_thumbnails_test.go !libvips. Same reasoning — these integration tests assert thumbnail derivation against a working processor. 3. Split testServerWithAttachments's processor wiring into two build-tagged helper files: * testimageprocessor_purego_test.go (//go:build !libvips) wires the real pure-Go processor. * testimageprocessor_libvips_test.go (//go:build libvips) is a no-op so the rest of the server test surface (uploads, downloads, auth, etc.) compiles + runs cleanly under -tags libvips. Verification: go build ./... — OK go build -tags libvips ./... — OK go test ./internal/attachments ./internal/server (default) — pass go test -tags libvips ./internal/server -run "TestUpload|TestDownload" — pass Phase 2 will introduce a real libvips test backend and drop the !libvips tags on the thumbnail tests. * fix(attachments): libvips binary boots cleanly per Codex review (round 4) Codex flagged that the libvips build still crashed at \`pad serve\` startup: cmd/pad/main.go calls attachments.NewProcessor() unconditionally, and the libvips stub was panicking — so any operator who built with -tags libvips today (Phase 2 isn't shipped yet) lost the entire server, not just image processing. Two minimal changes: 1. processor_libvips.go: stop panicking. Return nil + slog.Warn instead. Every call site already nil-checks the processor (the upload handler skips thumbnail derivation, the capabilities endpoint reports a degraded empty formats list), so the libvips-tagged binary now has the same runtime profile as a self-host build that opted out of image processing entirely — uploads succeed, originals display, only derived transformations are unavailable. The slog.Warn keeps the "this build doesn't have it yet" signal loud. 2. cmd/pad/main.go: skip srv.SetImageProcessor when NewProcessor returns nil, and log a "not wired" message in that branch. Distinguishes the wired vs. unwired states cleanly in the boot log. Phase 2 will replace processor_libvips.go's body with the real govips-v2-backed implementation; main.go's wiring is already shape- correct for that transition. Verification: go build ./... — OK go build -tags libvips ./... — OK go test ./... — pass (74s server tests included) go test -tags libvips ./internal/server -run "TestUpload|TestDownload|TestServerCapabilities_Public" — pass	2026-04-29 14:35:49 -04:00