Files
pad/internal/attachments/testdata
xarmian d83632e258 feat(attachments): extension trust — the audio/video split, the text family, the CFB office trio and RTF (TASK-2976 / BUG-2963 PR B) (#1309)
* test(attachments): record the Void-beyond-window limit, and correct three comments that claimed more than their evidence (TASK-2976)

The first commit of BUG-2963 PR B, before any extension-trust work: round
6's three items, deliberately kept out of PR A so its verified tip would
not move, plus the nix terminator-comment correction folded in from
BUG-2974's trail.

1. TestMatroskaDocTypeBeyondTheWindowIsWebM records a LIMITATION. A Void
   element is legal anywhere in an EBML header and may be any size; make
   one larger than the 512 bytes this door reads and the DocType behind
   it is not in the input at all, so the parse finds nothing and the
   stdlib's video/webm stands. Nothing got worse — the same file was
   video/webm before the DocType read existed, and video/webm's own
   allowlist entry permits inline serving — and no larger window fixes
   it, since Void may be larger still. The fixture is the ordinary
   FFmpeg Matroska with a 560-byte Void spliced into its header and the
   header size widened to match, the same construction as
   matroska-void-padded.head512; the complete file reads as
   matroska,webm under ffprobe, and what is committed is its first 512
   bytes, so the DocType is absent by construction rather than by
   truncation accident.

2. TestTarWinsAPrefixCollision no longer calls the collision
   "asymmetric". That was the round-4 premise round 5 refuted with
   flac-ustar-in-comment.head512 — a FLAC's Vorbis COMMENT tags are
   arbitrary UTF-8, so real audio carries "ustar" at offset 257 as
   readily as a real tar carries an audio marker at offset zero. The
   implementation and the fixtures already said so; the stale word
   survived in the place a reader looks first. The comment now gives the
   real reason tar leads the default order, which is weaker: its magic
   sits at a fixed offset rather than at a prefix.

3. The seven-byte textual-AAC input's comment says which of the two
   things it is. It is a valid ADTS sync and layer signature that the
   stdlib reads as text — not a decodable AAC, and nothing in the test
   establishes that it is. What review established is the case it stands
   in for, and the comment now separates the two.

4. The nix loop-terminator comments named the wrong mechanism
   (BUG-2974, observed day 62). The heal push cannot loop because a push
   made with the default GITHUB_TOKEN creates no workflow runs at all —
   `gh run list --branch main` spans cbfc073e -> 31d11e76 -> 0b16be49
   and skips the heal commit 249a8f87 entirely. The corrected-tree
   argument the comments gave is true and UNEXERCISED; it is now written
   as the backstop, with the tripwire that it becomes load-bearing the
   moment that push uses a PAT or app token. The per-commit query is not
   the instrument for this and the comment says so: it also returned
   nothing for 0b16be49, which has runs.

Also swept: the fixture README's "the two EBML entries" was already
false at four, and nix.yml's "THAT DISTINCTION" lost its referent once
paragraphs landed above it. The new fixture joins the FuzzSniffMIME
seeds.

go test ./internal/attachments/ green; nix/heal-vendor-hash_test.sh 20
passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* feat(attachments): the ISO-BMFF audio/video split — .mov and .m4a stop being refused (TASK-2976, BUG-2963 F4)

Both iPhone-default containers were refused at upload while their types
sat on the allowlist. Measured on 06ccabdd before the change:

  clip.mov  (qt   brand) -> sniff application/octet-stream -> mime_not_allowed
  clip.m4a  (isom brand) -> sniff video/mp4 -> mime_extension_mismatch
  clip.m4a  (M4A  brand) -> sniff video/mp4 -> mime_extension_mismatch

Three legs, per the day-62 per-family ruling on BUG-2963:

1. Major brand "qt  " -> video/quicktime. The stdlib's mp4 matcher wants
   a brand beginning "mp4" and a QuickTime file carries none, so it had
   no opinion at all. This ADDS a detection, exactly as the still-image
   brands do.

2. Major brand "M4A " -> audio/mp4. This is the one place this package
   OVERRIDES a type the standard library named: an .m4a's compatible
   brands routinely include "mp41", so the stdlib answers video/mp4 from
   a compatible brand, having no way to weigh it against the major one.
   The major brand is the file's own statement about itself. The
   override is bounded — both types are allowlisted, both render inline,
   and the only thing that moves is the CATEGORY, which decides whether
   the UI offers an audio player or a video one.

   It is read from the MAJOR brand alone and BEFORE the mp4-yield, which
   is the whole mechanism: the yield would otherwise return "" on that
   same mp41 brand before the check could speak. Both facts are stated
   in the code.

3. isom-branded bytes named .m4a -> audio/mp4, in ValidateUpload beside
   the zip+document branch and as the same kind of trust one family
   over: the BYTES establish the container, the FILENAME chooses only
   the spelling within it. No track reads, per the ruling.

sniffISOBMFFImage is now sniffISOBMFF. It answers for audio and video
now, and a name saying Image while it returns audio/mp4 is the exact
defect class this bug keeps producing.

The controls are the test. An accept-only test is satisfied by a rule
that trusts .m4a outright, which is a much larger claim than the one
granted, so each leg carries what the STDLIB said about the same bytes
(the thing separating "adds a detection" from "overrides one"), and the
extension leg carries four: the same bytes named .mp4 stay video, the
same bytes with no extension stay video, a WebM named .m4a is still
refused (the trust does not reach outside the MP4 family), and PNG bytes
named .m4a are still refused (the extension introduces no type). A
synthetic ftyp box with major isom and "M4A " among its COMPATIBLE
brands is refused too — reading a category out of the compatible list is
a wider rule than the one granted.

Fixtures: quicktime.head512, m4a-brand.head512, m4a-isom.head512 — real
FFmpeg n7.1 output, provenance in the fixture README. Swept prose that
said this pre-check is still-images-only (SniffMIME's doc comment, the
mime_isobmff.go header).

go test ./internal/attachments/ green. Mutation matrix next, then F5.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* refactor(attachments): the .m4a branch's category guard was dead — assert the invariant it stood for instead (TASK-2976)

A mutation run says the guard survives removal, and it is right: the
branch answers for exactly .m4a, so extEntry.Category == CategoryAudio
can never differ from the ext test beside it. (The zip branch's
identical-looking guard IS load-bearing, because that branch answers for
many extensions at once.)

What the branch actually depends on is that .m4a maps to an allowlisted
audio entry, since it returns that entry. That is now asserted directly
in TestBUG2963F4M4AExtensionTrust, so a remap of .m4a fails a test
instead of quietly retyping MP4 bytes. Dead code replaced by a tested
invariant, rather than kept as defence in depth it cannot provide.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* feat(attachments): the text family — the extension chooses which text, category unchanged (TASK-2976, BUG-2963 F5)

text/plain is the stdlib saying "these bytes are text" and nothing more;
it has no signature separating Markdown from YAML from JavaScript
because at the byte level there is none to have. So when the sniff is
text/plain and the extension maps to an allowlisted TEXT entry, that
entry is what gets stored. Per the day-62 ruling.

Category-preserving is the whole safety argument: nothing becomes an
image, an archive or a document by being renamed, and a mapping that is
not an allowlisted text entry does not fire — .svg still reaches
extension_blocked rather than being quietly stored as text.

TestTextFamilyStillStoresAsPlain is now TestBUG2963F5TextFamily. It
recorded this boundary and said in its own failure message that F5
landing should move it; it kept the same three inputs and changed the
expectation, which is what "move it" meant.

THE SERVING BUCKET MOVES FOR EVERY TYPE F5 TOUCHES, NOT ONLY .js.
text/plain is the only member of the text family in inlineSafe, so
choosing any other text spelling takes the file out of inline serving
and into Content-Disposition: attachment. The ruling weighed that for
.js, where the move to RenderForceDownload is the point. It applies
equally to .md, .csv, .json, .yaml and .toml, where it is a side effect
— and it turns out to be the honest one: the client's own DR-5 already
says every text/* subtype but plain text "is downloaded or rendered
inconsistently across browsers", and those types only reached the
browser-preview affordance because the server was storing them as
text/plain. Whether any of the five belongs in inlineSafe is an
allowlist decision and is not taken here.

The in-app markdown preview is unaffected: it fetch()es the bytes and
renders them itself, which an attachment disposition does not impede,
and it now matches on the MIME rather than falling back to the
extension. Its prose in web/src/lib/attachments/display.ts said an
uploaded .md "is stored as text/plain" as a present-tense measured
fact; that is now history, and the fallback it justifies is what
PRE-EXISTING rows still depend on, so the comment says that instead
(CONVE-23, and its mirror in display.test.ts).

Controls, since an accept-only test is satisfied by "trust text
extensions outright": .txt / an unmapped extension / no extension at all
stay text/plain; PNG bytes named .md are still a mismatch; text bytes
named .mp3 are still a mismatch (an allowlisted mapping in another
category is the dangerous shape and the one the category guard is for);
.svg still fails extension_blocked; and the .js leg asserts RenderMode
== RenderForceDownload against the same bytes named .txt, not merely
that the type changed.

go test ./internal/attachments/ green. Mutation matrix next.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* feat(attachments): the legacy Office trio and RTF — two more families the door refused (TASK-2976, BUG-2963)

CFB. The container legacy Office wrote .doc/.xls/.ppt into has no stdlib
signature, so every one of those files sniffed application/octet-stream
and was refused while all three types sat on the allowlist. The branch
reads eight fixed bytes and says only "this is a CFB container", because
that is all eight bytes can say: .msi installers and Visio files wear
the same header, and telling them apart is a directory-stream walk this
door does not do. So the extension chooses which of the three REVIEWED
types is stored, exactly as the zip+document branch does one container
family over.

The extension list is written out rather than derived from extMIMEMap.
The question is not "does this extension map to a document" — .vsd and
.msi are CFB too — it is "is this one of the three the allowlist
reviewed", and that is a list, not a predicate. .docx is the sharp
control: a document extension whose type is allowlisted but whose
container is zip, refused on a CFB header.

RTF. Printable ASCII, so the stdlib answers text/plain — an OPINION, not
the absence of one — and a .rtf was refused as a mismatch against its
own type. This one needs no extension: five fixed bytes at offset zero,
one of them a version digit the spec fixes, is a stronger signature than
the three-byte "ID3" the stdlib uses for audio/mpeg. So it is a
refinement in SniffMIME rather than extension trust in ValidateUpload,
and the tests assert it under three filenames including .bin, which is
what says so.

text/plain therefore joins the refinement switch, and the comment there
says what makes that case narrower than the octet-stream one: it admits
exactly one signature, and the bar for a second is the argument at
validRTFStream rather than the list's existence.

One behaviour change beyond the refusals: an EXTENSIONLESS RTF used to
store as text/plain and serve inline; it now stores as application/rtf
and serves as an attachment. More conservative, and named here because
it is the one leg that changes a file that was previously accepted.

CONVE-23 sweep, three sites, all of them prose this change made false:
SniffMIME's "two refinements", and the F6 comment claiming
text/javascript "is not reachable EITHER" — F5 made it reachable, which
is the whole point of including .js.

The CFB fixture is hand-built and the test says why that is sound here
and not elsewhere: the branch reads eight fixed bytes, so eight bytes
exercise all of it, unlike the ISO-BMFF and EBML fixtures where the code
walks structure an encoder produces.

go test ./internal/attachments/ green.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* fix(attachments): gate the CFB branch on the stdlib verdict, not the refined one (TASK-2976, BUG-2963)

A surviving mutant found this. Dropping the octet-stream gate changed
no test, which said the gate was either dead or untested; it is
untested, and it was also gating on the wrong value.

A CFB file's 512-byte header is sector data, so one can carry "ustar" at
offset 257 and be refined to application/x-tar by this package's own
magic table. Gated on the REFINED verdict, that file is refused under
its own .doc name — the false refusal BUG-2963 exists to remove. Gated
on the stdlib's, the question is whether anything IDENTIFIED the bytes,
and nothing did.

This is the reasoning the .aac branch already carries, in the same
function, for the same collision from the other side. The test is a CFB
header with ustar spliced into its sector data, and it asserts the
premise (these bytes really do refine to x-tar) before asserting the
outcome, so it cannot pass by the collision quietly failing to happen.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn
2026-09-09 19:23:18 -04:00
..

Attachment fixtures

Real files, not hand-built headers. A sniffer's whole job is to recognise what encoders actually emit, so a fixture I typed from a spec would test my reading of the spec (BUG-2961's tests, mime_isobmff_test.go).

file produced by ftyp brands
apple-still-heic.head512 sips -s format heic on macOS 26 — first 512 bytes only major heic; compatible mif1 MiPr miaf MiHB heic
apple-still.avif sips -s format avif on macOS 26, complete file (509 B) major avif; compatible MiPr avif miaf mif1
still.avif heif-enc -A -q 60 (libheif 1.20.2 / aom 3.12.1) on Linux, complete file major avif; compatible mif1 avif miaf
still-mif1.heif heif-enc -U (libheif 1.20.2, uncompressed codec) on Linux, complete file major mif1; compatible mif1 miaf

The .head512 file is a prefix because that is the entire input domain of SniffMIME — it reads at most 512 bytes and the ftyp box is the first box in the file. It is named for what it is so nobody mistakes it for a loadable image.

Two gaps, recorded rather than papered over:

  • No sips-produced HEIC beyond 512 bytes and no iPhone-camera capture. Both Apple fixtures come from sips, not from a camera: same container and brand set, different encoder pipeline.
  • No file whose MAJOR brand is mif1 from Apple. sips -s format heif writes no file on that machine. The libheif still-mif1.heif covers the major-mif1 shape from a different encoder instead of a guessed one.

BUG-2963 fixtures — formats the sniffer learned to recognise

Same rule as above: real encoder output, truncated to the 512 bytes SniffMIME actually reads. bzip2.head512 and sevenzip.head512 are shorter than 512 because the whole file is.

file produced by what it proves
tar.head512 GNU tar -cf (tar 1.34, Linux) ustar at offset 257 — the magic is not at the start, which is why no prefix matcher finds it
bzip2.head512 bzip2 -c (1.0.8), complete file (88 B) BZh plus the block-size digit
sevenzip.head512 py7zr 1.1.3, complete file (192 B) the six-byte 7z signature
flac.head512 libsndfile 1.2.2 via soundfile, FLAC format the fLaC stream marker
aac-adts.head512 FFmpeg 7.1 -c:a aac -f adts an ADTS sync (ff f1) — the one signature weak enough to need its extension
matroska.head512 FFmpeg 7.1 -f matroska EBML magic with DocType matroska at offset 24
webm.head512 FFmpeg 7.1 -f webm EBML magic with DocType webm at offset 24 — the control that stops the DocType read from answering Matroska for everything
avi.head512 FFmpeg 7.1 -f avi RIFF/AVI, which the stdlib names video/avi against the allowlist's video/x-msvideo

PR B fixtures — the audio/video split (F4)

file produced by what it proves
quicktime.head512 FFmpeg n7.1 -f mov major brand qt . The stdlib's mp4 matcher wants a brand beginning mp4 and this carries none, so it sniffs application/octet-stream — recognising it ADDS a detection rather than overriding one
m4a-brand.head512 FFmpeg n7.1 -f mp4 -brand "M4A " major brand M4A , compatible M4A iso2 mp41. The stdlib answers video/mp4 off that mp41, so this is the one fixture whose recognition OVERRIDES the standard library, on the major brand
m4a-isom.head512 FFmpeg n7.1 -f mp4 major brand isom, compatible isom iso2 mp41 — an audio-only MP4 whose brands name no category at all. It is the input for the extension-trust leg: the bytes reach video/mp4 and only the .m4a filename can choose the audio spelling

All three are one second of sine=frequency=440 encoded to AAC, truncated to the 512 bytes SniffMIME reads.

Fixtures for files the recogniser must ACCEPT

Real files that three rounds of structural validation refused. Each is a file a user legitimately has, and refusing them is the defect BUG-2963 exists to fix — reintroduced, for a while, by its own fix.

file produced by what it proves
tar-pax.head512 tar --format=pax -cf (tar 1.34) a PAX archive leads with a metadata header, so a full parse needs more blocks than this door ever reads
tar-gnu-longname.head512 tar --format=gnu -cf with a 144-character member name same shape, via GNU's long-name header
flac-zero-sample-rate.head512 flac.head512 with the sample-rate field zeroed RFC 9639 permits a zero sample rate for non-audio samples and still registers the result as audio/flac

Adversarial fixtures — the round-1 findings, kept as tests

These exist to be REFUSED or to be typed correctly against a check that used to get them wrong. Each was produced by an adversarial review round that broke the first version of these signatures.

file produced by what it proves
tar-bmp-firstmember.head512 GNU tar -cf on a file named BM.txt a structurally valid tar the stdlib reads as image/bmp — a real file exercising competing detections, refused before this change and still refused
elf-with-ustar-magic.head512 hand-built an ELF header with ustar at offset 257. It is ACCEPTED now — recognition is by magic — and the test says so. It was written when it was refused, and is kept because the finding it comes from used a real, executing binary
webm-void-says-matroska.head512 FFmpeg WebM with a Void element containing the string matroska, header size widened to match a substring search calls this Matroska; its DocType is webm
matroska-void-padded.head512 FFmpeg Matroska with 40 bytes of Void padding, header size widened to match DocType moves to offset 66, past any fixed leading window — the mistyping this change fixes, still live under a search
elf-with-valid-tar-checksum.head512 hand-built an ELF header carrying a well-formed tar header in its padding, checksum included. ACCEPTED, deliberately — and when this package still verified checksums, archive/tar's own reader accepted it too. It records a limitation rather than a defect
tar-flac-named-member.head512 GNU tar -cf on a file named fLaC.txt an ordinary archive carrying another format's prefix magic in its member NAME
flac-ustar-in-comment.head512 a real FLAC with a Vorbis COMMENT block whose vendor string places ustar at offset 257; decodes under libsndfile the same collision from the other side — Vorbis tags are arbitrary UTF-8, so audio can carry the tar magic as easily as a tar can carry an audio marker
matroska-void-beyond-window.head512 FFmpeg Matroska with a 560-byte Void spliced into its header, header size widened to match; the complete file reads as matroska,webm under ffprobe a Void larger than the sniff window puts DocType outside the input entirely, so the parse finds nothing and the stdlib's video/webm stands. A limitation, recorded — no window size fixes it, since Void may be larger still
matroska-nul-terminated-doctype.head512 sample.mkv with its DocType payload rewritten to matroska\0junk, declared length 13 a DocType value ends at its first NUL; trimming instead stored a real Matroska as WebM

Void elements are legal anywhere in an EBML header and their contents are meaningless by specification, which is why only a parse can tell payload from padding.

Where an EBML entry above says a muxer or ffprobe read a file, that is said of the file it was MADE FROM. What is committed is the first 512 bytes, as with every fixture here, so running ffprobe on the committed prefix fails with a premature EOF — that is the truncation, not the file. (This paragraph said "the two EBML entries" while there were four of them; it is a property of the whole family, not of a fixed pair.)

Ogg fixtures — kept for a format the sniffer does NOT recognise

Both are refused, and both are kept so the next person to reach for an application/ogg alias meets the evidence before writing one.

file produced by what it proves
ogg-opus.head512 FFmpeg 7.1 -c:a libopus -f ogg ordinary Ogg audio, refused — audio/ogg stays unreachable
ogg-vp8-video.head512 FFmpeg 7.1 -c:v libvpx -f ogg a real Ogg file whose first packet is OVP80: video in an Ogg container, which an unconditional alias accepted as inline audio

An alias was written, ruled in, and removed: whether an Ogg container is audio-only cannot be decided from its first page, because Ogg multiplexes and a second video stream's pages come later than any 512-byte sniff can see.

The FFmpeg used is the one bundled with Remotion (@remotion/compositor-linux-x64-gnu), transcoded from real project assets; there is no system FFmpeg on the machine these were made on.

Gap, recorded rather than papered over: the media fixtures come from one FFmpeg build, apart from flac-ustar-in-comment.head512, which is a FLAC assembled by hand from a real one and verified decodable by libsndfile. A second encoder would be worth having for the EBML pair in particular, since the DocType read is the only check here that depends on where a muxer places a string rather than on a fixed prefix.