Commit Graph

3 Commits

Author SHA1 Message Date
xarmian d83632e258 feat(attachments): extension trust — the audio/video split, the text family, the CFB office trio and RTF (TASK-2976 / BUG-2963 PR B) (#1309)
* test(attachments): record the Void-beyond-window limit, and correct three comments that claimed more than their evidence (TASK-2976)

The first commit of BUG-2963 PR B, before any extension-trust work: round
6's three items, deliberately kept out of PR A so its verified tip would
not move, plus the nix terminator-comment correction folded in from
BUG-2974's trail.

1. TestMatroskaDocTypeBeyondTheWindowIsWebM records a LIMITATION. A Void
   element is legal anywhere in an EBML header and may be any size; make
   one larger than the 512 bytes this door reads and the DocType behind
   it is not in the input at all, so the parse finds nothing and the
   stdlib's video/webm stands. Nothing got worse — the same file was
   video/webm before the DocType read existed, and video/webm's own
   allowlist entry permits inline serving — and no larger window fixes
   it, since Void may be larger still. The fixture is the ordinary
   FFmpeg Matroska with a 560-byte Void spliced into its header and the
   header size widened to match, the same construction as
   matroska-void-padded.head512; the complete file reads as
   matroska,webm under ffprobe, and what is committed is its first 512
   bytes, so the DocType is absent by construction rather than by
   truncation accident.

2. TestTarWinsAPrefixCollision no longer calls the collision
   "asymmetric". That was the round-4 premise round 5 refuted with
   flac-ustar-in-comment.head512 — a FLAC's Vorbis COMMENT tags are
   arbitrary UTF-8, so real audio carries "ustar" at offset 257 as
   readily as a real tar carries an audio marker at offset zero. The
   implementation and the fixtures already said so; the stale word
   survived in the place a reader looks first. The comment now gives the
   real reason tar leads the default order, which is weaker: its magic
   sits at a fixed offset rather than at a prefix.

3. The seven-byte textual-AAC input's comment says which of the two
   things it is. It is a valid ADTS sync and layer signature that the
   stdlib reads as text — not a decodable AAC, and nothing in the test
   establishes that it is. What review established is the case it stands
   in for, and the comment now separates the two.

4. The nix loop-terminator comments named the wrong mechanism
   (BUG-2974, observed day 62). The heal push cannot loop because a push
   made with the default GITHUB_TOKEN creates no workflow runs at all —
   `gh run list --branch main` spans cbfc073e -> 31d11e76 -> 0b16be49
   and skips the heal commit 249a8f87 entirely. The corrected-tree
   argument the comments gave is true and UNEXERCISED; it is now written
   as the backstop, with the tripwire that it becomes load-bearing the
   moment that push uses a PAT or app token. The per-commit query is not
   the instrument for this and the comment says so: it also returned
   nothing for 0b16be49, which has runs.

Also swept: the fixture README's "the two EBML entries" was already
false at four, and nix.yml's "THAT DISTINCTION" lost its referent once
paragraphs landed above it. The new fixture joins the FuzzSniffMIME
seeds.

go test ./internal/attachments/ green; nix/heal-vendor-hash_test.sh 20
passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* feat(attachments): the ISO-BMFF audio/video split — .mov and .m4a stop being refused (TASK-2976, BUG-2963 F4)

Both iPhone-default containers were refused at upload while their types
sat on the allowlist. Measured on 06ccabdd before the change:

  clip.mov  (qt   brand) -> sniff application/octet-stream -> mime_not_allowed
  clip.m4a  (isom brand) -> sniff video/mp4 -> mime_extension_mismatch
  clip.m4a  (M4A  brand) -> sniff video/mp4 -> mime_extension_mismatch

Three legs, per the day-62 per-family ruling on BUG-2963:

1. Major brand "qt  " -> video/quicktime. The stdlib's mp4 matcher wants
   a brand beginning "mp4" and a QuickTime file carries none, so it had
   no opinion at all. This ADDS a detection, exactly as the still-image
   brands do.

2. Major brand "M4A " -> audio/mp4. This is the one place this package
   OVERRIDES a type the standard library named: an .m4a's compatible
   brands routinely include "mp41", so the stdlib answers video/mp4 from
   a compatible brand, having no way to weigh it against the major one.
   The major brand is the file's own statement about itself. The
   override is bounded — both types are allowlisted, both render inline,
   and the only thing that moves is the CATEGORY, which decides whether
   the UI offers an audio player or a video one.

   It is read from the MAJOR brand alone and BEFORE the mp4-yield, which
   is the whole mechanism: the yield would otherwise return "" on that
   same mp41 brand before the check could speak. Both facts are stated
   in the code.

3. isom-branded bytes named .m4a -> audio/mp4, in ValidateUpload beside
   the zip+document branch and as the same kind of trust one family
   over: the BYTES establish the container, the FILENAME chooses only
   the spelling within it. No track reads, per the ruling.

sniffISOBMFFImage is now sniffISOBMFF. It answers for audio and video
now, and a name saying Image while it returns audio/mp4 is the exact
defect class this bug keeps producing.

The controls are the test. An accept-only test is satisfied by a rule
that trusts .m4a outright, which is a much larger claim than the one
granted, so each leg carries what the STDLIB said about the same bytes
(the thing separating "adds a detection" from "overrides one"), and the
extension leg carries four: the same bytes named .mp4 stay video, the
same bytes with no extension stay video, a WebM named .m4a is still
refused (the trust does not reach outside the MP4 family), and PNG bytes
named .m4a are still refused (the extension introduces no type). A
synthetic ftyp box with major isom and "M4A " among its COMPATIBLE
brands is refused too — reading a category out of the compatible list is
a wider rule than the one granted.

Fixtures: quicktime.head512, m4a-brand.head512, m4a-isom.head512 — real
FFmpeg n7.1 output, provenance in the fixture README. Swept prose that
said this pre-check is still-images-only (SniffMIME's doc comment, the
mime_isobmff.go header).

go test ./internal/attachments/ green. Mutation matrix next, then F5.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* refactor(attachments): the .m4a branch's category guard was dead — assert the invariant it stood for instead (TASK-2976)

A mutation run says the guard survives removal, and it is right: the
branch answers for exactly .m4a, so extEntry.Category == CategoryAudio
can never differ from the ext test beside it. (The zip branch's
identical-looking guard IS load-bearing, because that branch answers for
many extensions at once.)

What the branch actually depends on is that .m4a maps to an allowlisted
audio entry, since it returns that entry. That is now asserted directly
in TestBUG2963F4M4AExtensionTrust, so a remap of .m4a fails a test
instead of quietly retyping MP4 bytes. Dead code replaced by a tested
invariant, rather than kept as defence in depth it cannot provide.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* feat(attachments): the text family — the extension chooses which text, category unchanged (TASK-2976, BUG-2963 F5)

text/plain is the stdlib saying "these bytes are text" and nothing more;
it has no signature separating Markdown from YAML from JavaScript
because at the byte level there is none to have. So when the sniff is
text/plain and the extension maps to an allowlisted TEXT entry, that
entry is what gets stored. Per the day-62 ruling.

Category-preserving is the whole safety argument: nothing becomes an
image, an archive or a document by being renamed, and a mapping that is
not an allowlisted text entry does not fire — .svg still reaches
extension_blocked rather than being quietly stored as text.

TestTextFamilyStillStoresAsPlain is now TestBUG2963F5TextFamily. It
recorded this boundary and said in its own failure message that F5
landing should move it; it kept the same three inputs and changed the
expectation, which is what "move it" meant.

THE SERVING BUCKET MOVES FOR EVERY TYPE F5 TOUCHES, NOT ONLY .js.
text/plain is the only member of the text family in inlineSafe, so
choosing any other text spelling takes the file out of inline serving
and into Content-Disposition: attachment. The ruling weighed that for
.js, where the move to RenderForceDownload is the point. It applies
equally to .md, .csv, .json, .yaml and .toml, where it is a side effect
— and it turns out to be the honest one: the client's own DR-5 already
says every text/* subtype but plain text "is downloaded or rendered
inconsistently across browsers", and those types only reached the
browser-preview affordance because the server was storing them as
text/plain. Whether any of the five belongs in inlineSafe is an
allowlist decision and is not taken here.

The in-app markdown preview is unaffected: it fetch()es the bytes and
renders them itself, which an attachment disposition does not impede,
and it now matches on the MIME rather than falling back to the
extension. Its prose in web/src/lib/attachments/display.ts said an
uploaded .md "is stored as text/plain" as a present-tense measured
fact; that is now history, and the fallback it justifies is what
PRE-EXISTING rows still depend on, so the comment says that instead
(CONVE-23, and its mirror in display.test.ts).

Controls, since an accept-only test is satisfied by "trust text
extensions outright": .txt / an unmapped extension / no extension at all
stay text/plain; PNG bytes named .md are still a mismatch; text bytes
named .mp3 are still a mismatch (an allowlisted mapping in another
category is the dangerous shape and the one the category guard is for);
.svg still fails extension_blocked; and the .js leg asserts RenderMode
== RenderForceDownload against the same bytes named .txt, not merely
that the type changed.

go test ./internal/attachments/ green. Mutation matrix next.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* feat(attachments): the legacy Office trio and RTF — two more families the door refused (TASK-2976, BUG-2963)

CFB. The container legacy Office wrote .doc/.xls/.ppt into has no stdlib
signature, so every one of those files sniffed application/octet-stream
and was refused while all three types sat on the allowlist. The branch
reads eight fixed bytes and says only "this is a CFB container", because
that is all eight bytes can say: .msi installers and Visio files wear
the same header, and telling them apart is a directory-stream walk this
door does not do. So the extension chooses which of the three REVIEWED
types is stored, exactly as the zip+document branch does one container
family over.

The extension list is written out rather than derived from extMIMEMap.
The question is not "does this extension map to a document" — .vsd and
.msi are CFB too — it is "is this one of the three the allowlist
reviewed", and that is a list, not a predicate. .docx is the sharp
control: a document extension whose type is allowlisted but whose
container is zip, refused on a CFB header.

RTF. Printable ASCII, so the stdlib answers text/plain — an OPINION, not
the absence of one — and a .rtf was refused as a mismatch against its
own type. This one needs no extension: five fixed bytes at offset zero,
one of them a version digit the spec fixes, is a stronger signature than
the three-byte "ID3" the stdlib uses for audio/mpeg. So it is a
refinement in SniffMIME rather than extension trust in ValidateUpload,
and the tests assert it under three filenames including .bin, which is
what says so.

text/plain therefore joins the refinement switch, and the comment there
says what makes that case narrower than the octet-stream one: it admits
exactly one signature, and the bar for a second is the argument at
validRTFStream rather than the list's existence.

One behaviour change beyond the refusals: an EXTENSIONLESS RTF used to
store as text/plain and serve inline; it now stores as application/rtf
and serves as an attachment. More conservative, and named here because
it is the one leg that changes a file that was previously accepted.

CONVE-23 sweep, three sites, all of them prose this change made false:
SniffMIME's "two refinements", and the F6 comment claiming
text/javascript "is not reachable EITHER" — F5 made it reachable, which
is the whole point of including .js.

The CFB fixture is hand-built and the test says why that is sound here
and not elsewhere: the branch reads eight fixed bytes, so eight bytes
exercise all of it, unlike the ISO-BMFF and EBML fixtures where the code
walks structure an encoder produces.

go test ./internal/attachments/ green.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn

* fix(attachments): gate the CFB branch on the stdlib verdict, not the refined one (TASK-2976, BUG-2963)

A surviving mutant found this. Dropping the octet-stream gate changed
no test, which said the gate was either dead or untested; it is
untested, and it was also gating on the wrong value.

A CFB file's 512-byte header is sector data, so one can carry "ustar" at
offset 257 and be refined to application/x-tar by this package's own
magic table. Gated on the REFINED verdict, that file is refused under
its own .doc name — the false refusal BUG-2963 exists to remove. Gated
on the stdlib's, the question is whether anything IDENTIFIED the bytes,
and nothing did.

This is the reasoning the .aac branch already carries, in the same
function, for the same collision from the other side. The test is a CFB
header with ustar spliced into its sector data, and it asserts the
premise (these bytes really do refine to x-tar) before asserting the
outcome, so it cannot pass by the collision quietly failing to happen.

Claude-Session: https://claude.ai/code/session_01GqaEDuCtRiSJfa7eppWecn
2026-09-09 19:23:18 -04:00
xarmian cbfc073ef1 fix(attachments): make allowlisted formats reachable by recognising their magic (BUG-2963 PR A) (#1304)
* fix(attachments): types the allowlist names are reachable from their own bytes (BUG-2963 PR A)

BUG-2961 fixed one member of a class. The class, measured over 44 real files
covering 41 extensions: 25 of 48 upload-allowlist entries could never be the
type an upload was stored under, so the door refused — or silently retyped —
files every surface advertises as supported.

This is the sniff-side half. Nothing here trusts a filename to introduce a
type; the trust decisions (the text family, the audio/video category split,
the CFB office trio) are a separate change.

- F1: two spelling aliases, the shape audio/wave and application/x-gzip
  already have. video/avi is a pure spelling difference. application/ogg is
  not: Ogg is a container and the allowlist has no video/ogg, so aliasing
  types an Ogg video as audio. Ruled the better of two answers, since the
  alternative is refusing every Ogg, and the reason is on the alias line.

- F2: magic-byte pre-checks for tar (ustar at offset 257, which is why no
  prefix matcher finds it), bzip2, 7z and FLAC. Consulted ONLY where the
  stdlib returned application/octet-stream, so they can add a detection and
  never replace one. Raw AAC is resolved in ValidateUpload instead: twelve
  bits of sync is too weak to act on alone, so it is gated on the .aac
  extension as well — the bytes must still carry the sync, the extension only
  decides whether a weak signature may speak.

- F3: the one that was accepted rather than refused, and so survived the
  first pass. The mimesniff table maps the bare EBML magic to video/webm with
  no DocType check, so a Matroska file uploaded fine and was stored as WebM.
  A DocType read separates them; an EBML header carrying neither string keeps
  the stdlib verdict, so the fallback is the behaviour that shipped before.

- F6: application/javascript, text/yaml and application/xml leave the
  allowlist — no extension reaches those spellings and no sniff emits them.
  That deletion has a trap: extMIMEMap's values are looked up in `allowed`,
  and an extension naming a removed type is the mechanism that refuses .svg
  and .exe, so .xml now names text/xml. A test covers the .xml upload, not
  just the three lookups. audio/webm is equally unreachable and STAYS by
  ruling, with the comment a future tidy-up will read.

Fixtures are real encoder output truncated to the 512 bytes the door reads,
with provenance and one recorded gap in the testdata README. The measurement
harness ships in neither PR.

Refs: BUG-2963, BUG-2961

* docs(attachments): bring prose that F6 falsified back to true (BUG-2963)

Three client comments and one store filter described the server's category
for `application/javascript`, which stopped having one when F6 removed it from
the allowlist. Nothing in the previous commit's diff points at these lines,
which is the whole reason the sweep exists (team CONVE-23).

Behaviour is unchanged in all four. The web allowlists still exclude
`application/javascript` deliberately — the string can reach the client from
somewhere that is not our upload door — and the store's category filter still
matches it, because it buckets rows that EXIST rather than deciding what may
be created: a filter that stops matching a type costs a row nobody can find,
while one that matches a type no row carries costs nothing.

Refs: BUG-2963

* fix(attachments): admit a format by its own integrity check, not by its magic (BUG-2963)

Round 1 of adversarial review walked straight through the first version of
these signatures. A prefix match is not a format, and on a default-deny door
the difference is the whole point:

- a real, EXECUTING ELF binary with "ustar" in unused padding at offset 257
  was stored as application/x-tar
- the six-byte 7z signature, alone, was a 7z archive
- "fLaC\x00" was a FLAC stream; "BZh9\x00" was a bzip2 stream
- FF F1 00 — three bytes — was an AAC frame
- a WebM carrying "matroska" inside a legal Void element was stored as
  Matroska, and a Matroska with 40 bytes of Void padding was stored as WebM,
  leaving the mistyping this change exists to fix live for any padded file
- a real VP8-in-Ogg video was accepted as audio/ogg, category audio, INLINE

Every one was refused before the change and accepted after it. So each format
is now admitted only by the integrity check the format itself defines: tar's
header checksum, bzip2's block magic, 7z's start-header CRC, FLAC's mandatory
34-byte STREAMINFO, ADTS's reserved sampling-frequency index and layer bits.
The EBML DocType is PARSED — a real element walk that skips Void — rather than
searched for as a string in a fixed window. The Ogg alias is gated on the first
packet naming an audio codec (Vorbis, Opus, FLAC, Speex); Theora and VP8 in Ogg
stay refused exactly as before, and video/ogg is deliberately not added,
because admitting a format is a review and not a side effect.

Three comments asserted things that were false and are corrected rather than
softened: that these formats always sniff as octet-stream (a tar whose first
member is BM.txt sniffs image/bmp), that a .aac without a sync is refused (it
is not, if its bytes are another audio type), and that deleting one switch case
would fail one named test (this project's own mutation matrix already showed
otherwise, and the comment was written against that data).

Tests gained the cases that decide it: every near-miss body above as an exact
upload, the four EBML directions including both adversarial files, the ADTS
guards isolated so exactly one mutation kills each, and length boundaries at
the offsets each check indexes past — the previous negatives padded everything
to 512 bytes, which hid every length guard.

Refs: BUG-2963

* test(attachments): kill two mutants the round-1 tests left alive (BUG-2963)

Both survived the expanded matrix, and both for the same reason round 1 kept
finding: a negative case that fails for a reason other than the guard it names.

- The bzip2 near-miss was five bytes, so the LENGTH guard refused it and the
  block-magic check was never reached; removing that check left the suite
  green. The new sibling is long enough to reach it and NUL-padded so the
  stdlib still says octet-stream — without the padding the body sniffs as
  text and the magic table is never consulted at all, which the first attempt
  at this test demonstrated by passing for the wrong reason.

- The ADTS octet-stream gate had no discriminating input: the two-byte case
  fails validADTSHeader on length before the gate matters. FF F1 40 41 41 41
  41 is a structurally valid ADTS header — sync, layer 00, sampling index 0,
  frame length 2570 — whose every byte the stdlib reads as text, so it answers
  text/plain. That is the only shape that separates the gate from the
  structural check, and with the gate removed the file is stored as audio/aac.

Refs: BUG-2963

* test(attachments): fuzz the sniff path, since this change added a parser (BUG-2963)

Every other check in this package reads fixed offsets and is bounded by
construction. sniffEBMLDocType walks caller-supplied length fields, which is
the one shape here that can index out of range or fail to advance, so the
question 'does it survive malformed input' deserved an answer from running it
rather than from reading it.

Asserts the two properties a sniffer owes its caller — it returns, and it does
not panic — across sniffOpaqueMagic, sniffEBMLDocType, sniffOggAudio,
validADTSHeader, SniffMIME and ValidateUpload. What it returns for nonsense is
left to the table tests.

Seeds are the real fixtures, each truncated at fourteen offsets, plus the
shapes a walk breaks on first: unknown-size elements (the reserved all-ones
VINT), a child whose declared size exceeds the data, the invalid all-zero VINT
marker, a zero-length child that must still advance, and a truncated Ogg page
header. 5.56M executions, no panic and no hang.

Also collapses a duplicate fixture loader this file had grown alongside
mime_isobmff_test.go's readFixture.

Refs: BUG-2963

* fix(attachments): use the real parsers, drop Ogg, and stop overclaiming (BUG-2963)

Round 2 found 15 issues, two of them P1, and the important one is not any
single check — it is that round 1's answer was wrong in the same WAY round 1's
defect was. Round 1 defeated prefix matching with an ELF carrying "ustar" at
offset 257. The response was a checksum. Round 2 defeated the checksum with an
ELF carrying a CORRECT one.

Measured while responding, and it settles the question: archive/tar's own
Reader.Next accepts that file too. A 512-byte tar header is exactly those
fields, and nothing forbids another format's padding from containing them. The
two are not distinguishable at this size by anything in the standard library,
so this is a property of the formats and not a defect to fix. Adding a third
round of field checks would have been the same mistake a third time.

So the change is to the CLAIMS as much as to the code. These checks RECOGNISE
a format; they do not establish one. The safety property lives elsewhere and is
now stated once, at the top of mime_magic.go: whatever the bytes are, they are
stored opaquely, never executed or decompressed here, and served under a
reviewed type with nosniff, as an attachment for every type recognised here.
The checksum-correct ELF ships as a fixture and a test asserting it is
ACCEPTED — a limitation recorded where it cannot be rediscovered as a bug.

Where a real parser exists, it is now used: archive/tar and compress/bzip2,
which cannot drift from the parsers a consumer would use and handle GNU/pax
variants for free. bzip2 keeps its magic check alongside the decode because
neither subsumes the other — the magic refuses a five-byte stream the decoder
can only call truncated, the decode refuses an empty stream whose combined CRC
is wrong.

OGG IS REMOVED. The alias was ruled in, and review showed the question it has
to answer — is this container audio — cannot be answered from the head of the
file: an Ogg with Opus first and VP8 second passes a first-packet codec gate,
because Ogg multiplexes and the video pages come later. The gate was also
wrong in the other direction, refusing legitimate Skeleton-prefixed audio.
Ogg is refused exactly as before this branch; making it work means adding
video/ogg as a reviewed entry or demuxing, and neither belongs in a change
whose premise is that it adds no new trust. The reasoning is a comment where
the next person to reach for an alias will meet it.

Other round-2 correctness fixes: a DocType value ends at its first NUL, so a
real Matroska padded "matroska\x00junk" is no longer stored as WebM; reserved
all-ones EBML IDs are refused rather than acting as zero-length children;
FLAC checks the STREAMINFO BODY rather than its declaration; 7z checks that
the start header's arithmetic is representable, not only that its CRC agrees;
ADTS accounts for the two CRC bytes a protected frame declares.

Tests gained the direction they lacked entirely — legal variants that must NOT
be refused (an empty bzip2 stream, STREAMINFO carrying the last-block flag, a
CRC-protected ADTS frame) — and the truncation loop now asserts refusal below
each check's minimum instead of discarding its return value, which is why it
had been passing while flac.head512[:8] was accepted.

Refs: BUG-2963

* test(attachments): cover the five guards the rebuilt matrix found untested (BUG-2963)

Rebuilding the mutation matrix against the round-2 code turned up seven
survivors. Six were real gaps; each now has the exact input a review round
used, or the smallest one that separates the guard from its neighbours, and
two carry a control leg so the negative cannot pass for the wrong reason.

- 7z: a start header whose next-header offset is 2^64-1, with a recomputed
  valid CRC. The CRC proves the bytes are intended, not that they are possible.
- FLAC: STREAMINFO declaring a zero sample rate.
- bzip2: the head of a 200KB archive, which must be RECOGNISED — truncation is
  what a 512-byte head of any real archive looks like, and the guard that
  distinguishes it from a structural error had nothing testing it.
- EBML: a real Matroska whose DocType payload is 'matroska\x00junk'; and a
  header with a reserved all-ones ID before a valid DocType.

The seventh is an EQUIVALENT mutant and is recorded as one rather than fixed:
lowering validTarHeader's length guard from 512 to 262 changes no outcome,
because archive/tar refuses a short block by itself. Measured, then written
into the comment so the guard reads as clarifying rather than load-bearing.

Refs: BUG-2963

* fix(attachments): delete the bzip2 magic comparison a mutation proved dead (BUG-2963)

The check compared the block magic AND decoded, under a comment claiming each
caught what the other could not. A repaired mutation run refuted it: with the
comparison removed, every input the comment credited it with stopping is still
refused — the short ones by the length guard, the rest by the decoder. So the
line could not change an outcome while its comment said it did, which is the
failure this package's own SafeFallbackExtension comment warns about. Deleted
rather than demoted, and the length guard now says what it is for.

The mutants that found this had been scoring BUILD-FAIL, which is nothing at
all rather than a survivor — repairing them to compile is what surfaced three
untested guards, two of them FLAC's STREAMINFO block-size floor and ordering.

Refs: BUG-2963

* fix(attachments): recognise by magic, to the standard this door already uses (BUG-2963)

Three review rounds walked the structural-validation path to its end and the
ruling is to stop:

- Round 1 defeated magic matching with a real, executing ELF carrying "ustar"
  at offset 257.
- Round 2 defeated the checksum that answered round 1, with an ELF carrying a
  CORRECT one — and archive/tar's own Reader.Next accepts that file too. A
  512-byte tar header is exactly those fields; the two are not distinguishable
  at this size by anything in the standard library.
- Round 3 found the accumulated validation refusing REAL files: PAX and
  long-name GNU tars, legal randomized bzip2 blocks, FLAC declaring the zero
  sample rate RFC 9639 permits. That is this bug's own defect — refusing files
  people legitimately have — reintroduced by the fix for it.

Validation could not narrow what the door accepts and had started refusing
what it should take, so it is gone. Recognition is by defining magic: ustar at
257, BZh plus its digit, the 7z six bytes, fLaC, and for ADTS the syncword
plus the layer bits, which keeps its .aac extension gate because fourteen bits
is weaker than the rest.

The fact that settles the width, read from the stdlib rather than assumed:
http.DetectContentType recognises audio/mpeg from the three bytes "ID3"
(net/http/sniff.go), audio/mpeg is on the allowlist, and this door already
serves it inline. Every signature here is at least as wide, so this is the
door's EXISTING standard rather than a relaxation of it. A test asserts that
premise so it cannot rot.

Two refusals are deliberate and say so in the code: V7 tar has no magic
anywhere and cannot be recognised by this kind of check at all, and Ogg stays
refused for the reason a previous commit records.

bzip2 is no longer decompressed anywhere in the door. The safety paragraph is
rewritten from what the code does, for the third time and the last: nothing is
executed, nothing is decompressed, archives download while FLAC and AAC play
inline exactly as every other allowlisted audio type does, and nosniff means
recognition can move a file between reviewed types but never outside them.

Tests now assert the contract that exists. The widening is asserted rather
than described — every input the validation used to refuse is accepted, each
with its SERVING BUCKET checked, because that is the property that makes it
tolerable. The files validation used to refuse are asserted accepted. What
remains of the negatives is the only thing still true: the magic has to be
there, in the right place, in full.

Refs: BUG-2963

* test(attachments): restore two EBML properties the bulk cut dropped (BUG-2963)

Cutting the structural-validation tests wholesale took the DocType
NUL-termination and reserved-ID cases with them, though the DocType walk they
cover is untouched by the magic-only ruling. A mutation run is what noticed:
both mutations had gone from detected to surviving.

Worth recording as a shape rather than a slip — deleting a test file's worth of
obsolete assertions is exactly when live coverage leaves with them, and the
matrix is the only thing that says so.

Refs: BUG-2963

* fix(attachments): tar wins a prefix collision, and the AAC gate stops refusing real files (BUG-2963)

Round 4's two blocking findings were both REAL files of listed types turned
away — the direction the ruling's convergence bar names first.

**A tar whose first member is named fLaC.txt was refused.** Every recogniser
except tar's is a prefix test, and a tar header's first 100 bytes are its
member's FILENAME — arbitrary text a user chooses. So an ordinary archive
carrying "fLaC" or "BZh9" in a name was recognised as that format and then
refused for a category mismatch against its own .tar extension. Tar is tested
first now, and the order is load-bearing rather than arbitrary: the collision
is asymmetric. A real tar carrying a foreign prefix needs only a filename; a
real FLAC carrying "ustar" needs those five bytes at exactly offset 257 in
compressed data. Losing the first case costs ordinary uploads.

**Real AAC files were refused when their leading bytes looked textual.** The
gate ran on application/octet-stream alone, but a raw AAC frame whose ancillary
payload is printable makes the first 512 bytes read as text, so the stdlib
answers text/plain and a genuine, ffmpeg-decodable .aac was rejected for a
category mismatch. Neither verdict is a format detection; both mean "nothing
here identifies this", which is the condition under which a weak signature may
speak. A type the stdlib DOES recognise is still untouched, and a test pins
that with PNG bytes named .aac.

Also, the leftovers that keep being mine: comments still describing structural
validation that is gone, a test comment claiming production delegates to
archive/tar when it no longer does, a README row saying the ELF fixture exists
to be refused when it is now accepted, an unused fixture, and two fixtures the
README never listed.

Tests strengthened where a review round showed one example was standing in for
a whole signature: the EBML legs now ask sniffEBMLDocType directly, because
routed through SniffMIME the stdlib fallback supplied the same WebM answer and
the explicit mapping could be deleted with the suite green; the ADTS signature
is walked byte by byte; the bzip2 digit range is tested at both bounds and the
FLAC marker at its width and case.

Refs: BUG-2963

* test(attachments): give the ADTS verdict gate a control that actually controls (BUG-2963)

The PNG leg could not establish the gate it was named for: PNG bytes fail
validADTSHeader on the first byte, so they are refused with the gate removed
too. A mutation run said so — dropping the stdlib-verdict gate survived the
whole suite.

The only shape that separates the verdict gate from the structural check is an
input that PASSES validADTSHeader and is ALSO identified as something else: a
buffer opening with a valid ADTS header and carrying 'ustar' at offset 257,
named .aac. It is identified as a tar and must be refused for the category
mismatch it is; without the gate the AAC branch overwrites that and accepts it.

Recording the process failure alongside it, because it is one I have written
down before: this test was lost once between writing and committing, because
the mutation runner's restore is 'git checkout -- internal/attachments/' and
the work was still uncommitted. Committing is step one of running a control,
not step one of the unit.

Refs: BUG-2963

* fix(attachments): let the extension arbitrate a magic collision, both ways (BUG-2963)

Round 5 found the mirror image of round 4's finding, which is the useful part:
my fix for round 4 created it. Round 4 showed a tar whose first member is named
fLaC.txt being refused, so tar was ordered first. Round 5 then built complete,
decodable FLAC and AAC files carrying "ustar" at offset 257 in ordinary
metadata — a Vorbis COMMENT tag is arbitrary UTF-8 (RFC 9639 §8.6) — and those
were refused instead.

The premise I wrote into the ordering comment was false. I argued the collision
was asymmetric, that real audio could not plausibly carry "ustar" at a fixed
offset. It can, in a tag a user typed. Any total order refuses somebody.

So there is no winner by order. sniffOpaqueCandidates returns EVERY matching
type, and when the filename's extension names one of them it breaks the tie.
That is a narrower thing than extension trust and the code says so: every
candidate is a type the BYTES already matched, so the extension chooses among
readings rather than casting a vote, and a name for a type whose magic is
absent can never appear in the list. An extension naming none of them changes
nothing — asserted, including that a .zip name does not make colliding bytes a
zip.

The AAC branch is fixed by the same finding from the other direction: it was
gated on the REFINED sniff, so a real AAC with "ustar" in its payload — which
this package refines to application/x-tar — was refused under its own .aac
name. The gate now reads the standard library's verdict, which for those bytes
is "nothing identifies this". Kept rather than dropped as unfalsifiable,
because no mimesniff signature begins with 0xFF today and this is what stops a
fourteen-bit match overriding one that does.

Fixtures: the FLAC is real and decodes under libsndfile. The AAC counterpart is
NOT shipped — overwriting a real frame's bytes produces a file ffmpeg rejects,
so the test uses a synthetic buffer that reproduces the condition and says so
rather than claiming to be audio.

Also cleared, and reliably my own: comments still describing structural
validation, a test's why-strings still citing checksums and CRCs, a comment
claiming PAX and GNU archives are refused when they are recognised, and a
README gap note that no longer covered all the fixtures. Two test premises
tightened where they proved less than they said: the bzip2 prefix case also
failed the digit check, and the three-byte ID3 claim was tested with seven.

Refs: BUG-2963

* test(attachments): pin the default candidate order (BUG-2963)

Reordering sniffOpaqueCandidates changed no test, because extension
arbitration settles both known collisions whichever way the list runs. The
order still decides one case — colliding bytes whose extension names neither
candidate — and nothing asserted it, so the rationale in the comment was
unenforced.

Refs: BUG-2963

* docs(attachments): the fuzz comment still named the structural validators (BUG-2963)

They were removed by the magic-only ruling. Comment-only.

Refs: BUG-2963
2026-09-09 13:28:23 -04:00
xarmian 6a5eb3dee0 fix(attachments): HEIC, HEIF and AVIF are recognised from their bytes, not refused (BUG-2961) (#1291)
fix(attachments): HEIC, HEIF and AVIF are recognised from their bytes, not refused (BUG-2961)

image/heic, image/heif and image/avif have been on the upload allowlist all
along and every real file of those types was rejected 415 mime_not_allowed:
http.DetectContentType implements the WHATWG mimesniff table, which has no
signature for ISO base media file format still images, so they sniffed as
application/octet-stream and ValidateUpload's first rule refused them before
the extension cross-check was ever consulted. HEIC is the iPhone camera
default, so that was every photo shared straight from an Apple device, through
the web UI today and the mobile share sheet being built.

The fix stays inside rule 1 rather than around it. sniffISOBMFFImage reads the
ftyp box and returns the allowlisted MIME, so the sniffed type is still what
the allowlist is consulted with; it returns "" for everything else, so it can
only add detections. The alternative was to trust the filename for these three,
the way ValidateUpload already trusts it for zip-based Office documents, and
that was refused in triage because a renamed .heic would then pass on a name
alone. Both tests layers pin that: an executable named .heic is still refused,
and real HEIC bytes named .jpg are still stored as image/heic.

Four decisions, each argued at its site:

  - Every brand in the box, not just the major one. The measured Apple HEIC
    carries major heic with compatible mif1 MiPr miaf MiHB heic; the measured
    Apple AVIF carries major avif with compatible MiPr avif miaf mif1; libheif
    writes major mif1. A check on bytes 8..12 alone is right for one encoder
    and wrong for the next. A specific brand wins wherever it appears; mif1
    alone means image/heif.
  - Sequence brands (hevc/hevx/hevm/hevs, avis, msf1) are deliberately absent:
    they name image/*-sequence types, none of which is on the allowlist, so
    recognising them would produce a sniff ValidateUpload then refuses with a
    stranger message.
  - A container carrying an mp4 brand stays a video and is left to the stdlib.
  - A size-1 (64-bit largesize) ftyp box is declined, because the brands shift
    eight bytes and the fixed offsets would be reading the size field.

Fixtures are real encoder output with provenance recorded in testdata/README.md
(sips on macOS 26 for the Apple pair, libheif 1.20.2 here for the other two);
hand-built byte vectors appear only for shapes no available encoder produces.
Verified by negative control — with the pre-check disabled all four new
acceptance tests fail — and by a nine-mutant matrix in which the major-only
scan, the dropped mp4 yield, generic-over-specific preference, the minor
version read as a brand, the missing box bound, the removed largesize bail, a
dropped brand table entry and an unconsulted sniffer are each killed by a named
test. The one survivor is equivalent: an 8..11-byte buffer yields no whole
brand either way.

Codex found two more refusals this owed, both now guarded and mutation-checked:
a declared box size of 2..15 is impossible for an ftyp box, and a buffer that
stops inside the 16-byte header is a truncated file — in either case the bytes
at offset 8 are not brands of a box that exists, and classifying from them let
a caller-controlled payload be stored and served as an image.

Also corrects processor.go's claim that "display always works (browsers handle
WebP / AVIF / HEIC natively)", which the client's own viewer table contradicts
and which this change would have made load-bearing (CONVE-23). What it means
for rendering on builds that cannot derive a thumbnail is filed as BUG-2964;
the wider class of allowlist entries the sniffer cannot produce is BUG-2963.

Claude-Session: https://claude.ai/code/session_01HeChkgZVYb3NTgTcckF5KR
2026-09-08 13:30:42 -04:00