xarmian cbfc073ef1 fix(attachments): make allowlisted formats reachable by recognising their magic (BUG-2963 PR A) (#1304) ...

* fix(attachments): types the allowlist names are reachable from their own bytes (BUG-2963 PR A)

BUG-2961 fixed one member of a class. The class, measured over 44 real files
covering 41 extensions: 25 of 48 upload-allowlist entries could never be the
type an upload was stored under, so the door refused — or silently retyped —
files every surface advertises as supported.

This is the sniff-side half. Nothing here trusts a filename to introduce a
type; the trust decisions (the text family, the audio/video category split,
the CFB office trio) are a separate change.

- F1: two spelling aliases, the shape audio/wave and application/x-gzip
  already have. video/avi is a pure spelling difference. application/ogg is
  not: Ogg is a container and the allowlist has no video/ogg, so aliasing
  types an Ogg video as audio. Ruled the better of two answers, since the
  alternative is refusing every Ogg, and the reason is on the alias line.

- F2: magic-byte pre-checks for tar (ustar at offset 257, which is why no
  prefix matcher finds it), bzip2, 7z and FLAC. Consulted ONLY where the
  stdlib returned application/octet-stream, so they can add a detection and
  never replace one. Raw AAC is resolved in ValidateUpload instead: twelve
  bits of sync is too weak to act on alone, so it is gated on the .aac
  extension as well — the bytes must still carry the sync, the extension only
  decides whether a weak signature may speak.

- F3: the one that was accepted rather than refused, and so survived the
  first pass. The mimesniff table maps the bare EBML magic to video/webm with
  no DocType check, so a Matroska file uploaded fine and was stored as WebM.
  A DocType read separates them; an EBML header carrying neither string keeps
  the stdlib verdict, so the fallback is the behaviour that shipped before.

- F6: application/javascript, text/yaml and application/xml leave the
  allowlist — no extension reaches those spellings and no sniff emits them.
  That deletion has a trap: extMIMEMap's values are looked up in `allowed`,
  and an extension naming a removed type is the mechanism that refuses .svg
  and .exe, so .xml now names text/xml. A test covers the .xml upload, not
  just the three lookups. audio/webm is equally unreachable and STAYS by
  ruling, with the comment a future tidy-up will read.

Fixtures are real encoder output truncated to the 512 bytes the door reads,
with provenance and one recorded gap in the testdata README. The measurement
harness ships in neither PR.

Refs: BUG-2963, BUG-2961

* docs(attachments): bring prose that F6 falsified back to true (BUG-2963)

Three client comments and one store filter described the server's category
for `application/javascript`, which stopped having one when F6 removed it from
the allowlist. Nothing in the previous commit's diff points at these lines,
which is the whole reason the sweep exists (team CONVE-23).

Behaviour is unchanged in all four. The web allowlists still exclude
`application/javascript` deliberately — the string can reach the client from
somewhere that is not our upload door — and the store's category filter still
matches it, because it buckets rows that EXIST rather than deciding what may
be created: a filter that stops matching a type costs a row nobody can find,
while one that matches a type no row carries costs nothing.

Refs: BUG-2963

* fix(attachments): admit a format by its own integrity check, not by its magic (BUG-2963)

Round 1 of adversarial review walked straight through the first version of
these signatures. A prefix match is not a format, and on a default-deny door
the difference is the whole point:

- a real, EXECUTING ELF binary with "ustar" in unused padding at offset 257
  was stored as application/x-tar
- the six-byte 7z signature, alone, was a 7z archive
- "fLaC\x00" was a FLAC stream; "BZh9\x00" was a bzip2 stream
- FF F1 00 — three bytes — was an AAC frame
- a WebM carrying "matroska" inside a legal Void element was stored as
  Matroska, and a Matroska with 40 bytes of Void padding was stored as WebM,
  leaving the mistyping this change exists to fix live for any padded file
- a real VP8-in-Ogg video was accepted as audio/ogg, category audio, INLINE

Every one was refused before the change and accepted after it. So each format
is now admitted only by the integrity check the format itself defines: tar's
header checksum, bzip2's block magic, 7z's start-header CRC, FLAC's mandatory
34-byte STREAMINFO, ADTS's reserved sampling-frequency index and layer bits.
The EBML DocType is PARSED — a real element walk that skips Void — rather than
searched for as a string in a fixed window. The Ogg alias is gated on the first
packet naming an audio codec (Vorbis, Opus, FLAC, Speex); Theora and VP8 in Ogg
stay refused exactly as before, and video/ogg is deliberately not added,
because admitting a format is a review and not a side effect.

Three comments asserted things that were false and are corrected rather than
softened: that these formats always sniff as octet-stream (a tar whose first
member is BM.txt sniffs image/bmp), that a .aac without a sync is refused (it
is not, if its bytes are another audio type), and that deleting one switch case
would fail one named test (this project's own mutation matrix already showed
otherwise, and the comment was written against that data).

Tests gained the cases that decide it: every near-miss body above as an exact
upload, the four EBML directions including both adversarial files, the ADTS
guards isolated so exactly one mutation kills each, and length boundaries at
the offsets each check indexes past — the previous negatives padded everything
to 512 bytes, which hid every length guard.

Refs: BUG-2963

* test(attachments): kill two mutants the round-1 tests left alive (BUG-2963)

Both survived the expanded matrix, and both for the same reason round 1 kept
finding: a negative case that fails for a reason other than the guard it names.

- The bzip2 near-miss was five bytes, so the LENGTH guard refused it and the
  block-magic check was never reached; removing that check left the suite
  green. The new sibling is long enough to reach it and NUL-padded so the
  stdlib still says octet-stream — without the padding the body sniffs as
  text and the magic table is never consulted at all, which the first attempt
  at this test demonstrated by passing for the wrong reason.

- The ADTS octet-stream gate had no discriminating input: the two-byte case
  fails validADTSHeader on length before the gate matters. FF F1 40 41 41 41
  41 is a structurally valid ADTS header — sync, layer 00, sampling index 0,
  frame length 2570 — whose every byte the stdlib reads as text, so it answers
  text/plain. That is the only shape that separates the gate from the
  structural check, and with the gate removed the file is stored as audio/aac.

Refs: BUG-2963

* test(attachments): fuzz the sniff path, since this change added a parser (BUG-2963)

Every other check in this package reads fixed offsets and is bounded by
construction. sniffEBMLDocType walks caller-supplied length fields, which is
the one shape here that can index out of range or fail to advance, so the
question 'does it survive malformed input' deserved an answer from running it
rather than from reading it.

Asserts the two properties a sniffer owes its caller — it returns, and it does
not panic — across sniffOpaqueMagic, sniffEBMLDocType, sniffOggAudio,
validADTSHeader, SniffMIME and ValidateUpload. What it returns for nonsense is
left to the table tests.

Seeds are the real fixtures, each truncated at fourteen offsets, plus the
shapes a walk breaks on first: unknown-size elements (the reserved all-ones
VINT), a child whose declared size exceeds the data, the invalid all-zero VINT
marker, a zero-length child that must still advance, and a truncated Ogg page
header. 5.56M executions, no panic and no hang.

Also collapses a duplicate fixture loader this file had grown alongside
mime_isobmff_test.go's readFixture.

Refs: BUG-2963

* fix(attachments): use the real parsers, drop Ogg, and stop overclaiming (BUG-2963)

Round 2 found 15 issues, two of them P1, and the important one is not any
single check — it is that round 1's answer was wrong in the same WAY round 1's
defect was. Round 1 defeated prefix matching with an ELF carrying "ustar" at
offset 257. The response was a checksum. Round 2 defeated the checksum with an
ELF carrying a CORRECT one.

Measured while responding, and it settles the question: archive/tar's own
Reader.Next accepts that file too. A 512-byte tar header is exactly those
fields, and nothing forbids another format's padding from containing them. The
two are not distinguishable at this size by anything in the standard library,
so this is a property of the formats and not a defect to fix. Adding a third
round of field checks would have been the same mistake a third time.

So the change is to the CLAIMS as much as to the code. These checks RECOGNISE
a format; they do not establish one. The safety property lives elsewhere and is
now stated once, at the top of mime_magic.go: whatever the bytes are, they are
stored opaquely, never executed or decompressed here, and served under a
reviewed type with nosniff, as an attachment for every type recognised here.
The checksum-correct ELF ships as a fixture and a test asserting it is
ACCEPTED — a limitation recorded where it cannot be rediscovered as a bug.

Where a real parser exists, it is now used: archive/tar and compress/bzip2,
which cannot drift from the parsers a consumer would use and handle GNU/pax
variants for free. bzip2 keeps its magic check alongside the decode because
neither subsumes the other — the magic refuses a five-byte stream the decoder
can only call truncated, the decode refuses an empty stream whose combined CRC
is wrong.

OGG IS REMOVED. The alias was ruled in, and review showed the question it has
to answer — is this container audio — cannot be answered from the head of the
file: an Ogg with Opus first and VP8 second passes a first-packet codec gate,
because Ogg multiplexes and the video pages come later. The gate was also
wrong in the other direction, refusing legitimate Skeleton-prefixed audio.
Ogg is refused exactly as before this branch; making it work means adding
video/ogg as a reviewed entry or demuxing, and neither belongs in a change
whose premise is that it adds no new trust. The reasoning is a comment where
the next person to reach for an alias will meet it.

Other round-2 correctness fixes: a DocType value ends at its first NUL, so a
real Matroska padded "matroska\x00junk" is no longer stored as WebM; reserved
all-ones EBML IDs are refused rather than acting as zero-length children;
FLAC checks the STREAMINFO BODY rather than its declaration; 7z checks that
the start header's arithmetic is representable, not only that its CRC agrees;
ADTS accounts for the two CRC bytes a protected frame declares.

Tests gained the direction they lacked entirely — legal variants that must NOT
be refused (an empty bzip2 stream, STREAMINFO carrying the last-block flag, a
CRC-protected ADTS frame) — and the truncation loop now asserts refusal below
each check's minimum instead of discarding its return value, which is why it
had been passing while flac.head512[:8] was accepted.

Refs: BUG-2963

* test(attachments): cover the five guards the rebuilt matrix found untested (BUG-2963)

Rebuilding the mutation matrix against the round-2 code turned up seven
survivors. Six were real gaps; each now has the exact input a review round
used, or the smallest one that separates the guard from its neighbours, and
two carry a control leg so the negative cannot pass for the wrong reason.

- 7z: a start header whose next-header offset is 2^64-1, with a recomputed
  valid CRC. The CRC proves the bytes are intended, not that they are possible.
- FLAC: STREAMINFO declaring a zero sample rate.
- bzip2: the head of a 200KB archive, which must be RECOGNISED — truncation is
  what a 512-byte head of any real archive looks like, and the guard that
  distinguishes it from a structural error had nothing testing it.
- EBML: a real Matroska whose DocType payload is 'matroska\x00junk'; and a
  header with a reserved all-ones ID before a valid DocType.

The seventh is an EQUIVALENT mutant and is recorded as one rather than fixed:
lowering validTarHeader's length guard from 512 to 262 changes no outcome,
because archive/tar refuses a short block by itself. Measured, then written
into the comment so the guard reads as clarifying rather than load-bearing.

Refs: BUG-2963

* fix(attachments): delete the bzip2 magic comparison a mutation proved dead (BUG-2963)

The check compared the block magic AND decoded, under a comment claiming each
caught what the other could not. A repaired mutation run refuted it: with the
comparison removed, every input the comment credited it with stopping is still
refused — the short ones by the length guard, the rest by the decoder. So the
line could not change an outcome while its comment said it did, which is the
failure this package's own SafeFallbackExtension comment warns about. Deleted
rather than demoted, and the length guard now says what it is for.

The mutants that found this had been scoring BUILD-FAIL, which is nothing at
all rather than a survivor — repairing them to compile is what surfaced three
untested guards, two of them FLAC's STREAMINFO block-size floor and ordering.

Refs: BUG-2963

* fix(attachments): recognise by magic, to the standard this door already uses (BUG-2963)

Three review rounds walked the structural-validation path to its end and the
ruling is to stop:

- Round 1 defeated magic matching with a real, executing ELF carrying "ustar"
  at offset 257.
- Round 2 defeated the checksum that answered round 1, with an ELF carrying a
  CORRECT one — and archive/tar's own Reader.Next accepts that file too. A
  512-byte tar header is exactly those fields; the two are not distinguishable
  at this size by anything in the standard library.
- Round 3 found the accumulated validation refusing REAL files: PAX and
  long-name GNU tars, legal randomized bzip2 blocks, FLAC declaring the zero
  sample rate RFC 9639 permits. That is this bug's own defect — refusing files
  people legitimately have — reintroduced by the fix for it.

Validation could not narrow what the door accepts and had started refusing
what it should take, so it is gone. Recognition is by defining magic: ustar at
257, BZh plus its digit, the 7z six bytes, fLaC, and for ADTS the syncword
plus the layer bits, which keeps its .aac extension gate because fourteen bits
is weaker than the rest.

The fact that settles the width, read from the stdlib rather than assumed:
http.DetectContentType recognises audio/mpeg from the three bytes "ID3"
(net/http/sniff.go), audio/mpeg is on the allowlist, and this door already
serves it inline. Every signature here is at least as wide, so this is the
door's EXISTING standard rather than a relaxation of it. A test asserts that
premise so it cannot rot.

Two refusals are deliberate and say so in the code: V7 tar has no magic
anywhere and cannot be recognised by this kind of check at all, and Ogg stays
refused for the reason a previous commit records.

bzip2 is no longer decompressed anywhere in the door. The safety paragraph is
rewritten from what the code does, for the third time and the last: nothing is
executed, nothing is decompressed, archives download while FLAC and AAC play
inline exactly as every other allowlisted audio type does, and nosniff means
recognition can move a file between reviewed types but never outside them.

Tests now assert the contract that exists. The widening is asserted rather
than described — every input the validation used to refuse is accepted, each
with its SERVING BUCKET checked, because that is the property that makes it
tolerable. The files validation used to refuse are asserted accepted. What
remains of the negatives is the only thing still true: the magic has to be
there, in the right place, in full.

Refs: BUG-2963

* test(attachments): restore two EBML properties the bulk cut dropped (BUG-2963)

Cutting the structural-validation tests wholesale took the DocType
NUL-termination and reserved-ID cases with them, though the DocType walk they
cover is untouched by the magic-only ruling. A mutation run is what noticed:
both mutations had gone from detected to surviving.

Worth recording as a shape rather than a slip — deleting a test file's worth of
obsolete assertions is exactly when live coverage leaves with them, and the
matrix is the only thing that says so.

Refs: BUG-2963

* fix(attachments): tar wins a prefix collision, and the AAC gate stops refusing real files (BUG-2963)

Round 4's two blocking findings were both REAL files of listed types turned
away — the direction the ruling's convergence bar names first.

**A tar whose first member is named fLaC.txt was refused.** Every recogniser
except tar's is a prefix test, and a tar header's first 100 bytes are its
member's FILENAME — arbitrary text a user chooses. So an ordinary archive
carrying "fLaC" or "BZh9" in a name was recognised as that format and then
refused for a category mismatch against its own .tar extension. Tar is tested
first now, and the order is load-bearing rather than arbitrary: the collision
is asymmetric. A real tar carrying a foreign prefix needs only a filename; a
real FLAC carrying "ustar" needs those five bytes at exactly offset 257 in
compressed data. Losing the first case costs ordinary uploads.

**Real AAC files were refused when their leading bytes looked textual.** The
gate ran on application/octet-stream alone, but a raw AAC frame whose ancillary
payload is printable makes the first 512 bytes read as text, so the stdlib
answers text/plain and a genuine, ffmpeg-decodable .aac was rejected for a
category mismatch. Neither verdict is a format detection; both mean "nothing
here identifies this", which is the condition under which a weak signature may
speak. A type the stdlib DOES recognise is still untouched, and a test pins
that with PNG bytes named .aac.

Also, the leftovers that keep being mine: comments still describing structural
validation that is gone, a test comment claiming production delegates to
archive/tar when it no longer does, a README row saying the ELF fixture exists
to be refused when it is now accepted, an unused fixture, and two fixtures the
README never listed.

Tests strengthened where a review round showed one example was standing in for
a whole signature: the EBML legs now ask sniffEBMLDocType directly, because
routed through SniffMIME the stdlib fallback supplied the same WebM answer and
the explicit mapping could be deleted with the suite green; the ADTS signature
is walked byte by byte; the bzip2 digit range is tested at both bounds and the
FLAC marker at its width and case.

Refs: BUG-2963

* test(attachments): give the ADTS verdict gate a control that actually controls (BUG-2963)

The PNG leg could not establish the gate it was named for: PNG bytes fail
validADTSHeader on the first byte, so they are refused with the gate removed
too. A mutation run said so — dropping the stdlib-verdict gate survived the
whole suite.

The only shape that separates the verdict gate from the structural check is an
input that PASSES validADTSHeader and is ALSO identified as something else: a
buffer opening with a valid ADTS header and carrying 'ustar' at offset 257,
named .aac. It is identified as a tar and must be refused for the category
mismatch it is; without the gate the AAC branch overwrites that and accepts it.

Recording the process failure alongside it, because it is one I have written
down before: this test was lost once between writing and committing, because
the mutation runner's restore is 'git checkout -- internal/attachments/' and
the work was still uncommitted. Committing is step one of running a control,
not step one of the unit.

Refs: BUG-2963

* fix(attachments): let the extension arbitrate a magic collision, both ways (BUG-2963)

Round 5 found the mirror image of round 4's finding, which is the useful part:
my fix for round 4 created it. Round 4 showed a tar whose first member is named
fLaC.txt being refused, so tar was ordered first. Round 5 then built complete,
decodable FLAC and AAC files carrying "ustar" at offset 257 in ordinary
metadata — a Vorbis COMMENT tag is arbitrary UTF-8 (RFC 9639 §8.6) — and those
were refused instead.

The premise I wrote into the ordering comment was false. I argued the collision
was asymmetric, that real audio could not plausibly carry "ustar" at a fixed
offset. It can, in a tag a user typed. Any total order refuses somebody.

So there is no winner by order. sniffOpaqueCandidates returns EVERY matching
type, and when the filename's extension names one of them it breaks the tie.
That is a narrower thing than extension trust and the code says so: every
candidate is a type the BYTES already matched, so the extension chooses among
readings rather than casting a vote, and a name for a type whose magic is
absent can never appear in the list. An extension naming none of them changes
nothing — asserted, including that a .zip name does not make colliding bytes a
zip.

The AAC branch is fixed by the same finding from the other direction: it was
gated on the REFINED sniff, so a real AAC with "ustar" in its payload — which
this package refines to application/x-tar — was refused under its own .aac
name. The gate now reads the standard library's verdict, which for those bytes
is "nothing identifies this". Kept rather than dropped as unfalsifiable,
because no mimesniff signature begins with 0xFF today and this is what stops a
fourteen-bit match overriding one that does.

Fixtures: the FLAC is real and decodes under libsndfile. The AAC counterpart is
NOT shipped — overwriting a real frame's bytes produces a file ffmpeg rejects,
so the test uses a synthetic buffer that reproduces the condition and says so
rather than claiming to be audio.

Also cleared, and reliably my own: comments still describing structural
validation, a test's why-strings still citing checksums and CRCs, a comment
claiming PAX and GNU archives are refused when they are recognised, and a
README gap note that no longer covered all the fixtures. Two test premises
tightened where they proved less than they said: the bzip2 prefix case also
failed the digit check, and the three-byte ID3 claim was tested with seven.

Refs: BUG-2963

* test(attachments): pin the default candidate order (BUG-2963)

Reordering sniffOpaqueCandidates changed no test, because extension
arbitration settles both known collisions whichever way the list runs. The
order still decides one case — colliding bytes whose extension names neither
candidate — and nothing asserted it, so the rationale in the comment was
unenforced.

Refs: BUG-2963

* docs(attachments): the fuzz comment still named the structural validators (BUG-2963)

They were removed by the magic-only ruling. Comment-only.

Refs: BUG-2963

2026-09-09 13:28:23 -04:00

512 B

Raw History