`cargo xtask text-oracle` reported one number: word F1 against `pdftotext`. F1 is a bag of
words. It catches a character read wrong, a glyph not read at all, a font we cannot map -- and
it is blind to order, so a page emitted back to front scores a perfect 1.000. Reading order is
most of what text extraction has left to get wrong: a column read into the middle of its
neighbour (#2387), a figure's labels threaded through a paragraph (#2586), a form's value parted
from its label.
The hole is not hypothetical. Reviewing #2431, a change that moves text and alters nothing else,
**word F1 called 26 of the 29 files whose output changed identical**, while scoring the word
sequence separated 18 improvements from 6 regressions and named the shape of the regressions
(forms: `prefilled_f1040.pdf` 0.672 -> 0.481, a checkbox's number parted from its label). The
harness could not see the only thing that change does.
So a second score per file: the longest common subsequence of the two word sequences, as a share
of the longer one. It shows up in the first three corpus files -- `Test-plusminus.pdf` scores F1
1.000 and reading order 0.875, the same forty words in a title block that we read down the
column and `pdftotext` reads across.
Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages
is plenty of signal; the run says how many files that bit, so the cap cannot quietly become the
measurement. It reports; it does not gate, and like the pixel score it is a *disagreement* with
Poppler rather than a verdict -- in the #2431 measurement the same split scored as a win on one
file and a loss on another purely because `pdftotext` makes it in one place and not the other.
The function's doc comment, `crates/oracle/README.md` and `docs/gaps.md` gap 12 all say so.
`words` now takes the sequence rather than re-splitting the text, so both scores read the same
tokens. `docs/file-format-parity.md` claimed plain-text export was good "(word-F1 0.98)" in a row
whose own column is reading order; it now says the two are scored separately.
Verified: `cargo xtask text-oracle --limit 60` end to end; `cargo test -p xtask` 49 passed; fmt
and clippy clean. The unit test was watched failing with the score blinded to order (sorting both
sides made it 1.000 where it must be below 0.7). The corpus-wide first reading is not in this
commit: the machine is shared with a parallel build and the full sweep was stopped twice for
memory.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Keep RGBA decoding unchanged while satisfying the current Clippy lint.
Include the separately verified Rustls 0.23.46 lock prerequisite.
Co-authored-by: sanyamsingh001 <314051937+sanyamsingh001@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
The render oracle put four JBIG2 fixtures at the top of its disagreement list against Poppler, and
the obvious reading was that hayro-jbig2, which carries local patches, had a bug. Investigating it
found the reverse.
The 96 `bitmap-*.pdf` corpus files are a conformance suite: one small drawing encoded through every
corner of JBIG2 -- each generic-region template, custom AT pixels, TPGDON and TPGRON, refinement,
halftone with and without skip, symbol dictionaries with Huffman tables, MMR, the composition
operators, random access, unknown-height segments. Those features change how the bytes are
compressed, not what the picture is, so a correct decoder renders all 96 identically. That is a
real test with no reference image and no external oracle: the suite checks itself.
We render 1 distinct image across all 96. Poppler renders 8: it draws a blank page for the seven
halftone-skip and symbol fixtures, and its output changes when TPGRON is switched on, which a
correct decoder's cannot. So roughly a fifth of Poppler's JBIG2 output is wrong, the top of the
oracle's worst list is its bug rather than ours, and Poppler is not a usable oracle for JBIG2.
Watched failing by admitting one unrelated file to the set: the test fails and names it as the odd
one out rather than merely reporting a count.
The expectation holds of the pinned corpus. Whoever re-pins it to a newer pdf.js should re-check
this rather than assume it still holds, which the test's own documentation says.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Investigating the worst file of the first 250 turned the headline finding around. bug1552113.pdf
scores 84%, and the cause is Poppler: it paints the link's `/Border [0 0 112]` as a 112-unit band
across a 250x50 page, hiding the two lines of text. The fixture is named "this text should be
visible", and in our rendering it is. The oracle is not ground truth, and the top of its worst
list needs a person to look at both images before it is read as a bug of ours.
That led to a plausible change that the measurement rejected. Poppler synthesises appearances for
annotations that have none, where we draw only what the document carries -- `/AP`-less link borders
are already recorded as not covered under view.render-annotation-appearances -- so hiding
annotations looked like the way to measure page content alone. It is worse: it then scores Poppler
for every appearance we do draw. Over the same 250 files, shown gives mean 2.16%, p90 5.35% and 25
files within 0.1%; hidden gives 2.18%, 5.55% and 22, with annotation-tx3.pdf jumping to 45.64% and
annotation-tx2.pdf to 18.79%. The two biases nearly cancel.
So the default is unchanged and `--hide-annotations` is an opt-in for explaining a single file,
with the numbers written down so the experiment is not repeated. Verified that the default
reproduces the merged baseline exactly and that the flag moves it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gap 12 of docs/gaps.md: until rendering is measured, every claim about its quality is a judgement.
Text extraction had an oracle and written output had qpdf; rendering had nothing.
`cargo xtask render-oracle` renders page 1 of every corpus file with our CLI and with Poppler's
pdftoppm and reports how far apart they are. The metric is the point. A raw pixel comparison is
useless here -- two correct renderers disagree on every glyph edge, so it calls a perfect page
badly wrong and says nothing about whether a shading is missing. `pixels::compare` counts a pixel
as differing only when it differs from every pixel in the other image's 3x3 neighbourhood by more
than a tolerance, in both directions, taking the worse. A one-pixel shift is forgiven; absent or
miscoloured content is not.
First reading, 250 files at 72 dpi, tolerance 8: 229 compared, median 0.89%, mean 2.16%, p90
5.35%, 25 within 0.1%. It reports rather than gates -- the corpus is not on CI, and a first
reading of a borrowed renderer is a baseline to improve on, not a bar to clear. The worst results
cluster on JBIG2 `bitmap-*` files and on ZapfDingbats, which aims the renderer work at something
specific for the first time. 15 files could not be compared because the two renderers round the
page box to different pixel sizes; they are reported as unmeasurable rather than scored.
Verified against the knob's bounds end to end: tolerance 255 scores every file 0.00%, tolerance 0
takes the median to 0.95% and p90 to 26.36%.
Also turns on `extract_type3_shared_xobjects_matches_poppler_pixels`, which had never run anywhere.
Poppler is now installed on the Linux runner, and the test is version-robust by construction --
both sides of its comparison go through the same pdftoppm, so it is about our two documents rather
than about Poppler's output. Verified in all three states: runs and passes with pdftoppm present,
skips without it, fails without it when PDFCRAFT_ORACLES_REQUIRED is set.
`run_capture` keeps its exact behaviour, including leaving the buffer empty on non-UTF-8 output;
the byte-oriented capture an image needs is a second entry point on the same timeout logic.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every test that wanted an external oracle probed for it by hand and chose for itself what to do
when it was missing. The common choice -- print a line and return -- reports as a pass, so a suite
with no oracles installed looked exactly like one in which every oracle agreed.
That was not hypothetical. No CI runner installed qpdf, so `qpdf_agrees` had never executed on a
runner, while README.md said saved output was "independently checked with qpdf". For R2-R4 that
test is the only check of owner-password authentication (Algorithm 7) that does not go through our
own code.
`pdfcraft-oracle` holds the probes and one rule: a missing oracle is a declared skip or a failure,
never a silent pass. `PDFCRAFT_ORACLES_REQUIRED` picks which, on the same convention as
`CRAFT_FONTS_REQUIRED`. CI installs qpdf and sets it, so a skip notice there is now a red build.
The crate has no panicking path of its own; `or_skip!` expands its assert into the caller, which
is test code.
Verified by running the qpdf tests against qpdf 12.4.2 for the first time -- they pass, so the
README claim is now both enforced and true. Both gate directions were watched: absent and
unrequired skips with a notice, absent and required fails with the install command in the message.
Only qpdf is installed in CI, because it is the only tool the test job uses. The Poppler pixel
comparison in crates/organize stays ignored until a change turns it on and verifies it there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Giuseppe Silano