mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-10-26 03:10:21 +00:00
Tanish Manku
50cbf5fa19
`cargo xtask text-oracle` reported one number: word F1 against `pdftotext`. F1 is a bag of words. It catches a character read wrong, a glyph not read at all, a font we cannot map -- and it is blind to order, so a page emitted back to front scores a perfect 1.000. Reading order is most of what text extraction has left to get wrong: a column read into the middle of its neighbour (#2387), a figure's labels threaded through a paragraph (#2586), a form's value parted from its label. The hole is not hypothetical. Reviewing #2431, a change that moves text and alters nothing else, **word F1 called 26 of the 29 files whose output changed identical**, while scoring the word sequence separated 18 improvements from 6 regressions and named the shape of the regressions (forms: `prefilled_f1040.pdf` 0.672 -> 0.481, a checkbox's number parted from its label). The harness could not see the only thing that change does. So a second score per file: the longest common subsequence of the two word sequences, as a share of the longer one. It shows up in the first three corpus files -- `Test-plusminus.pdf` scores F1 1.000 and reading order 0.875, the same forty words in a title block that we read down the column and `pdftotext` reads across. Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages is plenty of signal; the run says how many files that bit, so the cap cannot quietly become the measurement. It reports; it does not gate, and like the pixel score it is a *disagreement* with Poppler rather than a verdict -- in the #2431 measurement the same split scored as a win on one file and a loss on another purely because `pdftotext` makes it in one place and not the other. The function's doc comment, `crates/oracle/README.md` and `docs/gaps.md` gap 12 all say so. `words` now takes the sequence rather than re-splitting the text, so both scores read the same tokens. `docs/file-format-parity.md` claimed plain-text export was good "(word-F1 0.98)" in a row whose own column is reading order; it now says the two are scored separately. Verified: `cargo xtask text-oracle --limit 60` end to end; `cargo test -p xtask` 49 passed; fmt and clippy clean. The unit test was watched failing with the score blinded to order (sorting both sides made it 1.000 where it must be below 0.7). The corpus-wide first reading is not in this commit: the machine is shared with a parallel build and the full sweep was stopped twice for memory. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
126 lines
6.5 KiB
Markdown
126 lines
6.5 KiB
Markdown
# pdfcraft-oracle
|
||
|
||
The external programs PdfCraft checks its own output against, and one rule: **a missing oracle is
|
||
either a declared skip or a failure, never a silent pass.**
|
||
|
||
- **Layer:** test support (`Class::Testkit` in `xtask/src/layers.rs`). A dev-dependency only, never
|
||
compiled into the product, and so usable even from the standalone crates (`cos`, `geom`,
|
||
`filters`, `crypt`, `arlington`) whose own tests need it.
|
||
- No panicking path of its own. The `or_skip!` macro expands its `assert!` into the caller, which
|
||
is test code.
|
||
|
||
## Why it exists
|
||
|
||
Every test that wanted an oracle used to probe for it by hand and decide on its own what to do when
|
||
it was missing. The common choice — print a line and return — reports as a **pass**, so a suite
|
||
with no oracles installed looked exactly like one in which every oracle agreed.
|
||
|
||
That was not hypothetical. No CI runner installed `qpdf`, so `cos/tests/encryption.rs::qpdf_agrees`
|
||
had never executed on a runner, while `README.md` said saved output was "independently checked with
|
||
qpdf". For R2–R4 that test is the *only* check of owner-password authentication (Algorithm 7) that
|
||
does not go through our own code.
|
||
|
||
## API
|
||
|
||
```rust
|
||
// In a test: bind the oracle, or return after saying why it was skipped.
|
||
let qpdf = pdfcraft_oracle::or_skip!(pdfcraft_oracle::qpdf());
|
||
let out = qpdf.command().arg("--check").arg(&path).output()?;
|
||
|
||
// Elsewhere (xtask), handle the absence yourself.
|
||
match pdfcraft_oracle::pdftotext() {
|
||
Ok(tool) => { /* tool.command() */ }
|
||
Err(absent) => bail!("{absent}"),
|
||
}
|
||
```
|
||
|
||
| Probe | Tool | Used for |
|
||
|---|---|---|
|
||
| `qpdf()` | qpdf | `--check` structural validation; the independent owner-password oracle |
|
||
| `pdftoppm()` | Poppler | page pixels, for comparing against our own raster output |
|
||
| `pdftotext()` | Poppler | text extraction (`cargo xtask text-oracle` scores word F1 and reading order against it) |
|
||
|
||
Each returns `Result<Oracle, Absent>`. `Oracle::command()` hands back a fresh `Command`;
|
||
`Absent` renders as an actionable sentence naming the install command.
|
||
|
||
## The gate
|
||
|
||
`PDFCRAFT_ORACLES_REQUIRED` (any value, including `0` — the same convention as
|
||
`CRAFT_FONTS_REQUIRED`) turns absence from a skip into a failure. CI sets it on the runners that
|
||
install the tools, so a skip notice there becomes a red build.
|
||
|
||
```
|
||
$ cargo test -p pdfcraft-cos --test encryption qpdf_agrees
|
||
test result: ok. 1 passed # qpdf absent: skipped, with a notice on stderr
|
||
|
||
$ PDFCRAFT_ORACLES_REQUIRED=1 cargo test -p pdfcraft-cos --test encryption qpdf_agrees
|
||
oracle `qpdf` is required but is not on PATH (program not found): PDFCRAFT_ORACLES_REQUIRED is
|
||
set, so this is a failure rather than a skip (apt install qpdf / brew install qpdf)
|
||
test result: FAILED. 1 failed
|
||
```
|
||
|
||
`or_skip!` expands to a `return`, so it belongs in a test returning `()`; a test returning `Result`
|
||
should match on the probe directly.
|
||
|
||
## Comparing pixels (`pixels`)
|
||
|
||
Rendering needs more than "did the tool run". A raw pixel-for-pixel difference is useless here:
|
||
two correct renderers disagree on every glyph edge, so a strict comparison calls a perfect page
|
||
badly wrong and tells you nothing about whether a shading is missing.
|
||
|
||
`pixels::compare` counts a pixel as differing only when it differs from **every** pixel in the
|
||
other image's 3x3 neighbourhood by more than a tolerance. A one-pixel shift or a differently
|
||
weighted edge finds a near match next door and is forgiven; a glyph that is not there, or a region
|
||
painted the wrong colour, has no near match and is counted. The test runs in both directions and
|
||
reports the worse result, so neither argument order flatters the blurrier renderer.
|
||
|
||
`decode_pgm` reads what `pdftoppm -gray` writes; `decode_pam_gray` reads what
|
||
`pdfcraft-cli render --out x.pam` writes, compositing premultiplied alpha over white first so a
|
||
transparent margin is not scored as black.
|
||
|
||
`cargo xtask render-oracle` drives it over the corpus. It reports; it does not gate.
|
||
|
||
A high score is a *disagreement*, not a verdict on us. The worst file of the first 250,
|
||
`bug1552113.pdf` at 84%, is Poppler painting a link's `/Border [0 0 112]` as a band over the text
|
||
the fixture exists to show; ours is the readable rendering. Look at both images before believing
|
||
the number. `--hide-annotations` separates annotation differences from page content, though the
|
||
default shows them: hiding is slightly worse overall, because it then scores Poppler for every
|
||
appearance we do draw.
|
||
|
||
**Poppler is not a usable oracle for JBIG2.** The 96 `bitmap-*.pdf` corpus fixtures encode one
|
||
picture through every corner of the format, so a correct decoder renders them all identically. We
|
||
render 1 distinct image across all 96; Poppler renders 8, drawing a blank page for seven of them.
|
||
`crates/render/tests/jbig2_invariants.rs` pins that invariant, and needs no oracle to do it.
|
||
|
||
## Comparing text
|
||
|
||
`cargo xtask text-oracle` scores two different things against `pdftotext`, because one number
|
||
hides the other.
|
||
|
||
**Word F1** is a bag of words: how much of what each side extracted the other also extracted,
|
||
with multiplicity. It catches a character read wrong, a glyph not read at all, a font we cannot
|
||
map. It is blind to *order* — a page emitted back to front scores 1.000.
|
||
|
||
**Reading order** is the longest common subsequence of the two word sequences, as a share of the
|
||
longer one. It catches the failures F1 cannot see: a column read into the middle of its
|
||
neighbour, a figure's labels threaded through a paragraph, a form's value parted from its label.
|
||
Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages
|
||
is plenty of signal; the run says how many files that bit.
|
||
|
||
The second score exists because of a measurement. Reviewing #2431, a change that moves text and
|
||
alters nothing else, word F1 called 26 of the 29 files whose output changed *identical*, while
|
||
the order score separated 18 improvements from 6 regressions — the regressions on forms, where
|
||
a checkbox's number was parted from its label.
|
||
|
||
Read it the same way as the pixel score, and then once more: Poppler's reading order is a second
|
||
opinion, not the truth, and the same split can score as a win on one file and a loss on another
|
||
purely because `pdftotext` happens to make it in one place and not the other.
|
||
|
||
## Still to do
|
||
|
||
- Acrobat itself as the reference, rather than other open renderers.
|
||
- Oracles that are not external processes (hayro, `lopdf`) are ordinary dev-dependencies and are
|
||
not routed through this crate.
|
||
- Two renderers sometimes round the same page box to different pixel sizes, and those files are
|
||
counted as unmeasurable rather than compared.
|