mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-10-25 18:59:48 +00:00
snipeTR
9035ae3c82
Text files were decoded with String::from_utf8_lossy wherever they are
read: Create PDF and Combine from a .txt file, the automation `create`
tool's text `path`, and Search & Redact's word list file. So
- a UTF-16 file (Windows Notepad's "Unicode") became NULs and replacement
characters ("??G?r?????e?" for "Grüße"),
- a UTF-8 byte order mark became a stray character at the start, and
- a Windows-1252 or 1254 file lost every accented letter ("caf�", "K��").
For a word list that means the words were never found and stayed
unredacted.
pdfcraft_create::decode_text picks UTF-8 or UTF-16 (LE/BE) by the byte
order mark and drops it, keeps valid UTF-8, and otherwise decodes the
Windows code page: 1254 when the bytes 1254 uses for Ğ İ Ş ğ ı ş appear
without á í ó ú (which Turkish never writes but Icelandic and Faroese,
the only Western languages using those bytes, always do), else 1252.
encoding_rs is already in the build through lopdf.
Co-authored-by: Brandon Thomas <echelon@gmail.com>